Introducing the Data ‘Products’ catalog

Sanjeev Mohan is a chief analyst at SanjMo. Get in touch with him LinkedIn.

My last businessroundups.org Council post in 2022 focused on exploring and defining a data product. As we enter 2023, we are already seeing more organizations building data products. Coinciding with the emergence of data products, the “data product catalog” will be introduced in 2023, which will further expand the range of current data catalogs used in a number of industries.

In my opinion, data catalogs themselves are experiencing an existential crisis, but the purpose they serve is undoubtedly essential. The name alone implies that its use is for data only and just a read-only metadata store. In reality, modern data catalogs manage metadata for a varied type of assets, including reports, dashboards, workflows, ML models, data quality metrics, transformations, spreadsheets, etc. Now, with the advent of data products, they have the ability to also store metadata related to the emerging space of data products.

Today’s modern stack is overly complex and has overwhelmed business users. Therefore, there is an overarching need for a unified metadata platform that serves diverse use cases, such as enabling DataOps activities such as orchestration, automation, and observability, data preparation using semantic layer and knowledge graphs, and data security, identity, and privacy. Therefore, a data product catalog is not a new stand-alone product, but an opportunity within the unified metadata platform that complements the data catalog.

What is a data product catalog?

A data product catalog is a repository of data products and associated metadata. It follows the entire data product creation lifecycle. Essentially, this catalog covers both the construction and operation of data products. It’s a single pane of glass to discover data products and associated metadata. A data consumer can use it to:

• Identify which data product is certified with a gold star and learn its quality aspects to decide which data product to use. For example, if there are 15 data products labeled “quarterly sales,” data consumers can determine which one is approved, so everyone is working with the same numbers. At the same time, data controllers can delete, rename or annotate the other 14 instances.

• Understand the use of data products and discover ‘data tribes’. This serves two purposes: to tell the user which data product is appropriate and which unused data products to retire. Automation marks data products as obsolete based on pattern and usage analysis, thereby reducing “data debt”.

• Explore operational metadata for data products, such as security access rights, data creators, version numbers, purpose, user consent, etc.

• Calculate productivity through “data telemetry” such as release frequency, number of data-related goals and objectives achieved, level of buy-in, and support for data strategy within the organization.

• Improve data-driven innovations, such as identifying new data products that can deliver new products or services.

Data producers or data engineers need to work with their business stakeholders to assemble high-quality datasets into logical data products, add business context and other metadata such as the datasheet, and publish it for discovery. As mentioned earlier, data product catalogs complement traditional data catalogs, which are used for business definition/glossary/lineage, metadata compilation, and discovering the occurrence of data across different sources.

Data product catalogs overlap with metadata management concepts.

This is not surprising, given that most new concepts take time to mature and develop standards. In fact, it is quite possible that the topics mentioned in this section will be refined or merged.

The idea of ​​data as a product got a huge boost when Zhamak Dehghani took it up four principles of data mesh, but the benefits extend far beyond just this. A data product is an evolution of self-service data access with governance and a published SLA on reliability, quality and trust. Self-service relies on raw data or a semantic layer, while data products take a more business value-oriented approach.

Successful implementation of data products is inseparable from the adoption of DataOps practices that define the lifecycle of buildings and operations. DataOps practices are used by developers and administrators of data products. Here, a data product catalog becomes a basis for the data producers and data consumers to merge their tasks. One of the most popular DataOps topics is data observability. A catalog of data products depends on operational statistics to provide meaningful use to the data consumers.

As mentioned earlier, a data product catalog serves as the marketplace of data products for internal and/or external users. Some organizations are already monetizing data products; hence the existence of a data marketplace has become essential. To avoid isolated metadata products, it is important that the data marketplace is a capability of the data product catalog.

Data contracts is another new topic that is gaining popularity. A data contract is defined by the data producer, who may not know who will use the data product. A data product catalog can be a single repository for data contracts so that they are applied consistently to all data consumers.

Finally, data access control is by no means a new topic, but data products can extend traditional rule-based and policy-based access control to a richer metadata-based approach that is dynamic and considers attributes, usage, and behavior. Since data products can be used by as yet unidentified data consumers, it is not always possible to determine security policies. As a result, the usage metrics and metadata-based approach may be the way to go for many.

Making data products mainstream can come closer to reality when combined with the idea of ​​a data products catalog. However, for these ideas to be successful, the market must create norms and governance processes or risk turning it into yet another overloaded yet vague concept.

---

businessroundups.org Business Council is the premier growth and networking organization for entrepreneurs and leaders. Am I eligible?

---