Why Can't I Find the Right Data?

The modern data stack has helped democratize the creation, processing, and analysis of data across organizations. However, it has also led to a new set of challenges thanks to the decentralization of the data stack. In this post, we’ll discuss one of the cornerstones of the modern data stack—data catalogs—and why they fall short of overcoming the fragmentation to deliver a fully self-served data discovery experience.

If you are a leaders of the data team at a company with 200+ employees, there is a high probability that you have

• started seeing data discovery issues at your company;
• tried one of the commercial or open-source data catalogs; or
• cobbled together an in-house data catalog.

If that’s the case, you’d definitely find this post highly relatable.

Pain Points

This post is based on our own experience of building DataHub at LinkedIn and the learnings from 100+ interviews with data leaders and practitioners at various companies. There may be many reasons why a company adopts a data catalog, but here are the pain points we often come across:

• Your data team is spending a lot of time answering questions about where to find data and what datasets to use.
• Your company is making bad decisions because data is inconsistent, poor in quality, delayed, or simply unavailable.
• Your data team can't confidently apply changes to, migrate, or deprecate data because there’s no visibility into how the data is being used.

The bottom line is that you want to empower your stakeholders to self-serve the data, and more importantly, the right data. The data team doesn't want to be bogged down by support questions as much as data consumers don't want to depend on the data team to answer their questions. Both of them share a common goal—True Self-service Data Discovery™.

First Reaction

In our research, we saw striking similarities in companies attempting to solve this problem themselves. The story often goes like this:

Create a database to store metadata.

Collect important metadata, such as schemas, descriptions, owners, usage, and lineage, from key data systems.

Make it searchable through a web app.

Voila! You now have a full self-service solution and proudly declare victory over all data discovery problems.‍

Initial Excitement‍

Let’s walk through what typically happened after this shiny new data catalog was introduced. It looked great on first impression. A handful of power users were super excited about the catalog and its potentials. They were thrilled about their newfound visibility into the whole data ecosystem and the endless opportunities to explore new data. They were optimistic that this was indeedThe Solution they’ve been looking for.‍

Reality Sets In‍

A few months after launching, you started noticing that the user engagement waned quickly. Customer’s questions in your data team’s Slack channel didn’t seem to go away either. If anything, they became even harder for the team to answer.

‍So what happened?

• People searched “revenue” hoping to find the official revenue dataset. Instead, they got 100s of similarly named results, such as “revenue”, “revenue_new”, ”revenue_latest”, “revenue_final”, “revenue_final_final”, and were at a complete loss.
• Even if the person knew the exact name of what they were looking for, the data catalog only provided technical information, e.g. SQL definition, column descriptions, linage, and data profile, without any explicit instructions on how to use it for a specific use case.
• Your data team has painstakingly tagged datasets as "core", "golden", "important", etc., but the customers didn't know what these tags mean or their importance. Worse yet, they started tagging things randomly and messed up the curation effort.

Is it really that hard to find the right data even with such advanced search capabilities and all the rich metadata? Yes! Because the answer to “what’s the right data” depends on who you are and what use cases you’re trying to solve. Most data catalogs only present the information from the producer’s point of view but fail to cater to the data consumers.

The Missing Piece

Providing the producer’s point of view through automation and integration of all the technical metadata is definitely a key part of the solution. However, the consumer’s point of view—trusted tables used by my organization, common usage patterns for various business scenarios, impact from upstream changes have on my analyses—is the missing piece that completes the data discovery & understandability puzzle.

Most of the data catalogs don't help users find the data they need; they help users find someone to pester, which often referred to as a “tap on the shoulder”. This is not true self-service.

The Solution

We believe that there are three types of information/metadata required to make data discovery truly self-serviceable:

1. Technical Metadata: This refers to all metadata originating from the data systems, including schemas, lineage, SQL/code, description, data profile, data quality, etc. Automation and integration would keep the information at the user’s fingertips.
   Challenges: There is no standard for metadata across data platforms. Worst yet, many companies build their own custom systems that hold or produce key metadata. How to integrate these systems at scale to ingest metadata accurately, reliably, and timely is an engineering challenge.

3. Business Metadata: Each business function operates based on a set of common business definitions, often referred to as “business terms”. Examples include Active Customers, Revenue, Employees, Churn, etc. As a data-driven organization relies heavily on these definitions to make key business decisions, it is paramount for data practitioners to correctly translate between the physical data and business terms.
   ‍Challenges: Many companies lack the tools, processes, and disciplines to govern and communicate these business terms. As a result, when serving a business ask, data practitioners often struggle to find the right data for a particular business term or end up producing results that contradict each other.
4. Behavioral Metadata: Surfacing the association between people and data is critical to effective data discovery. Users often place their trust in data based on who created or used it. They also prefer to learn how to do their analyses from more experienced “power users”. To that end, we need to encourage the sharing of these data learnings/insights across the company. This would also improve your organization’s data literacy, provide a better understanding of the business, and reduce inconsistencies.
   ‍Challenges: People interact with the data in different ways. Some query using Snowflake console, Notebooks, R, and Presto, while others explore using BI tools, dashboards, or even spreadsheets. As a result, the learnings and insights often spread across multiple places and make it difficult to associate people with data.

It should be fairly clear by now that discovering the right data and understanding what it means is not a mere technical problem. It requires bringing technical, business, and behavioral metadata together. Doing this without creating an onerous governance process will boost your organization’s data productivity significantly and bring true data-driven culture to your company.