<div>&nbsp;</div>
   <p>Active metadata is like hot gossip. Here&rsquo;s why.</p>
   <p id="4e49">
     Just like data mesh or the metrics layer,&nbsp;<em class="mj"
       >active metadata</em
     >
     is the latest hot topic in the data world. As with every other new concept
     that gains popularity in the data stack, there&rsquo;s been a sudden explosion
     of vendors rebranding to &ldquo;active metadata&rdquo;, ads following you
     everywhere, and&hellip; confusion.
   </p>
   <p id="22a1">
     With everyone talking about active metadata, it must be pretty easy to
     understand, right?
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go mk">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*oj53dlEWig_Hcxmtj4GKJA.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go ml">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*bF3m6G6XM4D4UrYgMD8NsQ.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <p id="5481">
     Apparently not! I&rsquo;ve been talking about active metadata for over a year
     now, but I still see questions like these all the time.
   </p>
   <p id="c0f1">
     Active metadata can sound a bit scary, but it doesn&rsquo;t have to be. It is
     a must-have tool in the modern data toolbox, so if you&rsquo;re still
     wondering what it means, this article is for you.
   </p>
   <p id="76b6">
     I&rsquo;ve broken down the ideas behind active metadata with as little jargon
     as possible. Keep reading to learn what active metadata is, what it looks
     like, how you can actually use it, how it fits into the modern data stack, and
     why it even matters.
   </p>
   <h3 id="6b2d">What is active metadata?</h3>
   <p id="569c">
     I could start dropping some jargon here, but then both you and I will be
     asleep in seconds. So let&rsquo;s jump into an analogy instead.
   </p>
   <p id="4dd5">
     Imagine that you got your hands on the juiciest piece of tech gossip &mdash;
     Apple is expanding into recreational marijuana to literally help people
     &ldquo;think different&rdquo;.
   </p>
   <p id="aee1">
     There&rsquo;s no way you&rsquo;re going to keep something this exciting a
     secret. The world has to know. So you post it on your blog,&nbsp;<em
       class="mj"
       >blogspot.applefansunite.com</em
     >. All done, right?
   </p>
   <p id="b968">
     Just like&nbsp;<a
       class="mds-text-primary"
       href="https://www.vice.com/en/article/n7n74d/traffic-jams-are-possible-in-elon-musks-tunnels-apparently"
       target="_blank"
       rel="noopener ugc nofollow"
       >a car in the Hyperloop tunnel</a
     >, we all know that&rsquo;s not going anywhere. You can&rsquo;t just put the
     story somewhere and hope people will find it. You have to actually deliver it
     into people&rsquo;s hands.
   </p>
   <p id="47ef">
     You sharpen your PR chops, blast the news to tech reporters and news sites,
     and lo and behold it&rsquo;s&nbsp;<em class="mj">everywhere</em> in no time.
     It&rsquo;s already been mummified, and your grandfather just asked why apple
     farmers are talking about this Molly girl on your group chat.
   </p>
   <blockquote class="nj">
     <p id="988d" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt">Metadata is like this information. If it sits&nbsp;</em
       >passively<em class="nt"
         >&nbsp;in its own little world, with no one seeing or sharing it, does it
         even matter? But if it&nbsp;</em
       >actively<em class="nt"
         >&nbsp;moves to the places where people already are, it becomes part of
         and adds context to a larger conversation.</em
       >
     </p>
   </blockquote>
   <h3 id="36ed">What is the difference between active and passive metadata?</h3>
   <p id="e3db">
     <strong>Passive metadata</strong>&nbsp;is the standard way of aggregating and
     storing metadata into a static data catalog. This usually covers basic
     technical metadata &mdash; schemas, data types, models, etc.
   </p>
   <p id="4780">
     Think of passive metadata as putting out information on a personal blog. Every
     so often, it&rsquo;ll get picked up and go viral on Hacker News. But most of
     the time it&rsquo;s just going to sit unseen and unused, even when people
     actually need to know it.
   </p>
   <p id="c214">
     <strong>Active metadata</strong>&nbsp;makes it possible for metadata to flow
     effortlessly and quickly across the entire data stack, embedding enriched
     context and information in every tool in the data stack. It is usually more
     complex than passive metadata, covering operational, business, and social
     metadata along with basic technical information.
   </p>
   <p id="16a6">
     Think of active metadata as a viral story. It shows up everywhere you already
     live in what seems like seconds. It&rsquo;s immediately cross-checked against
     and combined with other information, bringing together a network of related
     context into a larger trend or story. And it sparks conversations, making
     everyone more knowledgable and informed in the end.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go nx">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*h71nNqp0b03WgO_XI2o8fw.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt">The four main types of metadata. (Image by author.)</em>
     </figcaption>
   </figure>
   <h3 id="7541">Why does active metadata matter?</h3>
   <blockquote class="nj">
     <p id="302a" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt"
         >To put it simply, no one wants to go to another website to &ldquo;browse
         the metadata&rdquo;.</em
       >
     </p>
   </blockquote>
   <p id="ca3d">
     As we embraced the internet and data exploded in the early aughts, companies
     realized they needed to manage all their new data.
   </p>
   <p id="091a">
     We entered a golden age of metadata management. New companies like&nbsp;<a
       class="mds-text-primary"
       href="https://www.informatica.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Informatica</a
     >,&nbsp;<a
       class="mds-text-primary"
       href="https://www.collibra.com/us/en"
       target="_blank"
       rel="noopener ugc nofollow"
       >Collibra</a
     >, and&nbsp;<a
       class="mds-text-primary"
       href="https://www.alation.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Alation</a
     >&nbsp;were created, and they hyped the importance of data catalogs. People
     needed a way to sort through all their options, so we got reports like
     Gartner&rsquo;s&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/3993025"
       target="_blank"
       rel="noopener ugc nofollow"
       >Magic Quadrant for Metadata Management</a
     >. Billion-dollar companies emerged, and companies spent hundreds of millions
     of dollars on metadata management.
   </p>
   <p id="41e1">
     Yet just last year, Gartner released their&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/4006759"
       target="_blank"
       rel="noopener ugc nofollow"
       >Market Guide for Active Metadata</a
     >&nbsp;and declared that &ldquo;Traditional metadata practices are
     insufficient&hellip;&rdquo;
   </p>
   <p id="5d60">
     That&rsquo;s because passive data catalogs solve the &ldquo;too many
     tools&rdquo; problem by adding&hellip; another tool. They aggregate metadata
     from different parts of the data stack, and it stagnates there. User adoption
     suffers, and&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-future-of-data-catalogs-868ffbe7354d"
       target="_blank"
       rel="noopener"
       >these exciting tools turn into expensive shelfware</a
     >.
   </p>
   <p id="2761">
     Active metadata sends metadata back into every tool in the data stack, giving
     the humans of data context wherever and whenever they need it &mdash; inside
     the BI tool as they wonder what a metric actually means, inside Slack when
     someone sends the link to a data asset, inside the query editor as try to find
     the right column, and inside Jira as they create tickets for data engineers or
     analysts.
   </p>
   <h3 id="050b">How does active metadata fit into the modern data stack?</h3>
   <p id="68c9">
     Active metadata functions as a layer on top of the modern data stack.
   </p>
   <p id="bdd2">
     It leverages open APIs to connect all the tools in your data stack and ferry
     metadata back and forth in a two-way stream. This is what allows active
     metadata to bring context, say, from Snowflake into Looker, Looker into Slack,
     Slack into Jira, and Jira back into Snowflake.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go od">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*53oiNCxKcmY9bLFNCKNnfg.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Active metadata in the modern data stack. (Image by author.)</em
       >
     </figcaption>
   </figure>
   <h3 id="dbd9">4 characteristics of active metadata</h3>
   <p id="e3b1">
     According to&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-gartner-magic-quadrant-for-metadata-management-was-just-scrapped-d84b2543f989"
       target="_blank"
       rel="noopener"
       >Gartner&rsquo;s new Market Guide for Active Metadata</a
     >, active metadata is an always-on, intelligence-driven, action-oriented,
     API-driven system, the opposite of its passive, static predecessor.
   </p>
   <p id="0837">
     This can be broken down into the four key characteristics of active metadata.
   </p>
   <ul>
     <li id="4667">
       <strong>Always on</strong>: Active metadata is always on. Rather than
       waiting for people to manually enter or parse metadata, this means
       continually collecting metadata at every stage of the modern data stack
       &mdash; logs, query history, usage statistics, and more.
     </li>
     <li id="fd88">
       <strong>Intelligent</strong>: Active metadata isn&rsquo;t just about
       collecting metadata. It&rsquo;s about constantly processing metadata to
       connect the dots and create intelligence from it. This means that with
       active metadata, the system will only get smarter over time as people use it
       more and it observes more metadata.
     </li>
     <li id="c801">
       <strong>Action-oriented</strong>: Active metadata doesn&rsquo;t just stop at
       intelligence. It should drive action by curating recommendations, generating
       alerts, and making it easier for people to make decisions &mdash; or even
       automatically making decisions without human intervention, like stopping
       downstream pipelines when data quality issues are detected.
     </li>
     <li id="3a52">
       <strong>Open by default</strong>: Active metadata platforms use APIs to hook
       into every piece of the modern data stack. This makes magical user
       experiences possible by saving data practitioners from the endless tool- and
       context-switching. This is called embedded collaboration, which is when work
       happens where you are with the least amount of effort.
     </li>
   </ul>
   <h3 id="a46d">5 use cases of active metadata</h3>
   <p id="b89a">
     There are dozens, if not hundreds, of use cases of active metadata. (Enough
     for several articles of their own &mdash; coming soon!) Let&rsquo;s go through
     a few of my favorites.
   </p>
   <ul>
     <li id="e13d">
       <strong>Purge stale or unused assets</strong>: Use active metadata to
       periodically calculate when each asset (e.g. a data table, dashboard, etc)
       was last used and/or how many people used it. If it was used within the last
       30 days, great! If an asset hasn&rsquo;t been used in the last 60 days,
       automatically archive it. If no one has touched it in the last 90 or 120
       days, purge it entirely.
     </li>
     <li id="a52a">
       <strong>Allocate compute resources dynamically</strong>: Imagine that 90% of
       users log in to a BI tool during the last week of a financial quarter.
       Active metadata can be used to automatically scale up compute resources just
       before that week and scale them down again afterward.
     </li>
     <li id="34bd">
       <strong>Enrich user experience in BI tools</strong>: Instead of switching
       between a BI tool and data catalog, use active metadata to bring context
       into dashboards. Relevant metadata (like business terms, descriptions,
       owners, and lineage) can be pushed into the BI tool. Then when someone is
       looking at each table, they can understand who owns it, where the data came
       from, etc. This information could even be used as labels in auto-generated
       reports.
     </li>
   </ul>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go os">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*aLuvXmyV2tyIr1YKbIbNZA.gif"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Enriching BI tools with active metadata. (Image from author.)</em
       >
     </figcaption>
   </figure>
   <ul>
     <li id="2ef1">
       <strong>Identify popular assets</strong>: Use active metadata to create a
       custom relevance score for each asset. This can be based on usage
       information from places like query logs, lineage, and BI dashboards. Then
       the most popular, relevant assets should be surfaced more frequently in
       search and checked more frequently for data quality issues.
     </li>
     <li id="d9f9">
       <strong>Notify downstream consumers</strong>: It&rsquo;s awful if the CEO
       ends up seeing a broken dashboard before the data team. Active metadata can
       be used to check for issues when a data store changes and notify downstream
       data users about potential issues. For example, when a data store is
       crawled, the new metadata would be compared against previous metadata. If
       there are any potential breaking changes (e.g. the addition or removal of a
       column), lineage could be used to find who owns this data store and notify
       them in Slack, Jira, email, etc.
     </li>
   </ul>
   <h3 id="aac4">The future of active metadata</h3>
   <p>
     As metadata becomes big data and big data becomes a behemoth, active metadata
     isn&rsquo;t just a wonderful dream. It&rsquo;s a necessity &mdash; the only
     way to understand today&rsquo;s data.
   </p>
   <p id="cb3d">
     Managing, processing, and analyzing metadata is the new normal for modern data
     teams. Doing this passively and manually, though, isn&rsquo;t possible.
     That&rsquo;s why it&rsquo;s been so exciting to see active metadata take shape
     in the last year and become the de facto standard for what people expect out
     of modern metadata.
   </p>
   <p id="f4b4">
     All of these use cases &mdash; like auto-tuned pipelines, automated data
     quality alerts, and continuously validated calculations &mdash; would have
     sounded wildly impossible just a few years ago. Today, they&rsquo;re actually
     in reach. I couldn&rsquo;t be more excited to see the intelligent data dream
     become a reality as active metadata continues to evolve in the coming years.
   </p>    
   

What Is Active Metadata, and Why Does It Matter?

<p><a class="mds-text-primary" href="https://blog.getdbt.com/future-of-the-modern-data-stack/">The modern data stack</a>&nbsp;has helped democratize the creation, processing, and analysis of data across organizations. However, it has also led to&nbsp;<a class="mds-text-primary" href="https://benn.substack.com/p/the-modern-data-experience">a new set of challenges</a>&nbsp;thanks to the decentralization of the data stack. In this post, we&rsquo;ll discuss one of the cornerstones of the modern data stack&mdash;data catalogs&mdash;and why they fall short of overcoming the fragmentation to deliver a fully self-served data discovery experience.</p>
    <p>If you are a leaders of the data team at a company with 200+ employees, there is a high probability that you have</p>
    <ul role="list">
    <li>started seeing data discovery issues at your company;</li>
    <li>tried one of the commercial or open-source data catalogs; or</li>
    <li>cobbled together an in-house data catalog.</li>
    </ul>
    <p>If that&rsquo;s the case, you&rsquo;d definitely find this post highly relatable.</p>
    <h3><strong>Pain Points</strong></h3>
    <p>This post is based on our own experience of building&nbsp;<a class="mds-text-primary" href="https://engineering.linkedin.com/blog/2019/data-hub">DataHub</a>&nbsp;at LinkedIn and the learnings from 100+ interviews with data leaders and practitioners at various companies. There may be many reasons why a company adopts a data catalog, but here are the pain points we often come across:</p>
    <ul role="list">
    <li>Your data team is spending a lot of time answering questions about where to find data and what datasets to use.</li>
    <li>Your company is making bad decisions because data is inconsistent, poor in quality, delayed, or simply unavailable.</li>
    <li>Your data team can't confidently apply changes to, migrate, or deprecate data because there&rsquo;s no visibility into how the data is being used.</li>
    </ul>
    <p>The bottom line is that you want to empower your stakeholders to self-serve the data, and more importantly, the&nbsp;<em>right</em>&nbsp;data. The data team doesn't want to be bogged down by&nbsp;<em>support questions</em>&nbsp;as much as data consumers don't want to depend on the data team to answer their questions. Both of them share a common goal&mdash;True Self-service Data Discovery&trade;.</p>
    <h3><strong>First Reaction</strong></h3>
    <p>In our research, we saw striking similarities in companies attempting to solve this problem themselves. The story often goes like this:</p>
    <li>Create a database to store metadata.</li>
    <li>Collect important metadata, such as schemas, descriptions, owners, usage, and lineage, from key data systems.</li>
    <li>Make it searchable through a web app.</li>
    </ol>
    <p>Voila! You now have a full self-service solution and proudly declare victory over all data discovery problems.<strong>&zwj;</strong></p>
    <h3><strong>Initial Excitement</strong>&zwj;</h3>
    <p>Let&rsquo;s walk through what typically happened after this shiny new data catalog was introduced. It looked great on first impression. A handful of power users were super excited about the catalog and its potentials. They were thrilled about their newfound visibility into the whole data ecosystem and the endless opportunities to explore new data. They were optimistic that this was indeed<em>The Solution</em>&nbsp;they&rsquo;ve been looking for.<strong>&zwj;</strong></p>
    <h3><strong>Reality Sets In</strong>&zwj;</h3>
    <p>A few months after launching, you started noticing that the user engagement waned quickly. Customer&rsquo;s questions in your data team&rsquo;s Slack channel didn&rsquo;t seem to go away either. If anything, they became even harder for the team to answer.</p>
    <p>&zwj;So what happened?</p>
    <ul role="list">
    <li>People searched &ldquo;revenue&rdquo; hoping to find the official revenue dataset. Instead, they got 100s of similarly named results, such as &ldquo;revenue&rdquo;, &ldquo;revenue_new&rdquo;, &rdquo;revenue_latest&rdquo;, &ldquo;revenue_final&rdquo;, &ldquo;revenue_final_final&rdquo;, and were at a complete loss.</li>
    <li>Even if the person knew the exact name of what they were looking for, the data catalog only provided&nbsp;<em>technical information</em>, e.g. SQL definition, column descriptions, linage, and data profile, without any explicit instructions on how to use it for a specific use case.</li>
    <li>Your data team has painstakingly tagged datasets as "core", "golden", "important", etc., but the customers didn't know what these tags mean or their importance. Worse yet, they started tagging things randomly and messed up the curation effort.</li>
    </ul>
    <p>Is it really that hard to find the right data even with such advanced search capabilities and all the rich metadata?&nbsp;<em>Yes!</em>&nbsp;Because the answer to &ldquo;what&rsquo;s the right data&rdquo; depends on who you are and what use cases you&rsquo;re trying to solve. Most data catalogs only present the information from the producer&rsquo;s point of view but fail to cater to the data consumers.</p>
    <h3><strong>The Missing Piece</strong></h3>
    <p>Providing the producer&rsquo;s point of view through automation and integration of all the technical metadata is definitely a key part of the solution. However, the consumer&rsquo;s point of view&mdash;trusted tables used by my organization, common usage patterns for various business scenarios, impact from upstream changes have on my analyses&mdash;is the missing piece that completes the data discovery &amp; understandability puzzle.</p>
    <p>Most of the data catalogs don't help users find the data they need; they help users find someone to pester, which often referred to as a &ldquo;tap on the shoulder&rdquo;. This is not true self-service.</p>
    <h3><strong>The Solution</strong></h3>
    <p>We believe that there are three types of information/metadata required to make data discovery truly self-serviceable:</p>
    <ol role="list">
    <li><strong>Technical Metadata</strong>: This refers to all metadata originating from the data systems, including schemas, lineage, SQL/code, description, data profile, data quality, etc. Automation and integration would keep the information at the user&rsquo;s fingertips.<br /><strong>Challenges</strong>: There is no standard for metadata across data platforms. Worst yet, many companies build their own custom systems that hold or produce key metadata. How to integrate these systems at scale to ingest metadata accurately, reliably, and timely is an engineering challenge.</li>
    </ol>
    <ol role="list" start="3">
    <li><strong>Business Metadata</strong>: Each business function operates based on a set of common business definitions, often referred to as &ldquo;business terms&rdquo;. Examples include Active Customers, Revenue, Employees, Churn, etc. As a data-driven organization relies heavily on these definitions to make key business decisions, it is paramount for data practitioners to correctly translate between the physical data and business terms.<br />&zwj;<strong>Challenges</strong>: Many companies lack the tools, processes, and disciplines to govern and communicate these business terms. As a result, when serving a business ask, data practitioners often struggle to find the right data for a particular business term or end up producing results that contradict each other.</li>
    <li><strong>Behavioral Metadata</strong>: Surfacing the association between people and data is critical to effective data discovery. Users often place their trust in data based on who created or used it. They also prefer to learn how to do their analyses from more experienced &ldquo;power users&rdquo;. To that end, we need to&nbsp;<a class="mds-text-primary" href="https://locallyoptimistic.com/post/share-your-data-insights-to-engage-your-colleagues/">encourage</a>&nbsp;the sharing of these data learnings/insights across the company. This would also improve your organization&rsquo;s data literacy, provide a better understanding of the business, and reduce inconsistencies.<br />&zwj;<strong>Challenges</strong>: People interact with the data in different ways. Some query using Snowflake console, Notebooks, R, and Presto, while others explore using BI tools, dashboards, or even spreadsheets. As a result, the learnings and insights often spread across multiple places and make it difficult to associate people with data.</li>
    </ol>
    <p>It should be fairly clear by now that discovering the right data and understanding what it means is not a mere technical problem. It requires bringing technical, business, and behavioral metadata together. Doing this without creating an onerous governance process will boost your organization&rsquo;s data productivity significantly and bring true data-driven culture to your company.</p>

metadata

What Is Active Metadata, and Why Does It Matter?

Why Can't I Find the Right Data?

Tags

Top contributor