Metrics Store

What is a metrics store? Why your data team should define business metrics in code

A metrics store is a centralized and governed place for organizations to store key metrics. It is a way of centralizing organizational knowledge, creating a repository for stakeholders to access key metrics in a repeatable way, regardless of how they plan to consume them downstream.

Until recently, metrics stores were built in-house. Companies with large data teams (like Airbnb) built internal metrics stores to standardize metrics definitions across data sets and data tools. This helped streamline experimentation and at a larger scale, it helped get everyone on the same page when it comes to defining important indicators like “revenue,” “new users,” or “customer growth.” Today, data teams can purchase this functionality from off-the-shelf SaaS tools.

In this article, we’ll cover:

• Benefits of a metrics store and why you need one
• How a metrics store fits into the modern data stack
• How data teams use metrics stores
• Use cases for data and business teams

Benefits of a metrics store: consistency, access, and productivity

Metrics stores are starting to gain traction as part of the modern data stack. That's because having a metrics store can provide some major benefits. With a metrics store, you can:

• Have a centralized place to consume data off the data warehouse.
• Ensure that the data fueling your metrics is accurate and consistent.
• Provide access to accurate data so that people see the same results when they analyze the data, regardless of the tool they’re using.

To understand the value of a metrics store, you need to understand the context. Most organizations have indicators to track success and progress towards goals. Some organizations call them key performance indicators (KPIs) or leading indicators, while others call them metrics. The premise is the same—organizations need to understand if their decisions (or actions) are helping them achieve their objectives. Usually, data teams are responsible for building accurate data sets for the business to track these metrics in downstream business intelligence (BI) tools like Tableau, Looker, or Mode. But not every team uses the same tool, and the underlying data used to define a metric like revenue might live in a few different tables in the warehouse. That means that reporting “revenue” can become a battle of questions like...

“Which definition is right?” Why are there multiple definitions?” “Can I even trust this data?” Even simple questions like “How many sales did we make last week?” can lead to confusion.

By building metrics in code (in a metrics store), data teams have a centralized location to build, measure, and identify business metrics. This also makes your data teams more efficient and strategic because they don’t have to build data sets over and over. Instead, they become enablers, helping business leaders define, share, and champion their metrics across the organization.

How does a metrics store fit into the modern data stack?

Companies design their data infrastructure based upon how they expect to drive decision-making through analytical systems. Typically, these companies manage vast amounts of data, flowing from various different applications or third-party source systems. Data is captured and stored in a data warehouse or lake, depending on the type of analytical or operational workflows that the data is intended for. Some transformation may occur before the data flows to the warehouse (think ETL), but more often than not companies are opting for an ELT approach, where datasets are loaded into the warehouse, then prepared for analytical application consumption downstream. Another approach is when denormalization is performed at the application layer itself, sequestering the metric logic within those bespoke tools (see diagram below).

A metrics store sits conveniently in between an organization’s data warehouse and other downstream tools. Essentially, it acts as a proxy to the warehouse, translating metric requests into SQL queries to the warehouse. In this way, transformation occurs at the 'metric level' —consistently defined in code, accessible to all downstream tools, and centrally governed to maximize insights.

How does a data and analytics team use a metrics store?

Although metrics consistency is one critical piece of an organization’s success, data teams also use a metrics store for a variety of other important tasks. As Nick Handel explained on the Data Engineering Podcast, teams use a metrics store to:

1. Interface with metric knowledge: Explore and set metrics definitions and see metrics lineage. Get everyone on the same page when it comes to defining important metrics like “active users,” “net churn,” or “monthly recurring revenue.” That also includes the lineage of how each metric is built, including information at the data source level.
2. Meet consumers in the interfaces they already use: Exposing metrics to all the different places where they’re consumed, like CRMs, experimentation tools, BI tools, and data quality tools. That means that a salesperson looking at a “opportunities created” metric in a BI tool would see the same numbers as a marketer.
3. Manage metrics lifecycles: Track changes to metrics over time and alert metrics owners when things change. If the lineage or definition of a metric changes, the owner is immediately notified and can see why and how this change occurred. If a spike or a dip in the data is present, the owner already knows it happened and has answers for stakeholders. No more tracking down anomalies in the data warehouse.

Who is a metrics store made for? Use cases for data and business teams

A metrics store has direct benefits for data and analytics teams, but more importantly, it has a direct impact on the organization as a whole. When everyone is speaking the same language of data, you have less tension over what a metric means or which number is accurate. Everyone has a single source of truth.

The primary people who use a metrics store are:

Data producers (Data engineers, analytics engineers, data analysts) - People who build normalized data sets in the warehouse and who build and test hypotheses. These producers are responsible for providing clean data sets for business users.

• How they use a metrics store: Data producers, typically roles within the data and analytics team, use a metrics store to centralize definitions for metrics, calculate metrics, and then surface these metrics to data consumers wherever they like to consume data. This alleviates a lot of repeatable work.

Data consumers (Line-of-business owners) - People who like to consume and explore data. They want everyone to use the same definitions for metrics and they want to see metrics sliced by different dimensions.

• How they use a metrics store: Data consumers could be in more technical roles, like a product manager, or they could be a business stakeholder like a head of sales. These consumers can use a metrics store to see an overview of how metrics are performing or they can access these calculated metrics in other tools, like a business intelligence tool, for example.

Every data team needs a metrics store

Data is only helpful when it’s accessible and accurate. Up until recently, there was no centralized tool for this purpose. Analysts used a combination of BI tools and documentation tools to record metrics definitions and lineage. Adopting a metrics store puts all of your metrics information in one place. It becomes a knowledge base for metrics for the entire organization. This means end users have more context and detail so they get more insights out of your data. It also exposes metrics to a broader range of applications and a larger group of people, making your data team more efficient and strategic since they don’t have to build and rebuild data sets for analyses. We see a future where metrics stores are available to everyone, regardless of your organization’s size or industry.