      <p>The CRM is no longer seen as the definitive source of trust for enterprises when it comes to collecting customer data. Instead, it has become just another SaaS tool that is unable to handle the complex data architectures that modern enterprises have created.</p>
      <p>Warehouses have emerged as the new system of records, revolutionizing the way</p>
      <p>we approach data management. The CDW industry has grown from $36B to $80B in the last 5 years. One of the main benefits of a data warehouse is its flexibility, security and ability to grow with your business.</p>
      <p>Several apps have realized the true potential of data warehouses and have harnessed their power by building on top of them, enabling business applications to go beyond just business intelligence.</p>
      <h3>What are the benefits of data warehousing?</h3>
      <p>In the last 5 years, the&nbsp;number of SaaS tools used by companies has been multiplied by 10.</p>
      <p><img style="width: 100%;" title="" src="https://lh4.googleusercontent.com/HGPG2FX4GIugtslhQ3oa4SCVTRYz4hB_GJoX7NeRNxFhqHocpoy6fVwY_xbltRVJ-6LsYPUzLlEWCa4Aq8fQ59HEZwHhrWjoJ1ggmO-aJs2YpVyNPa5r4J-iT7SLHAzjZwhhOwvWU3O455gU4NJ0B6tkqthfeY2YBTIDh4bE3Kv9WMQwaE8BEKefzEzide-ifz-srzaEtw" alt="" /></p>
      <p>To face the ever-growing complexity and volume of their data, businesses are increasingly adopting data warehouses as their source of truth, and shifting to modern data stack technologies.</p>
      <p>It helps organizations to centralize and manage large amounts of data from various sources, such as transactional databases, log files, and external data sources (ie: third-party tools).</p>
      <p><img style="width: 100%;" title="" src="https://lh6.googleusercontent.com/9o41IbX0SEU6CTai5mZsrh5TQN8PbDQSXfjyW30bZk2PiPh7esAwfCgpbpkkUCez8k76nLz6lNRhpGsVR_16t7tVNzFJpW5xU_zFBEKyEZDXeYEmt-hjXYxPf45aU90Ac6ggTkTyp2nJKoZqT278D38pU8byk31ZwhiUKPVNTwmer-1wmVU965Nlz-XOP15yDD9UDiuF3g" alt="" /></p>
      <p>Some specific benefits of data warehousing include:</p>
      <ol start="1">
      <li>Improved data access:&nbsp;Data warehousing provides a single, centralized repository for data from various sources, making it easier for users to access and analyze the data.</li>
      <li>Enhanced data quality:&nbsp;Data warehousing helps to ensure the quality and</li>
      </ol>
      <p>integrity of the data by enforcing data standards, performing data cleansing, and providing a consistent view of the data.</p>
      <ol start="3">
      <li>Increased efficiency:&nbsp;Data warehousing enables users to access and analyze data faster, as the data is pre-processed and organized in a structured format. This can help to reduce the time and resources required to perform data analysis.</li>
      <li>Improved decision-making:&nbsp;Data warehousing provides a comprehensive view of an organization's data, enabling users to identify trends, patterns, and relationships that may not be apparent in individual data sources. This can support better decision-making processes.</li>
      </ol>
      <h3>What are the most common data warehouse applications?</h3>
      <p>There are several common applications of data warehousing, including:</p>
      <ol start="1">
      <li>Business intelligence:&nbsp;Data warehousing is often used to support business intelligence (BI) and reporting activities. The modern BI tools like Looker or Metabase sits directly on top of the warehouse, enabling businesses to get</li>
      </ol>
      <p>insights faster and to drive the business with a data driven approach, not intuition.</p>
      <ol start="2">
      <li>Analytics:&nbsp;Data warehousing enables organizations to perform advanced analytics, such as data mining, scoring, and predictive modeling, on large datasets. This can help organizations to identify trends, patterns and relationships in the data that can be used to optimize business processes and improve decision-making.</li>
      <li>Customer segmentation:&nbsp;Data warehousing can be used to segment customers based on their characteristics and behavior, enabling organizations to tailor marketing campaigns and other initiatives to specific groups of customers. The 360&deg; view of the customer we&rsquo;ve all dreamt about is finally here!</li>
      </ol>
      <h3>What is a data warehouse native app?</h3>
      <p><img style="width: 100%;" title="" src="https://lh5.googleusercontent.com/u3pKPTtWQk9-xQ675DQv-XoI-ZAsnCULU5Kj4E0iVfHotVjMKjmJQHo8uGlFZsVcOacE2seuJKHp-Kc0V3iPJ1dyIlM4lLGXKPyxLuNFqwlrCSQx-oziQcL_uWzIgRMaLJt4Uk0jeFdDE75AAYxlWITEmHsjAFwhH006i4WGglyOWv8-5cSQH5Fa5BDs3q_YNKy2s2YsJw" alt="" /></p>
      <p>Data warehouse native applications are the go-to softwares for maximizing the potential of data warehouses, the place where all your data lives. They are optimized for working with large amounts of data, and custom business entities.</p>
      <p>These applications are used for data analysis and reporting and more recently, for orchestrating sales processes and customer engagement.</p>
      <p>Different arena of tools understood the potential of this new generation of SaaS, and start leveraging the CDW for specific usecases. Some example includes the BI tools that were originally among the first to do it or more recently tools like Pocus in the</p>
      <p>PLG industry.</p>
      <h3>What are the benefits of data warehouse native apps?</h3>
      <p>They are software applications that does not have its own data backend, are just an application layer on top of your company data. No need to set new data pipelines to sync data back to your destination app.</p>
      <p>Why should you care about this new wave?</p>
      <p>Glad you asked.</p>
      <p>Well, one of the biggest advantages is that the data doesn&rsquo;t need to be synced or replicated on the vendor&rsquo;s application, which is typically the case with traditional SaaS solutions.</p>
      <p>We shouldn't have to pay for the same data multiple times. We should be able to get it once and use it for any purpose without paying more.. Or I hope you don&rsquo;t have 100 tools&nbsp;😅</p>
      <p>Another advantage of these apps is that, because they don't replicate your data, they don't have ownership of it. This means you have complete control over your data, as well as enhanced</p>
      <ol start="1">
      <li>Observability:&nbsp;The current model of transferring data to a third-party system is problematic because it completely removes visibility and the ability to trace dependencies within the pipeline.</li>
      <li>Security:&nbsp;You own your data and you keep it in a environment you can control.</li>
      </ol>
      <ol start="3">
      <li>Cost efficiency:&nbsp;Around 75% cost reduction compare to the sync cost of pushing your data to an external platform (+ most of those platform make you pay record based, so you end up paying multiple time for the same data). With CDW native apps, you don't have to replicate your data.</li>
      </ol>
      <p>They also benefit from the data warehouse qualities like greater data accessibility for end users, reliable data integrity that also help bridging the gap between data teams and business folks.</p>
      <p>Data engineers ensure data quality, business folks leverage it to drive revenue.</p>
      <p>Ultimately, you don't have to be tied to a fixed schema, you can create and own your data models &amp; business entities definition.</p>
      <h3>Introducing Cargo: The retool for go-to-market teams</h3>
      <p><img style="width: 100%;" title="" src="https://lh4.googleusercontent.com/nyTQeh3eM6vZV8Uju9SSMVISbSUDFRTZAr-3ycL4xCvjkv-DlgnGldDYIszKxiHFQn87N5sICVfgIysd25V7zSROkhLSjtfMIEc_ShlzLTzlKyp3UvnHoHEd5qPCyD8-6UL3bpryXT1z7UW0x68EjZrnij4okcCoQQPpXAVMHQ44n3exnyTGP5y0p3Ot3tE_M4E-d_9zPQ" alt="" /></p>
      <p>As we said earlier, there are several tools leveraging the power of the data</p>
      <p>warehouse. Those are idiosyncratic softwares having a standalone approach. Conversely, at Cargo</p>
      <p>We are unopinionated by design - Opinionated by usecases.</p>
      <p>Cargo let mature organizations build their own internal go-to-market apps on their own, fully customizable and way more effective to tackle their use-cases.</p>
      <p>Those mid-market &amp; enterprise companies deal with high volume and ever-growing complexity of data. By sitting on top of their warehouse, we make it easy to build applications that answer their business-specific needs.</p>
      <p>The main usecase today for enterprise companies is lifecycle marketing like sending a personalized promotional or product update email to a specific segment, as well as building nurturing automation based on users behaviors.</p>
      <p>For mid-market and SaaS companies, the primary use-cases are around building</p>
      <p>scoring system, for account grading or churn scoring, or doing lead routing according to their own territories and rules.</p>
      <p>Interested to learn more? Join the movement&nbsp;<a href="https://www.google.com/url?q=https://www.getcargo.io/&amp;sa=D&amp;source=editors&amp;ust=1673427255784271&amp;usg=AOvVaw1aSN-G0xPo1r2zMd5oHyJS">here</a></p>
      

Data warehouse native applications

<p>Salesforce has been considered for a long time as the source of truth. However, in the last 5 years, the number of SaaS tools used by companies has been multiplied by 10. Data are disparate across more tools than it was before (analytics data, product data, marketing data, financial data &hellip;) and Salesforce became less and less the reference for business and customer data.</p>
    <p><img style="width: 100%;" src="https://lh4.googleusercontent.com/0JhLEfep2e4inseeeM6QGwWypGrDtGzUisl-AVGu6Wh3W0CcFA8llRi2i3deZtYscRqtySs9f3Amc5AamTI-jIFqVz3OallwjDBhH4zFi5USJVZzNX07xUIaKsRm-CatGayVYez7TN0J1P1O4KXcIRNhuLmEw0sYemVDkBOVGkdaNz7lUV_Ooo_selwz_OMgSCMviRU-Rw" alt="" /></p>
    <p>CDP Stack</p>
    <p>The first generation of tools called CDPs tried to solve these problems. Their main value proposition was simple: Reconsolidate data (mainly analytics data and customer data) to have a clear overview of the customer journeys and sync them back into your company&rsquo;s CRM. They were promising to own the source of truth again. But they actually made the same mistake than CRMs did. They couldn&rsquo;t easily adapt and fit the data complexity of your organization. You ended up with an infinite loop, data you couldn&rsquo;t unmerge and automatic workflows you couldn&rsquo;t control.</p>
    <p>The second generation of tools called Reverse ETL tackled the problem in a smarter way taking advantage of the rise of cloud data warehouse adoption. As the data warehouse was becoming central in companies for data visualization, they leveraged the work done by analytics engineers to sync it back into your CRM.</p>
    <p><img title="" style="width: 100%;" src="https://lh5.googleusercontent.com/kZGIPeHo7w1yYZl1tGSj8OdLSWLJMzcb7YBPAckoZ-GIB-ubZm4Kg-SeP_6a0oTD-rnMszVNBoCGMaYVGCdJ45_Z94nEJr2GFi4sc7bCcBtah9Cn-wlXuoT_q46DLuNWDPznVz0hVYl4BQUV9dy14Foc26S-qCvtFQFeSSuBYPNo8Ki63bwgQDAS_0rBD_OpfSLiXx9Fpg" alt="" /></p>
    <p>Current Modern Data Stack</p>
    <p>But finally, if we step back, what are these 2 tool categories are trying to do? &hellip; They are just patching CRM limitations.</p>
    <p>Let&rsquo;s list some of these limitations here:</p>
    <p><strong>Opinionated</strong> - CRMs enforce a proprietary data model to represent your business entities and you will have to hack it to match your own definition or push other than customer or sales data</p>
    <p><strong>Inflexible</strong> - Models can&rsquo;t be easily recomputed as your organization grows. You have to be cautious when starting to build your CRM properly otherwise you&rsquo;ll end up with messy and unusable data and have to start again from the beginning</p>
    <p><strong>Locked up</strong> - APIs were great for the old world but limited when we are talking about synchronizing/updating a big amount of data. Data is managed and owned by our CRM and can&rsquo;t be easily accessible for other providers/services.</p>
    <p>And who is the best applicant to solve all these limitations? &hellip; Your data warehouse. Instead of trying to push it and format it in another tool you don&rsquo;t master/control. Why not just use these models you or your team built while remaining in the warehouse.</p>
    <p>That&rsquo;s why we truly believe that the data warehouse and especially Snowflake is going to be a dangerous competitor for Salesforce in the coming years.</p>
    <p>And both of them know their strengths and weaknesses. Snowflake owns the data and Salesforce, the sales interface. On one side, Salesforce signed a partnership for easy data sharing. On the other side, Snowflake announced the launch of Unistore and the Native Application Framework that show their interest in not only being considered as a data warehouse anymore. Their ambition is to become a data operation system with a marketplace and to define a new standard for software to communicate together and access the data.</p>
    <p><img style="width: 100%;" title="" src="https://lh6.googleusercontent.com/9iRaFwOXV1VwfuAumqEBofmLEdLyq17EsC9IMAg7CuZwckyYs7nxYiJnnKuoF_N8NsC7mRjcrRub6pOTKIppiByBmNmQjyrnpbVxbTR2O8NcMvD2linM183wgRU-MxR8ugapn3-EVsEceJopcnP0I7gHinwa4i2yrRUwELBIPnYF6eQh5O7TWACPvIKCCTdbiqM8Gie_eA" alt="" /></p>
    <p>Future Modern Data Stack</p>
    <p>Semantic layer will become a new standard for SaaS apps to communicate together. Data engineering will provide a unified data business layer that apps could consume. APIs won&rsquo;t be necessary anymore to sync data between tools. SaaS tools read and write data in the data warehouse and won&rsquo;t own your data.</p>
    <p>CRM was the first software to be a SaaS, it was the first software to own a marketplace, and the first software to provide a public API.</p>
    <p>At Cargo, we want to lead this revolution and define the future of CRMs.</p>

Snowflake is the new Salesforce

<section>
    <div>
    <p>Spotify is the world&rsquo;s most popular audio streaming service with 433M users, and probably unwarranted of a further introduction. While the company has continued to grow extensively for the last years, so has the need for a fast and scalable infrastructure to support that growth. At our last Heroes of Data Meetup, Spotify&rsquo;s Sonja Ericsson joined us to talk about how they are migrating from Luigi to Flyte in order to build a next generation workflow platform to power all of their 20,000+ daily workflows.</p>
    </div>
    </section>
    <section>
    <div>
    <h3>Who&rsquo;s Sonja?</h3>
    <p><a class="mds-text-primary" href="https://www.linkedin.com/in/sonja-ericsson/" target="undefined" rel="noopener noreferrer">Sonja Ericsson</a>&nbsp;has been working as a Backend Engineer at Spotify for four years and has a Master&rsquo;s in Computer Science and Engineering from KTH Royal Institute of Technology in Stockholm, Sweden. Prior to joining Spotify, her experiences include software engineering at Epidemic Sound as well as data analytics and integrations at Zimpler, a Stockholm based fintech company.</p>
    </div>
    </section>
    <section>
    <div>
    <figure><img style="width: 100%" src="https://images.prismic.io/validio/929ef9d3-4cb6-4d04-9b33-238aa3696591_Screenshot+2022-10-06+at+15.15.18.png?auto=compress,format" alt="" />
    <figcaption>
    <p><em>Sonja Ericsson is a Backend Engineer at Spotify.</em></p>
    </figcaption>
    </figure>
    </div>
    </section>
    <section>
    <div>
    <h3>What Does Workflow Orchestration at Spotify Look Like?</h3>
    <p>Sonja&rsquo;s team is responsible for the platform that handles scheduling, orchestration and deployment of all data pipelines at Spotify &mdash; that&rsquo;s 20,000+ batch data pipelines running daily, defined in 1,000+ repositories, owned by 300+ teams. For many years, most of these pipelines have relied on a tool called Luigi, which was built in-house by Spotify and open-sourced in 2012. In essence, it is a client-side orchestration framework (with a server scheduler) used to build data pipelines in Python.</p>
    <p>In the old stack, users write their workflow code in the Luigi framework and use platform tasks provided by Spotify and open source Luigi through libraries. They would then build their workflow image based on a base image, also provided by Spotify, with additional dependencies. The complete workflow, including all tasks and dependencies, would get packaged into the image, and finally scheduled for deployment on&nbsp;<a class="mds-text-primary" href="https://kubernetes.io/" target="undefined" rel="noopener noreferrer">Kubernetes</a>.</p>
    </div>
    </section>
    <section>
    <div>
    <figure><img style="width: 100%" src="https://images.prismic.io/validio/fc013898-f98e-4c19-89ce-721a68caa401_Screenshot+2022-10-06+at+15.20.23.png?auto=compress,format" alt="" />
    <figcaption>
    <p><em>Simplified view of the old workflow stack at Spotify.</em></p>
    </figcaption>
    </figure>
    </div>
    </section>
    <section>
    <div>
    <h3>Why Move From Luigi To Flyte?</h3>
    <p>Luigi has been serving Spotify well over the years, and has been widely adopted as a workflow orchestration standard in the data engineering community. In recent years however, Spotify has identified areas of improvement where Luigi was struggling to meet the company&rsquo;s large scale orchestration demands. To stay competitive, it&rsquo;s increasingly important for Spotify to have tooling that can stay fast and scalable while the organization is growing.&nbsp;</p>
    <p>After evaluating different alternatives to Luigi for about a year,&nbsp;<a class="mds-text-primary" href="https://engineering.atspotify.com/2022/03/why-we-switched-our-data-orchestration-service/" target="undefined" rel="noopener noreferrer">Spotify decided to go with Flyte</a>&nbsp;which was built by Lyft and open-sourced in 2020. Flyte&rsquo;s orchestration framework had the extensibility to integrate Spotify tooling and needs, great scalability, and support for multiple languages.&nbsp;</p>
    <h3>The Challenges Of Luigi And How Flyte Solves Them</h3>
    <p>When using Luigi, Spotify usually factors in four main challenges; low feature penetration, dependency conflicts, inaccurate platform insights, and limited extensibility. Let&rsquo;s go through these one by one, and how Flyte solves them for Spotify.</p>
    <h4>1. Low Feature Penetration</h4>
    <p>In Luigi, one complete workflow would be frozen within one Docker image, meaning any upgrade requires that image to be rebuilt. In turn, 1,000 pull requests would be opened for a single upgrade to happen. Opening these PRs was usually automatic but since PRs were often not merged, this caused low feature penetration.</p>
    <h4><strong>The Solution</strong></h4>
    <p>✅ In Flyte, tasks can be executed by backend plugins. This allows for upgrades without user intervention, improving feature penetration.</p>
    <h4>2. Dependency Conflicts</h4>
    <p>In Luigi, all of the tasks were packaged within one Docker image and consequently share dependencies. These tasks usually have pretty complex dependencies which often results in dependency conflicts for the Luigi users.</p>
    <h4><strong>The Solution</strong></h4>
    <p>✅ In Flyte, each task can be isolated in its own image or executed by a backend plugin without involving a container. This reduces the problem of dependency conflicts.</p>
    <h4>3. Inaccurate Platform Insights</h4>
    <p>In Luigi, there is also a lack of easily retrievable structured entity information about the workflows or tasks, which means it&rsquo;s hard to know what happens within an image. They parse code to figure out usages of tasks and arguments.&nbsp;</p>
    <h4><strong>The Solution</strong></h4>
    <p>✅ In Flyte, entities are structured, type-safe and versioned. This allows for supporting many features such as understanding usages, workflow introspection, and caching.</p>
    <p>✅ Entities can also easily be shared and reused. Spotify is planning to have a task catalog of reusable tasks instead of shipping tasks as libraries.</p>
    </div>
    </section>
    <section>
    <div>
    <figure><img style="width: 100%" src="https://images.prismic.io/validio/b00a656b-674d-420a-8a54-7b376fd42e78_Screenshot+2022-10-06+at+15.20.45.png?auto=compress,format" alt="" />
    <figcaption>
    <p><em>A screenshot of the workflow introspection feature in Flyte.</em></p>
    </figcaption>
    </figure>
    </div>
    </section>
    <section>
    <div>
    <h4>4. Limited Extensibility</h4>
    <p>Luigi is a client-side framework, but Spotify often needs direct server-side interactions, for managing jobs in different systems without relying on containers. Spotify is also in frequent need for Java, while Luigi only supports Python. To combat this, they used to have a Java orchestration framework as well. However, any feature shipped in one language would then have to be shipped in the other, which leads to a lot of maintenance problems.</p>
    <h4><strong>The Solution</strong></h4>
    <p>✅ Due to a protobuf interface, a Flyte SDK can be implemented using any language. This makes it possible to leverage varying languages for different use cases and mix tasks from different languages in a workflow.</p>
    <p>✅ The Flyte backend is extensible through plugins, of which there are many available open source.</p>
    <p>✅ Going from client-side framework to a platform approach provides more control, extensibility, and ability to introduce abstractions.</p>
    </div>
    </section>
    <section>
    <div>
    <figure><img style="width: 100%" src="https://images.prismic.io/validio/8c912a6f-c79d-4b18-baec-5f00663ba6c6_Screenshot+2022-10-06+at+15.20.59.png?auto=compress,format" alt="" />
    <figcaption>
    <p><em>In Flyte, it&rsquo;s possible to leverage varying languages, like Java and Python, for different use cases.</em></p>
    </figcaption>
    </figure>
    </div>
    </section>
    <section>
    <div>
    <h3>Closing Words</h3>
    <p>Spotify is currently using Flyte to run ~4,000 workflows each day, and ~100,000 executions across 175 teams. They are continuously working on integrating Flyte with Spotify&rsquo;s internal ecosystem and achieving feature parity with Luigi. Spotify has also been working on features like authorization and lineage. The goal is to successfully migrate all of their 20,000+ workflows to Flyte.</p>
    <p><em>Heroes of Data is an initiative by the data community for the data community. We share the stories of everyday data practitioners and showcase the opportunities and challenges that arise daily. Sign up for the Heroes of Data newsletter on&nbsp;<a class="mds-text-primary" href="https://https//heroesofdata.substack.com/" target="undefined" rel="noopener noreferrer">Substack</a>&nbsp;and follow Heroes of Data on&nbsp;<a class="mds-text-primary" href="https://www.linkedin.com/company/heroes-of-data/" target="undefined" rel="noopener noreferrer">Linkedin</a>. This article is summarized by Emil Bring, based on a presentation by Sonja Ericsson from Spotify at a Heroes of Data meetup in September 2022.</em></p>
    </div>
    </section>

Why Spotify moved from Luigi to Flyte to power their 20,000+ daily workflows

<p>If you have spent any time in the data space in the last 10 years, you&rsquo;ll know that data job titles have gotten hilariously complicated and confusing. There are Data Analysts, Data Scientists, Analytics Engineers, Data Engineers, Business Analysts, Business&nbsp;Intelligence&nbsp;Analysts, Product Analysts, Product Data Scientists, Data Product Managers (?), ML Engineers, Data Enthusiasts, People Who Just Really Love Counting, and dozens of other titles floating around.</p>
    <p>This proliferation of Data nouns &ndash;&nbsp;and their confusing mapping to a core set of tasks &ndash;&nbsp;has been a through line of my own career. When I started as an Analyst, I spent most of my time trying to produce basic counts to help make decisions ("how many WAUs?" could take weeks to figure out). I often worked with Data Engineers who built complex pipelines with &ldquo;real code&rdquo; to take data from source to the warehouse, but that was about the extent of my involvement in building data systems.</p>
    <p>At some point in the mid-2010s, &ldquo;Data Scientist&rdquo; started to enter into the zeitgeist. As best I could tell, it meant they either knew more statistics than me, or just did the exact same work but used Python instead of SQL. I&rsquo;m pretty sure they got paid more than me, though.</p>
    <p>And then a few years ago,&nbsp;<a class="mds-text-primary" href="https://www.google.com/url?q=https://www.getdbt.com/author/claire-carroll/&amp;sa=D&amp;source=editors&amp;ust=1668008209107045&amp;usg=AOvVaw3QyTX0UdQ3HX2mXlCsE5dU">our pal Claire</a>&nbsp;and the good folks at dbt&nbsp;<a class="mds-text-primary" href="https://www.google.com/url?q=https://www.getdbt.com/what-is-analytics-engineering/&amp;sa=D&amp;source=editors&amp;ust=1668008209107361&amp;usg=AOvVaw3xGFS2lQU8icie4AJLB6gk">brought&nbsp;</a><a class="mds-text-primary" href="https://www.google.com/url?q=https://www.getdbt.com/what-is-analytics-engineering/&amp;sa=D&amp;source=editors&amp;ust=1668008209107530&amp;usg=AOvVaw0NxxzXoRoBFMW-1EZou_cK">another</a><a class="mds-text-primary" href="https://www.google.com/url?q=https://www.getdbt.com/what-is-analytics-engineering/&amp;sa=D&amp;source=editors&amp;ust=1668008209107707&amp;usg=AOvVaw3mSMvbamDgbvtlmqDA9VRl">&nbsp;role into the mix:&nbsp;</a><a class="mds-text-primary" href="https://www.google.com/url?q=https://www.getdbt.com/what-is-analytics-engineering/&amp;sa=D&amp;source=editors&amp;ust=1668008209107867&amp;usg=AOvVaw3x5xoL9JW1ANUL0GNLmjKE">Analytics Engineer</a>. This felt new, exciting, and different from Analyst roles &ndash;&nbsp;like a more approachable Data Engineer. But I was confused, as the technical skillset appeared to overlap with traditional Analysts quite a bit. You just do SQL, plus some Git and Jinja? Isn&rsquo;t that what I do all day?</p>
    <p>Part of this confusion is the consequence of new tooling and capabilities. Historically, the lines between these roles were mostly drawn based on how &ldquo;technical&rdquo; you were&mdash; how well you could code. But in the modern data world, there are&nbsp;<a class="mds-text-primary" href="https://www.google.com/url?q=https://hex.tech/blog/how-to-build-a-dashboard-in-python/&amp;sa=D&amp;source=editors&amp;ust=1668008209108283&amp;usg=AOvVaw2tvuQwxALAzY2jeI4BYzTH">Python people putting together charts and analyses</a>, and&nbsp;<a class="mds-text-primary" href="https://www.google.com/url?q=https://dataform.co/blog/consider-sql-when-writing-your-next-processing-pipeline&amp;sa=D&amp;source=editors&amp;ust=1668008209108488&amp;usg=AOvVaw3_q0yRAI2zu2V_1nfAbq7-">SQL people building complex data pipelines</a>&mdash;&nbsp;the languages they use or how familiar they are with the command line no longer tells us much about what they actually spend their days building doing.</p>
    <p>And (for once!) it turns out I wasn&rsquo;t alone in my confusion. Today, I talk to folks all the time who are interested in data work and think they might be a &ldquo;data person&rdquo;, but just don&rsquo;t know how to chart a path through the thicket of data and ML titulature. It&rsquo;s unnecessarily complicated, and that makes it really hard for individuals to build careers and for leaders to grow data teams.</p>
    <p><img style="width: 100%" title="" src="https://lh4.googleusercontent.com/qLGQ9q5ZDRsgN4VSEpeclwcT4tliTU_FBsTiLtlqdXyDjU8UYtE6W0TRh8cgD3coJV0TZYvOJj6jaasPq5jpVY2VTJm16jP38qOsFCNzlZK9HniMPNc9T5lwZjnLE_-Aj1L4XSK48c2ZUaaFbkFD1ZQDlh4y9RqzPWElXJJAVPUqSKy_ZNc_S8bT9WtQ1KB76sPVjfRLvQ" alt="" /></p>
    <p>But why Analytics Engineers?</p>
    <p>Working at&nbsp;<a class="mds-text-primary" href="https://www.google.com/url?q=https://hex.tech/&amp;sa=D&amp;source=editors&amp;ust=1668008209108982&amp;usg=AOvVaw0Wi1a_xEmiW2jafkFNJKE6">Hex</a>&nbsp;has given me a unique vantage point on this. Every day, thousands of data people use our product, which gives us some insight into who is doing data work, what &ldquo;data work&rdquo; means to them, and what they call themselves. And as I&rsquo;ve set out to build our own internal Data team, I&rsquo;ve spent hours reflecting and pattern-matching to better understand what kinds of people are best suited for each type of role.</p>
    <p>Systems vs. Storytelling</p>
    <p>What I have found is that there are really only two data roles: System Builders and Storytellers. All of the other titles, from Data Scientist to Analytics Engineer, are really just variations of these same two themes, and there&rsquo;s one simple question to pick a path:&nbsp;do you love building systems, or telling stories?&nbsp;Or, in the famous words of Marie Kondo, what&nbsp;<a href="https://www.google.com/url?q=https://konmari.com/about-the-konmari-method/&amp;sa=D&amp;source=editors&amp;ust=1668008209109512&amp;usg=AOvVaw30sGD_YBy0ErIIcKCnmT-o">sparks joy</a>?</p>
    <p>📊&nbsp;Storytellers</p>
    <p>You can hoard terabytes of data, but there&rsquo;s no point if you aren&rsquo;t using it to make decisions. This is where Storytellers come in. Their fundamental mission is to use data to understand the world, and share that insight to influence others.&nbsp;Analysts, Data Scientists, and most other people using data every day fall into the &ldquo;Storytelling&rdquo; bucket.</p>
    <p>Here are some questions to figure out if you should choose a career on the Storytelling path:</p>
    <ul>
    <li>Do you feel satisfaction driving change by having the best argument, knowing the decisions you&rsquo;re framing help to bring clarity and focus to an organization/business/product?</li>
    <li>Do you love helping other people make the right decisions using data?</li>
    <li>Do you feel proud navigating narrative chaos to create clarity for others?</li>
    <li>Have the best moments in your data career come from using data to change someone&rsquo;s mind, modifying a team&rsquo;s roadmap, or shifting an org's strategy?</li>
    </ul>
    <p>If you&rsquo;re nodding and shouting 'yes!&rsquo;,&nbsp;you&rsquo;d be a great Analyst or Data Scientist&nbsp;- someone who finds decisions to be made, frames them for the right decision maker, and drives change based on the outcome.</p>
    <p>🛠&nbsp;System Builders</p>
    <p>Storytellers might build the pretty charts and write the good words, but there wouldn&rsquo;t be any data behind them if not for the System Builders. Their fundamental mission is to make integrated, reliable data sets available to others.&nbsp;Analytics Engineers, Data Engineers, and Data Product Managers fall into the &ldquo;Systems&rdquo; bucket.</p>
    <p>Here are some questions to figure out if you&rsquo;re a System Builder at heart:</p>
    <ul>
    <li>Do you feel proud navigating and abstracting away technical complexity to build something that empowers others?</li>
    <li>Do you feel satisfaction wrangling order from chaos?</li>
    <li>Do you love helping other people answer their quantitative questions?</li>
    <li>Have the best moments in your data career come from taking raw and unintelligible data sets and turning them into something that more removed stakeholders can consume?</li>
    </ul>
    <p>If so, you&rsquo;d be a great&nbsp;Analytics Engineer or Data Engineer&nbsp;- someone who builds systems and processes around data to enable operational excellence and decision making.</p>
    <p>Building a data team</p>
    <p>If you&rsquo;re a data and analytics leader, you can apply this same framing to clarify your roles both internally and externally.</p>
    <p>🙇&nbsp;Assess your current team</p>
    <p>I often see Systems folks being asked to spend time on one-off data pulls or other non-scalable work, and Storytellers not having a clear role in the decision-making process.</p>
    <p>Ask yourself:</p>
    <ul>
    <li>Are the people on my team spending a majority of their time on Systems or Storytelling today?</li>
    <li>What would each person on the team like to do more of, in an ideal world?</li>
    <li>Are the major bottlenecks to data team impact on the systems side (not having right data available) or storytelling (not driving analysis and decision-making)?</li>
    </ul>
    <p>✍️&nbsp;Realign your job descriptions</p>
    <p>Evaluate what your actual needs are, and adjust your JDs to focus on the applicable function and title. Make them clearer, in both title and detail, about what it is you&rsquo;re looking for.</p>
    <ul>
    <li>Is a role focused on creating scaled data systems, or digging into problems and bringing perspective to decision making?</li>
    <li>Are your JDs written in a way that implies both Systems and Storytelling responsibilities? This could alienate both archetypes.</li>
    </ul>
    <p>🐝&nbsp;Structure your team in a way that fosters cross-pollination and collaboration</p>
    <p>I don&rsquo;t recommend splitting your Storytellers and System Builders into two separate organizations &mdash; there&rsquo;s incredible value from having tight feedback loops between the two roles.</p>
    <p>At Hex, both Storytellers (Analysts) and System Builders (Analytics Engineers) report into BizOps. This way, Systems folks are getting the rapid feedback that will help them build more effectively, and the Storytellers are getting the data they need. They also work directly alongside our Revenue Operations folks, who both contribute to and leverage the Systems that our Analytics Engineers build.<a class="mds-text-primary" href="https://www.google.com/url?q=https://hex.tech/blog/storytellers-and-system-builders/%23fn-1&amp;sa=D&amp;source=editors&amp;ust=1668008209112791&amp;usg=AOvVaw1z5dj_kiqLxck5d1Wd0O_a">1</a></p>
    <p>Data teams of the future</p>
    <p>It&rsquo;s almost too obvious to write, but when people do things that make them happy,&nbsp;<a class="mds-text-primary" href="https://www.google.com/url?q=https://www.wellsteps.com/blog/2022/05/17/happy-employees-more-productive/&amp;sa=D&amp;source=editors&amp;ust=1668008209113351&amp;usg=AOvVaw3cVJR-hFAgnHXS0J_Mz07O">they&rsquo;re more productive</a>. If we start mapping career growth with this &ldquo;System Builder vs. Storyteller&rdquo; paradigm instead of arguing about the minutae of hip new titles, we will see efficiency gains and happier people across the board.</p>
    <p>Adopting this mindset at Hex has made our hiring cycles more effective, because candidates have more clarity around what they&rsquo;re applying for. Everyone in the org, not just the data team, knows what these mythical Data People do and how to engage with them. There is tighter alignment between what each member of the team does and the tasks that bring them joy.</p>
    <p>And that joy translates into greater productivity from these brilliant and rare Storytellers and System Builders.</p>
    <p>Try it out on your own team! I&rsquo;d really love to hear about how it goes:&nbsp;<a class="mds-text-primary" href="mailto:hello@hex.tech">hello@hex.tech</a>.</p>

Storytellers and System Builders

<p>Why this gap is relevant and how to close it</p>
    <p>Showing what&rsquo;s happening in dashboards is informative, not insightful. It&rsquo;s useful to know if a key business metric is going up or down, but it&rsquo;s not actionable. Only the &ldquo;why&rdquo; behind these changes can drive recommendations and actions.</p>
    <p>However, I often hear how teams look at dashboards in weekly/monthly reviews and comment that metrics went up or down, without clear answers to root causes or actionable insights. This leads to missed opportunities to drive business impact through data and analytics, hindering the agility needed to stay ahead of the competition.</p>
    <p>Why is this relevant?</p>
    <p>Analytics is still primarily descriptive in most organizations (i.e., WHAT&rsquo;s happening).</p>
    <p>Few data and business teams have nailed diagnostic analytics (i.e., WHY this happened).</p>
    <p>As external events affect the business and stakeholders take action, metrics change as a reflection. Businesses need to understand what drives KPIs and why they are changing to make better decisions. This can only be achieved by nailing diagnostic analytics: providing comprehensive insights into metrics changes at the speed of business.</p>
    <p>But oftentimes, stakeholders and data practitioners come up with hypotheses on why metrics changed based on their domain expertise, intuition, and experience. These reasons are typically high-level and just scratching the surface by testing 2 or 3 usual suspects, leading to missed opportunities.</p>
    <h3>The analytics landscape</h3>
    <p>On a high-level, diagnostic analytics is often overlooked as it lies between data and the business. It&rsquo;s helpful to compare it against the other types of analytics types.</p>
    <p><img style="width: 100%" title="" src="https://lh4.googleusercontent.com/wMeV-VK6O6jnrEf4MYCCZVm6ScOWmPctKGR2JuaJdSaCLvzlB2pADLUTYn5WyMAy1bW2XMOscBur-t0hhw1JVtzgahk3Yv7GFB43kNAxJ8BIIitMU4w-ThhYuhkxvdaUXK7FZoXt5SvDVpNceQdwHbbQp28Qcvqau6eqqk2U7UY2ZOgc2AT6hI5syWgUyA" alt="" /></p>
    <p class="text-muted">(Image by the author)</p>
    <p>● Descriptive analytics&nbsp;is self-served through dashboards. Data teams manage governance and infrastructure and enable business teams. There are clear processes, ownership, and tools.</p>
    <p>● Diagnostic analytics&nbsp;lies between the business and data. It&rsquo;s nearly impossible to self-serve diagnostic analytics through common dashboards (although many data teams fall into this fallacy). There aren&rsquo;t clear processes and the data culture varies significantly.</p>
    <p>● Predictive and prescriptive analytics&nbsp;are managed by the data teams with clear data products (e.g., recommendation systems, and forecasting models) used by the business folks. They are technically complex, but very clear concerning processes, people, and culture.</p>
    <p>What&rsquo;s causing this gap</p>
    <p>As data grows in volume and complexity and businesses change faster than ever, the gap between business needs and delivery keeps widening. Most organizations overlook this gap for various reasons. A deep understanding of the gap requires a holistic approach based on the 4 key elements: people, culture, tools, and processes.</p>
    <p>Let&rsquo;s go deeper into each one of the 4 key elements.</p>
    <p>&nbsp;<img title="" style="width: 100%" src="https://lh5.googleusercontent.com/Qt9MyR4Ibu7s4c8M6_04YJsoe6OVqmXj0EdYX4nuA0Peb-MCTGVfYePbC7vWbP1CmaeQb5Y-axnDp0kMNMpeAtaw-i2JrbxnBB6fqe3Iv4kkGJ05qVZJ0Q0uKTC-xn5Ic04mtrC3U4oU_fAXfHUOqmimrI7_-GcbLvWyEyHLHrwuiJk2p7CTdCmwDlQiMQ" alt="" /></p>
    <p class="text-muted">(Image by the author)</p>
    <h3>How to close this gap?</h3>
    <p>Ideally, business and data teams would proactively and regularly analyze metrics changes. Even if the change is minor, there are typically opposing effects. Identifying them is crucial so that teams can quickly differentiate what&rsquo;s working and what&rsquo;s not working and take actions to maximize business impact.</p>
    <p>Where to focus depends on the starting point. It requires a comprehensive overview of the 4 elements described before. Based on conversations with dozens of data and business leaders, I often see three scenarios:</p>
    <p>The 4 states of diagnostic analytics</p>
    <p><img title="" style="width: 100%" src="https://lh3.googleusercontent.com/Y8r3KHB_5Rxj5m_jeGCaq32LzCR5__k1VHx5cCPAmq9DAhxGPF66NWyjx-66IOD4syVz2d8BUviX64IcQTvd83sKGUqnkRQ9A_CEsOKEpEK1bKuZxBvKuujmsVXZqNlvJYp97mTARTWKePdW1--Lg5POCg_EiANQnDZa421hMAPCA6sV9IQgyXpooB6aNw" alt="" /></p>
    <p class="text-muted">(Image by the author)</p>
    <p><strong>State 1-&nbsp;Stuck in the &ldquo;what&rdquo;</strong></p>
    <p>Companies lack the data culture to drill down into the why. Teams monitor metrics on dashboards, but rarely identify data-driven reasons as to why they changed.</p>
    <p><span style="text-decoration: underline;">Actions</span>:&nbsp;Foster data culture and show the value of diagnostic analytics to business stakeholders</p>
    <p><span style="text-decoration: underline;">How</span>:</p>
    <ul>
    <li>Pick 2 or 3 use cases where you see significant potential to deliver business value. Proactively investigate changes in these business metrics and share your findings in a structured and impact-driven manner</li>
    <li>Continue until you achieve a couple of success stories. This should be the base to further expand into additional use cases and start the change management process</li>
    </ul>
    <p><strong>State 2 -&nbsp;The usual suspects</strong></p>
    <p>Teams end up only scratching the surface of WHY metrics change and only test a couple of usual suspects regularly.</p>
    <p>This is the tricky one because it lacks symptoms. Business stakeholders just test their high-level hypotheses, leading to missed opportunities. Data teams believe that they are doing a great self-service analytics job.</p>
    <p><span style="text-decoration: underline;">Actions</span>:&nbsp;Raise awareness that not drilling down to the why is leading to missed opportunities</p>
    <p><span style="text-decoration: underline;">How</span>:</p>
    <ul>
    <li>&nbsp;Pick 2 or 3 critical use cases where you know that business teams are only looking at the usual suspects. Proactively drill down to the why and share your findings by comparing comprehensive analysis vs previous analyses that were only scratching the surface<br /><br /></li>
    <li>Build out success stories and expand into more use cases</li>
    </ul>
    <p><strong>State 3 -&nbsp;Need for speed</strong></p>
    <p>Business is aware of the value of drilling down to the why but struggles with the speed-comprehensiveness trade-off.</p>
    <p>In this case, data culture is already advanced and teams are only missing the last mile of the marathon. Most teams are aware there is business value to be unlocked, but struggle to accelerate diagnostic analytics. Performing comprehensive root-cause analysis with existing workflows is too complex and time-consuming. Teams struggle to deliver insights at the speed of business.</p>
    <p><span style="text-decoration: underline;">Action</span>: Augment current workflows to accelerate speed to insight and eliminate the speed - comprehensiveness trade-off.</p>
    <p><span style="text-decoration: underline;">How</span>:</p>
    <ul>
    <li>Consider augmented analytics platforms to accelerate speed to insight and eliminate the trade-off. These tools leverage ML to automate hypothesis testing and driver scoring. This way teams can get insights at the speed of business and you can scale analytics without throwing more analysts at the problem.<br /><br /></li>
    <li>Review current diagnostic analytics procedures and ensure a proper way to conduct these analyses, share insights and organize them.<br /><br /></li>
    <li>Increase the collaboration between data analysts and the business for the most important use cases to develop domain expertise</li>
    </ul>
    <p><strong>State 4 -&nbsp;Full Force</strong></p>
    <p>Teams proactively diagnose metrics changes for constant improvement. By augmenting traditional workflows teams eliminate the speed- comprehensiveness trade-off, share actionable insights proactively and capture a wide range of opportunities.</p>
    <p>This state can be showcased as a best practice. The teams work collaboratively and proactively, processes are well-defined and clear, and the right set of tools are implemented to drill down to the why and find actionable insights. As a result, the ROI of data analytics is clear and there is strong trust in the data culture.</p>
    <p>&mdash;</p>
    <p><strong>Conclusion</strong>:</p>
    <p>Showing what&rsquo;s happening in dashboards is not enough. Drilling down to the why comprehensively and quickly is the only way to deliver actionable insights that increase the value of analytics. This is the way to conquer the last mile of the data journey.</p>
    <p>&ndash;</p>
    <p>Thoughts? Reach out to<a class="mds-text-primary" href="https://www.google.com/url?q=https://www.linkedin.com/in/joaoantoniosousa/&amp;sa=D&amp;source=editors&amp;ust=1667399433078351&amp;usg=AOvVaw3hwpUa9O2U0kyQ29HissKf">Jo&atilde;o Sousa</a>. Joao is the Director of Growth at Kausa.</p>

The diagnostic analytics gap

<p><a class="mds-text-primary" href="https://blog.getdbt.com/future-of-the-modern-data-stack/">The modern data stack</a>&nbsp;has helped democratize the creation, processing, and analysis of data across organizations. However, it has also led to&nbsp;<a class="mds-text-primary" href="https://benn.substack.com/p/the-modern-data-experience">a new set of challenges</a>&nbsp;thanks to the decentralization of the data stack. In this post, we&rsquo;ll discuss one of the cornerstones of the modern data stack&mdash;data catalogs&mdash;and why they fall short of overcoming the fragmentation to deliver a fully self-served data discovery experience.</p>
    <p>If you are a leaders of the data team at a company with 200+ employees, there is a high probability that you have</p>
    <ul role="list">
    <li>started seeing data discovery issues at your company;</li>
    <li>tried one of the commercial or open-source data catalogs; or</li>
    <li>cobbled together an in-house data catalog.</li>
    </ul>
    <p>If that&rsquo;s the case, you&rsquo;d definitely find this post highly relatable.</p>
    <h3><strong>Pain Points</strong></h3>
    <p>This post is based on our own experience of building&nbsp;<a class="mds-text-primary" href="https://engineering.linkedin.com/blog/2019/data-hub">DataHub</a>&nbsp;at LinkedIn and the learnings from 100+ interviews with data leaders and practitioners at various companies. There may be many reasons why a company adopts a data catalog, but here are the pain points we often come across:</p>
    <ul role="list">
    <li>Your data team is spending a lot of time answering questions about where to find data and what datasets to use.</li>
    <li>Your company is making bad decisions because data is inconsistent, poor in quality, delayed, or simply unavailable.</li>
    <li>Your data team can't confidently apply changes to, migrate, or deprecate data because there&rsquo;s no visibility into how the data is being used.</li>
    </ul>
    <p>The bottom line is that you want to empower your stakeholders to self-serve the data, and more importantly, the&nbsp;<em>right</em>&nbsp;data. The data team doesn't want to be bogged down by&nbsp;<em>support questions</em>&nbsp;as much as data consumers don't want to depend on the data team to answer their questions. Both of them share a common goal&mdash;True Self-service Data Discovery&trade;.</p>
    <h3><strong>First Reaction</strong></h3>
    <p>In our research, we saw striking similarities in companies attempting to solve this problem themselves. The story often goes like this:</p>
    <li>Create a database to store metadata.</li>
    <li>Collect important metadata, such as schemas, descriptions, owners, usage, and lineage, from key data systems.</li>
    <li>Make it searchable through a web app.</li>
    </ol>
    <p>Voila! You now have a full self-service solution and proudly declare victory over all data discovery problems.<strong>&zwj;</strong></p>
    <h3><strong>Initial Excitement</strong>&zwj;</h3>
    <p>Let&rsquo;s walk through what typically happened after this shiny new data catalog was introduced. It looked great on first impression. A handful of power users were super excited about the catalog and its potentials. They were thrilled about their newfound visibility into the whole data ecosystem and the endless opportunities to explore new data. They were optimistic that this was indeed<em>The Solution</em>&nbsp;they&rsquo;ve been looking for.<strong>&zwj;</strong></p>
    <h3><strong>Reality Sets In</strong>&zwj;</h3>
    <p>A few months after launching, you started noticing that the user engagement waned quickly. Customer&rsquo;s questions in your data team&rsquo;s Slack channel didn&rsquo;t seem to go away either. If anything, they became even harder for the team to answer.</p>
    <p>&zwj;So what happened?</p>
    <ul role="list">
    <li>People searched &ldquo;revenue&rdquo; hoping to find the official revenue dataset. Instead, they got 100s of similarly named results, such as &ldquo;revenue&rdquo;, &ldquo;revenue_new&rdquo;, &rdquo;revenue_latest&rdquo;, &ldquo;revenue_final&rdquo;, &ldquo;revenue_final_final&rdquo;, and were at a complete loss.</li>
    <li>Even if the person knew the exact name of what they were looking for, the data catalog only provided&nbsp;<em>technical information</em>, e.g. SQL definition, column descriptions, linage, and data profile, without any explicit instructions on how to use it for a specific use case.</li>
    <li>Your data team has painstakingly tagged datasets as "core", "golden", "important", etc., but the customers didn't know what these tags mean or their importance. Worse yet, they started tagging things randomly and messed up the curation effort.</li>
    </ul>
    <p>Is it really that hard to find the right data even with such advanced search capabilities and all the rich metadata?&nbsp;<em>Yes!</em>&nbsp;Because the answer to &ldquo;what&rsquo;s the right data&rdquo; depends on who you are and what use cases you&rsquo;re trying to solve. Most data catalogs only present the information from the producer&rsquo;s point of view but fail to cater to the data consumers.</p>
    <h3><strong>The Missing Piece</strong></h3>
    <p>Providing the producer&rsquo;s point of view through automation and integration of all the technical metadata is definitely a key part of the solution. However, the consumer&rsquo;s point of view&mdash;trusted tables used by my organization, common usage patterns for various business scenarios, impact from upstream changes have on my analyses&mdash;is the missing piece that completes the data discovery &amp; understandability puzzle.</p>
    <p>Most of the data catalogs don't help users find the data they need; they help users find someone to pester, which often referred to as a &ldquo;tap on the shoulder&rdquo;. This is not true self-service.</p>
    <h3><strong>The Solution</strong></h3>
    <p>We believe that there are three types of information/metadata required to make data discovery truly self-serviceable:</p>
    <ol role="list">
    <li><strong>Technical Metadata</strong>: This refers to all metadata originating from the data systems, including schemas, lineage, SQL/code, description, data profile, data quality, etc. Automation and integration would keep the information at the user&rsquo;s fingertips.<br /><strong>Challenges</strong>: There is no standard for metadata across data platforms. Worst yet, many companies build their own custom systems that hold or produce key metadata. How to integrate these systems at scale to ingest metadata accurately, reliably, and timely is an engineering challenge.</li>
    </ol>
    <ol role="list" start="3">
    <li><strong>Business Metadata</strong>: Each business function operates based on a set of common business definitions, often referred to as &ldquo;business terms&rdquo;. Examples include Active Customers, Revenue, Employees, Churn, etc. As a data-driven organization relies heavily on these definitions to make key business decisions, it is paramount for data practitioners to correctly translate between the physical data and business terms.<br />&zwj;<strong>Challenges</strong>: Many companies lack the tools, processes, and disciplines to govern and communicate these business terms. As a result, when serving a business ask, data practitioners often struggle to find the right data for a particular business term or end up producing results that contradict each other.</li>
    <li><strong>Behavioral Metadata</strong>: Surfacing the association between people and data is critical to effective data discovery. Users often place their trust in data based on who created or used it. They also prefer to learn how to do their analyses from more experienced &ldquo;power users&rdquo;. To that end, we need to&nbsp;<a class="mds-text-primary" href="https://locallyoptimistic.com/post/share-your-data-insights-to-engage-your-colleagues/">encourage</a>&nbsp;the sharing of these data learnings/insights across the company. This would also improve your organization&rsquo;s data literacy, provide a better understanding of the business, and reduce inconsistencies.<br />&zwj;<strong>Challenges</strong>: People interact with the data in different ways. Some query using Snowflake console, Notebooks, R, and Presto, while others explore using BI tools, dashboards, or even spreadsheets. As a result, the learnings and insights often spread across multiple places and make it difficult to associate people with data.</li>
    </ol>
    <p>It should be fairly clear by now that discovering the right data and understanding what it means is not a mere technical problem. It requires bringing technical, business, and behavioral metadata together. Doing this without creating an onerous governance process will boost your organization&rsquo;s data productivity significantly and bring true data-driven culture to your company.</p>

Why Can't I Find the Right Data?

<p>This is a long due post, predominantly because there is a lot of confusion around data lineage , data observability and the interdependencies.</p>
      <p>As many of us know, Data Lineage is one of the most discussed topics today and so is data observability. This article covers the applicability of data lineage in Data Observability.</p>
      <p>PS: If you are specifically looking to understand lineage better, these are few of my favorites on this topic.</p>
      <p>Building Data Lineage: <a class="mds-text-primary" href="https://medium.com/netflix-techblog/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977">netflix</a> &amp; Leveraging Apache spark for Data Lineage : <a class="mds-text-primary" href="https://engineeringblog.yelp.com/2022/08/spark-data-lineage.html">Yelp</a></p>
      <h3><strong>What is Data Lineage?</strong></h3>
      <p>Data lineage tracks the changes and transformations impacting any data record. Data lineage tells us where data is coming from, where it is going, and what happens as it flows from data sources and pipeline workflows to downstream data marts and dashboards.</p>
      <p>In short, it enables better data governance by giving you a more complete, top-down picture of your data and analytics ecosystem.</p>
      <h3><strong>Why is it so important for data quality?</strong></h3>
      <p>Data lineage is essential for understanding the health of the data because it ensures transparency around the source of data, ownership, access, and its transformation. These are excellent indicators of the reliability and trust of the data. However unlike other data quality indicators like <strong>Completeness, Validity, Accuracy, Consistency, Uniqueness, and Timeliness</strong>.(Read <a class="mds-text-primary" href="https://docs.telm.ai/academy/">here</a>) Data lineage can not be metricized.</p>
      <p>The insights provided by Lineage enable data users to solve all kinds of problems encountered within mass quantities of interconnected data like Data troubleshooting, Impact analysis, Discovery and trust, Data privacy regulation (GDPR and PII mapping), and Data asset cleanup/technology migration, and Data valuation.</p>
      <h3><strong>What is Data Observability?</strong></h3>
      <p>Data Observability is a set of measures that can help predict, identify, and resolve the data issues. This is often done by leveraging statistical analysis and machine learning.</p>
      <p>Data Observability aims to reduce the mean time to detect (MTTD) and mean time to resolve (MTTR) data issues.</p>
      <h3><strong>Why is Lineage important for data Observability?</strong></h3>
      <p>I MG SRC : Telm.ai</p>
      <p>Data lineage can be very effective in getting good Observability outcomes.The insights provided by Lineage enable data users to quickly find the root cause of issues , conduct impact analysis and also invoke a remediation.</p>
      <p>So your data observability should leverage lineage information.</p>
      <h3><strong>How is data lineage used in Data Observability?</strong></h3>
      <p>Today, most data observability companies leverage Data Lineage for mean time to resolve (MTTR) issues by doing two things,</p>
      <p><strong>Root cause analysis (RCA)</strong></p>
      <p>Once a monitoring system detects an issue, Lineage can help investigate the previous steps in the pipeline and changes. Specifically, if you are using Datawarehouse monitoring tools like Montecarlo data, Metaplane etc you can find table/view lineage that can help shrink the root cause analysis time.</p>
      <p><strong>Downstream Impact Analysis</strong></p>
      <p>Often Data Lineage can help find downstream impact and invoke remediation flow. Remediation by definition is an reactive approach and can become a nightmare.</p>
      <p>If you have a datamesh architecture, and have multiple consuming products using the data, leveraging lineage helps you identify which products are impacted and who own those products/systems. Now you can systematically notify those users. Another use case is using Lineage where you can identify the impacted dashboards.</p>
      <p>The above methods are <strong>definitely useful</strong> but still suboptimal for data observability.</p>
      <p>Data Lineage for RCA and Impact analysis</p>
      <p>&zwj;</p>
      <h3><strong>What is the correct usage of Lineage in Observability?</strong></h3>
      <p>Data Observability, by definition, should be <strong>proactive</strong>. Data engineers should use <strong>Lineage to check downstream impact</strong> before adding any changes. For Example, am I making schema changes? Let me automatically check the downstream implications.</p>
      <p>For data in motion a full pipeline monitoring approach will not only help with (resolve) MTTR but also MTTD(detect) issues. Design a metric monitoring system that monitors every step in the data pipeline. So users get alerted when there is a drift or outlier in a specific step of the pipeline, and multiple data points will help detect issues even before they have downstream impact.</p>
      <p>Example : Get alerted that the CRM system got partial data at 2pm PST.Such an alert can automatically be sent to the CRM system owner, so data engineers don't have to find out about these issues inside Snowflake or BigQuery and reverse track them back to the CRM system. Often this approach will also help orchestrate the date pipeline flow refer: <a class="mds-text-primary" href="https://www.telm.ai/blog/who-has-control-of-your-data-pipeline">Circuit breaker pattern</a>.&nbsp;</p>
      <p>A good analogy would be monitoring vs. tracing and logs. If you have a lot of monitoring, you may not need tracing and log review because tracking at every level will highlight the same problems but sooner. This is because all methods highlight the same issues just from different angles. So I would say that Lineage is a good tool for root cause analysis and impact analysis (MTTR) but your observability tools should focus higher on for mean time to detect (MTTD) issues i.e be more proactive.</p>
      <h3><strong>Summary</strong></h3>
      <p>Data Lineage is a crucial aspect of Data reliability. However, to effectively reduce both the MTTD and MTTR data issues :</p>
      <p>1: Leverage lineage to understand the downstream impact <strong>before</strong> making changes.</p>
      <p>2: Monitor important data metrics every step of the pipeline and leverage Lineage to identify exactly where(step of the pipeline) the which metric has drifted.</p>
      <p>This approach enables users to reduce both time to detect and time to resolve data issues.</p>
      <p>&zwj;</p>
      

How important is data lineage in terms of data observability ?

What Is Data Lineage?

<div>
      <div>
      <div>
      <p>Data matters more than ever &ndash; we all know that. But at a time when being a data-driven business is so critical, how much can we trust data and what it tells us? That&rsquo;s the question behind data reliability, which focuses on having complete and accurate data that people can trust. This article will explore everything you know about data reliability and the important role of data observability along the way, including:</p>
      <ul>
      <li>What is data reliability?</li>
      <li>Why is it important?</li>
      <li>How do you measure it?</li>
      <li>Data quality vs. data reliability: What&rsquo;s the difference?</li>
      <li>What is a data quality framework?</li>
      <li>How can observability help improve data reliability?</li>
      <li>Top data reliability testing tools</li>
      </ul>
      </div>
      </div>
      <h3>What is data reliability?</h3>
      <div>
      <div>
      <p>Data reliability looks at the completeness and accuracy of data, as well as its consistency across time and sources. The consistency piece is particularly important, as data needs to be consistent to be truly reliable, that way it&rsquo;s always trustworthy.</p>
      <p>Data reliability is one element of data quality.</p>
      <p>Specifically, it helps build trust in data. It&rsquo;s what allows us to make data-driven decisions and take action confidently based on data. The value of that trust is why more and more companies are introducing Chief Data Officers &ndash; with the number doubling among the top publicly traded companies between 2019 and 2021,&nbsp;<a class="mds-text-primary" href="https://www.pwc.com/us/en/tech-effect/ai-analytics/building-trust-in-data.html" target="_blank" rel="noopener">according to PwC</a>.</p>
      </div>
      </div>
      <h3>Why is data reliability important?</h3>
      <div>
      <div>
      <p>Measuring data reliability requires looking at three core factors:</p>
      <ol>
      <li><strong>Is it valid?&nbsp;</strong>Validity of data looks at whether or not it&rsquo;s stored and formatted in the right way. This is largely a data quality check.</li>
      <li><strong>Is it complete?&nbsp;</strong>Completeness of data identifies if anything is missing from the information. While data can be valid, it might still be incomplete if critical fields are not present that could change someone&rsquo;s understanding of the information.</li>
      <li><strong>Is it unique?&nbsp;</strong>The uniqueness of data checks for any duplicates in the data set. This uniqueness is important to avoid over-representation, which would be inaccurate.</li>
      </ol>
      <p>To take it one step further, some teams also consider factors like:</p>
      <ul>
      <li>If and when the data source was modified</li>
      <li>What changes were made to data</li>
      <li>How often the data has been updated</li>
      <li>Where the data originally came from</li>
      <li>How many times the data has been used</li>
      </ul>
      <p>Overall, measuring data reliability is essential to not just help teams trust their data, but also to identify potential issues early on. Regular and effective data reliability assessments based on these measures can help teams quickly pinpoint issues to determine the source of the problem and take action to fix it. Doing so makes it easier to resolve issues before they become too big and ensures organizations don&rsquo;t use unreliable data for an extended period of time.</p>
      </div>
      </div>
      <h3>Data quality vs. data reliability: What&rsquo;s the difference?</h3>
      <div>
      <div>
      <p>All of this information begs the question: What&rsquo;s the difference between&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/what-is-good-data-quality-for-data-engineers/" target="_blank" rel="noreferrer noopener">data quality</a>&nbsp;vs. data reliability?</p>
      <p>Quite simply, data reliability is part of the bigger data quality picture. Data quality takes on a much bigger focus than reliability, looking at elements like completeness, consistency, conformity, accuracy, integrity, timeliness, continuity, availability, reproducibility, searchability, comparability, and &ndash; you guessed it &ndash; reliability.</p>
      <p>For&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/whats-the-difference-data-engineer-vs-data-scientist-vs-analytics-engineer/" target="_blank" rel="noreferrer noopener">data engineers</a>, there are typically four data quality dimensions that matter most:</p>
      <ul>
      <li><strong>Fitness:&nbsp;</strong>Is the data fit for its intended use, which considers accuracy and integrity throughout its lifecycle.</li>
      <li><strong>Lineage:&nbsp;</strong>Where and when did the data come from and where did it change, which looks at source and origin.</li>
      <li><strong>Governance:&nbsp;</strong>Can you control the data, which takes into account what should and shouldn&rsquo;t be controllable and by whom, as well as privacy, regulations, and security.</li>
      <li><strong>Stability:&nbsp;</strong>Is the data complete and available in the right frequency, which includes consistency, dependability, timeliness, and bias.</li>
      </ul>
      <p>Fitness, lineage, and stability all have elements of data reliability throughout them. Although taken as a whole, data quality clearly encompasses a much larger picture than data reliability.</p>
      </div>
      </div>
      <h3>What is a data quality framework?</h3>
      <div>
      <div>
      <p>A&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/data-quality-framework/" target="_blank" rel="noreferrer noopener">data quality framework</a>&nbsp;allows organizations to define relevant data quality attributes and provide guidance for processes to continuously ensure data quality meets expectations. For example, using a data quality framework can build trust in data by ensuring what team members view is always accurate, up to date, ready on time, and consistent.</p>
      <p>A good data quality framework is actually a cycle, which typically involves six steps largely led by data engineers:</p>
      <ol>
      <li><strong>Qualify:&nbsp;</strong>Understand a list of requirements based on what the end consumers of the data need.</li>
      <li><strong>Quantify:&nbsp;</strong>Establish quantifiable measures of data quality based on the list of requirements.</li>
      <li><strong>Plan:&nbsp;</strong>Build checks on those data quality measures that can run through a data observability platform.</li>
      <li><strong>Implement:&nbsp;</strong>Put the checks into practice and test that they work as expected.</li>
      <li><strong>Manage:&nbsp;</strong>Confirm the checks also work against historical pipeline data and, if so, put them into production.</li>
      <li><strong>Verify:&nbsp;</strong>Check with data engineers and data scientists that the work has improved performance and delivers the desired results, and check that the end consumers of the data are getting what they need.</li>
      </ol>
      </div>
      </div>
      <h3>How can observability help improve data reliability?</h3>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://databand.ai/data-observability/" target="_blank" rel="noreferrer noopener">Data observability</a>&nbsp;is about understanding the health and state of data in your system. It includes a variety of activities that go beyond just describing a problem. Data observability can help identify, troubleshoot, and resolve data issues in near real-time.</p>
      <p>Importantly, data observability is essential to getting ahead of&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/the-impact-of-bad-data-and-why-observability-is-now-imperative/" target="_blank" rel="noreferrer noopener">bad data</a>&nbsp;issues, which sit at the heart of data reliability. Looking deeper, data observability encompasses activities like monitoring, alerting, tracking, comparisons, analyses, logging, and SLA tracking, all of which work together to understand end-to-end data quality &ndash; including data reliability.</p>
      <p>When done well, data observability can help improve data reliability by making it possible to identify issues early on to respond faster, understand the extent of the impact, and restore reliability faster as a result of this insight.</p>
      </div>
      </div>
      <h3>Top data reliability testing tools</h3>
      <div>
      <div>
      <p>Understanding the importance of data reliability, how it sits within a broader data quality framework, and the importance of data observability is a critical first step. Next, taking action to invest in it requires the right technology.</p>
      <p>With that in mind, here&rsquo;s a look at the top data reliability testing tools available to data engineers. It&rsquo;s also important to note that some of these solutions are often referred to as data observability tools since better observability leads to better reliability.</p>
      </div>
      </div>
      <h4>1) Databand</h4>
      <div>
      <div>
      <p>Databand is a&nbsp;<a class="mds-text-primary" href="https://databand.ai/platform/" target="_blank" rel="noreferrer noopener">data observability platform</a>&nbsp;that helps teams monitor and control data quality by isolating and triaging issues at their source. With Databand, you can know what to expect from data by identifying trends, detecting anomalies, and visualizing data reads. This allows a team to easily alert the right people in real time about issues like missing data deliveries, unexpected data schemes, and irregular data volumes and sizes.</p>
      </div>
      </div>
      <h4>2) Datadog</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://www.datadoghq.com/" target="_blank" rel="noopener">Datadog&rsquo;s</a>&nbsp;observability platform provides visibility into the health and performance of each layer of your environment at a glance. It allows you to see across systems, apps, and services with customizable dashboards that support alerts, threat detection rules, and AI-powered anomaly detection.</p>
      </div>
      </div>
      <h4>3) Great Expectations</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://greatexpectations.io/" target="_blank" rel="noopener">Great Expectations</a>&nbsp;offers a shared, open standard for data quality. It makes data documentation clean and human-readable, all with the goal of helping data teams eliminate pipeline debt through data testing, documentation, and profiling.</p>
      </div>
      </div>
      <h4>4) New Relic</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://newrelic.com/platform/applied-intelligence" target="_blank" rel="noopener">New Relic&rsquo;s</a>&nbsp;data observability platform offers full-stack monitoring of network infrastructure, applications, machine learning models, end-user experiences, and more, with AI assistance throughout. They also have solutions specifically geared towards AIOps observability.</p>
      </div>
      </div>
      <h4>5) Bigeye</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://www.bigeye.com/" target="_blank" rel="noopener">Bigeye</a>&nbsp;offers a data observability platform that focuses on monitoring data, rather than data pipelines. Specifically, it monitors data freshness, volume, formats, categories, outliers, and distributions in a single dashboard. It also uses machine learning to set forecasting for alert thresholds.</p>
      </div>
      </div>
      <h4>6) Datafold</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://www.datafold.com/data-observability" target="_blank" rel="noopener">Datafold</a>&nbsp;offers data reliability with features like regression testing, anomaly detection, and column-level lineage. They also have an&nbsp;<a class="mds-text-primary" href="https://github.com/datafold/data-diff/" target="_blank" rel="noopener">open-source</a>&nbsp;command-line tool and Python library to efficiently diff rows across two different databases.</p>
      <p>In addition to these five tools, others available include PagerDuty, Datafold, Monte Carlo, Cribl, Soda, and Unravel.</p>
      </div>
      </div>
      <h3>Make Data Reliability a Priority</h3>
      <div>
      <div>
      <p>The risks of bad data combined with the competitive advantages of quality data mean that data reliability must be a priority for every single business. To do so, it&rsquo;s important to understand what&rsquo;s involved in assessing and improving reliability (hint: it comes down in large part to data observability) and then to set clear responsibilities and goals for improvement.</p>
      </div>
      </div>
      </div>

What Is Data Reliability And How Observability Can Help ?

<div>
      <div>
      <div>
      <p>Data is the most valuable asset for most businesses today. Or at least it has the potential to be. But to realize the full value, organizations must manage their data correctly. This management covers everything from how it&rsquo;s collected to how it&rsquo;s maintained and analyzed. And a big component of that is data governance.</p>
      <p>Data governance refers to the policies, processes, roles, and technology that businesses use to ensure data availability, usability, integrity, and security. This article will explore everything you need to know about data governance, including:</p>
      <ul>
      <li>What is it?</li>
      <li>What&rsquo;s the difference between data governance vs. data management?</li>
      <li>Why is it important?</li>
      <li>What are the components of effective data governance?</li>
      <li>What are the key roles involved in it?</li>
      <li>What are its best practices?</li>
      </ul>
      </div>
      </div>
      <h3>What is Data Governance?</h3>
      <div>
      <div>
      <p>Data governance is a core component of any big data management strategy that organizations introduce to drive insights. Effective data governance ensures quality and consistency in the data used to power critical business decisions.</p>
      <p>At a high level, it can refer to data roles and responsibilities, data accessibility, data policies and processes, data creation procedures, data flows, and more. Digging deeper, it defines the architecture for decision-making and access rights around data, answering questions like:</p>
      <ul>
      <li>How do we define data?</li>
      <li>Where does data come from?</li>
      <li>How do we confirm the quality of data?</li>
      <li>How do we use data?</li>
      <li>Where do we store data?</li>
      <li>How do we protect data?</li>
      <li>How do we organize data?</li>
      <li>How do we connect data across systems?</li>
      <li>How do we maintain a current inventory of our data?</li>
      <li>How accurate does that data inventory need to be?</li>
      </ul>
      <p>Software for data governance can either be purpose-built or baked into applications that make up the<a class="mds-text-primary" href="https://www.kdnuggets.com/2021/11/modern-data-stack-leaving-behind.html" target="_blank" rel="noopener">&nbsp;modern data stack</a>.</p>
      </div>
      </div>
      <h3>What&rsquo;s the Difference Between Data Governance vs. Data Management?</h3>
      <div>
      <div>
      <p>Data governance and data management are often used interchangeably, however, the two terms refer to different practices.</p>
      <p>It sets the strategy by introducing policies and procedures throughout the data lifecycle. Meanwhile, data management is the practice of enforcing those policies and procedures so that the data is ready for use.</p>
      <p>In short, it is the cornerstone of all data management initiatives.</p>
      </div>
      </div>
      <h3>Why is Data Governance Important?</h3>
      <div>
      <div>
      <p>In today&rsquo;s data-driven world, organizations need effective data governance to be able to trust in the quality and consistency of their data.</p>
      <p>A strong approach to data governance benefits the entire organization by giving individuals a clear way to access data, shared terminology to discuss data, and a standard way to understand data and make it meaningful.</p>
      <p>Some of the key benefits of data governance include:</p>
      <ul>
      <li>Introducing a clear&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/data-quality-framework/" target="_blank" rel="noreferrer noopener">data quality framework</a>&nbsp;to bring together data and create a shared understanding for better insights and decisions</li>
      <li>Improving consistency of data across systems and processes, for efficient&nbsp;<a class="mds-text-primary" href="https://databand.ai/mad-data-podcast/modern-data-love-why-data-observability-data-integration-belong-together/" target="_blank" rel="noreferrer noopener">data integration</a></li>
      <li>Clearly defining policies and procedures around data-related activities to ensure standardization across the entire organization</li>
      <li>Outlining roles and responsibilities in terms of data management and data access for clarity among stakeholders</li>
      <li>Improving compliance by allowing for faster response and resolution to data incidents</li>
      </ul>
      <p>On the flip side, poor data governance can hamper regulatory compliance initiatives, which can create problems for companies when it comes to satisfying new data privacy and protection laws.</p>
      </div>
      </div>
      <h3>What are the Components of Effective Data Governance?</h3>
      <div>
      <div>
      <p>In order for it to be effective, it must encompass several key components that support the follow-on data management activities. These components include:</p>
      <h4>Data Standards</h4>
      <p>It should set explicit data standards for consistency across the entire organization. These standards should assess and verify data quality and should be transparent to everyone in the company. As a result, they should help teams better comprehend and use data.</p>
      <p>Data standards should also allow any third-party auditors to easily see how the organization handles sensitive data, how that data gets used, and why it gets used in that way. This transparency is essential for compliance, especially in the case of a data breach.</p>
      <h4>Data Integration</h4>
      <p><a class="mds-text-primary" href="https://databand.ai/blog/what-is-data-integration/" target="_blank" rel="noreferrer noopener">Data integration</a>&nbsp;brings together data from diverse sources to make data more readily available and power deeper insights. Good data governance requires a complete understanding of how data gets integrated across systems and processes. Specifically, the data governance program should define the tools, policies, and procedures used to pass data across systems and combine information.</p>
      <p>As a best practice, these data integration guidelines should be clear and easy to follow to ensure every new system adheres to them. Additionally, the team responsible for data governance should assist in reviewing these guidelines during any new technology implementations.</p>
      <h4>Data Security</h4>
      <p>Protecting the security of data is essential, as any unauthorized access to data or even loss of data can pose serious risks &ndash; from dangers to the subjects of data to financial loss to reputational damage. A data governance framework outlines a variety of elements related to data security, including where data is stored, how it&rsquo;s accessed, and what level of availability it has.</p>
      <p>Specifically, it should detail defenses like authentication tools and encryption algorithms that need to be implemented to protect the data network. Then, any teams working on data governance should partner closely with IT security to ensure adequate protection measures are in place based on those guidelines.</p>
      <h4>Data Lifecycle Management</h4>
      <p>Understanding the organization&rsquo;s data lifecycle means knowing where data resides at any given time as it moves through systems until it eventually gets discarded. Good data governance allows you to quickly discover and isolate data at any point in the lifecycle.</p>
      <p>This concept, also known as data lineage, allows analysts to trace data back to its source to confirm trustworthiness.</p>
      <h4>Data Observability</h4>
      <p><a class="mds-text-primary" href="https://databand.ai/data-observability/" target="_blank" rel="noreferrer noopener">Data observability</a>&nbsp;allows you to understand the health and state of data in your system to identify and resolve issues in near real-time. It includes a variety of activities that go beyond just describing the problem, providing context to also resolve the problem and work to prevent it from recurring.</p>
      <p>Data governance helps set the framework for data observability, setting guidelines for what to monitor and when and what thresholds should set off alerts when something isn&rsquo;t right. A good&nbsp;<a class="mds-text-primary" href="https://databand.ai/platform/" target="_blank" rel="noreferrer noopener">data observability platform</a>&nbsp;can handle these activities, making it important to choose a platform that can meet the requirements for identifying, troubleshooting, and resolving problems outlined in your strategy.</p>
      <h4>Metadata Management</h4>
      <p>Another critical component of data governance is metadata management, which focuses on maintaining consistent definitions of data across systems. This consistency is important to ensure data flows smoothly across integrated solutions and that everyone has a shared understanding of the data.</p>
      <p>The framework should include details on data definition, data security, data usage, and data lineage. In doing so, it should make it possible to clearly identify and classify all types of data in a standardized way across the organization.</p>
      <h4>Data Stewardship</h4>
      <p>Data stewardship is the practice that guarantees your organization&rsquo;s data is accessible, usable, secure, and trustworthy. While the data governance strategy determines your organization&rsquo;s goals, risk tolerance, security standards, and strategic data needs to set high-level policies, data stewardship focuses on making sure those policies get implemented.</p>
      <p>To achieve this follow-through, data stewardship assigns clear roles and responsibilities for various initiatives outlined in the strategy.</p>
      </div>
      </div>
      <h3>What are the Key Roles Involved in Data Governance?</h3>
      <div>
      <div>
      <p>Data governance programs can only succeed if they have clearly defined roles and responsibilities. As a result, it&rsquo;s important to identify the right people within your organization to take on this ownership and establish their roles in the program.</p>
      <p>Specifically, every data governance program requires people in three critical roles, each of which must be filled with qualified individuals who understand their specific responsibilities and how they contribute to the bigger picture. These roles include:</p>
      <h4>Chief Data Officer</h4>
      <p>The Chief Data Officer is the data governance leader. This person is responsible for overseeing the entire program, including enforcing and implementing all policies and procedures and leading the data committee and data stewards.</p>
      <h4>Data Committee</h4>
      <p>The data committee is a group of individuals that sets data governance policies and procedures, including rules for how data gets used and who can access it. They also resolve any disputes that arise regarding data usage or its role within the organization. The committee&rsquo;s purpose is to promote&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/what-is-good-data-quality-for-data-engineers/" target="_blank" rel="noreferrer noopener">data quality</a>&nbsp;and ensure that data owners and data stewards have what they need at every point in the data lifecycle to do their jobs effectively.</p>
      <h4>Data Stewards</h4>
      <p>The data stewards are responsible for carrying out the data governance policies set by the data committee. They oversee data, making sure everything adheres to policies throughout the entire data lifecycle from creation to archival. The data stewards also train new staff on policies.</p>
      <p>In some cases, data stewards might also be the data owners. In other cases, those might be two separate groups. Either way, the data owners are the people who manage the systems that create and house data.</p>
      </div>
      </div>
      <h3>What are Data Governance Best Practices?</h3>
      <div>
      <div>
      <p>When it comes to getting data governance off the ground (or improving what your organization already has in place) there are several best practices to consider:</p>
      <h4>Get Buy-In from the Top</h4>
      <p>As with any initiative, buy-in for data governance needs to start at the top. This top-down buy-in is important to make sure that everyone in the organization adheres to data governance policies and that those who are in a position to influence that acceptance understand the importance of your work.</p>
      <p>To achieve this buy-in, share with executives how your data governance plan can help them realize their strategic objectives. The more you can highlight the advantages of the program and how it relates to their work, the easier it will be.</p>
      <h4>Communicate Often</h4>
      <p>Communication beyond top-level executives is essential to effective data governance. To ensure everyone is aware of what your team is doing around data governance and why it matters, make a list of everyone in the organization who has a stake in or would be affected by that work.</p>
      <p>Then establish regular communications to share updates about program changes, roadblocks, and successes, that way everyone knows where to go for updates and can stay informed on a regular basis.</p>
      <h4>Combine Long-Term Goals with Short-Term Gains</h4>
      <p>When it comes to data governance, you won&rsquo;t be able to tackle everything at once. Instead, it should be a continuous effort to support data-driven decision-making and open up new opportunities for people throughout the organization.</p>
      <p>As a result, your long-term plan needs to include smaller, short-term initiatives that you can weave into the day-to-day operations of your company for immediate wins. This approach ensures that you see progress quickly and can help uncover any potential roadblocks faster. It also opens the door to new ideas that can even improve your long-term plan.</p>
      <h4>Assign Clear Responsibility &ndash; and Train People Accordingly</h4>
      <p>You can&rsquo;t simply assign someone the role of data steward and hope for the best. You need to make sure that anyone playing a role in your data governance program takes their part seriously, and that means you need to take their responsibilities just as seriously.</p>
      <p>This means you need to be clear about the responsibilities that data stewards and data committee members take on and offer training to support those people in their data governance roles. This training should cover everything from why it is so important to what&rsquo;s expected from people in different roles.</p>
      <h4>Audit Process Adoption</h4>
      <p>A big part of data governance involves developing processes for how the company will handle data, especially when it comes to sensitive information. Auditing how these processes are actually living in your organization and how well people are adopting them can be extremely informative as you continue to make program improvements.</p>
      <p>That&rsquo;s because even the best processes won&rsquo;t do your organization any good if no one adheres to them.</p>
      </div>
      </div>
      <h3>Regularly Measure Progress and Keep an Eye Toward Improvements</h3>
      <div>
      <div>
      <p>Finally, remember that data governance is not a one-and-done effort. It&rsquo;s a program that must continuously evolve based on factors like adoption and changing business needs.</p>
      <p>As a result, it&rsquo;s important to regularly check in on how policies are faring and the impact on data quality. The more you can measure that progress, the better you can manage the situation and identify what&rsquo;s working well and what needs to be improved.</p>
      </div>
      </div>
      </div>

What Is Data Governance And Where Observability Fits In ?

The Rise of DataOps

<div>
      <p>Everyone is talking about the modern data stack (MDS) nowadays. I am a data system person. I started building core database systems in the big data era, and have witnessed the birth and prosperity of cloud computing over the last decade. But the first time I came across the term &ldquo;modern data stack,&rdquo; I felt confused - is it just yet another buzzword that the cloud service vendors created to attract people&rsquo;s eyeballs? There are so many articles online, but most of them are quite markety and salesy. After running a startup building core systems in the modern data stack domain for a while, I would like to share my thoughts. In this article, I will explain &ldquo;modern data stack&rdquo; to you in simple terms, and discuss why modern data stack can really matter in companies.</p>
      <h3 tabindex="-1">What is a modern data stack?</h3>
      <p>The most general (and perhaps most unclear) explanation of the modern data stack is that it is a set of tools built around a data warehouse to simplify data integration. The fundamental starting point of a modern data stack is to save data engineers and data analysts time. Here, data integration essentially is to put data from different data sources together, and then analyze and operate. For example, in an e-commerce website, there must be data sources such as user access data, user order data, and commodity information data. By putting these data together in certain ways, we can get some knowledge, such as which kinds of users are interested in which products, which types of products are best to sell, and so on. The modern data stack, in general, is all about helping people turn data into knowledge.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/fivetran_mds_f4bb971fab.png" alt="" /></p>
      <center>An illustration of the modern data stack from Fivetran</center>
      <p>The image above is from the&nbsp;<a class="mds-text-primary" href="https://youtu.be/Zag5yeTPtPU?t=446">sharing</a>&nbsp;of&nbsp;<a class="mds-text-primary" href="https://www.fivetran.com/">Fivetran</a>&nbsp;CEO&nbsp;<a class="mds-text-primary" href="https://twitter.com/frasergeorgew">George Fraser</a>&nbsp;at Tableau Conference 2019. In the sharing George explains what a modern data stack looks like. The data warehouse takes input from various data sources, transform it internally inside the warehouse, and provides well-modeled data to support business intelligence,</p>
      <p>After reading the above definition of the modern data stack, you may ask: Why is this architecture "modern"? Does it sound like a buzzword for marketing gimmicks? It is true that the term "modern data stack" sounds so marketing-oriented that people might even dislike it. Before we dive deeper into the modern data stack, let's explore who invented the term first.</p>
      <h3 tabindex="-1">Who invented the term "modern data stack"?</h3>
      <p>Frankly speaking, it is very difficult to find a very clear answer regarding its roots due to existing ambiguities. However, it is certain that the term was popularized by&nbsp;<a class="mds-text-primary" href="https://www.fivetran.com/">Fivetran</a>,&nbsp;<a class="mds-text-primary" href="https://www.getdbt.com/">dbt</a>, and the venture capitals behind them.</p>
      <p>By searching on the internet, we can find that the first time this word first became popular after 2020. The most famous speech/article is from the second half of 2020 by dbt CEO&nbsp;<a class="mds-text-primary" href="https://twitter.com/jthandy">Tristan Handy</a>. In his article, &ldquo;<a class="mds-text-primary" href="https://www.getdbt.com/blog/future-of-the-modern-data-stack/">The Modern Data Stack: Past, Present, and Future</a>", the development of the modern data stack is into three eras: Cambrian Explosion I (2012-2016), Deployment (2016-2020), and Cambrian Explosion II (2020-2025). I suggest readers read the original article for the exact definitions of the three eras. In the same year, another report from&nbsp;<a class="mds-text-primary" href="https://a16z.com/">a16z</a>, a well-known venture capital, was published, titled "<a class="mds-text-primary" href="https://future.com/emerging-architectures-for-modern-data-infrastructure-2020/">Emerging Architectures for Modern Data Infrastructure: 2020</a>". This article explains the data infrastructure used by various tech companies in the United States in 2020 from VC&rsquo;s perspective.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/google_trend_MDS_0b1b6b4089.png" alt="" /></p>
      <center>The search popularity of &ldquo;modern data stack&rdquo; on Google Trends</center>
      <p>In fact, after a deeper search, you will find that&nbsp;<a class="mds-text-primary" href="https://twitter.com/frasergeorgew">George Fraser</a>, CEO of Fivetran, mentioned this concept in&nbsp;<a class="mds-text-primary" href="https://www.fivetran.com/blog/competitive-analytics-requires-modern-architecture">a speech in 2019</a>. On Twitter, Tristan also said that he proposed a similar idea of the &ldquo;modern BI stack&rdquo; back in 2016.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/twitter_yingjun_a3e172def3.png" alt="" /></p>
      <center>Reply from dbt CEO Tristan Handy on Twitter.</center>
      <h3 tabindex="-1">Why do we need a modern data stack?</h3>
      <p>Knowing the history of the modern data stack, we can go back to the previous question: why is this architecture "modern"? Why do we need a "modern" data stack? Was the previous data stack not perfect? To answer these questions, I think we should focus on "data", not "modern". It is data that lies at the heart of the modern data stack, not technology. The essence of this terminology is that it changes the way companies use technology: companies no longer build a stack for a certain technology product, but build a stack for their own data.</p>
      <p>Let&rsquo;s think back on how companies managed their data 20 years ago. Twenty years ago, corporate data was basically stored in database systems sold by Oracle, IBM, and Microsoft. Some typical characterizations in these use cases are the following.</p>
      <ul>
      <li>First, the database systems sold by tech giants such as Oracle, IBM, and Microsoft are expensive, and not all companies can afford those database systems.</li>
      <li>Second, database vendors provide consulting and support services, not operation and maintenance services. Usually, enterprises need to hire a dedicated DBA team to operate and maintain these databases and build applications on top of these databases.</li>
      <li>Third, neither the size of data nor the number of applications was large.</li>
      </ul>
      <p>Because of these reasons, when an enterprise tried to manage its own data, they were not really building a framework for the data. Instead, they were building a framework for the database.</p>
      <p>Now time is different. In the past 20 years, the scale of data and the number of applications of enterprises have grown exponentially. A number of database products have emerged from the needs of modern enterprises. The development and popularization of cloud computing have further stimulated more enterprises to use databases. On the other hand, the explosion of the market has also led to sustainable developments in the database field. Compared with 20 years ago:</p>
      <ul>
      <li>
      <p>The price and the usage bar of databases have been largely reduced.</p>
      </li>
      <li>
      <p>Database vendors do not only sell software, but services.</p>
      </li>
      <li>
      <p>Databases no longer compete for performance, but for ease of use.</p>
      </li>
      </ul>
      <p>When facing massive data and applications, if the price of data software is low enough and the performance is good enough, companies can save their time from the dirty work of dealing with the databases, and focus on how to make data management easier. That is exactly the original intention behind the modern data stack: making data management easier.</p>
      <h3 tabindex="-1">How to make data management easier?</h3>
      <p>As mentioned above, the advocates of the modern data stack include dbt, Fivetran, a16z, other data software companies, and many venture capitals. What they all want to change is the way how companies prepare data for analytics: from the traditional ETL (Extract, Transform, Load) to the ELT (Extract, Load, Transform). Below is an illustration also from George Fraser's&nbsp;<a class="mds-text-primary" href="https://youtu.be/Zag5yeTPtPU?t=488">sharing</a>&nbsp;at Tableau Conference 2019 explaining the difference between ETL and ELT. Here ETL means that to ingest the data source into the data warehouse, it must go through three steps: data extraction, transformation, and loading. On the contrary, what ELT means is that only data extraction and data loading is needed before loading data into the cloud data warehouses. Data transformation is carried out directly inside the data warehouse.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/fivetran_ELT_ELT_7c8954776f.png" alt="" /></p>
      <center>The difference between ETL and ELT</center>
      <p>Turning the traditional ETL into the new type of ELT allows the complex computation of data to be moved from outside the data warehouse to the inside. This movement can simplify the management of the entire data. This still sounds a bit confusing: how does ELT simplify our data management? Can you turn your company's "old tech stack" into a "modern data stack" in one simple step?</p>
      <p>Imagine what our data management would be like without ELT. Some employees of a company want to analyze raw data, they use ETL tools to build a pipeline to clean and process the data into table X in the data warehouse, and then analyze table X in the data warehouse. In the process of analysis, they find that they have not extracted a column of data, or the data processing method was wrong, or the unit of the data was wrong. At this time, what should she/he do? The only way is to re-fetch the data from the data source, rebuild the ETL pipeline, and re-analyze the data. However, the data in the data source is often temporary. Usually, the enterprise only saves the data source for seven days or 30 days. That is saying, when she/he wants to re-extract the data, build the pipeline, and perform analysis, the original data may be already lost.</p>
      <p>ELT solves this problem in a brute-force way: store all the original data in the data warehouse. When the original data is stored in the data warehouse, there is no so-called data loss problem, and all historical records can be found. When we want to process data, we do not build any pipelines for processing but directly write SQL and use the computing power of the data warehouse itself for processing.</p>
      <p>It sounds really simple, and doesn't seem like something that only appeared in the 2020s. Why was ELT not used 20 years ago? Why has ELT been promoted only in recent years? I think there are several reasons for this.</p>
      <ul>
      <li>Firstly, the amount of data 20 years ago was relatively small, the data format was relatively simple compared to today, and the processing requirements were not high. Even if it was troublesome for everyone to develop applications, they could all be solved by stack engineers.</li>
      <li>Secondly, database systems from 20 years ago were simply too expensive to buy or not want to spend a lot of money on storing large amounts of data.</li>
      <li>Thirdly, the performance of the database system at that time was still relatively weak. Today, the cost of engineers is high, the applications are complex and diverse, and it is no longer possible to simply rely on engineers to solve problems; the emergence of cloud computing has greatly reduced the cost of data warehouses, making it possible to store massive amounts of historical data cheaply; The performance has been greatly improved, and it is no longer difficult to process large amounts of data in the data warehouse.</li>
      </ul>
      <p>If we think about it carefully, we will find that the essence of ELT is actually to turn the data warehouse into the center of data management, and to use the capabilities of the data warehouse to deal with data problems as much as possible. Now back to the topic of this article, this way of building the stack centered on the data warehouse is the way advocated by the modern data stack: the extensive usage of data warehouse capabilities. ELT is only one part of the modern data stack. In fact, the modern data stack also includes other aspects, including data visualization, metadata information management, data discovery, data sharing, etc. We will not cover these topics in this article.</p>
      <h3 tabindex="-1">What does the modern data stack have to do with cloud computing?</h3>
      <p>Modern data stacks and cloud computing are strongly correlated but not necessarily causal. In fact, cloud computing has indeed catalyzed the development of the modern data stack. The essence of the modern data stack is to simplify data management, while cloud computing is to reduce the cost of data management; hence the starting point is different. Nevertheless, without a low-cost solution, there would be no way to make data management easier.</p>
      <h3 tabindex="-1">Will there be a one-size-fits-all system on the modern data stack?</h3>
      <p>The one-size-fits-all system here refers to a system that can do everything: it can support operational processing, analytical processing, stream processing, data visualization, data sharing, data governance, etc. All tasks are done with one system. I think it's an ideal vision, but practically speaking, we haven't reached the stage where one size can fit all. The key question here is not whether it is technically feasible, but whether the product can be widely accepted.</p>
      <p>I think this question can be approached from both a product competition and user needs perspective.</p>
      <ul>
      <li>From the product competition, modern data software competition is more than &ldquo;hard&rdquo; level competitions, e.g., performance and price. Vendors have expanded the competition to more points in the &ldquo;soft&rdquo; level, for example, security and usability. It is often difficult for a one-size-fits-all system to compete with a product specializing in a certain task. A good example is Zoom. Although many office software, including Slack, Microsoft Teams, etc., can do video conferencing, Zoom has become a worthy winner in this field through its professionalism and ease of use.</li>
      <li>In terms of user needs, users rarely need to use all the features in a one-size-fits-all system. Small and medium-sized enterprises tend to use only two or three functions, and naturally, they may prefer the system that does the best job of these two or three functions. In a large company, users do use a variety of functions, but when trying to convince such users to replace their existing systems, they are often deterred by the high cost of migration.</li>
      </ul>
      <p>Of course, various viewpoints justify the existence of a one-size-fits-all system. The most common argument is that enterprise-level users often only want to solve problems but do not want to spend time choosing solutions. The existence of a one-size-fits-all system can significantly save their time in selecting products. Furthermore, such a system can provide a unified user experience that is more suitable for enterprise users.</p>
      <p>I don't think there is a standard answer to this question, and I believe that the future trend (at least in the next 3-5 years) is coexistence. Nevertheless, the one-size-fits-all system will still be a minority. Whether it is a Swiss Army knife, a fruit knife, or a utility knife, they all have their own living space, so that the Swiss Army Knife will not occupy the market of other kinds of knives.</p>
      <h3 tabindex="-1">Summary</h3>
      <p>The modern data stack serves users' data, and it is completely changing the situation in which users are bound with a few tech giants. The goal of modern data stacks is to greatly simplify the difficulty of managing data for users, allowing users to care more about the data itself, rather than the software. Of course, the modern data stack is still quickly evolving. In the next article, I will explain the current landscape of modern data stack, and what's next in the near future.</p>
      </div>

Everything You Wanted to Know but were Afraid to Ask about Modern Data Stack (part I)

How to Create a Data Governance Team? 3 Essential Steps

Getting Started with Data Observability

Building a Modern Data Team: From Analytics to AI

Data Governance - A Thought Leader's Perspective

Decoding the Modern Data Stack

<p>&ldquo;There must be something wrong with Excel. I can't get these numbers to make sense.&rdquo; For anyone who has had a similar experience of staring at a spreadsheet for far too long, we have news for you: Excel isn&rsquo;t the problem; your data is.</p>
      <p>Growing businesses encounter this issue as they scale their offerings and onboard new technologies. The more information you gather and store, the more complex your data infrastructure becomes. One manual error &mdash; like a coding typo or selecting the wrong data type for a new CRM field &mdash; can cause entire data sets to be inaccurate or incomplete. </p>
      <p>Mistakes will inevitably happen. That&rsquo;s why it&rsquo;s important to have data reliability tools and procedures in place that prevent bad data from moving further through your tech stack. This enables your organization to be confident that the data being used for analysis, insights, and decision-making is accurate and up-to-date. </p>
      <p>In this guide, you&rsquo;ll learn why data reliability is important, how it affects business outcomes, and the tools companies can employ to strengthen reliability and ensure consistency.</p>
      <h3>Why is data reliability important?</h3>
      <p>&ldquo;<a class="mds-text-primary" href="https://www.mozartdata.com/post/how-to-be-a-data-driven-company" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>Data-driven decisions</u></a>&rdquo; has become quite the buzzword, but don&rsquo;t let that distract from the fact that making business decisions based on accurate data is vitally important. Over the last decade, businesses have gained unprecedented access to quantitative and qualitative data about their operations, customers, and prospects. With all this information at our fingertips, it&rsquo;s become best practice for every team, from marketing to customer success, to make data-driven decisions. Why trust your gut or talk about hypotheticals if you can use hard facts and figures to guide your next move?</p>
      <p>In some ways, access to this information has helped level the playing field among small, medium, and more mature businesses. The data is out there if you&rsquo;re able to tap into it and know how to use it. But this also presents a problem. </p>
      <p>You may be collecting and storing a sizable amount of information within your <a class="mds-text-primary" href="https://www.mozartdata.com/post/data-stack" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>data stack</u></a>, but are you certain that these data sets &mdash; individually and when combined for analysis &mdash; are always complete, accurate, and up-to-date? If your answer is not a confident &ldquo;yes,&rdquo; then your business can&rsquo;t reliably make data-driven decisions. </p>
      <p>Data reliability is the foundation for confident decision-making and a successful data team.</p>
      <h3>What tools can improve data reliability?</h3>
      <p>Proactive, automated alerts and transform tests are two crucial data reliability tools companies utilize to proactively catch and debug issues, ensuring their data is accurate and reliable as it flows through the data pipeline. </p>
      <h3>Alerts</h3>
      <p>If organizations don't monitor their data, they're setting themselves up to miss important outcomes. This could be flaws in the data, like duplicate fields, or important events, such as revenue goals. Proactive, automated alerts can be raised the moment that specific conditions &mdash; which are predefined by the business &mdash; are met. </p>
      <p>Data reliability should be an ongoing consideration, and data alerts can help maintain diligence. There are two notable points in time when data monitoring should lead to an alert, if necessary: as the data is loaded into a <a class="mds-text-primary" href="https://www.mozartdata.com/post/data-warehouse" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>data warehouse</u></a> and post-transform.</p>
      <p>When data flows from its source, such as a marketing automation platform, into a data warehouse, it&rsquo;s still considered to be raw data. At this point, it&rsquo;s likely the data will contain flaws like duplicates, missing values, and incorrect formatting. A <a class="mds-text-primary" href="https://www.mozartdata.com/post/data-transformation" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>transform</u></a> can clean the data, but many of these issues can and should be fixed before getting to that point. </p>
      <p>Alerts that identify issues as data enters the warehouse allow you to take corrective action before defective data is used downstream. Corrective action might involve editing the data in the source platform, or letting an engineering team know that data is not being stored or collected correctly in a database, so that more involved troubleshooting can be prioritized. These types of actions help build confidence that the transformed data teams are working with is as complete and accurate as possible. </p>
      <p>Additionally, the results of a transform should be monitored to help businesses track specified outcomes. Alerts raised post-transform might highlight potential discrepancies, like unusually low or high returned values, or monitor discrete metrics for irregularities, long-term trends, milestones, etc. </p>
      <h4>Tests</h4>
      <p>Related to alerts are tests. In the context of maintaining data reliability, this would often look like a transform test. A transform test runs before the actual transform and can be used to flag specific outcomes. After the test runs and identifies an issue, it can either prevent the transform from running or send out an alert. </p>
      <p>Transform tests can be an important tool to help prevent data errors from reaching others, proactively identify data collection issues or unexpected positive outcomes, and maintain everyone&rsquo;s trust in your data. </p>
      <h3>How to make your data more reliable with Mozart Data</h3>
      <p>Mozart Data&rsquo;s modern data platform provides data alerts, transform tests and many other tools to help you ensure your data is accurate and reliable. Maintaining trustworthy data gives business leaders confidence in their team&rsquo;s reports and analyses, and enables them to make strategic, data-driven decisions. </p>
      <p>Data reliability is only possible if you have a clear view of your entire data pipeline though. This is achieved through what&rsquo;s known as data observability. <a class="mds-text-primary" href="https://www.mozartdata.com/post/what-is-data-observability" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>Read this post</u></a> for a deep dive into this topic and how data observability is linked to business growth. </p>

What is Data Reliability?

<p>Do you know the current status &mdash; quality, reliability, and uptime &mdash; of your data and data systems? Not last month or last week, but where they stand at this moment. As businesses grow, being able to confidently answer this question becomes more important. That&rsquo;s because data needs to be clean, accurate, and up-to-date to be considered reliable for analysis and decision-making. This confidence comes through what&rsquo;s known as data observability.</p>
      <p>In this guide, we&rsquo;ll review what data observability is, why it&rsquo;s a key driver of business growth, and what tools are needed to support the process.</p>
      <p>Advanced data teams automate the movement of raw data from data sources (like a CRM, advertising platform, and marketing automation tool) to a modern data platform, where data sets are <a class="mds-text-primary"  href="https://www.mozartdata.com/post/what-is-the-best-way-to-clean-up-a-large-data-set" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>cleaned</u></a>, organized, and centralized for storage in a data warehouse. It&rsquo;s from this point that <a class="mds-text-primary"  href="https://www.mozartdata.com/post/data-transformation" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>transformed data</u></a> can be synchronized with business intelligence tools for reporting, creating visualizations, conducting analyses, and generating insights. The flow of data from disparate origin sources to the destination, whether it&rsquo;s a data warehouse or business intelligence tool, is known as the data pipeline.</p>
      <p><strong>Data observability is the ability to see what&rsquo;s happening in your entire data pipeline, optimize performance, and monitor the health of your data.</strong> Without data observability, businesses are only viewing data at one point in time and won&rsquo;t have a real-time, transparent view of the workflow, nor insight into previous versions of saved data sets. Data observability, therefore, is critical for rapidly detecting, troubleshooting, and resolving problems, as well as ensuring that the data flowing into your <a class="mds-text-primary"  href="https://www.mozartdata.com/post/is-a-business-intelligence-tool-enough-for-your-business" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>business intelligence tool</u></a> is reliable for analysis, detecting patterns, and developing insights. </p>
      <h3>Why is data observability important? </h3>
      <p>Data observability is a key driver of business growth, as it ensures data quality and enables you to be agile and make sound, data-driven decisions in real time. </p>
      <p>Imagine, for example, a business called Intrepidly runs a B2B subscription-based productivity tool that&rsquo;s accessible to users on its desktop website, Android app, and iOS app. Intrepidly has already launched a new feature on the web, and next it&rsquo;ll be released in the iOS app. The updated version has been deployed and appears to be running fine. New subscriber profiles are being created in the CRM via the iOS point of sale. But Intrepidly&rsquo;s revenue operations team notices that key profile fields, such as subscription type and discount level, are being populated with NULL values. </p>
      <p>Where along the data pipeline is the issue occurring? Is it before or after the transform? Are these key pieces of missing information retrievable? Depending on what went wrong and where, Intrepidly could experience significant effects on revenue and the customer experience. </p>
      <p>Errors and omissions in the data pipeline can lead to downtime, which is when data is inaccurate or unavailable. In businesses that lack data observability, it could take multiple developers hours to answer these questions and rectify the issue. Troubleshooting data issues is not where you want your high-value employees spending their time. </p>
      <p>Data observability could also mean the difference between catching an error right after the version release versus moments before the start of an all-hands presentation when the data in the deck looks &ldquo;off." In other scenarios, data issues can delay product launches, internal technology implementation, reporting obligations, and decision-making. </p>
      <p>Maintaining data observability helps prevent &ldquo;drop everything to find the cause&rdquo; situations because it takes a proactive approach to monitoring the health of a business&rsquo;s data pipelines.</p>
      <h3>Data observability tools</h3>
      <p>There are specific tools, including data lineage and run history, that provide businesses with a transparent view of their data pipeline. Although these capabilities may not prevent human errors that result in situations like that mentioned above, businesses that have complete visibility into their data pipelines will be able to detect issues early and significantly accelerate the resolution process. </p>
      <p>Data lineage is one tool that&rsquo;s used to monitor the pipeline so businesses can see what information their transforms and reports are relying on and quickly trace errors back to the source. Mozart Data&rsquo;s modern data platform infers lineage so businesses don&rsquo;t have to set it up themselves.</p>
      <p>Run history makes troubleshooting failed transforms easier by showing whether they were run manually or automatically. To get to the root of broken transforms, you can use version history to see when changes were made, what those changes were, and who made them.</p>
      <p>Snapshots record historical information by capturing transform or source table information at a specific point in time and at a set frequency &mdash; typically daily. These are helpful for performing automated checks. For instance, when you need to track down the date that a particular field goes missing or automatically compare relative changes (e.g., is the amount we're billing this month close to what we billed last month).</p>
      <h3>How Mozart Data increases visibility into your data</h3>
      <p>Mozart Data&rsquo;s modern data platform provides data lineage, run history, version history, snapshots, and other data observability tools to give you visibility into your data systems and processes. This enables teams to streamline workflows, optimize their pipelines, and realize operational efficiency gains. It&rsquo;s also key to maintaining trustworthy data that org leaders feel confident using for analyses, insights, and business decisions. </p>

What is Data Observability?

The Modern Data Stack Ecosystem: Spring 2022 Edition

What is Data Onboarding? And 3 Ways It's Overwhelming Your Teams

What's the Difference Between Data Wrangling vs Data Cleansing vs Data Transformations

What is Reverse ETL? The Definitive Guide

Change Data Capture: What It Is and How to Use It

How to Build and Operate an Open-Source Data Stack on Kubernetes

<div>
        <p data-selectable-paragraph="">As data became more and more available to companies, data integration became one of
            the most crucial challenges for organizations. In the past decades, ETL (extract, transform, load) and later ELT
            (extract, load, transform) emerged as data integration methods that transfer data from a source to a data
            warehouse. Despite their similarities, the two differ in fundamental ways.</p>
        <p data-selectable-paragraph="">This blog aims to give an overview of ETL and ELT processes, describe the difference
            between them and outline which tools are now available for organizations. We do so by covering the following
            points:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>A brief history of data integration</strong></li>
            <li data-selectable-paragraph=""><strong>What is ETL?</strong></li>
            <li data-selectable-paragraph=""><strong>Why do we need ETL tools?</strong></li>
            <li data-selectable-paragraph=""><strong>ETL vs. ELT &mdash; What&rsquo;s the difference?</strong></li>
            <li data-selectable-paragraph=""><strong>When should you use ELT instead of ETL?</strong></li>
            <li data-selectable-paragraph=""><strong>What characteristics should your ETL/ELT tool have?</strong></li>
            <li data-selectable-paragraph=""><strong>What are the best ETL/ELT tools available right now?</strong></li>
        </ul>
    </div>
    <div>
        <h3><strong>A brief history of data integration</strong></h3>
        <p data-selectable-paragraph="">Today, businesses require data integration to centralize, access, and activate their
            data. As businesses start leveraging data from hundreds of different sources across continents and business
            departments and teams to make data-driven decisions, data integration has never been more critical. Although
            this heavily fractured landscape might seem new, it is not. In fact, the first challenges related to data
            integration date back to the 1960s when IBM and other companies pioneered the first Database Management Systems
            (DBMS), whose later advancements led to the sharing of data between computers. The process of integrating data
            with external machines soon became a challenge. This was challenging because organizations integrate with
            systems that they cannot control, and this often fails. So, it becomes really hard to create systems that
            don&rsquo;t break when you do data integration.</p>
        <p data-selectable-paragraph="">Data integration can be defined as the practice of consolidating data from separated
            source systems to transfer it to a data warehouse. Integration starts at the ingestion phase and includes steps
            like cleansing, ETL mapping, and transformation. ETL plays a central role in transforming raw data into usable
            data &mdash; enabling data teams to trust their data and make informed business decisions. Without data
            integration, accurate analytics is impossible to achieve as making decisions with incomplete information most
            likely leads to undesirable outcomes. And if these decisions involve hundreds of data sources, terabytes of
            data, and millions of dollars, integration becomes especially important.</p>
        <p data-selectable-paragraph="">ETL was the first standardized approach to dealing with data integration which
            appeared in the 1970s. It first emerged as a response to the adoption of relational databases and the increasing
            need to move information between them efficiently. Then, in the 1980s, data integration became even more
            critical with the rise of data warehouses. Data warehouses could integrate data from different sources. However,
            they typically required custom ETLs for each data source. This led to the emergence of a wide variety of ETL
            tools. And by the end of the 1990s, these solutions were affordable for both enterprises and mid-market
            businesses.</p>
        <p data-selectable-paragraph="">Finally, with the emergence of cloud computing in the 2000s, cloud data warehouses
            and data lakes brought forward a new development: ELT, which, as we will see later in this blog, allows
            businesses to load unlimited raw data into a cloud data warehouse.</p>
    </div>
    <div>
        <h3><strong>What is ETL?</strong></h3>
        <p data-selectable-paragraph="">As organizations collect data from multiple sources &mdash; such as CRM systems,
            Cloud applications, and more &mdash; the ETL process aims to gather data from separate sources and create a
            centralized database. Practically, ETL pipelines have the role of extracting raw data from its source,
            transforming it, and finally loading it in the warehouse &mdash; which is a centralized database.</p>
        <p data-selectable-paragraph="">Let&rsquo;s dive into each one of these steps:</p>
        <p data-selectable-paragraph=""><strong>Extract</strong></p>
        <p data-selectable-paragraph="">The first step is the process of data extraction. Here, data pipelines extract data
            from the different sources and gather it in a staging area. The staging area is vital because it allows the
            extraction of data from various sources at different times, and it prevents the simultaneous performance of
            extractions and transformations from happening &mdash; therefore averting the risk of overwhelming data sources.
        </p>
        <p data-selectable-paragraph=""><strong>Transform</strong></p>
        <p data-selectable-paragraph="">The second step is the data transformation process &mdash; in other words, making
            the raw data fit for analysis. This consists of&nbsp;<strong>improving the quality</strong>&nbsp;of the data by
            performing actions like sorting, joining, reformatting, filtering, merging, aggregation, and so on,
            and&nbsp;<strong>restructuring and reformatting</strong>&nbsp;the data to adapt it to the specific business need
            it will be used for.</p>
        <p data-selectable-paragraph=""><strong>Load</strong></p>
        <p data-selectable-paragraph="">The last step is loading the transformed data into the warehouse. The loading can be
            full &mdash; so the data is fully loaded into the warehouse at once &mdash; or incremental &mdash; where the
            data is loaded at different scheduled intervals using batch or stream loading. Batch loading consists of the
            scheduled extraction of different batches of data from a source system. In contrast, stream loading consists of
            data ingestion by pipelines as soon as they are made available by the source system.</p>
    </div>
    <div>
        <h3><strong>Why do we need ETL tools?</strong></h3>
        <p data-selectable-paragraph="">The ETL process of migrating transformed data into a warehouse enables organizations
            to break down silos and makes it easier for data users to turn the available information into business
            intelligence.</p>
        <p data-selectable-paragraph="">ETL is beneficial for organizations because it combines diverse data into a unified
            view. This enables data users to find and access information more efficiently, consequently improving
            productivity.</p>
        <p data-selectable-paragraph=""><strong>The advantages of ETL</strong></p>
        <p data-selectable-paragraph="">Implementing ETL into your data stack can bring many benefits, including:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>Speed:&nbsp;</strong>There is no waiting involved when using ETL tools,
                as they instantly load data into your data warehouse, where they are ready for transformation.</li>
            <li data-selectable-paragraph=""><strong>Flexibility:&nbsp;</strong>As transformations do not need to be defined
                at the beginning, it is pretty straightforward to integrate additional data sources into the ELT process.
            </li>
            <li data-selectable-paragraph=""><strong>Low initial cost:&nbsp;</strong>ETL automates the process of onboarding
                data, and as you don&rsquo;t need to define transformations, the initial cost is relatively low.</li>
            <li data-selectable-paragraph=""><strong>Low maintenance requirements:&nbsp;</strong>As the process is more
                straightforward and automated, it will require less maintenance. It is also easier to fix bugs since
                transformation only happens in the last step.</li>
            <li data-selectable-paragraph=""><strong>Scalability:&nbsp;</strong>As the amount of data you use increases, you
                can expand your storage in the cloud.</li>
        </ul>
    </div>
    <div>
        <h3><strong>Challenges to ETL and the rise of ELT</strong></h3>
        <p data-selectable-paragraph="">The main characteristic of ETL is the order in which the process is performed
            &mdash; specifically, the data goes through transformation before being loaded into the warehouse. This specific
            order is the reason behind some of the difficulties organizations face when using ETL. These are the following:
        </p>
        <ul>
            <li data-selectable-paragraph=""><strong>Dealing with heavy data loads:&nbsp;</strong>Volume and demand for data
                have been dramatically growing in the past few years. Unfortunately, many ETL processes are failing to scale
                to match these heavy loads appropriately. When dealing with large amounts of data, some common mistakes are
                loading irrelevant data, bottlenecks due to insufficient CPU and serial data processing instead of parallel
                data processing.</li>
            <li data-selectable-paragraph=""><strong>Maintenance of the pipelines:&nbsp;</strong>In ETL, pipelines extract
                and transform data all at once, breaking as soon as schema changes in the data source. For instance, when a
                field is removed. This compromises the code needed to transform the data, and it has to be re-written by
                data engineers.</li>
            <li data-selectable-paragraph=""><strong>Data security risks:&nbsp;</strong>When loading large amounts of raw
                data into your storage, data security can become a concern. To deal with this challenge, managing user
                access to the raw data stored in the data warehouse is crucial.</li>
            <li data-selectable-paragraph=""><strong>Low compliance with regulations:&nbsp;</strong>Since data is stored
                with minimal processing, there may be some additional steps to ensure that you comply with the main security
                protocols, including GDPR and HIPAA.</li>
        </ul>
    </div>
    <div>
        <h3><strong>ETL vs. ELT &mdash; What&rsquo;s the difference?</strong></h3>
        <p data-selectable-paragraph="">Traditional ETL is characterized by the performance of the transformation process
            before the loading into the warehouse. The reason for this is that, back in the day when ETL was created,
            storage, computation, and bandwidth were costly. Hence the need to reduce the volume of data before it gets to
            the data warehouse.</p>
        <p data-selectable-paragraph="">As storage and computation cost dramatically decreased, cloud storage grew
            exponentially. This led to the entrance into the market of data warehouses like Amazon Redshift and Google
            BigQuery. Cloud data warehouses are infinitely more scalable compared to traditional warehouses, allowing
            organizations to fit enormous amounts of data.</p>
        <p data-selectable-paragraph="">The emergence of cloud data warehouses called for a radical change in ETL processes
            too. If ETL processes were fine for traditional warehouses, their performance did not improve in a cloud
            environment. The reason for this is that the scalability issues were inherent to the process of moving data into
            the warehouse. At the same time, due to the arrival of cloud storage, organizations started loading large
            volumes of untransformed data into the warehouse without having to worry about its cost. From this moment on, it
            has been possible to extract data from the source systems and load it into the cloud warehouse without needing
            to go through the transformation process. The Extract-Load-Transform (ELT) process replaced traditional ETL.</p>
        <p data-selectable-paragraph="">Let&rsquo;s take a side-by-side look at the differences between ETL and ELT, their
            pros and cons, and how they can work together to provide a holistic data integration solution for businesses.
            Here are the ten critical differences between ETL and ELT.</p>
        <figure>
            <div tabindex="0" role="button">
                <div><img role="presentation" src="https://miro.medium.com/max/1400/1*nY2lYvuPTC_ywQOs0nny7Q.png" alt="" width="700" height="628"></div>
            </div>
            <figcaption data-selectable-paragraph="">Source: Benedetta Cittadin</figcaption>
        </figure>
    </div>
    <div>
        <h3><strong>What characteristics should your ETL/ELT tools have?</strong></h3>
        <p data-selectable-paragraph="">If you&rsquo;ve made it so far, it probably means you are interested in integrating
            ETL or ELT in your data stack. But what characteristics should these tools have to meet your standards?
            We&rsquo;ve gathered some general, critical requirements that any of these tools should have:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>Built-in integrations</strong>: You will need your ETL/ELT tools to
                interact with many different programs and APIs to use your data.</li>
            <li data-selectable-paragraph=""><strong>Ease of use:&nbsp;</strong>The tool should have an intuitive interface
                that assists you in simplifying as many functions as possible as well as making it possible to train
                non-developers to use it.</li>
            <li data-selectable-paragraph=""><strong>Automation:&nbsp;</strong>The more tasks you can automate, the faster
                and easier it will be to get your work done. And if you adopt a tool, you don&rsquo;t want to have to
                program SQL workarounds every time you need to get something done. This is why your ETL/ELT tool should have
                automation in place.</li>
        </ul>
        <h3><strong>What are the best ETL tools available right now?</strong></h3>
        <p data-selectable-paragraph="">With companies collecting increasingly high amounts of data daily, there is a
            growing need to distribute data from the source to the data warehouse. This is why we have recently seen a
            blowing up of ETL/ELT tools in the market. In this section, I&rsquo;ll briefly outline the different ETL tools
            that are currently being used by organizations.</p>
        <p data-selectable-paragraph="">ETL can be grouped into different categories based on their infrastructure.
            I&rsquo;ll briefly go through three of these groups.</p>
        <p data-selectable-paragraph=""><strong>Enterprise software ETL</strong></p>
        <p data-selectable-paragraph="">Developed by commercial organizations, these solutions are often the most mature
            ones in the marketplace. These tools include graphical user interfaces (GUIs) for architecting ETL pipelines,
            support for relational and non-relational databases, and extensive documentation and user groups.</p>
        <p data-selectable-paragraph="">Although they offer more functionality, enterprise ETL tools are usually more
            expensive and require more employee training because of their complexity.</p>
        <p data-selectable-paragraph="">Think of tools like IBM DataStage and Oracle Data Integrator.</p>
        <p data-selectable-paragraph=""><strong>Open-source</strong></p>
        <p data-selectable-paragraph="">Many free ETL tools offer GUIs for data-sharing processes and monitoring the flow of
            information. An advantage of open-source tools is that organizations can access the source code and extend its
            capabilities.</p>
        <p data-selectable-paragraph="">Think of tools like Airbyte and Talend.</p>
        <p data-selectable-paragraph=""><strong>Custom ETL tools</strong></p>
        <p data-selectable-paragraph="">Cloud service providers (CSPs) now offer ETL tools built on their infrastructure. An
            advantage of this type of solution is that if the organization also stores its data using the same CSP, the
            pipeline is optimized because all processes occur within a single infrastructure. A disadvantage, however, is
            that cloud-based ETL tools only work within the CSP environment and don&rsquo;t support data stored in other
            locations.</p>
        <p data-selectable-paragraph="">Think of tools like Azure Data Factory and Google Cloud Dataflow.</p>
        <h3><strong>Conclusion</strong></h3>
        <p data-selectable-paragraph="">ETL and ELT can serve your data integration purpose in different ways. So, to choose
            the solution that&rsquo;s right for you, you need to consider factors like</p>
        <ul>
            <li data-selectable-paragraph="">The data you have</li>
            <li data-selectable-paragraph="">The type of storage you use</li>
            <li data-selectable-paragraph="">The (long-term) needs of your business</li>
        </ul>
    </div>

Survival of the Fittest: ETL vs. ELT



        <h3>&zwj;<strong>What is data product management</strong>?</h3>
        <p>In our experience at Secoda working with many data teams, we've seen most data teams do not have the tools they need
            to succeed. For growing organizations, the data function is usually an afterthought. The first data hire is brought
            on before raising a Series A and is expected to manage the workload that comes afterward with little to no support.
            This means that this first data hire usually spends all of their time answering other people&rsquo;s questions.
            These teams tend to be overworked and are likely to remain a reactive function for the foreseeable future.</p>
        <p>This is because most data teams are understaffed and misunderstood. Instead of reacting to employee data requests,
            working with data requires the data team to treat employees as their customers and their data as the product.</p>
        <p>An organization's data product is its collection of all of its data knowledge, along with the resources used for
            accessing, creating, and analyzing that data. Making the data product more self comprehensive, accessible,
            trustworthy, accurate, and secure should be the goal of the data team and the data team.&nbsp;</p>
        <p>Additionally, the data PM should help direct resources towards the highest leverage areas of the business. When
            products are built with everyone in mind, they usually don&rsquo;t accomplish what they need to for any specific
            subset of users. Data PMs should think about their data products the same way. Building data products for specific
            levels of technical capabilities and specific&nbsp;<em>persona&rsquo;s&nbsp;</em>of data consumers should be the
            goal of the data PM.&nbsp;</p>
        <h3>What makes a good data product manager</h3>
        <p>A good data product manager understands the company ecosystem of people and data. It is crucial to understand the
            different segments of the organization and document the requirements and knowledge from both data producers and
            consumers to generate and manage clean, reliable, and meaningful data. A good data product manager will apply
            product management practices to guide the roadmap for the data team by displaying some of the competencies below:
        </p>
        <ul role="list">
            <li>Conducting employees interviews to understand how they want to use data</li>
            <li>Running data-design sprints (How do we get a minimum version of this product to the employees in the shortest
                amount of time)&nbsp;</li>
            <li>Data product prioritization and road map planning.</li>
            <li>Communicating with the product team and updating the data team on changes that may affect their work.</li>
            <li>Performing vendor assessments.</li>
            <li>Translating business-to-data requirements, and vice versa.</li>
            <li>Defining and tracking key metrics for each stakeholder.</li>
            <li>Sharing knowledge and tracking data product usage metrics.</li>
        </ul>
        <p>We think about data PMs as the bridge between the data team and the rest of the organization. It takes time, tooling,
            and measurement to make sure that all stakeholders can access the right data, but when it&rsquo;s achieved,
            organizations can increase enterprise value by up to 5%.</p>
        <h3>Role requirements</h3>
        <ul role="list">
            <li><strong>Create Data Requirements</strong>&nbsp;&mdash; Data PMs should create clear data product requirements
                that satisfy the preferences of the end-users and data developers they work with. Even though data product
                requirements are still being defined, we like the development, release, and measurement model for building
                useful data products and determining how they are used.&nbsp;</li>
        </ul>
        <ul role="list">
            <li><strong>Employee feedback</strong>&nbsp;&mdash; Data PMs should bring together basic employee insights through
                interviews, understanding department OKR&rsquo;s, taking inbound questions, etc. Based on an explicit set of
                requests from the employees for whom the data product is being developed, they meet their baseline expectations
                (e.g. the employee says a part of the data is broken).</li>
        </ul>
        <ul role="list">
            <li><strong>Data insights</strong>- Data PMs should have a data proficiency where they can look at the data related
                to their data product (very meta) and use that data to make decisions about which products are performing well
                and which should be depreciated. They should have a working knowledge of data languages like SQL and know how to
                use popular tools like dbt, Looker, Metabase, Snowflake, Fivetran, and so on.</li>
        </ul>
        <ul role="list">
            <li><strong>Run Meetings</strong>&nbsp;&mdash; Data PMs should also run effective meetings with the data team and
                advocate for the team's resources to leadership. That includes setting the agenda and tracking action items as
                well as making sure the data team is well staffed and able to contribute to the data product the way that
                employees require.&nbsp;</li>
        </ul>
        <ul role="list">
            <li><strong>Vendor evaluation</strong>&nbsp;&mdash; The data PM should work closely with the engineering, sales, and
                operations teams to understand which products can give their team the best visibility and transparency into the
                data. By evaluating the right tools with other department leads, the data PM can help accelerate their team's
                data literacy and data maturity.&nbsp;</li>
        </ul>
        <h3>Data analytics</h3>
        <p>Think about how difficult it would be for a product manager to manage features without critical information about how
            people are using the product. Today, many data teams are operating in this same way. It&rsquo;s very unclear which
            tables, dashboards, metrics, and charts are used frequently and which should be depreciated. PMs can empower
            themselves and their teams with the information they need to make decisions to improve data products by adopting a
            tool that enables them to measure the most used data knowledge.&nbsp;</p>
        <p>It should be possible for data PMs to see how often teams use data, whether they are using the right data, and
            whether they aren't using any data. By understanding how people use analytics', data PMs can focus on the most
            important tasks that drive data literacy across their company.&nbsp;</p>
        <h3>The future of the Data PM</h3>
        <p>Over the next three to five years, we see the product-management role continuing to evolve toward a deeper focus on
            data as a product. Data product managers of the future will work with the data team to produce high-quality data
            products that can help drive self-service across the organization.&nbsp;</p>
        <p>To communicate data with the rest of the organization, data PM's might choose to implement tools like dbt, Looker,
            and Secoda. As we continue to help data PMs for the organization share their knowledge, we hope to be able to offer
            a greater depth of assistance and support to all employees trying to use data to make decisions. Data interactions
            are becoming increasingly complex, and Secoda hopes to simplify the interaction with data for every employee.</p>

Data Product Manager: The role and best practices for beginners

<p>From personal experience, I have always found it interesting to learn how to create an organized catalog of data.
        However, this interest was transformed into a passion when I began to realize the amount of time and effort it could
        save me within my job responsibilities. Creating a data catalog can greatly help you with organizing the data they
        collect, therefore making it easier to find what you need when you need it.</p>
    <p>By the end of this post, we should&rsquo;ve shed some light on the following questions:</p>
    <p><strong>Why all data teams need data discovery tools?</strong></p>
    <p><strong>What kind of data catalogue should teams look for?</strong></p>
    <p><strong>How to get buy in for a data catalogue?</strong></p>
    <p><strong>Which tools are here to stay?</strong></p>
    <h3>Why do you need a data catalog?</h3>
    <p>Simple data cataloging starts with a great organization. A data catalog is a collection of metadata and documentation
        that helps make sense of the data sprawl that exists in most growing companies. Getting together and starting to use
        a data catalog is a simple process, but starting to get adoption and having the&nbsp;<a
            href="https://www.secoda.co/blog/create-a-data-dictionary" style="color: #f66640;">catalog</a>&nbsp;exist as
        part of your workflow is a
        little bit more difficult.&nbsp;</p>
    <p>Even though it may seem like an easy task, getting different stakeholders to change their routines and start using a
        new tool can be very challenging. An example of the data catalog problems shared by one of the delivery companies we
        spoke with. At this company, it was difficult to get aligned on which tables were commonly used, joined, how they
        were used together and what columns meant. Similarly, it&rsquo;s difficult to monitor the number of data assets that
        exist across different departments, especially when the number of resources grows at a faster rate than people. Why
        is this the case?&nbsp;</p>
    <p>Data is becoming more decentralized through concepts like the data mesh. As more teams outside of the data function
        start to use data in their day-to-day, different tables, dashboards and definitions are being created at an almost
        exponential rate. Data catalogs are important because they help you organize your data whether you are working with
        structured or unstructured data. They help you identify what kind of data you have, how it is related to each other
        and what the best means to store it is so that you can quickly find it when needed.</p>
    <p>It happens to all data-driven companies as they grow both their data and their people. While it sounds like an easy
        problem, which might require a meeting, converging the business and data to eliminate confusion can be a very
        challenging task. That's why a data catalog can be one of the most valuable tools that a data team can use to
        start&nbsp;<a href="https://www.secoda.co/blog/net-promoter-score-for-data-teams" style="color: #f66640;">measuring
            their impact.&nbsp;</a>
    </p>
    <p>Below are the steps that teams need to take when creating a data catalog:</p>
    <h3>1. Gather sources from across the organization&nbsp;</h3>
    <p>The first step data teams need to take is to collect the different resources that are scattered across different
        tools in the organization. This may require multiple meetings and stakeholders to come together and figure out which
        resources need to be in the catalog. This collection could be done in a spreadsheet with an ongoing list of all
        resources and how they connect.</p>
    <p>It is common for teams to work with a tool that connects to multiple sources and collects the metadata from those
        sources into a central location. Using&nbsp;<a
            href="https://www.secoda.co/blog/keep-all-your-data-knowledge-in-one-place" style="color: #f66640;">data
            catalogs</a>&nbsp;makes this
        extremely easy and is a vital component of any data-driven organization. With the volume of data collected by
        businesses increasing, properly organizing it is becoming increasingly important. Priority should always be given to
        ensuring that the data is useful, but we should also take into account gains to be reaped through improved
        efficiency and collaboration.</p>
    <h3>2. Give each resource an owner</h3>
    <p>After data teams have identified all the resources from across the company that they would like to include in their
        data catalog, we recommend assigning ownership to each resource. Teams that we&rsquo;ve worked within the past have
        assigned ownership based on the source, schema or even domain. Teams that start assigning ownership should look for
        people who are familiar with the data knowledge they are responsible for managing and are willing to help others who
        want to learn how to use it.&nbsp;</p>
    <p>Ownership doesn&rsquo;t have to reside in the data team. Many products, operations and growth leaders will likely
        want to own certain pieces of&nbsp;<a
            href="https://www.secoda.co/blog/the-ultimate-guide-to-increasing-your-teams-data-literacy"
            style="color: #f66640;">company data
            knowledge</a>. By empowering these leaders as data stewards, the data team can start automating the repetitive
        questions they get from employees about how to use company data.&nbsp;</p>
    <h3>3. Get support and sign off</h3>
    <p>Once these meetings conclude and owners are on the same page, have the owners sign off on their responsibilities. The
        owners should be in alignment with the documentation and feel like the data team worked&nbsp;<a
            href="https://www.secoda.co/blog/how-to-champion-the-shift-towards-a-collaborative-data-culture"
            style="color: #f66640;">collaboratively</a>&nbsp;with
        them to come to this ownership structure. One effective strategy is to involve the leadership team in the exercise
        early to make sure that their team leads are signing off on the owners of data. This way, leadership can see how
        widespread the understanding of data is across the company. If the team leadership team sees&nbsp;<a
            href="https://www.secoda.co/blog/rethinking-the-data-catalog" style="color: #f66640;">the value of a data
            catalog</a>, this can move at
        a much faster pace.</p>
    <h3>4. Integrate the catalog base into your workflow</h3>
    <p>After data teams have received support for their data documentation process, they should look for ways to integrate
        this tool into their workflow. This step is critical for maintenance and upkeep. Without a tool that allows
        teammates to receive notifications on Slack, it will likely be forgotten. By creating a process around the data
        catalog, teams can ensure that it is not left behind as the team grows</p>
    <p>One additional piece of context that teams should look for is the usage and adoption of the data catalog across the
        company. If a tool can display the way that employees are searching and using the knowledge base, it can be a great
        source of information for the data team as they continue to iterate on the information in this central repository.
        By ensuring that there are triggers and notifications that prompt users to enter the application, teams can make
        sure that they are taking advantage of the&nbsp;<a
            href="https://www.secoda.co/blog/data-product-manager-the-role-and-best-practices-for-beginners"
            style="color: #f66640;">data catalog
            and improving it like a product.&nbsp;</a></p>
    <h3>5. Upkeep the data catalog</h3>
    <p>Although the documentation should be stable,&nbsp;<a
            href="https://www.secoda.co/blog/analytics-teams-need-more-than-traditional-documentation"
            style="color: #f66640;">it may need to change
            over time</a>. One instance that might require documentation to change is when a new revenue stream is
        introduced or when the pricing of an existing revenue line changes. These changes traditionally come from the
        business team and might require the data team to implement the changes into the data catalog.</p>
    <p>Documenting your data should be essential to any data-driven business process. It allows data practitioners to see
        how their work is used, by whom, and for what purpose. Just the act of documenting the data will help you understand
        what you have, where it is, and how it can be used. It may seem like an intimidating task at first, but I encourage
        you to start small, with a few datasets here and there, until the process becomes second nature. And once it does,
        you'll never look back!</p>
    <h3>Final thoughts</h3>
    <p>To sum it up, data catalogs are valuable for any organization, especially those growing quickly. No matter the size,
        data documentation is important in the pursuit of data operations. Having a tool that can track, document, store and
        provide metadata insights for systems beyond just your current data stack will help you identify your blind spots in
        your current architecture and integration points moving forward.&nbsp;</p>
    <p>Teams that invest the time to get alignment using a data catalog can see major benefits in the long term as they make
        faster decisions as a team. Creating a data catalog is not a small undertaking,&nbsp;<a
            href="https://www.secoda.co/blog/create-a-data-dictionary" style="color: #f66640;">that's why we make it easier
            with Secoda</a>. The
        process requires patience and alignment with leadership.&nbsp;</p>

How to Create A Data Catalog, A Step-by-Step Guide

Things I Wish I Knew When I Was Building a Data Team

<p>Let&rsquo;s not mince words. Product led growth (PLG) isn&rsquo;t something that happens overnight. It has to infuse company culture and involves commitment from every team - not just the go-to-market teams on the front lines.&nbsp;</p>
      <p>PLG introduces a lot of new data about how customers are using the product that GTM teams can leverage, but this data presents new challenges to data teams who are tasked with figuring out how to model data in a scalable and reliable way. It doesn&rsquo;t help that data leaders are being constantly bombarded with one-off analytics requests.&nbsp;</p>
      
      <p>Although the situation sounds dire, enabling PLG data will pay dividends. Revenue teams at product led companies need to leverage product data in order to run the most effective, scalable go-to-market (GTM) motions.&nbsp;</p>
      <ul role="list">
      <li>Sales teams can use product data to identify the best customers to connect with so that they are spending time on the customers most likely to buy.&nbsp;</li>
      <li>Marketing teams can use product data to refine nurture campaigns to provide a more personalized, relevant brand interaction with users.&nbsp;</li>
      <li>Customer Success teams can use product data to identify superstar customers or proactively reach out to customers who are at risk to churn.&nbsp;</li>
      </ul>
      
      <p>We&rsquo;ve spoken to hundreds of companies implementing PLG or looking to transition to PLG, and throughout all these conversations, we&rsquo;ve identified 5 common obstacles data teams need to address when it comes to enabling GTM teams with product data. In this blog post, we&rsquo;ll elaborate on these 5 challenges and provide suggestions on how to address them.</p>
      
      <h3 id="toc-0">The 5 common challenges for PLG data teams</h3>
      <ul role="list">
      <li>Data is generated by multiple teams across your company</li>
      <li>Data isn't owned by the stakeholders who use it (i.e. your revenue team)</li>
      <li>Data is surfaced in platforms that teams don&rsquo;t use on a daily basis</li>
      <li>Data that is surfaced in downstream platforms is overwhelming in scope</li>
      <li>Data that is converted into PQLs or Scores become black boxes that are rarely actionable</li>
      </ul>
      
      <h3 id="toc-1">Data is generated by multiple teams across your company</h3>
      <p>Every team across your company is generating data about customers on a daily basis.&nbsp;</p>
      <ul role="list">
      <li>Sales teams are manually inputting data about opportunities in their CRMs, tracking activity, or automatically enriching accounts and contacts with tools like Clearbit and ZoomInfo.&nbsp;</li>
      <li>Marketing teams are automatically adding leads into CRMs and tracking marketing activity.&nbsp;</li>
      <li>Product teams are tracking product usage data in data warehouses.&nbsp;</li>
      <li>Customer Success teams are working with customers through tools like Intercom.&nbsp;</li>
      </ul>
      <p>The data inputs are endless and are not uniformly collected, which means that everything lives in different data silos. It&rsquo;s difficult to stitch all this data together into a single view for all teams to consume.</p>
      <p>Further, when different teams are generating data, they are generating it for their own use cases rather than thinking about how they can scale their data sets across the organization. We often observe this when sales teams get access to Segment data, which is often generated by Marketing or Product. Sales teams don&rsquo;t understand what various button clicks mean, making the barrier to adopting Segment data in sales processes that much higher. This is a challenge we navigated over at Correlated as well! For example, we track the various actions that users automate in our product and build alerts around &ldquo;exceptions&rdquo;. However, &ldquo;exceptions&rdquo; are labeled differently for different actions, so we have to catch all the labels correctly in order to get a complete alert for all &ldquo;exceptions&rdquo;. Obviously, if we threw this raw data at our sales team, they would have absolutely no idea how to use it.</p>
      <h4>Suggestion: Leverage a data warehouse as your source of truth and invest in enabling technologies in your data stack</h4>
      <p>Ultimately, the customers who are most successful in implementing a PLG GTM motion with us are the ones who have a data warehouse. This is because a data warehouse can act as your source of truth for all your customer and user data. It is certainly a challenge to sync all of your data sources with your data warehouse, but it is not an insurmountable one. ETL tools like&nbsp;<a class="mds-text-primary"  href="https://www.fivetran.com/">Fivetran</a>,&nbsp;<a class="mds-text-primary"  href="https://www.stitchdata.com/" target="_blank" rel="noopener">Stitch</a>, and&nbsp;<a class="mds-text-primary"  href="https://airbyte.com/">Airbyte</a>&nbsp;make it possible to do this using a single platform.&nbsp;</p>
      <p>Once all the data is loaded into your data warehouse, you can leverage all the features and functionalities of data warehouses to shape and model your data. We highly recommend&nbsp;<a class="mds-text-primary"  href="https://www.getdbt.com/">dbt</a>&nbsp;as a great option for those looking to scale their data modeling across their organization.</p>
      <p>Leveraging a data warehouse is not without its limitations. It requires deep institutional knowledge and expertise from data teams in order to unblock key business challenges, and of course this becomes a cost center. Investing in tools that allow teams to self-serve their way into pre-built models is a great option, and there are many tools in the market today that can help. For example, Reverse ETL solutions like&nbsp;<a class="mds-text-primary"  href="https://www.getcensus.com/">Census</a>&nbsp;and&nbsp;<a class="mds-text-primary"  href="https://hightouch.io/">Hightouch</a>&nbsp;connect into the models your data teams have built and make it easy for downstream applications to consume that data. This makes it possible for your data teams to scale their efforts to support multiple use cases. BI solutions like&nbsp;<a class="mds-text-primary"  href="https://www.tableau.com/">Tableau</a>,&nbsp;<a class="mds-text-primary"  href="https://www.looker.com/">Looker</a>, and more recently,&nbsp;<a class="mds-text-primary"  href="https://hex.tech/">Hex</a>, provide business users with access to pre-modeled data. Tools like&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/">Correlated</a>&nbsp;sit on top of your data warehouse to enable business users to build automated playbooks.&nbsp;</p>
      
      <h3 id="toc-2">Data isn't owned by the stakeholders who use it</h3>
      <p>One of the biggest challenges with implementing a data warehouse as described above is that you have to figure out how to enable the various personas who need that data. If all you do is dump data into a data warehouse, none of the teams who rely on that data to do their jobs will be able to access it. If left to fester, data teams and GTM teams can ultimately reach a stalemate where data teams have implemented processes that prevent GTM teams from getting what they want. Alternatively, GTM teams could be requesting data team resources all day every day, completely inundating data teams with requests. We&rsquo;ve seen both situations at companies, and it is ugly.</p>
      <h4>Suggestion: Focus on driving alignment between data teams and business goals. Make cross-team collaboration part of every day processes.</h4>
      <p>Companies should view their data teams as a pooled resource that supports the entire company. As a result, data team resources should be allocated and scheduled similarly to how engineering teams schedule and allocate time to work on product roadmap. It&rsquo;s important, however, to not treat the data team as a separate silo where requests are sent (and often added to the backlog). Instead, scheduling and allocation of resources should happen hand in hand with the business units who will benefit from data team resources. This is the model that we&rsquo;ve seen work best in companies with successful PLG GTM motions that leverage data.</p>
      <p>One of the reasons why it&rsquo;s important to intertwine business use cases with data team projects is that we&rsquo;ve rarely seen companies successfully implement a product-led data strategy without doing so. If you send the data team off to &ldquo;solve our usage tracking problems,&rdquo; they&rsquo;ll end up coming up with a solution that might not work with the downstream tools your business teams use, or they&rsquo;ll try to find a solution that works for everything, essentially boiling the ocean on all possibilities. This ends up taking upwards of a year to implement, leaving GTM teams flying blind in the meantime.&nbsp;</p>
      <p>Instead, focus on use cases that you as a business want to accomplish and build out the infrastructure, models, and views that can support those use cases. Leveraging data is a core strategic competency for SaaS businesses, so it&rsquo;s not realistic to expect to never have to touch your existing data models again. This is why the tools you choose to use are so important - although you will have to adjust your data models from time to time, it doesn&rsquo;t have to be painful every single time.</p>
      
      <h3 id="toc-3">Data is surfaced in platforms that revenue teams don&rsquo;t use on a daily basis</h3>
      <p>Typically when companies start scratching the surface on leveraging product data to run a product led growth motion, they&rsquo;ll take an initial approach of driving &ldquo;visibility&rdquo;. This is essentially Looker, Mode, Tableau, or some other business intelligence (BI) dashboard that shows things like how an account is using the product or a list of &ldquo;product qualified leads&rdquo;. This is a worthwhile exercise because it allows your data team to build out the data models that your GTM teams can benefit from.&nbsp;</p>
      <p>However, time and time again, we&rsquo;ve heard feedback that GTM teams simply do not look at dashboards. In the rare cases that they do, they find that dashboards are rarely actionable. For example, let&rsquo;s say that you have a dashboard that lists out PQLs. An AE can&rsquo;t just take an email off that dashboard and email them. That&rsquo;s because (1) they have to figure out if they&rsquo;ve already emailed the user before (check Salesforce/Hubspot), (2) they have to figure out what to say to the user (check another dashboard on product usage), (3) they aren&rsquo;t sure why the user is even a PQL in the first place (Slack data team to ask how a PQL is triggered).</p>
      <h4>Solution: Pipe data into downstream applications.</h4>
      <p>Rather than expecting GTM teams to look at dashboards and analyze the data, send data directly to the downstream applications that they live in on a daily basis. We&rsquo;ll talk about some of the downsides to this approach in the next &ldquo;problem,&rdquo; but some approaches we&rsquo;ve seen companies take is to build very specific views in Salesforce that contain all the information an AE (account executive) or CSM (customer support manager) needs to reach out in an intelligent way. For example, on the Account view, they might include things like: seats sold, seats utilized, number of active users. This allows the revenue team to stay in the tools they use, but still see the data they need to reach out to customers appropriately.</p>
      
      <h3 id="toc-4">Data that is surfaced in revenue team platforms is overwhelming in scope</h3>
      <p>This problem is linked in many ways to the prior problem - when companies discover that their teams are not using dashboards, their initial response is to pipe&nbsp;<em>data</em>&nbsp;into downstream applications. Now, I highlight&nbsp;<em>data</em>&nbsp;here because data is messy, there is a lot of it, and it requires analysis. We&rsquo;ve seen companies pipe so much data into Salesforce that they literally run out of custom fields. As an AE or CSM looking at Salesforce, those views simply become information overload. You end up giving so much information to your GTM teams that they literally cannot do anything with it.</p>
      <h4>Solution: Strive to pipe actionable insights over data.</h4>
      <p>A better approach is to think about the end goal you want to achieve by sending data to downstream applications. Are you trying to surface a lead? Are you trying to suggest a next-best-action? That should determine what data you send and how you send it.&nbsp;</p>
      <p>For example, let&rsquo;s dig into the &ldquo;seats sold&rdquo; and &ldquo;seats utilized&rdquo; example. You could pipe that information into Salesforce and surface that as a custom field on the Account object. However, wouldn&rsquo;t it be better if you created a lead in Salesforce, assigned it to an AE, and added a note that said - this customer is hitting 80% license utilization, add them to the &ldquo;Upsell&rdquo; Outreach sequence? You are sending insights that are immediately actionable.&nbsp;</p>
      <p>Some patterns we&rsquo;ve seen that we really like are: auto-creating Salesforce Leads, auto-creating Salesforce Opportunities, adding users to Outreach or Salesloft Sequences / Cadences, adding users to lists in Hubspot. The opportunities are endless. By thinking about insights and expected action over just data, you can ensure that all your efforts to build out a product-led data strategy are actually adopted by the GTM teams who have to put them into practice.</p>
      
      <h3 id="toc-5">Data that is converted into PQL Scores becomes a black box</h3>
      <p>Final topic: PQL Scoring. People really love PQL Scores recently, but it&rsquo;s something that you should definitely approach with care. In some ways, PQL Scores are the ultimate effort to convert data into insights. E.g. I crunched all the data for you, and this customer is great to reach out to. The problem is that GTM teams need to know how to reach out, which means they have to understand the &ldquo;<em>why&rdquo;</em>&nbsp;behind PQL Scores. If they don&rsquo;t understand the why, they are not going to adopt the score. You&rsquo;ll spend 6-12 months building out a score that ultimately no one uses.&nbsp;</p>
      <p>This is not to say scores are not useful - they certainly are, particularly when you&rsquo;re trying to get an overall directional feel of if someone is worth reaching out to. However, they should only be used as one piece of the puzzle and should not be viewed as a solution to PLG.</p>
      <h4>Solution: Think about the playbooks you want to enable after receiving an insight</h4>
      <p>There are a lot of vendors out there claiming to surface leads &ldquo;magically&rdquo; and with &ldquo;machine learning&rdquo;. We ourselves run machine learning models to help our customers identify which Signals might be interesting to go after, so it&rsquo;s a viable strategy and can surface some interesting insights. However, again, insights are more valuable if they drive a downstream action.&nbsp;</p>
      <p>If you&rsquo;re thinking of using a score, we&rsquo;d suggest tying it to a playbook you want to execute. For example, create an &ldquo;Upsell&rdquo; score, and align it with an &ldquo;Upsell&rdquo; playbook. Create an &ldquo;Onboarding Self-Serve&rdquo; score, and align it with an appropriate playbook. If you simply use one score, your results won&rsquo;t be as good as focusing on personalized, actionable playbooks that drive real results.&nbsp;</p>
      
      <h3 id="toc-6">Time to dive in!</h3>
      <p>We&rsquo;ve explored a lot of surface area in this blog post, from data warehouses to delivering that data in actionable ways to GTM teams. This may sound like a lot, but we&rsquo;ve seen a lot of companies get to success by taking things one step at a time. What we&rsquo;ve seen work best is actually a more vertical approach. Rather than trying to solve problem 1, then moving onto problem 2, etc etc, pick one important business challenge that aligns with a top-line goal and solve all the problems vertically.</p>
      <p>For example, let&rsquo;s say that your business wants to drive self-serve conversion. Next, identify the &ldquo;actions&rdquo; you want your GTM teams to take, the &ldquo;data&rdquo; you need to make that possible, and the &ldquo;tools&rdquo; that you&rsquo;ll need to tie everything together. Finally, go forth and do it! By aligning data teams with actionable business outcomes, you&rsquo;ll be able to get the backing and support you need across teams to get things done.&nbsp;</p>
      <p>If you'd like to learn how Correlated can assist your product-led journey,&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/get-started">get started</a>&nbsp;for free or&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/demo-request">schedule a call</a>&nbsp;with our team.</p>

5 challenges Data Teams face supporting PLG revenue teams and how to address them

<div>&nbsp;</div>
   <p>Active metadata is like hot gossip. Here&rsquo;s why.</p>
   <p id="4e49">
     Just like data mesh or the metrics layer,&nbsp;<em class="mj"
       >active metadata</em
     >
     is the latest hot topic in the data world. As with every other new concept
     that gains popularity in the data stack, there&rsquo;s been a sudden explosion
     of vendors rebranding to &ldquo;active metadata&rdquo;, ads following you
     everywhere, and&hellip; confusion.
   </p>
   <p id="22a1">
     With everyone talking about active metadata, it must be pretty easy to
     understand, right?
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go mk">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*oj53dlEWig_Hcxmtj4GKJA.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go ml">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*bF3m6G6XM4D4UrYgMD8NsQ.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <p id="5481">
     Apparently not! I&rsquo;ve been talking about active metadata for over a year
     now, but I still see questions like these all the time.
   </p>
   <p id="c0f1">
     Active metadata can sound a bit scary, but it doesn&rsquo;t have to be. It is
     a must-have tool in the modern data toolbox, so if you&rsquo;re still
     wondering what it means, this article is for you.
   </p>
   <p id="76b6">
     I&rsquo;ve broken down the ideas behind active metadata with as little jargon
     as possible. Keep reading to learn what active metadata is, what it looks
     like, how you can actually use it, how it fits into the modern data stack, and
     why it even matters.
   </p>
   <h3 id="6b2d">What is active metadata?</h3>
   <p id="569c">
     I could start dropping some jargon here, but then both you and I will be
     asleep in seconds. So let&rsquo;s jump into an analogy instead.
   </p>
   <p id="4dd5">
     Imagine that you got your hands on the juiciest piece of tech gossip &mdash;
     Apple is expanding into recreational marijuana to literally help people
     &ldquo;think different&rdquo;.
   </p>
   <p id="aee1">
     There&rsquo;s no way you&rsquo;re going to keep something this exciting a
     secret. The world has to know. So you post it on your blog,&nbsp;<em
       class="mj"
       >blogspot.applefansunite.com</em
     >. All done, right?
   </p>
   <p id="b968">
     Just like&nbsp;<a
       class="mds-text-primary"
       href="https://www.vice.com/en/article/n7n74d/traffic-jams-are-possible-in-elon-musks-tunnels-apparently"
       target="_blank"
       rel="noopener ugc nofollow"
       >a car in the Hyperloop tunnel</a
     >, we all know that&rsquo;s not going anywhere. You can&rsquo;t just put the
     story somewhere and hope people will find it. You have to actually deliver it
     into people&rsquo;s hands.
   </p>
   <p id="47ef">
     You sharpen your PR chops, blast the news to tech reporters and news sites,
     and lo and behold it&rsquo;s&nbsp;<em class="mj">everywhere</em> in no time.
     It&rsquo;s already been mummified, and your grandfather just asked why apple
     farmers are talking about this Molly girl on your group chat.
   </p>
   <blockquote class="nj">
     <p id="988d" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt">Metadata is like this information. If it sits&nbsp;</em
       >passively<em class="nt"
         >&nbsp;in its own little world, with no one seeing or sharing it, does it
         even matter? But if it&nbsp;</em
       >actively<em class="nt"
         >&nbsp;moves to the places where people already are, it becomes part of
         and adds context to a larger conversation.</em
       >
     </p>
   </blockquote>
   <h3 id="36ed">What is the difference between active and passive metadata?</h3>
   <p id="e3db">
     <strong>Passive metadata</strong>&nbsp;is the standard way of aggregating and
     storing metadata into a static data catalog. This usually covers basic
     technical metadata &mdash; schemas, data types, models, etc.
   </p>
   <p id="4780">
     Think of passive metadata as putting out information on a personal blog. Every
     so often, it&rsquo;ll get picked up and go viral on Hacker News. But most of
     the time it&rsquo;s just going to sit unseen and unused, even when people
     actually need to know it.
   </p>
   <p id="c214">
     <strong>Active metadata</strong>&nbsp;makes it possible for metadata to flow
     effortlessly and quickly across the entire data stack, embedding enriched
     context and information in every tool in the data stack. It is usually more
     complex than passive metadata, covering operational, business, and social
     metadata along with basic technical information.
   </p>
   <p id="16a6">
     Think of active metadata as a viral story. It shows up everywhere you already
     live in what seems like seconds. It&rsquo;s immediately cross-checked against
     and combined with other information, bringing together a network of related
     context into a larger trend or story. And it sparks conversations, making
     everyone more knowledgable and informed in the end.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go nx">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*h71nNqp0b03WgO_XI2o8fw.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt">The four main types of metadata. (Image by author.)</em>
     </figcaption>
   </figure>
   <h3 id="7541">Why does active metadata matter?</h3>
   <blockquote class="nj">
     <p id="302a" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt"
         >To put it simply, no one wants to go to another website to &ldquo;browse
         the metadata&rdquo;.</em
       >
     </p>
   </blockquote>
   <p id="ca3d">
     As we embraced the internet and data exploded in the early aughts, companies
     realized they needed to manage all their new data.
   </p>
   <p id="091a">
     We entered a golden age of metadata management. New companies like&nbsp;<a
       class="mds-text-primary"
       href="https://www.informatica.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Informatica</a
     >,&nbsp;<a
       class="mds-text-primary"
       href="https://www.collibra.com/us/en"
       target="_blank"
       rel="noopener ugc nofollow"
       >Collibra</a
     >, and&nbsp;<a
       class="mds-text-primary"
       href="https://www.alation.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Alation</a
     >&nbsp;were created, and they hyped the importance of data catalogs. People
     needed a way to sort through all their options, so we got reports like
     Gartner&rsquo;s&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/3993025"
       target="_blank"
       rel="noopener ugc nofollow"
       >Magic Quadrant for Metadata Management</a
     >. Billion-dollar companies emerged, and companies spent hundreds of millions
     of dollars on metadata management.
   </p>
   <p id="41e1">
     Yet just last year, Gartner released their&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/4006759"
       target="_blank"
       rel="noopener ugc nofollow"
       >Market Guide for Active Metadata</a
     >&nbsp;and declared that &ldquo;Traditional metadata practices are
     insufficient&hellip;&rdquo;
   </p>
   <p id="5d60">
     That&rsquo;s because passive data catalogs solve the &ldquo;too many
     tools&rdquo; problem by adding&hellip; another tool. They aggregate metadata
     from different parts of the data stack, and it stagnates there. User adoption
     suffers, and&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-future-of-data-catalogs-868ffbe7354d"
       target="_blank"
       rel="noopener"
       >these exciting tools turn into expensive shelfware</a
     >.
   </p>
   <p id="2761">
     Active metadata sends metadata back into every tool in the data stack, giving
     the humans of data context wherever and whenever they need it &mdash; inside
     the BI tool as they wonder what a metric actually means, inside Slack when
     someone sends the link to a data asset, inside the query editor as try to find
     the right column, and inside Jira as they create tickets for data engineers or
     analysts.
   </p>
   <h3 id="050b">How does active metadata fit into the modern data stack?</h3>
   <p id="68c9">
     Active metadata functions as a layer on top of the modern data stack.
   </p>
   <p id="bdd2">
     It leverages open APIs to connect all the tools in your data stack and ferry
     metadata back and forth in a two-way stream. This is what allows active
     metadata to bring context, say, from Snowflake into Looker, Looker into Slack,
     Slack into Jira, and Jira back into Snowflake.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go od">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*53oiNCxKcmY9bLFNCKNnfg.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Active metadata in the modern data stack. (Image by author.)</em
       >
     </figcaption>
   </figure>
   <h3 id="dbd9">4 characteristics of active metadata</h3>
   <p id="e3b1">
     According to&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-gartner-magic-quadrant-for-metadata-management-was-just-scrapped-d84b2543f989"
       target="_blank"
       rel="noopener"
       >Gartner&rsquo;s new Market Guide for Active Metadata</a
     >, active metadata is an always-on, intelligence-driven, action-oriented,
     API-driven system, the opposite of its passive, static predecessor.
   </p>
   <p id="0837">
     This can be broken down into the four key characteristics of active metadata.
   </p>
   <ul>
     <li id="4667">
       <strong>Always on</strong>: Active metadata is always on. Rather than
       waiting for people to manually enter or parse metadata, this means
       continually collecting metadata at every stage of the modern data stack
       &mdash; logs, query history, usage statistics, and more.
     </li>
     <li id="fd88">
       <strong>Intelligent</strong>: Active metadata isn&rsquo;t just about
       collecting metadata. It&rsquo;s about constantly processing metadata to
       connect the dots and create intelligence from it. This means that with
       active metadata, the system will only get smarter over time as people use it
       more and it observes more metadata.
     </li>
     <li id="c801">
       <strong>Action-oriented</strong>: Active metadata doesn&rsquo;t just stop at
       intelligence. It should drive action by curating recommendations, generating
       alerts, and making it easier for people to make decisions &mdash; or even
       automatically making decisions without human intervention, like stopping
       downstream pipelines when data quality issues are detected.
     </li>
     <li id="3a52">
       <strong>Open by default</strong>: Active metadata platforms use APIs to hook
       into every piece of the modern data stack. This makes magical user
       experiences possible by saving data practitioners from the endless tool- and
       context-switching. This is called embedded collaboration, which is when work
       happens where you are with the least amount of effort.
     </li>
   </ul>
   <h3 id="a46d">5 use cases of active metadata</h3>
   <p id="b89a">
     There are dozens, if not hundreds, of use cases of active metadata. (Enough
     for several articles of their own &mdash; coming soon!) Let&rsquo;s go through
     a few of my favorites.
   </p>
   <ul>
     <li id="e13d">
       <strong>Purge stale or unused assets</strong>: Use active metadata to
       periodically calculate when each asset (e.g. a data table, dashboard, etc)
       was last used and/or how many people used it. If it was used within the last
       30 days, great! If an asset hasn&rsquo;t been used in the last 60 days,
       automatically archive it. If no one has touched it in the last 90 or 120
       days, purge it entirely.
     </li>
     <li id="a52a">
       <strong>Allocate compute resources dynamically</strong>: Imagine that 90% of
       users log in to a BI tool during the last week of a financial quarter.
       Active metadata can be used to automatically scale up compute resources just
       before that week and scale them down again afterward.
     </li>
     <li id="34bd">
       <strong>Enrich user experience in BI tools</strong>: Instead of switching
       between a BI tool and data catalog, use active metadata to bring context
       into dashboards. Relevant metadata (like business terms, descriptions,
       owners, and lineage) can be pushed into the BI tool. Then when someone is
       looking at each table, they can understand who owns it, where the data came
       from, etc. This information could even be used as labels in auto-generated
       reports.
     </li>
   </ul>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go os">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*aLuvXmyV2tyIr1YKbIbNZA.gif"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Enriching BI tools with active metadata. (Image from author.)</em
       >
     </figcaption>
   </figure>
   <ul>
     <li id="2ef1">
       <strong>Identify popular assets</strong>: Use active metadata to create a
       custom relevance score for each asset. This can be based on usage
       information from places like query logs, lineage, and BI dashboards. Then
       the most popular, relevant assets should be surfaced more frequently in
       search and checked more frequently for data quality issues.
     </li>
     <li id="d9f9">
       <strong>Notify downstream consumers</strong>: It&rsquo;s awful if the CEO
       ends up seeing a broken dashboard before the data team. Active metadata can
       be used to check for issues when a data store changes and notify downstream
       data users about potential issues. For example, when a data store is
       crawled, the new metadata would be compared against previous metadata. If
       there are any potential breaking changes (e.g. the addition or removal of a
       column), lineage could be used to find who owns this data store and notify
       them in Slack, Jira, email, etc.
     </li>
   </ul>
   <h3 id="aac4">The future of active metadata</h3>
   <p>
     As metadata becomes big data and big data becomes a behemoth, active metadata
     isn&rsquo;t just a wonderful dream. It&rsquo;s a necessity &mdash; the only
     way to understand today&rsquo;s data.
   </p>
   <p id="cb3d">
     Managing, processing, and analyzing metadata is the new normal for modern data
     teams. Doing this passively and manually, though, isn&rsquo;t possible.
     That&rsquo;s why it&rsquo;s been so exciting to see active metadata take shape
     in the last year and become the de facto standard for what people expect out
     of modern metadata.
   </p>
   <p id="f4b4">
     All of these use cases &mdash; like auto-tuned pipelines, automated data
     quality alerts, and continuously validated calculations &mdash; would have
     sounded wildly impossible just a few years ago. Today, they&rsquo;re actually
     in reach. I couldn&rsquo;t be more excited to see the intelligent data dream
     become a reality as active metadata continues to evolve in the coming years.
   </p>    
   

What Is Active Metadata, and Why Does It Matter?

<p><em>The modern data stack is on the rise. Many companies use raw data from their SaaS analytics tools as input for their data warehouse, but this introduces problems downstream. Are there better ways?</em></p>
      <h3 id="popularization-of-the-all-in-one-saas-analytics-tool" class="anchor anchorWithStickyNavbar_mojV">Popularization of the all-in-one SaaS analytics tool<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#popularization-of-the-all-in-one-saas-analytics-tool">​</a></h3>
      <p>When Google Analytics launched nearly 20 years ago, it took the market by storm. Its success created the mold for SaaS-based analytics offerings that included everything in one box: data collection, storage, analysis &amp; visualization.</p>
      <p>Ease-of-use and low barrier of entry were two of its main drivers. Instrumentation typically required little in terms of engineering resources and the included analyses, although somewhat limited, were ready to be used by a broad audience.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/urchin.jpeg" alt="Urchin Analytics, to become Google Analytics after acquisition." />
      <p class="imageCaption___5l">Urchin Analytics, to become Google Analytics after acquisition.</p>
      </div>
      <h3 id="the-big-shift-towards-data-warehousing" class="anchor anchorWithStickyNavbar_mojV">The big shift towards data warehousing<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-big-shift-towards-data-warehousing">​</a></h3>
      <p>While SaaS analytics tools are still very popular and widely used, their wild growth has left the data analysis landscape highly fragmented. Most companies have numerous tools collecting data, and each of them have a different way of analyzing &amp; consuming it. As a result, it&rsquo;s very hard to get a holistic view of what all that combined data is telling you.</p>
      <p>We&rsquo;re seeing a major shift where companies export data from their SaaS analytics tools to use as input for their data warehouse. Not only to create a single source of truth for all data consumers, but also to unlock the potential of raw data. By combining full raw data sets, you can go way beyond the scope of your SaaS analytics tools. You&rsquo;re technically only limited by the quality and completeness of the data.</p>
      <h3 id="the-rise-of-the-modern-data-stack" class="anchor anchorWithStickyNavbar_mojV">The rise of the modern data stack<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-rise-of-the-modern-data-stack">​</a></h3>
      <p>The big shift towards data warehousing is represented by the upsurge of companies that operate in this space and how fast they&rsquo;re growing.&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://snowplowanalytics.com/" target="_blank" rel="noopener noreferrer">Over 10.000 companies now use Snowplow</a>&nbsp;to process events at scale and send them straight into the DWH. dbt now has over&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://www.getdbt.com/blog/next-layer-of-the-modern-data-stack/" target="_blank" rel="noopener noreferrer">9.000 weekly active data transformation projects</a>.&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://www.rudderstack.com/" target="_blank" rel="noopener noreferrer">13.000+ sites and apps are running RudderStack</a>&nbsp;to pull data from SaaS tools and send it into data warehouses. The list goes on.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/dbt-cloud.png" alt="dbt Cloud in action" />
      <p style="text-align: center;" class="imageCaption___5l">dbt Cloud in action</p>
      </div>
      <h3 id="-but-saas-analytics-tools-arent-great-input-for-your-dwh" class="anchor anchorWithStickyNavbar_mojV">... but SaaS analytics tools aren&rsquo;t great input for your DWH<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#-but-saas-analytics-tools-arent-great-input-for-your-dwh">​</a></h3>
      <p>Most data warehouses are fed with data that&rsquo;s collected by all-in-one-box SaaS analytics tools. While historically explainable, this often causes problems downstream.</p>
      <h3 id="the-data-they-collect-is-not-designed-for-advanced-modeling" class="anchor anchorWithStickyNavbar_mojV">The data they collect is not designed for advanced modeling<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-data-they-collect-is-not-designed-for-advanced-modeling">​</a></h3>
      <p>There is a big difference between what data teams want their raw data to look like and what it actually looks like when it comes from SaaS analytics tools. The data sets they collect were never designed for advanced modeling on the raw data. It is often incomplete or overcomplete, unstructured and ambiguous. Significant grunt work is typically required before it can be used for modeling.</p>
      <h3 id="data-duplication-frustrates-adoption-of-a-single-source-of-truth" class="anchor anchorWithStickyNavbar_mojV">Data duplication frustrates adoption of a single source of truth<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#data-duplication-frustrates-adoption-of-a-single-source-of-truth">​</a></h3>
      <p>By extracting data from SaaS tools and loading it into a data warehouse, you&rsquo;re effectively duplicating a source.</p>
      <p>SaaS analytics tools all offer their own way to consume the collected data, and chances are many of your company&rsquo;s data consumers still rely on the dashboards of these tools to inform their daily decisions. Either out of habit, or simply because it offers an easy &amp; fast way to get data.</p>
      <p>In practice, this often leads to discussions about why numbers of different teams don&rsquo;t seem to be adding up, as the data team and data consumers both use different versions of the same source of data as input.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/before-saas.png" alt="" />
      <p class="imageCaption___5l">&nbsp;</p>
      </div>
      <h3 id="they-operate-as-black-boxes" class="anchor anchorWithStickyNavbar_mojV">They operate as black boxes<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#they-operate-as-black-boxes">​</a></h3>
      <p>SaaS analytics tools typically don&rsquo;t disclose the actual models behind an analysis. As a result, rebuilding an analysis to work with the data set in your data warehouse requires significant reverse engineering effort, and troubleshooting &amp; debugging can be painful when anything is off.</p>
      <hr />
      <h3 id="unbundling-saas-analytics-tools" class="anchor anchorWithStickyNavbar_mojV">Unbundling SaaS analytics tools<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#unbundling-saas-analytics-tools">​</a></h3>
      <p>The way data is used has evolved, and the all-in-one-box model that SaaS analytics tools use no longer fits. In order to effectively combine and analyze data from multiple sources and truly serve all data consumers from a single source of truth, we need to unbundle the black box and separate data collection, storage, modeling and visualization.</p>
      <h3 id="moving-all-data-consumption-downstream" class="anchor anchorWithStickyNavbar_mojV">Moving all data consumption downstream<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#moving-all-data-consumption-downstream">​</a></h3>
      <p>By separating data collection, all collected data can be sent straight into the data warehouse without stopovers. This eliminates source duplication and moves data consumption downstream. It enables&nbsp;<em>all</em>&nbsp;analysis &amp; visualization to occur on a single source of truth: the data warehouse.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/after-saas.png" alt="" />
      <p class="imageCaption___5l">&nbsp;</p>
      </div>
      <h3 id="shifting-from-vendor-locked-tools-to-raw-data-and-code" class="anchor anchorWithStickyNavbar_mojV">Shifting from vendor-locked tools to raw data and code<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#shifting-from-vendor-locked-tools-to-raw-data-and-code">​</a></h3>
      <p>Unbundling the black boxes also provides a big opportunity for the adoption of open standards. Shifting from vendor-locked tools to raw data and code enables data teams to adopt generic ways of structuring data to share and reuse each others&rsquo; tools, models and analyses.</p>
      <p>We think the future of analytics is open and data warehouse-native, and there are clear signals we&rsquo;re already on the way. It&rsquo;s time to enable meaningful collaboration and to unbundle SaaS analytics.</p>

The Unbundling of SaaS Analytics

<p>Breaking down some of the problems I’ve seen in data collaboration and offering advice on how to make better, faster decisions with collaborative analytics.</p>
<p>We all know how analytics can help make businesses make better decisions. Being data-driven just simply isn&rsquo;t an option anymore. Startups are collecting more data than ever before and are increasingly using that data to&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://a16z.com/2011/08/20/why-software-is-eating-the-world/">eat the world</a>. We&rsquo;ve seen the rise of&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://www.whatmatters.com/articles/ted-talk">OKR goal-setting</a>, reporting tabs in nearly every SaaS tool, and metrics dashboards lining the walls in office hallways.<br /><br />Yet for all of this progress, the ways we share insights and make decisions together haven&rsquo;t evolved over the last decade. Too many companies spend too long collecting, preparing, and reporting on data to improve. In this post, I&rsquo;ll break down some of the problems I&rsquo;ve seen first-hand in data collaboration and offer advice on how you can create a better environment for creating, sharing, and measuring data together.</p>
<h2 id="the-problems">The problems</h2>
<h3 id="finding-a-source-of-truth"><br />Finding a Source of Truth</h3>
<p>Startups now have a plethora of dashboards in our tools to find data. But it&rsquo;s still too challenging to find the data you need. Legacy BI tools suffer from&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://venturebeat.com/2022/04/10/7-easy-ways-to-fight-the-data-sprawl/">dashboard sprawl</a>&nbsp;and poor search experiences. It&rsquo;s challenging to agree upon and standardize metrics to get common questions answered in a scalable way.</p>
<p>To fix this, we&rsquo;ve tried creating data catalogs so operators can more easily answer FAQs and find the data they&rsquo;re looking for. We&rsquo;ve tried to provide definitions for data to help operators understand what tables entail, how metrics are calculated, and what a particular column means. These efforts result in incremental progress towards becoming data-driven, but more is needed.</p>
<h3 id="data-breadlines">Data Breadlines</h3>
<p>Too few employees have access to real-time data to make everyday decisions. When operators need to query the database or ask a question involving data from multiple sources, they&rsquo;re&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/why-were-building-canvas-a-data-tool-for-non-technical-teams/">forced to ask for help</a>&nbsp;in a shared analytics channel or submit a ticket to their data or engineering teams.</p>
<p>This process results in a painful waiting period until the technical team can prioritize the request. Even when the data or engineering team can prioritize a request, there&rsquo;s typically a frustrating back-and-forth of questions and answers across Slack, Email, and Zoom.</p>
<p>Operators shouldn&rsquo;t have to wait days or even weeks for the answers they need, while technical teams should be able to focus on strategic work without constant pings for help on&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/enabling-ad-hoc-analysis/">tactical, ad-hoc questions.</a></p>
<h3 id="siloed-broken-workflows">Siloed, Broken Workflows</h3>
<p>Stop me if you&rsquo;ve gone through this process before. You need to prepare for a presentation. You go to your apps, pull CSVs, build some metrics, screenshot some charts, and throw them into a deck. You present the information and get some live questions about the data. Unfortunately, because the chart is a static screenshot, you can&rsquo;t dive into the data live, so you promise to follow up afterward.</p>
<p>The hours-long workflow we endure to prepare presentations across a suite of apps is fundamentally broken. There&rsquo;s no connective tissue between the data sources, the analysis tools, the presentation, and the feedback on the data.</p>
<p>Because of these silos, we may get fewer eyeballs from peers before a presentation. We might forget to include a killer insight or make a potential error in our analyses. Because it's too difficult to answer questions together live during a presentation, we can get caught flat-footed and look unprepared. And because it's so difficult to measure and discuss progress on metrics together, we fail to monitor updates closely and act quickly and decisively as teams.</p>
<h2 id="how-collaborative-analytics-helps">How Collaborative Analytics Helps</h2>
<p>To help teams move quickly to get the answers they need, you should consider investing in collaborative processes and tools. See below for some key components you should consider.</p>
<h3 id="the-warehouse-as-a-source-of-truth">The Warehouse as a Source of Truth</h3>
<p>Having a single source of truth that&rsquo;s accurate and up-to-date ensures your teams can all work off the correct data independently and confidently. Best-in-class companies typically use a data warehouse and ETL tools to centralize data from multiple sources in one place. If you don&rsquo;t know where to start, check out my guide on navigating the tools in the modern data stack.</p>
<p>Suppose you don&rsquo;t have the bandwidth or technical experts to implement a data stack correctly. In that case (time for a quick plug),&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/no-warehouse-no-problem/">Canvas provides a managed data stack</a>&nbsp;for centralizing your apps and modeling your data in one place.</p>
<h3 id="reusable-datasets-for-fast-discovery">Reusable Datasets for Fast Discovery</h3>
<p>With your data in one place, you&rsquo;ll need to organize it, provide context, and make it easy to find. Key metrics and common areas should be searchable and social proof should help teams quickly identify the right data to answer their questions.</p>
<p>You should have clear owners for building and maintaining dashboards on the company and team levels. They should define and document data clearly and ensure a clear process for getting help on questions. You should be aware of which dashboards are frequently used and invest accordingly, while continually improving or pruning dashboards that are ghost towns.</p>
<h3 id="collaborative-workspaces-for-exploration">Collaborative Workspaces for Exploration</h3>
<p>Shared, collaborative workspaces are critical in breaking down silos and helping teams share and discuss insights with others.</p>
<p>Yes, having dashboards are a must. But as I pointed out, your teams will still have everyday questions that dashboards don&rsquo;t answer. Instead of a ticketing system, you should have a way for your operators to answer these questions without knowing SQL.</p>
<p>No-code and spreadsheet-like interfaces help your operators gain self-serve access without going days or weeks for technical teams to answer strategic and tactical questions. And when they have a question, operators and technical teams can collaborate and comment right where the data lives. This way, you can ditch the screenshots, snippets, and Slack channels and start making decisions together from anywhere.</p>
<h2 id="canvas-the-collaborative-analytics-platform">Canvas, the Collaborative Analytics Platform</h2>
<p>At&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/">Canvas</a>, we&rsquo;re thrilled to be working on the future of data collaboration. Unlike legacy BI tools, Canvas has:</p>
<ul>
<li>Integrations to over a hundred apps so you can centralize and explore your data without a warehouse</li>
<li>A spreadsheet-like interface for your teams to pivot, write formulas, join, and create charts without SQL</li>
<li>Figma-like collaboration lets your teams communicate where the data lives</li>
<li>Out-of-the-box usage metrics inform data teams of who is using specific models so they can invest in popular areas or fix problems affecting less helpful models</li>
<li>Role-based access control so you can make sure the right people have access to the right data</li>
<li>A native dbt integration so your data models and documentation about the tables and columns are automatically synced and surfaced directly to operators</li>
</ul>

How to Make Better Decisions Together with Collaborative Analytics

<p class="p2">
    <span class="s1"
      >The term &ldquo;observability&rdquo; means many things to many people. A
      lot of energy has been spent&mdash;particularly among vendors offering an
      observability solution&mdash;in trying to define what the term means in one
      context or another.</span
    >
  </p>
  <p class="p2">
    <span class="s1"
      >But instead of getting bogged down in the &ldquo;what&rdquo; of
      observability, I think it&rsquo;s more valuable to address the
      &ldquo;why.&rdquo; What are we trying to accomplish with observability? What
      is the end goal?&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >At&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/why-unravel/"
        target="_blank"
        rel="noopener"
        >Unravel</a
      >, I&rsquo;m not only the co-founder and CTO but also the head of our
      Customer Success team, so when thinking about modern data stack
      observability, I look at the issue through the lens of customer problems and
      what will alleviate their pain and solve their challenges.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >I start by considering the ultimate end goal or Holy Grail: autonomous
      systems. Data teams just want things to work; they want everything to be
      taken care of, automatically. They want the system itself to be watching
      over issues, resolving problems without any human intervention at all.
      It&rsquo;s the true spirit of AI: all they have to do is tell the system
      what they want to achieve. It&rsquo;s going from reactive to
      proactive.&nbsp;</span
    ><span class="s3"
      >No one ever has to troubleshoot the problem, because it was
      &ldquo;fixed&rdquo; before it ever saw the light of day. The system
      recognizes that there will be a problem and auto-corrects the issue
      invisibly.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >As an industry, we&rsquo;re not completely at that point yet for the modern
      data stack. But we&rsquo;re getting closer.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >We are on a continuous and incremental spectrum of progressively less toil
      and more automation: going from the old ways of manually stitching together
      logs, metrics, traces, and events from disparate systems; to accurate
      extraction and correlation of all that data in context; to automatic
      insights and identification of significant patterns; to AI-driven actionable
      recommendations; to autonomous governance.</span
    >
  </p>
  <img
    style="width: 100%"
    src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-observability-spectrum-750x299.png"
    loading="lazy"
    alt=""
  />
  <p></p>
  <h3 class="p5">Beyond observability: to optimization and on to governance</h3>
  <p class="p2">
    <span class="s1"
      >Let&rsquo;s say you have a data pipeline from Marketing Analytics called
      ML_Model_Update_and_Scoring that&rsquo;s missing its SLA. The SLA specifies
      that the pipeline must run in less than 20 minutes, but it&rsquo;s taking 25
      minutes. What happened? Why is the data pipeline now too slow? And how can
      you tune things to meet the SLA? This particular application is pretty
      complex, with multiple jobs processing in parallel and several components
      (orchestration, compute, auto-scaling clusters, streaming platforms,
      dashboarding), so the problem could be anywhere along the line.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >It&rsquo;s virtually impossible to manually pore through t</span
    ><span class="s3"
      >he thousands of logs that are generated at each stage of the pipeline from
      the various tools</span
    ><span class="s2"
      >&mdash;Spark and Kafka and Airflow logs, Databricks cluster logs,
      etc.&mdash;to &ldquo;untangle the wires&rdquo; and figure out where the
      slowdown could be. But where should you even start? Even if you have an idea
      of what you&rsquo;re looking for, it can take hours&mdash;even days or weeks
      for highly complex workflows&mdash;to stitch together all the raw
      metrics/events/logs/traces data to figure out what&rsquo;s meaningful to why
      your data pipeline is missing its SLA. Just a single app can run 10,000
      containers on 10,000 nodes, with 10,000 logs.&nbsp;</span
    >
  </p>
  <img style="width: 100%" src="https://miro.medium.com/max/1200/1*XYQOw2kNJKZGKzBOrR0E7A.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s2">That&rsquo;s where observability comes in.</span>
  </p>
  <h3>
    <span id="Observability_tells_you_what"
      ><span class="s1">OBSERVABILITY TELLS YOU &ldquo;WHAT&rdquo;</span></span
    >
  </h3>
  <p class="p2">
    <span class="s1"
      >Instead of having to sift through reams of logs and cobble together
      everything manually,&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/dataops/"
        target="_blank"
        rel="noopener"
        >full-stack observability</a
      >&nbsp;extracts all the relevant raw data from all the various components of
      your data stack and correlates it to paint a picture of what&rsquo;s going
      on. All the information captured by telemetry data (logs, metrics, events,
      traces) is pulled together in context and in the &ldquo;language
      you&rsquo;re speaking&rdquo;&mdash;the language of&nbsp;</span
    ><span class="s4">data applications and pipelines</span
    ><span class="s1">.</span>
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-observability-before-2-750x264.png" loading="lazy" alt="" />
  <p></p>
  <p id="attachment_9257" class="wp-caption alignnone">
    In this case, observability shows you that while the application completed
    successfully, it took almost 23 minutes (22m 57s)&mdash;violating the SLA of
    20 minutes. Here, Unravel takes you to the exact spot in the complex pipeline
    shown earlier and, on the left, has pulled together granular details about the
    various jobs processing in parallel. You can toggle on a Gantt chart view to
    get a better view of the degree of parallelism:
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-gantt-750x544.png" loading="lazy" alt="" />
  <p></p>
  
  <p>
    So now you know what caused the pipeline to miss the SLA and where it happened
    (jobs #0 and #3), but you don’t know why. You’ve saved a lot of investigation
    time—you get the relevant data in minutes, not hours—and can be confident that
    you haven’t missed anything, but you still need to do some detective work to
    analyze the information and figure out what went wrong.
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-chart-1-750x171.png" loading="lazy" alt="" />
  <h3 class="p7">
    <span id="Optimization_tells_you_whyand_what_to_do_about_it"
      ><span class="s1"
        >OPTIMIZATION TELLS YOU &ldquo;WHY&rdquo;&mdash;AND WHAT TO DO ABOUT
        IT</span
      ></span
    >
  </h3>
  <p class="p2">
    <span class="s4"
      >The better&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/dataops/"
        target="_blank"
        rel="noopener"
        >observability tools</a
      >&nbsp;also point out, based on patterns, things you should pay attention
      to&mdash;or areas you don&rsquo;t need to investigate.&nbsp;</span
    ><span class="s1"
      >By applying machine learning and statistical algorithms, it essentially
      throws some math at the correlated data to identify significant
      patterns</span
    ><span class="s4"
      >&mdash;what&rsquo;s changed, what hasn&rsquo;t, what&rsquo;s different.
      This pipeline runs regularly; why was it slow this time?&nbsp;</span
    ><span class="s1"
      >It&rsquo;s the same kind of analysis a human expert would do, only done
      automatically with the help of ML and statistical algorithms.</span
    >
  </p>
  <p class="p2">
    <span class="s1"
      >While it would certainly be helpful to get some generalized insight into
      why the pipeline slowed</span
    ><span class="s4"
      >&mdash;it&rsquo;s a memory issue&mdash;what you really need to know is what
      to do about it.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >AI-enabled observability goes beyond telling you what happened and why, to
      pinpointing the root cause and providing recommendations on exactly what you
      need to do to fix it.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-ai-reco-750x468.png" loading="lazy" alt="" />
  <p class="p4">
    <span class="s2"
      ><span class="s3"
        >AI-driven recommendations provide specific configuration parameters that
        need to be applied in order for the pipeline to run faster</span
      >&nbsp;and meet the 20-minute SLA. After the AI recommendations are
      implemented, we can see that the pipeline now runs in under 11
      minutes&mdash;a far cry from the almost 23 minutes before.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-after-750x103.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s3"
      >Too often getting to the point of actually fixing the problem is another
      time-consuming effort of trial and error. Actionable AI recommendations
      won&rsquo;t fix things automatically for you</span
    ><span class="s2"
      >&mdash;taking the action still requires human intervention&mdash;but all
      the Sherlock Holmes work is done for you. The fundamental questions of what
      went wrong, why, and what to do about it are answered automatically.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >Beyond optimizing performance, AI recommendations can also identify
      where&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/cluster-resource-optimization/"
        target="_blank"
        rel="noopener"
        >cost could be improved</a
      >. Say your pipeline is completing within your SLA commitments, but
      you&rsquo;re spending much more on cloud resources than you need to. An AI
      engine can determine how many or what size containers you actually need to
      run each individual component of the job&mdash;vs. what you currently have
      configured. Most enterprises soon realize that they&rsquo;re overspending by
      as much as 50%.&nbsp;</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-cost-1-1024x476.png" loading="lazy" alt="" />
  <p></p>
  <h3 class="p7">
    <span id="Governance_helps_avoid_problems_in_the_first_place"
      ><span class="s1"
        >GOVERNANCE HELPS AVOID PROBLEMS IN THE FIRST PLACE</span
      ></span
    >
  </h3>
  <p class="p4">
    <span class="s2"
      >These capabilities save a ton of time and money, but they&rsquo;re still
      relatively reactive. What every enterprise wants is a more proactive
      approach to making their data applications and pipelines run better, faster,
      cheaper. Spend less time firefighting because there was no fire to put out.
      The system itself would understand that the memory configurations were
      insufficient and automatically take action so that pipeline performance
      would meet the SLA.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >For a whole class of data application problems, this is already happening.
      AI-powered recommendations and insights lay the groundwork for putting in
      place some automated governance policies that take action on your
      behalf.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-chart-2-750x141.png" loading="lazy" alt="" />
  <p></p>
  <p>
    Governance is really all about converting the AI recommendations and insights
    into impact. In other words, have the system run automatic responses that
    implement fixes and remediations for you. No human intervention is needed.
    Instead of reviewing the AI recommendation and then pulling the trigger, have
    the system apply the recommendation automatically.&nbsp;
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/new-alert.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s2"
      >Policy-based governance rules could be as benign as sending an alert to the
      user if the data table exceeds a certain size threshold or as aggressive as
      automatically requesting a configuration change for a container with more
      memory.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >This is true&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/aiops-for-big-data/"
        target="_blank"
        rel="noopener"
        >AIOps</a
      >. You don&rsquo;t have to wait until after the fact to address an issue or
      perpetually scan the horizon for emerging problems. The system applies AI/ML
      to all the telemetry data extracted and correlated from everywhere in your
      modern data stack to not only tell you what went wrong, why, and what to do
      about it, but it predicts and prevents problems altogether without any human
      having to touch a thing.</span
    >
  </p>
  

Beyond Observability for the Modern Data Stack

<p>
      A
      <a
        style="color: #f66640"
        href="https://venturebeat.com/2021/08/04/97-of-execs-say-data-science-is-crucial-to-maintaining-profitability/"
        target="_blank"
        >majority</a
      >
      of business leaders believe data insights are key to the success of their<br />business
      in a digital environment. However, many companies
      <a
        style="color: #f66640"
        href="https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard"
        target="_blank"
        >struggle</a
      >
      to build a<br />data-driven culture, with a key reason being the lack of a
      sound data democratization<br />strategy.
    </p>
    <p>
      A survey by Google Cloud and Harvard Business Review shows
      <a
        style="color: #f66640"
        href="https://cloud.google.com/resources/hbr-turn-data-into-business-value-report"
        target="_blank"
        >97%</a
      >
      of companies<br />recognize that democratizing data is critical to their
      success, but only
      <a
        style="color: #f66640"
        href="https://cloud.google.com/resources/hbr-turn-data-into-business-value-report"
        >60%</a
      >
      are<br />"effectively" moving toward that goal.
    </p>
    <p>But what's data democratization?</p>
    <p>
      Democratizing data in an organization is much more than just giving employees
      access<br />to data. It's a process that enables everyone in an organization,
      regardless of their<br />technical skills, to be comfortable working with
      data, feel confident talking about it, and<br />make data-driven decisions
      that can positively impact various aspects of the business.<br />It's an
      ongoing process because it goes hand in hand with data literacy, which is also
      a<br />continuous process.
    </p>
    <h3>The rise of all-in-one data solutions</h3>
    <p>
      Until recently, data was only used by a few technical specialists in
      organizations who<br />would set up and operate the systems needed to make
      data available and accessible to<br />the rest of the team. But as the data
      democratization trend grows, new tools are being<br />developed for
      non-technical professionals to provide organization-wide access to data<br />and
      analytics.
    </p>
    <p>
      Of the data software available on the market, all-in-one tools that offer a
      comprehensive<br />range of solutions through a single platform are the most
      popular options for many<br />organizations looking to extract more value from
      their data.
    </p>
    <p>
      There are two main challenges that all-in-one data tools are designed to
      address,<br />namely the gap between data and commercial teams, and the
      expensive and<br />time-consuming nature of the process of data
      maturity&mdash;a measurement of how capable<br />an organization is at making
      the most of its data.
    </p>
    <h3>The gap between data and commercial teams</h3>
    <p>
      Data and commercial teams are often siloed functions within a company, and
      it's not<br />uncommon for them to fail to communicate with each other and
      even get in each other's<br />way. Different training backgrounds, different
      goals, or lack of context are just some of<br />the reasons.
    </p>
    <p>
      This inevitably leads to inefficiencies as each department takes a different
      direction on a<br />given project. In fact,
      <a
        style="color: #f66640"
        href="https://blog.bit.ai/collaboration-statistics/"
        target="_blank"
        >97%</a
      >
      of employees and executives believe lack of communication<br />and alignment
      within an organization impacts the outcome of a task or project.
    </p>
    <p>
      Better collaboration and cross-functional coordination between different
      departments is<br />therefore a key success factor, especially in today's
      world of remote work where a<br />company's growth and survival depends more
      than ever on effective teamwork.
    </p>
    <h3>Data maturity being an expensive and long process</h3>
    <p>
      Data maturity can be a costly and lengthy process for many organizations. A
      <a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        >Gartner</a
      ><br /><a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        >survey</a
      >
      reveals that 87.5% of companies have low levels of data and analytics
      maturity.<br />This means that individual business units are pursuing their
      own data and analytics<br />initiatives as stand-alone projects, and there's
      no common structure that would enable<br />the enterprise to efficiently
      leverage advanced analytics.
    </p>
    <p>
      According to
      <a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        target="_blank"
        >Gartner</a
      >, low maturity can be the result of a lack of vision and skills, a lack<br />of
      experience in strategic planning and execution, limited budgets, and primitive
      or<br />outdated infrastructure. As a result, organizations suffer from
      incomplete or inconsistent<br />data sets, poor data quality, duplicate data
      processes, data security issues, and poor<br />coordination across the
      enterprise.
    </p>
    <p>
      For data democratization to reach its full potential, organizations must first
      improve their<br />maturity, but they may not have enough time and financial,
      technological, and human<br />resources to do so. It can also be difficult to
      bring on board skilled data experts&mdash;who<br />are in high demand
      today&mdash;or to get buy-in from leadership and senior management to<br />advance
      data initiatives.
    </p>
    <p>
      This is where all-in-one data tools can shine. They eliminate the need to
      invest in<br />multiple external tools or in-house solutions, consolidate and
      centralize all data, and<br />save teams the time and effort of searching
      multiple data sources.
    </p>
    <h3>Benefits of all-in-one data tools</h3>
    <p>
      Let's now take a closer look at how commercial teams, and companies in
      general, can<br />access and benefit from all-in-one data tools.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >1. Improved marketing performance</strong
      >
    </p>
    <p>
      According to a study,
      <a
        style="color: #f66640"
        href="https://get.treasuredata.com/rs/714-XIJ-402/images/the-state-of-the-customer-journey-arm-treasure-data.pdf"
        target="_blank"
        >47%</a
      >
      of marketers say that data silos are their biggest challenge<br />when it
      comes to gaining insights from data.
    </p>
    <p>
      A single data platform allows companies to create a journey map for customers
      that<br />shows how they interact with the brand over time. This journey map
      helps marketers<br />identify key touchpoints that can be improved or better
      leveraged. By being able to<br />segment users by their behavior, location, or
      device, companies can better target them.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >2. Improved customer experience</strong
      >
    </p>
    <p>
      A Forrester report shows that 72% of companies find it challenging to manage
      multiple<br />customer relationship management (CRM) tools across geographies
      and technology<br />silos.
    </p>
    <p>
      An all-in-one data platform enables the collection of customer feedback from
      multiple<br />channels, providing a holistic view of customer sentiment and
      satisfaction. This<br />feedback helps commercial teams identify what
      customers want and find ways to<br />improve the overall experience for their
      customers, while data teams get to reclaim their<br />time and energy to
      in-depth, impactful work.
    </p>
    <p>
      This is especially important because
      <a
        style="color: #f66640"
        href="https://go.emplifi.io/us-uk-customer-expectations-report.html"
        target="_blank"
        >61%</a
      >
      of people are willing to pay at least 5% more<br />if they know they'll have a
      good customer experience, and
      <a style="color: #f66640;" href="https://go.emplifi.io/us-uk-customer-expectations-report.html"
        >56%</a
      >
      say that the quality of<br />customer service has a greater impact on how
      positively they view a brand than any<br />other factor.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >3. Better customer lifetime value and retention</strong
      >
    </p>
    <p>
      The ability to identify key metrics in real time using a single data platform
      enables rapid<br />decision making and helps increase customer loyalty and
      engagement. This also<br />improves customer retention as companies can tailor
      their products or services to meet<br />customers' needs.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >4. Reduced costs and time allocated to data operations</strong
      >
    </p>
    <p>
      As all-in-one data tools help companies identify their customers' needs in
      real time, they<br />can help reduce costs, while increasing profits, and time
      spent on data operations.<br />That's because they can become data-driven
      without a full team of specialists, and if<br />and when they do hire a data
      expert or team, they&rsquo;ll be well equipped to dive right into<br />the
      work. Additionally, companies can identify when the cost of customer
      acquisition<br />outweighs the benefits, allowing for rapid growth or
      discontinuation of programs that<br />aren't working.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >5. Higher efficiency and revenue</strong
      >
    </p>
    <p>
      When a company has access to information about customer needs through a
      single<br />data platform, it can streamline its operations and achieve
      greater efficiency by<br />delivering products or services in the shortest
      possible time. This, in turn, enables faster<br />revenue growth.
    </p>
    <h3>Future-proofing your company through Data</h3>
    <p>
      All-in-one data solutions can facilitate the seismic shift to a data-driven
      culture. They<br />can help organizations launch their data operations in a
      matter of hours instead of<br />months or even years. They enable businesses
      to become data-driven without a full<br />team of specialists, and even if
      they do hire a data expert or team, they're well-equipped<br />to get started
      right away.
    </p>
    <p>
      Meanwhile, teams in each department can focus on important tasks instead of
      sifting<br />through and pulling data from multiple data sources. Easy access
      to data-driven insights<br />would lead to greater efficiency, productivity,
      and profits as well as higher data security.<br />Customers will also have a
      much better experience. So it's a win-win situation.
    </p>
    <p>
      In today's borderless digital world where data is the lingua franca, data
      democratization<br />is no longer a nice-to-have, but a necessity for
      organizations to achieve their goals and<br />gain a competitive advantage.
    </p>
    <p>
      To succeed, executives should invest in all-in-one data platforms and take a
      coordinated<br />and holistic approach to their efforts toward data
      democratization, which has triggered a<br />seismic shift in the business
      world.
    </p>
    

How All-in-One Tools Are Accelerating Data Democratization

<div>
      <p class="lead-paragraph blog-post"></p>
      <div class="cms-rich-text blog-post w-richtext">
          <p>The Modern Data Stack has been the hot topic of the data world for quite some time now. In the 12 years since
              Redshift was initially released, we’ve seen a mass explosion of capital and innovation that has enabled
              organizations to transform how they think about and leverage data.</p>
          <p>For this article, I’d like to discuss one of the major advantages of the MDS — the shift from ETL to ELT —
              and the primary users who are served and underserved in this paradigm. Today, I believe that the MDS
              provides a massive unlock in value for data engineers and analytic engineers but lacks the easy-to-use and
              intuitive toolkits for the data analyst, citizen data scientist, and knowledge worker that sits closest to
              the decision.</p>
          <p>The purpose of this blog is not to discuss the past, present, or future of the Modern Data Stack. There are
              many articles that provide a fantastic history lesson on the transition of on-prem legacy systems to the
              truly cloud native, modern data stack most of us know and love today. If you’re new to the MDS and
              interested in learning more, dbt’s CEO, Tristan Handy, wrote a great post, <a
              class="mds-text-primary"
                  href="https://www.getdbt.com/blog/future-of-the-modern-data-stack/" target="_blank">The Modern Data
                  Stack: Past, Present, and Future</a>, in December 2020, that perfectly articulates the transformation at
              hand.</p>
          <h3><strong>What is ELT and why is it important?</strong></h3>
          <p>You’ve likely heard about ELT — Extract Load and Transform… the Modern Data Stack’s evolution on ETL. This is
              a game changer by nature in that it enables organizations to ingest raw data into the data warehouse and
              transform it later. ELT gives end-users access to the entirety of the datasets they need by circumventing
              downstream issues of missing data that could prevent a specific business question from being answered.</p>
          <p>ELT isn’t a new topic. I remember the early days of my career in the Big Data/Hadoop space. Originally
              introduced by Google in 2005, the promise of Hadoop was, more or less, ELT. Don’t transform data on its way
              into the data platform and risk losing critical data that you need down the road. Instead, ship the entirety
              of the raw data into Hadoop, and you’ll have it…. for whenever you need it.</p>
          <p>The promise was great! Truly an amazing sales pitch, but unfortunately, it just didn’t work out. Data
              processing was too slow. It frustrated end-users, was not usable by the knowledge workers, and didn’t enable
              agility in analytically driven decision making. Spark made huge progress in reducing the speed of
              processing, but a better solution has emerged.</p>
          <p>This is where the Modern Data Stack, and the Cloud Data Warehouse (the core foundation of the MDS), differ.
              By separating storage from compute and enabling elastic computation in the cloud, tools such as Snowflake,
              BigQuery, Delta Lake, and Redshift have changed this paradigm. Data teams can now extract and load raw data
              directly into cloud storage for a super low cost per terabyte, and the users who need prepared data for
              analysis can get lightning-fast speeds throughout their data analysis workflow.</p>
          <h3><strong>What are the components, and who are the major players powering ELT?</strong></h3>
          <p>To make things simple, I’ve outlined a few of the major technologies being leveraged today. Again, for a
              complete overview of the MDS, check out Tristan Handy’s blog that I posted above.</p>
          <ul role="list">
              <li>Cloud Data Warehouse — Snowflake, BigQuery, Delta Lake, Redshift, and Synapse</li>
              <li>Data Integration — Fivetran, Airbyte, and Segment</li>
              <li>Data Transformation — dbt (the clear winner in the data transformation game)</li>
              <li>Reverse ETL — Census and Hightouch</li>
          </ul>
          <figure  >
              <div><img
  style="width:100%" src="https://uploads-ssl.webflow.com/6276a6c8de031667550a3847/62866cc38480f6788747d7c9_mds.png"
                      loading="lazy" alt=""></div>
              <figcaption>The modern data stack with leading vendors in each Silo</figcaption>
          </figure>
          <h3><strong>Who are the main users being served? Who are the underserved users?</strong></h3>
          <p>To answer this question, it’s first important to level set on the key users we see in the end to end data
              stack. I like to think of this by breaking down users into <em>data producers</em> and <em>data
                  consumers</em>. Data producers are the individuals who are integrating and modeling data on behalf of
              the organization. Data consumers consume the cleaned and modeled data to answer critical business questions
              from that data.</p>
          <figure  >
              <div><img
  style="width:100%"
                      src="https://uploads-ssl.webflow.com/6276a6c8de031667550a3847/62867609a729465066769724_personas.png"
                      loading="lazy" alt=""></div>
              <figcaption>Professionals who use the modern data stack and descriptions</figcaption>
          </figure>
          <p>In working in this space for years, I’ve seen firsthand that these tools (mentioned above) enable the data
              engineer and the (newly minted job title) analytic engineer super well. They provide modern, easy-to-use,
              and cloud native code-first tools for the ELT workflow and provide the organizations that employ these users
              with better speed and governance of their data assets.</p>
          <p>Said differently, the Modern Data Stack is awesome in that it enables centralization, speed, agility, and
              performance. But, today, it is a walled garden for users who are highly proficient in SQL development.</p>
          <p>So who is left out in this world? In Tristan’s article on the past, present, and future of the Modern Data
              Stack (posted above), he argues that one of the biggest gaps still to be closed in the MDS is the bridge to
              the data consumer.</p>
          <p>Tristan goes as far as saying, “Data consumers were actually more self-serve <em>prior</em> to the advent of
              the modern data stack: Excel skills are widely dispersed through the population of knowledge workers. There
              has not yet been an analogous interface where all knowledge workers can seamlessly interact with data in the
              modern data stack in a horizontal way.”</p>
          <p>I couldn’t agree more! In my viewpoint, the data consumer, in the analytics workflow, is not only underserved
              but arguably the most essential user who needs a solution. After all, this user is the one who sits closest
              to the business decision that can drive differentiated advantage in the marketplace.</p>
          <p>This knowledge user is different. They typically understand SQL conceptually, and many are proficient at
              writing SQL code, but the true power of this user is that they act as both the data SME and the business SME
              that truly understands the problem at hand and the business context of what they are trying to solve. Said
              differently, the critical value of the knowledge user is domain expertise, not programming expertise.</p>
          <p>This user needs to answer questions from data. They need insights from data…. fast. They are typically on the
              hook for preparing critical data insights that lead to executive and senior management decisions on the
              direction of the company. Important stuff!</p>
          <h3><strong>So… what does ELT mean for this knowledge user? Envisioning a new world.</strong></h3>
          <p>In my opinion, ELT represents a transformational advantage for this user. Traditional BI tools such as
              Tableau do an amazing job at enabling visualization of data to answer questions but rely on the data
              engineer or analytic engineer to first prepare raw data into an analytical table that Tableau can query.</p>
          <p>This means that the visual insights from raw data workflow is, today, inherently not self-service. An analyst
              needs to partner with the analytics engineering team to get the data they need in the format they need. This
              slows down decision making to a halt and typically annoys the analytic engineering team with ad hoc requests
              that fall outside the scope of their prioritized sprint.</p>
          <p>Analytic engineers own production-level data pipelines that the business relies on. These pipelines are
              static, hardened, and scalable. This is a fundamentally different requirement than ad hoc data insights
              which are agile, iterative, and experimental. A new solution paradigm is needed.</p>
          <p>This requirement is not new. Alteryx built a massive business around no-code data preparation, but it was
              built for the traditional, on-premise world. It does not give users the advantages of ELT within the Modern
              Data Stack. It is, instead, ETL, forcing users to extract data from the central warehouse, impacting
              performance, speed, and auditability of the transformation pipeline.</p>
          <p>In another blog from Tristan, <a class="mds-text-primary" href="https://roundup.getdbt.com/p/code-and-no-code-having-it-both-ways?s=r"
                  target="_blank">“Code and No-Code, Having it Both Ways”</a>, he describes a major gap in the marketplace
              requiring no-code tools that, behind the scenes, write Modern Data Stack native SQL for the end user that is
              automatically pushed down to data in the Cloud Data Warehouse.</p>
          <p>A solution like this would enable data consumers to capture the value of ELT in the Modern Data Stack without
              forcing them to become an expert-level SQL developer. It would enable a rapid up-leveling of skill sets
              within a user community overnight.</p>
          <h3><strong>No-Code on the Modern Data Stack — A new paradigm to enable the knowledge worker</strong></h3>
          <p>At Rasgo, this is a problem we’ve thought a lot about. We’ve worked with hundreds of data analysts, business
              users, and citizen data scientists who have heard a lot about the Modern Data Stack and the value their data
              engineering teams have gleaned, but haven’t yet been able to fully capture the value of the cloud data
              ecosystem themselves.</p>
          <p>From my viewpoint, a new breed of tools purpose built for these end users needs to emerge to truly bring data
              consumers into the MDS ecosystem. These tools need to be UI first, enable a more iterative workflow, and be
              fully native to the underlying dbt transformation engine. They need to enable business users to answer
              ad-hoc questions from data as they come up, whether or not these questions rely on metrics that have already
              been created by the Analytic Engineering team. Without this, data consumers will always be at the mercy of
              their engineering counterparts, and the engineering teams will always be responding to ad hoc requests from
              the business, instead of thinking about modeling data for the holistic needs of the enterprise.</p>
          <p>Lately, I’ve been describing this to folks within the industry as “The UI for the Modern Data Stack.” It’s
              something we at Rasgo are heavily committed to perfecting, and we believe that, without this, true data
              driven decision making will never fully be realized.</p>
          <p>Really appreciate you reading!! Super exciting times to be in the data world and we look forward to what’s
              ahead :)</p>
      </div>
  </div>

ELT for the Data Consumer

<div>
      <h3>The first step towards being data-driven</h3>
      <figure>
      <p data-selectable-paragraph="">I started my career as a first-generation analyst focusing on writing SQL scripts, learning R, and publishing dashboards. As things progressed, I graduated into Data Science and Data Engineering where my focus shifted to managing the life-cycle of ML models and data pipelines. 2022 is my 16th year in the data industry and I am still learning new ways to be productive and impactful. Today, I am now the head of a data science &amp; data engineering function in one of the unicorns and I would like to share my findings and where I am heading next.</p>
      <p data-selectable-paragraph="">When I look at the big picture, I realised that the problems most companies face are quite similar. Their vision towards being data-driven has turned into a BHAG &mdash; pronounced &ldquo;bee hag&rdquo; (<strong>B</strong>ig&nbsp;<strong>H</strong>airy&nbsp;<strong>A</strong>udacious&nbsp;<strong>G</strong>oal). We data folks like patterns, so here are my findings:</p>
      <ol>
      <li data-selectable-paragraph="">During 5 out of 10 review meets, I have witnessed people question the reliability of the data/report/dashboard. Additionally, HODs will also try to convince others that their data is the most accurate or reliable :)</li>
      <li data-selectable-paragraph="">A lot of times, HOD comes and says that the data is not updated. The data team is already working to fix the report/data table.</li>
      <li data-selectable-paragraph="">A new product got launched the week before, however, we are yet to figure out the performance. The data team is working on a query change and will soon update the CXO team.</li>
      <li data-selectable-paragraph="">Everyone has built expertise around writing complicated ML (machine-learned) models, however very few talk about or deploy inference monitoring. There is a high probability of model drift or performance drift in the coming weeks/months if not monitored or observed efficiently.</li>
      <li data-selectable-paragraph="">Very few companies deploy solutions or models to detect performance anomalies.</li>
      </ol>
      <p data-selectable-paragraph="">The list is long, I am sure you can relate or add more to this.</p>
      <p data-selectable-paragraph="">In a nutshell, I found that data reliability is a BIG challenge and there is a need for a solution that is easy to use, understand, deploy, and also not heavy on investment.</p>
      <p data-selectable-paragraph="">Hello, I am&nbsp;<strong>Jatin Solanki</strong> who is on a mission to build and develop a solution to make your data reliable.</p>
      </div>
      <div>
      <h3><strong>What is needed to make your data more reliable?</strong></h3>
      <p data-selectable-paragraph="">Complexities around data infrastructure are surging as companies gear to get a competitive edge and out-of-the-box offerings.</p>
      <p data-selectable-paragraph="">Every company goes through a data maturity matrix. In order to reach a level where you deploy AI models or self-service models, you need to invest in a robust foundation.</p>
      <blockquote>
      <p data-selectable-paragraph="">In my opinion, the foundation begins with a reliable data source or defining source of truth. Your data models won&rsquo;t be impactful if it&rsquo;s ingested with bad data. You know it&rsquo;s garbage&nbsp;<code>in</code>&nbsp;garbage&nbsp;<code>out</code></p>
      </blockquote>
      <p data-selectable-paragraph="">On a high level, here are a few checks you can implement to ensure data reliability:</p>
      <ul>
      <li data-selectable-paragraph=""><strong>Volume:</strong>&nbsp;It ensures all the row/events are captured or ingested.</li>
      <li data-selectable-paragraph=""><strong>Freshness:&nbsp;</strong>Recency of the data. If your data gets updated every xx mins, this test will ensure its updated and raises an incident if not.</li>
      <li data-selectable-paragraph=""><strong>Schema Change</strong>: If there is a schema change or a new feature that was launched, your data team needs to be aware to update the scripts.</li>
      <li data-selectable-paragraph=""><strong>Distribution</strong>: All the events are in an acceptable range. e.g if a critical shouldn&rsquo;t contain&nbsp;<em>null&nbsp;</em>values, then this test ensures to raise an alert for any&nbsp;<em>null or missing values</em>.</li>
      <li data-selectable-paragraph=""><strong>Lineage</strong>: This is a must-have module, however, we always underplay these ones. Lineage provides a handy info to the data team of the upstream and downstream.</li>
      <li data-selectable-paragraph=""><strong>Reconciliation</strong>: I would say recon or finding deltas between two given datasets. This could be used to understand the difference between&nbsp;<code>staging</code>and&nbsp;<code>production</code>&nbsp;OR between&nbsp;<code>source</code>&nbsp;and&nbsp;<code>destination</code>&nbsp;. This could be effective in running some financial recon too, like payment gateway to the sales table.</li>
      </ul>
      </div>
      <div>
      <h3>What next? How do we implement this?</h3>
      <p data-selectable-paragraph="">The most common question people face with:</p>
      <blockquote>
      <p data-selectable-paragraph="">Build versus Buy</p>
      </blockquote>
      <p data-selectable-paragraph="">I am a big fan of open source tech, however, in some critical modules, I prefer buying an out-of-the-box solution because it&rsquo;s scalable and already tested in the market. Developing in-house might cost you around US2k per month and it includes a few hours of engineer&rsquo;s time along with cloud cost.</p>
      <p data-selectable-paragraph="">If you are inclined toward buying an out-of-the-box solution, here are a few factors that should be part of your checklist.</p>
      <ul>
      <li data-selectable-paragraph="">Should be able to connect to popular sources which require minimal config.</li>
      <li data-selectable-paragraph="">Extract information automatically without the need for additional code.</li>
      <li data-selectable-paragraph="">No-code or CLI (I leave it to you)</li>
      <li data-selectable-paragraph="">Lineage and Catalog module.</li>
      <li data-selectable-paragraph="">Data Reconciliation along with scheduling feature.</li>
      <li data-selectable-paragraph="">Anomaly detection</li>
      <li data-selectable-paragraph="">Of course, Of course, all the tests we discussed earlier along with alerts should be in a position to tell where to&nbsp;<code>debug</code>.</li>
      </ul>
      <figure>
      <div tabindex="0" role="button">
      <div><picture><source srcset="https://miro.medium.com/max/640/1*NtdfA_0ur1qp5JWpTkQTMg.webp 640w, https://miro.medium.com/max/720/1*NtdfA_0ur1qp5JWpTkQTMg.webp 720w, https://miro.medium.com/max/750/1*NtdfA_0ur1qp5JWpTkQTMg.webp 750w, https://miro.medium.com/max/786/1*NtdfA_0ur1qp5JWpTkQTMg.webp 786w, https://miro.medium.com/max/828/1*NtdfA_0ur1qp5JWpTkQTMg.webp 828w, https://miro.medium.com/max/1100/1*NtdfA_0ur1qp5JWpTkQTMg.webp 1100w, https://miro.medium.com/max/1400/1*NtdfA_0ur1qp5JWpTkQTMg.webp 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" /><source srcset="https://miro.medium.com/max/640/1*NtdfA_0ur1qp5JWpTkQTMg.png 640w, https://miro.medium.com/max/720/1*NtdfA_0ur1qp5JWpTkQTMg.png 720w, https://miro.medium.com/max/750/1*NtdfA_0ur1qp5JWpTkQTMg.png 750w, https://miro.medium.com/max/786/1*NtdfA_0ur1qp5JWpTkQTMg.png 786w, https://miro.medium.com/max/828/1*NtdfA_0ur1qp5JWpTkQTMg.png 828w, https://miro.medium.com/max/1100/1*NtdfA_0ur1qp5JWpTkQTMg.png 1100w, https://miro.medium.com/max/1400/1*NtdfA_0ur1qp5JWpTkQTMg.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og" /><img style="width:100%;"  role="presentation" src="https://miro.medium.com/max/1400/1*NtdfA_0ur1qp5JWpTkQTMg.png" alt="" width="700" height="389" /></picture></div>
      </div>
      <figcaption data-selectable-paragraph="">A robust platform provides easy access to all the incidents and also evaluates the data health.</figcaption>
      </figure>
      <p data-selectable-paragraph="">It should be in a position to automatically detect my critical data assets and apply hygiene checks.</p>
      <figure>
      <div tabindex="0" role="button">
      <div><picture><source srcset="https://miro.medium.com/max/640/1*Wx8QYn9b-ajTiFcVsV5glw.webp 640w, https://miro.medium.com/max/720/1*Wx8QYn9b-ajTiFcVsV5glw.webp 720w, https://miro.medium.com/max/750/1*Wx8QYn9b-ajTiFcVsV5glw.webp 750w, https://miro.medium.com/max/786/1*Wx8QYn9b-ajTiFcVsV5glw.webp 786w, https://miro.medium.com/max/828/1*Wx8QYn9b-ajTiFcVsV5glw.webp 828w, https://miro.medium.com/max/1100/1*Wx8QYn9b-ajTiFcVsV5glw.webp 1100w, https://miro.medium.com/max/1400/1*Wx8QYn9b-ajTiFcVsV5glw.webp 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" /><source srcset="https://miro.medium.com/max/640/1*Wx8QYn9b-ajTiFcVsV5glw.png 640w, https://miro.medium.com/max/720/1*Wx8QYn9b-ajTiFcVsV5glw.png 720w, https://miro.medium.com/max/750/1*Wx8QYn9b-ajTiFcVsV5glw.png 750w, https://miro.medium.com/max/786/1*Wx8QYn9b-ajTiFcVsV5glw.png 786w, https://miro.medium.com/max/828/1*Wx8QYn9b-ajTiFcVsV5glw.png 828w, https://miro.medium.com/max/1100/1*Wx8QYn9b-ajTiFcVsV5glw.png 1100w, https://miro.medium.com/max/1400/1*Wx8QYn9b-ajTiFcVsV5glw.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og" /><img style="width:100%;"  role="presentation" src="https://miro.medium.com/max/1400/1*Wx8QYn9b-ajTiFcVsV5glw.png" alt="" width="700" height="658" /></picture></div>
      </div>
      <figcaption data-selectable-paragraph="">Only platform to group alerts instead of pushing 100+ alerts.</figcaption>
      </figure>
      <p data-selectable-paragraph="">At last, the solution should help you reduce data quality incidents and make your data more&nbsp;<em>reliable.</em></p>
      </div>
      <div>
      <h3><strong>So, do I need a data observability platform?</strong></h3>
      <p data-selectable-paragraph="">If your answer to any of the below questions or scenarios is &ldquo;Yes&rdquo;, then you should procure or deploy a data observability solution right away.</p>
      <ol>
      <li data-selectable-paragraph="">Dashboard not getting updated on regular basis?</li>
      <li data-selectable-paragraph="">Don&rsquo;t know which report is accurate?</li>
      <li data-selectable-paragraph="">Business stakeholders are the first to learn about data incidents.</li>
      <li data-selectable-paragraph="">Questions during a meeting on the performance stats.</li>
      <li data-selectable-paragraph="">Have at least 2 members in the data team.</li>
      <li data-selectable-paragraph="">Deployed a business intelligence tool.</li>
      </ol>
      <blockquote>
      <p data-selectable-paragraph="">As software developers have leveraged on DataDog, Dynatrace, etc kind of solutions to ensure web/app uptime, data leaders should invest in data observability solutions to ensure data reliability.</p>
      </blockquote>
      </div>

What is Data observability, do I need it?

10 tips for building an advanced data platform

<p>ELT is becoming the default choice for data architectures and yet, many best practices focus primarily on &ldquo;T&rdquo;: the transformations.</p>
      <p>But the extract and load phase is where data quality is determined for transformation and beyond. As the saying goes, &ldquo;garbage in, garbage out.&rdquo;</p>
      <p>Robust EL pipelines provide the foundation for delivering accurate, timely, and error-free data.</p>
      <p>Luckily we have a community full of data experts that have worked with Meltano, Stitch, Airbyte, Fivetran, and all the big extract and load tools on the market. So we asked them to provide their most important extract and load practices!</p>
      <p>We distilled 7 data practices used &amp; loved by the community that will drive up quality for all your data sets, no matter what tool you use.</p>
      <p><em>But wait, why aren&rsquo;t these &ldquo;best practices&rdquo;? Because we consider them pick and choose. If you&rsquo;re tackling a new project, or don&rsquo;t yet have many practices inside your extract and load processes, you implement all of them. If you do already have some, complement them with the ones that make sense in your situation.</em></p>
      <h3>Setting the Stage: We need E&amp;L practices, because &ldquo;copying raw data&rdquo; is more complex than it sounds.</h3>
      <p>The idea of ELT as a pattern sounds easy, &ldquo;just copy over the source data first and then run your transformations over the raw data inside your own space&rdquo;. However, both the word &ldquo;copy&rdquo; as well as &ldquo;raw data&rdquo; are two words with hidden obstacles.&nbsp;</p>
      <p>&ldquo;Copying&rdquo; sounds easy. But source data changes, and unless you know what changed, &ldquo;copying&rdquo; turns out to be more complicated than you think. Imagine a production table of 150 million &ldquo;orders&rdquo; that does come with a &ldquo;timestamp&rdquo; but not with a &ldquo;modified&rdquo; data. And yes, these exist, all over the place. How do you know that orders got modified, and if so, which ones? For instance, how would you know which orders got &ldquo;canceled&rdquo;, an operation that usually takes place in the same data record and just &ldquo;modifies&rdquo; it in place.</p>
      <p>&ldquo;Raw data&rdquo; sounds clear. And yet the idea of extracting &amp; loading implicitly means that usually, you copy between two different technical systems A and B, where you need to adjust the data to match system, B. You ingest from REST APIs and put it into Snowflake, or from an Oracle database into Redshift. Every time you change systems, you will need to modify the &ldquo;raw data&rdquo; to adhere to the rules of the new system. You need to do type casting, you need to think about whether you want to &ldquo;flatten JSONs&rdquo;, or whether you want to add additional metadata to your data.&nbsp;</p>
      <p>Simply &ldquo;copying raw data&rdquo; will bring up new questions every time you add a new data source or data target to your list. Even if it is just a new table from the same production database you&rsquo;ve always been copying from.&nbsp;</p>
      <p>These practices will be your guide whenever you take on a new source for ingesting data into your data system.</p>
      <h3>1. Make each EL run uniquely identifiable &ndash; timestamp everything.</h3>
      <p>We start with arguably the most important best practice: Make every bit of data you load into your data system identifiable and traceable back to the process that got it there.</p>
      <p>Typical ways of doing this is to include metadata values that capture:</p>
      <ul>
      <li>Ingestion time: the timestamp indicating when the load process started.</li>
      <li>Ingestion process: a unique identifier representing the load process and its instance.</li>
      <li>Source system: Metadata about where the data was extracted from.</li>
      </ul>
      <figure><img style="width:100%"  src="https://meltano.com/wp-content/uploads/2022/12/Pic01-1024x331.png" sizes="(max-width: 1024px) 100vw, 1024px" srcset="https://meltano.com/wp-content/uploads/2022/12/Pic01-1024x331.png 1024w, https://meltano.com/wp-content/uploads/2022/12/Pic01-300x97.png 300w, https://meltano.com/wp-content/uploads/2022/12/Pic01-768x249.png 768w, https://meltano.com/wp-content/uploads/2022/12/Pic01-1536x497.png 1536w, https://meltano.com/wp-content/uploads/2022/12/Pic01.png 1928w" alt="" width="1024" height="331" /></figure>
      <p>Add any or all of these metadata to each row/entry of the data you ingest. We recommend you use the starting time of your ingestor as the &ldquo;ingestion time&rdquo; as it simplifies the process. The &ldquo;identifier of your ingestion instance&rdquo; should be clear. Don&rsquo;t just provide the &ldquo;Airflow-OracleDB-Ingester&rdquo; as a process, but the &ldquo;Airflow-OracleDB-Ingester-ID1244&rdquo; where ID1244 clearly identifies the specific run of ingestion.&nbsp;</p>
      <p>One nice benefit of having the source system as metadata is that you&rsquo;re able to quickly debug problems in downstream dashboards and identify its source. This is also useful metadata for other use cases.</p>
      <p>If you have a legacy and a new customer registration component, you can provide the source as a filter option inside dashboards, allowing users to filter for customers from just one system.</p>
      <figure><img style="width:100%"  src="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-1-1024x600.jpg" sizes="(max-width: 1024px) 100vw, 1024px" srcset="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-1-1024x600.jpg 1024w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-1-300x176.jpg 300w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-1-768x450.jpg 768w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-1-1536x900.jpg 1536w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-1-2048x1200.jpg 2048w" alt="" width="1024" height="600" /></figure>
      <h3>2. Deduplicate data at a level beyond the raw level.</h3>
      <p>There are usually three cases of duplicate data hitting your data systems you will want to &ldquo;deduplicate&rdquo;. But no matter the case, don&rsquo;t do it at the raw/landing level!</p>
      <p>The first case is intentional duplicate data, where a source system contains something your end-users or you consider to be duplicates. For instance your CRM system might have two entries for a certain customer that canceled and signed up again. If you deduplicate at the raw level, this means either merging the two or deleting one. Both of which will delete data that is present in the source system.</p>
      <p>The second case is unintentional duplicate data, where the source system either deleted a record you still have in your data warehouse, or the source system unintentionally produces duplicate data it will likely delete in the future. Even though this is an &ldquo;error&rdquo;, I don&rsquo;t recommend deleting this data in your raw ingestion area but rather filter it further down the line, for instance in the next stage of your modeling. Otherwise you add logic to your ingestion that is hard to follow up on later.&nbsp;</p>
      <p>The third case is duplication happening due to technical restrictions. It might be the case that your ingestion tooling prefers a &ldquo;at least once delivery&rdquo; strategy or it might even be a bug in an ingestion process. With &ldquo;at least once delivery&rdquo; incremental load strategies, you&rsquo;re ensuring to get all data rows, but might duplicate some. Again we recommend to keep the duplicate data at the raw level, and filter it down at a later level.</p>
      <figure><img style="width:100%"  src="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-2-1024x560.jpg" sizes="(max-width: 1024px) 100vw, 1024px" srcset="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-2-1024x560.jpg 1024w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-2-300x164.jpg 300w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-2-768x420.jpg 768w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-2-1536x840.jpg 1536w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-2-2048x1121.jpg 2048w" alt="" width="1024" height="560" /></figure>
      <p>No matter the case, don&rsquo;t deduplicate on load. Load everything, and keep it there as it is. Do your deduplication later at the next stage.</p>
      <h3>3. Don&rsquo;t flatten during EL, do it one stage later.</h3>
      <p>A lot of the source systems you ingest will return arrays, JSONs, or other nested objects with some kind of hierarchy you will want to break down for further processing. But not at the ingestion level. Take the raw data and dump it as it is in your data system, then have a process do your &ldquo;flattening&rdquo;.&nbsp;</p>
      <p>A very typical example are JSON objects. Your source might contain a large JSON object and you would like to have it processed into individual columns inside your Snowflake database. This practice suggests to first have a raw table with just your metadata columns and one column &ldquo;JSON_blob&rdquo; containing the JSON object. In a secondary step you can then process this data into columns.</p>
      <figure><img style="width:100%"  src="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-3-1024x596.jpg" sizes="(max-width: 1024px) 100vw, 1024px" srcset="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-3-1024x596.jpg 1024w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-3-300x174.jpg 300w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-3-768x447.jpg 768w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-3-1536x893.jpg 1536w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-3-2048x1191.jpg 2048w" alt="" width="1024" height="596" /></figure>
      <p>The reason for this is that flattening involves business logic. It involves you knowing what properties are &ldquo;always there&rdquo;. If you flatten on ingestion, your ingestion process might break because one JSON object is NULL, or one JSON object doesn&rsquo;t come with one expected value. It is always easier to take the already ingested data and rerun your flattener than it is to run your ingestion + flattening process together.</p>
      <p><em>Additional tip: The same practice leads to avoiding type casting (if at all possible) on ingestion. We recommend doing typecasting after ingestion.</em></p>
      <h3>4. Have an immutable raw level.</h3>
      <p>At some point, you&rsquo;re going to switch to incremental updates on your data. It&rsquo;s the time to really keep in mind to have an &ldquo;immutable raw level&rdquo; an area where you never, I repeat never, modify or delete data.&nbsp;</p>
      <p>Not deduplicating is one part of it (see rule 2), but there is more: In your immutable raw level, you do not remove records removed upstream, you don&rsquo;t modify data modified upstream. You&rsquo;re simply loading new data in, that is it.</p>
      <p>A great example I still painfully remember is a north star metric dashboard. It showed the current development of the north star metric I worked at, based on customer behavior over the last couple of months. The dashboard and the numbers looked great, moving upwards. Product &amp; management decisions were made on this basis. New records of the north star metric were broadcasted each week.</p>
      <p>Then suddenly, one day, the dashboard looked different. Our north star metric shaved 10% of its value, and 30% in a specific segment.</p>
      <p>Turns out, one huge customer left, and the records got completely wiped in the regarding system.&nbsp;</p>
      <p>Since we were modifying our raw data, we completely messed up the north star metric irreversibly.</p>
      <p>From that day on, we used&nbsp;<a target="_blank" rel="noopener"  class="mds-text-primary"  href="https://docs.getdbt.com/docs/build/snapshots">snapshots</a>&nbsp;on all the raw data that might be subject to change.&nbsp;</p>
      <figure><img style="width:100%"  src="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-4-1024x639.jpg" sizes="(max-width: 1024px) 100vw, 1024px" srcset="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-4-1024x639.jpg 1024w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-4-300x187.jpg 300w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-4-768x479.jpg 768w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-4-1536x959.jpg 1536w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-4-2048x1278.jpg 2048w" alt="" width="1024" height="639" /></figure>
      <p>Do yourself a favor, and have your raw level immutable.</p>
      <p><em>Note: immutable staging areas are effectively what you create with full table syncs if you do not delete data. Also for GDPR and privacy concerns, you might need to make an exception here.</em></p>
      <h3>5. Do not transform data on ingestion, not even slightly unless you have to.</h3>
      <p>There are good reasons to transform data on the fly on ingestion, but almost every case you will think of also works without. There are two good reasons for transforming data on the fly: legal and security. For all other reasons, you should try to first ingest data, and then run a small transformation on the ingested data.&nbsp;</p>
      <p>If you do opt to do a &ldquo;on the fly&rdquo; transformation while ingesting data, make sure you make it fail proof. Try to only add data or subtract data, not modify it.&nbsp;</p>
      <p>By default, if you do want to do transformations, it is always better to either create a mapping table or ingest a mapping table and do the join there.</p>
      <figure><img style="width:100%"  src="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-5-1024x823.jpg" sizes="(max-width: 1024px) 100vw, 1024px" srcset="https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-5-1024x823.jpg 1024w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-5-300x241.jpg 300w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-5-768x617.jpg 768w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-5-1536x1234.jpg 1536w, https://meltano.com/wp-content/uploads/2022/12/BP_-EL-Practices-Frame-5-2048x1646.jpg 2048w" alt="" width="1024" height="823" /></figure>
      <p>You can do so by using mechanisms like &ldquo;<a target="_blank" rel="noopener"  class="mds-text-primary"  href="https://docs.getdbt.com/docs/build/seeds">dbt seeds</a>&rdquo; or ingest Google Sheets maintained by an external contributor.&nbsp;</p>
      <p>These practices will ensure your pipelines are robust and thus serve high quality and recent data.</p>


Modern Data Stack - Sitemap