<div>
      <h3>The first step towards being data-driven</h3>
      <figure>
      <p data-selectable-paragraph="">I started my career as a first-generation analyst focusing on writing SQL scripts, learning R, and publishing dashboards. As things progressed, I graduated into Data Science and Data Engineering where my focus shifted to managing the life-cycle of ML models and data pipelines. 2022 is my 16th year in the data industry and I am still learning new ways to be productive and impactful. Today, I am now the head of a data science &amp; data engineering function in one of the unicorns and I would like to share my findings and where I am heading next.</p>
      <p data-selectable-paragraph="">When I look at the big picture, I realised that the problems most companies face are quite similar. Their vision towards being data-driven has turned into a BHAG &mdash; pronounced &ldquo;bee hag&rdquo; (<strong>B</strong>ig&nbsp;<strong>H</strong>airy&nbsp;<strong>A</strong>udacious&nbsp;<strong>G</strong>oal). We data folks like patterns, so here are my findings:</p>
      <ol>
      <li data-selectable-paragraph="">During 5 out of 10 review meets, I have witnessed people question the reliability of the data/report/dashboard. Additionally, HODs will also try to convince others that their data is the most accurate or reliable :)</li>
      <li data-selectable-paragraph="">A lot of times, HOD comes and says that the data is not updated. The data team is already working to fix the report/data table.</li>
      <li data-selectable-paragraph="">A new product got launched the week before, however, we are yet to figure out the performance. The data team is working on a query change and will soon update the CXO team.</li>
      <li data-selectable-paragraph="">Everyone has built expertise around writing complicated ML (machine-learned) models, however very few talk about or deploy inference monitoring. There is a high probability of model drift or performance drift in the coming weeks/months if not monitored or observed efficiently.</li>
      <li data-selectable-paragraph="">Very few companies deploy solutions or models to detect performance anomalies.</li>
      </ol>
      <p data-selectable-paragraph="">The list is long, I am sure you can relate or add more to this.</p>
      <p data-selectable-paragraph="">In a nutshell, I found that data reliability is a BIG challenge and there is a need for a solution that is easy to use, understand, deploy, and also not heavy on investment.</p>
      <p data-selectable-paragraph="">Hello, I am&nbsp;<strong>Jatin Solanki</strong> who is on a mission to build and develop a solution to make your data reliable.</p>
      </div>
      <div>
      <h3><strong>What is needed to make your data more reliable?</strong></h3>
      <p data-selectable-paragraph="">Complexities around data infrastructure are surging as companies gear to get a competitive edge and out-of-the-box offerings.</p>
      <p data-selectable-paragraph="">Every company goes through a data maturity matrix. In order to reach a level where you deploy AI models or self-service models, you need to invest in a robust foundation.</p>
      <blockquote>
      <p data-selectable-paragraph="">In my opinion, the foundation begins with a reliable data source or defining source of truth. Your data models won&rsquo;t be impactful if it&rsquo;s ingested with bad data. You know it&rsquo;s garbage&nbsp;<code>in</code>&nbsp;garbage&nbsp;<code>out</code></p>
      </blockquote>
      <p data-selectable-paragraph="">On a high level, here are a few checks you can implement to ensure data reliability:</p>
      <ul>
      <li data-selectable-paragraph=""><strong>Volume:</strong>&nbsp;It ensures all the row/events are captured or ingested.</li>
      <li data-selectable-paragraph=""><strong>Freshness:&nbsp;</strong>Recency of the data. If your data gets updated every xx mins, this test will ensure its updated and raises an incident if not.</li>
      <li data-selectable-paragraph=""><strong>Schema Change</strong>: If there is a schema change or a new feature that was launched, your data team needs to be aware to update the scripts.</li>
      <li data-selectable-paragraph=""><strong>Distribution</strong>: All the events are in an acceptable range. e.g if a critical shouldn&rsquo;t contain&nbsp;<em>null&nbsp;</em>values, then this test ensures to raise an alert for any&nbsp;<em>null or missing values</em>.</li>
      <li data-selectable-paragraph=""><strong>Lineage</strong>: This is a must-have module, however, we always underplay these ones. Lineage provides a handy info to the data team of the upstream and downstream.</li>
      <li data-selectable-paragraph=""><strong>Reconciliation</strong>: I would say recon or finding deltas between two given datasets. This could be used to understand the difference between&nbsp;<code>staging</code>and&nbsp;<code>production</code>&nbsp;OR between&nbsp;<code>source</code>&nbsp;and&nbsp;<code>destination</code>&nbsp;. This could be effective in running some financial recon too, like payment gateway to the sales table.</li>
      </ul>
      </div>
      <div>
      <h3>What next? How do we implement this?</h3>
      <p data-selectable-paragraph="">The most common question people face with:</p>
      <blockquote>
      <p data-selectable-paragraph="">Build versus Buy</p>
      </blockquote>
      <p data-selectable-paragraph="">I am a big fan of open source tech, however, in some critical modules, I prefer buying an out-of-the-box solution because it&rsquo;s scalable and already tested in the market. Developing in-house might cost you around US2k per month and it includes a few hours of engineer&rsquo;s time along with cloud cost.</p>
      <p data-selectable-paragraph="">If you are inclined toward buying an out-of-the-box solution, here are a few factors that should be part of your checklist.</p>
      <ul>
      <li data-selectable-paragraph="">Should be able to connect to popular sources which require minimal config.</li>
      <li data-selectable-paragraph="">Extract information automatically without the need for additional code.</li>
      <li data-selectable-paragraph="">No-code or CLI (I leave it to you)</li>
      <li data-selectable-paragraph="">Lineage and Catalog module.</li>
      <li data-selectable-paragraph="">Data Reconciliation along with scheduling feature.</li>
      <li data-selectable-paragraph="">Anomaly detection</li>
      <li data-selectable-paragraph="">Of course, Of course, all the tests we discussed earlier along with alerts should be in a position to tell where to&nbsp;<code>debug</code>.</li>
      </ul>
      <figure>
      <div tabindex="0" role="button">
      <div><picture><source srcset="https://miro.medium.com/max/640/1*NtdfA_0ur1qp5JWpTkQTMg.webp 640w, https://miro.medium.com/max/720/1*NtdfA_0ur1qp5JWpTkQTMg.webp 720w, https://miro.medium.com/max/750/1*NtdfA_0ur1qp5JWpTkQTMg.webp 750w, https://miro.medium.com/max/786/1*NtdfA_0ur1qp5JWpTkQTMg.webp 786w, https://miro.medium.com/max/828/1*NtdfA_0ur1qp5JWpTkQTMg.webp 828w, https://miro.medium.com/max/1100/1*NtdfA_0ur1qp5JWpTkQTMg.webp 1100w, https://miro.medium.com/max/1400/1*NtdfA_0ur1qp5JWpTkQTMg.webp 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" /><source srcset="https://miro.medium.com/max/640/1*NtdfA_0ur1qp5JWpTkQTMg.png 640w, https://miro.medium.com/max/720/1*NtdfA_0ur1qp5JWpTkQTMg.png 720w, https://miro.medium.com/max/750/1*NtdfA_0ur1qp5JWpTkQTMg.png 750w, https://miro.medium.com/max/786/1*NtdfA_0ur1qp5JWpTkQTMg.png 786w, https://miro.medium.com/max/828/1*NtdfA_0ur1qp5JWpTkQTMg.png 828w, https://miro.medium.com/max/1100/1*NtdfA_0ur1qp5JWpTkQTMg.png 1100w, https://miro.medium.com/max/1400/1*NtdfA_0ur1qp5JWpTkQTMg.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og" /><img style="width:100%;"  role="presentation" src="https://miro.medium.com/max/1400/1*NtdfA_0ur1qp5JWpTkQTMg.png" alt="" width="700" height="389" /></picture></div>
      </div>
      <figcaption data-selectable-paragraph="">A robust platform provides easy access to all the incidents and also evaluates the data health.</figcaption>
      </figure>
      <p data-selectable-paragraph="">It should be in a position to automatically detect my critical data assets and apply hygiene checks.</p>
      <figure>
      <div tabindex="0" role="button">
      <div><picture><source srcset="https://miro.medium.com/max/640/1*Wx8QYn9b-ajTiFcVsV5glw.webp 640w, https://miro.medium.com/max/720/1*Wx8QYn9b-ajTiFcVsV5glw.webp 720w, https://miro.medium.com/max/750/1*Wx8QYn9b-ajTiFcVsV5glw.webp 750w, https://miro.medium.com/max/786/1*Wx8QYn9b-ajTiFcVsV5glw.webp 786w, https://miro.medium.com/max/828/1*Wx8QYn9b-ajTiFcVsV5glw.webp 828w, https://miro.medium.com/max/1100/1*Wx8QYn9b-ajTiFcVsV5glw.webp 1100w, https://miro.medium.com/max/1400/1*Wx8QYn9b-ajTiFcVsV5glw.webp 1400w" type="image/webp" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" /><source srcset="https://miro.medium.com/max/640/1*Wx8QYn9b-ajTiFcVsV5glw.png 640w, https://miro.medium.com/max/720/1*Wx8QYn9b-ajTiFcVsV5glw.png 720w, https://miro.medium.com/max/750/1*Wx8QYn9b-ajTiFcVsV5glw.png 750w, https://miro.medium.com/max/786/1*Wx8QYn9b-ajTiFcVsV5glw.png 786w, https://miro.medium.com/max/828/1*Wx8QYn9b-ajTiFcVsV5glw.png 828w, https://miro.medium.com/max/1100/1*Wx8QYn9b-ajTiFcVsV5glw.png 1100w, https://miro.medium.com/max/1400/1*Wx8QYn9b-ajTiFcVsV5glw.png 1400w" sizes="(min-resolution: 4dppx) and (max-width: 700px) 50vw, (-webkit-min-device-pixel-ratio: 4) and (max-width: 700px) 50vw, (min-resolution: 3dppx) and (max-width: 700px) 67vw, (-webkit-min-device-pixel-ratio: 3) and (max-width: 700px) 65vw, (min-resolution: 2.5dppx) and (max-width: 700px) 80vw, (-webkit-min-device-pixel-ratio: 2.5) and (max-width: 700px) 80vw, (min-resolution: 2dppx) and (max-width: 700px) 100vw, (-webkit-min-device-pixel-ratio: 2) and (max-width: 700px) 100vw, 700px" data-testid="og" /><img style="width:100%;"  role="presentation" src="https://miro.medium.com/max/1400/1*Wx8QYn9b-ajTiFcVsV5glw.png" alt="" width="700" height="658" /></picture></div>
      </div>
      <figcaption data-selectable-paragraph="">Only platform to group alerts instead of pushing 100+ alerts.</figcaption>
      </figure>
      <p data-selectable-paragraph="">At last, the solution should help you reduce data quality incidents and make your data more&nbsp;<em>reliable.</em></p>
      </div>
      <div>
      <h3><strong>So, do I need a data observability platform?</strong></h3>
      <p data-selectable-paragraph="">If your answer to any of the below questions or scenarios is &ldquo;Yes&rdquo;, then you should procure or deploy a data observability solution right away.</p>
      <ol>
      <li data-selectable-paragraph="">Dashboard not getting updated on regular basis?</li>
      <li data-selectable-paragraph="">Don&rsquo;t know which report is accurate?</li>
      <li data-selectable-paragraph="">Business stakeholders are the first to learn about data incidents.</li>
      <li data-selectable-paragraph="">Questions during a meeting on the performance stats.</li>
      <li data-selectable-paragraph="">Have at least 2 members in the data team.</li>
      <li data-selectable-paragraph="">Deployed a business intelligence tool.</li>
      </ol>
      <blockquote>
      <p data-selectable-paragraph="">As software developers have leveraged on DataDog, Dynatrace, etc kind of solutions to ensure web/app uptime, data leaders should invest in data observability solutions to ensure data reliability.</p>
      </blockquote>
      </div>

Data Engineering

What is Data observability, do I need it?

Tags

Top contributor