What data engineering practices that you will recommend to bridge the gap between data producers and data consumers?

To bridge the gap between data producers and consumers, we should see this from a social and technical perspective. Social: be a "people" person with data consumers. Don't get into the technical details. They don't care. Ask them about the problems and questions they have. Keep asking why, why, why! Almost like a therapist. Be a "geek" with the data producers and manage the conversation towards what the consumers care about in order to keep technical focus and avoid boiling the ocean. Technical: with data consumers, data modeling and knowledge management is key. You should be able to translate what the consumer is saying and draw it on the white board, and push into the details of the semantics (what does X really mean). You should be able to manage meetings with different stakeholders, and integrate the results and present them to a wider group. With data producers, you need to ground the semantics from the data models into the reality of the data. This means defining the mappings/transforms at a granular level, proactive find issues (is there conflicting meaning in the data?) and surface this to the right stakeholders. Bridging the gap is less about technology and more about the people skills and process. Tools are need such as data modeling (can be sophisticated ones or just lucid charts) and a data catalog to inventory the data and knowledge work that is being done. Finally, if you enjoy bridging this gap, you may not be a data engineer. I've been calling this role the knowledge scientist. See: https://www.knowledgescientist.org/

The founders of Avo developed the "Purpose Meeting" back when they were at QuizUp. Before every feature release, they’d gather the stakeholders for 30 minutes – an engineer from each development platform (iOS, Android, web, backend), a PM, a designer, a data specialist – and they’d go through these steps: [1] Align on goals: What does success look like? [2] Commit to metrics: How will we measure how successful we are? [3] Design data: What analytics events will we need for our success metrics? ‍The impact of doing this for every feature release went way beyond ensuring they had the highest-impact-lowest-effort tracking in place to understand the success of their releases. It turned out to be a fundamentally helpful process to ensure alignment on why they were shipping what they were shipping.  Purpose meetings revolutionized engineering buy-in for analytics at QuizUp. Engineers would request purpose meetings before they kicked off feature development because it made their work more efficient and purposeful. With analytics defined before writing any feature code, iOS and Android would always be in sync. It eradicated the need to refactor code to add analytics after-the-fact. Analytics went from being a “chore” that was done “for the data team” and “not for the end user”, to something that devs were interested in to understand how successful their release was. The developer conversion from looking up release metrics in Amplitude sky rocketed – from a couple of developers, to the majority of developers being curious.More here: https://www.avo.app/blog/tracking-the-right-product-metrics - Edited