Abhi Sivasailam's guest appearances

Abhi Sivasailam

Data Engineering Podcast

Catalog & Cocktails

The Analytics Engineering Podcast

Data Mesh Radio

OwlTail

Metrics are one of the most important and durable primitives in a company. And if data teams are looking to create value FAST… they have to learn how their businesses work.To figure out how you business works, you have to find the right starting point; back to the METRICS.Join Tim, Juan and special guest Abhi Sivasailam from Levers Lab on this week’s episode of Catalog &amp; Cocktails to hear more about the most valuable use cases. Episode highlights: [00:09 - 02:45] Introductions &amp; Cheers[02:49 - 03:42] Warmup: What are things in life that you wish happened faster?[03:49 - 04:57] Abhi's talk about data teams and company growth models[04:59 - 07:21] There's too many data people[07:24 - 08:57] What are people wasting their time on: variance and standardization[09:01 - 11:07] Challenges to understanding the business and growth models[11:12 - 13:03] Metrics form the core of well-defined end states[13:05 - 15:03] What Abhi thinks of as a metric, and how it ties into the knowledge graph[15:04 - 16:37] A plan of attack improves semantic knowledge and avoiding building as you go[16:47 - 17:51] The accretive development problem[17:53 - 22:48] A core structure of standards with uniqueness layered on top[22:53 - 25:30] Focus on specificities instead of repeated work[25:46 - 27:24] Not a bridge too far to say the semantic layer doesn't need to be that creative, but more standardized[27:32 - 29:09] Abhi thinks automated integration is impossible[29:19 - 32:29] Arbitrary uniqueness vs being coupled to an open standard [32:31 - 33:40] SOMA[33:42 - 36:22] How does SOMA affect a data person's life, and how does a company adopt the standard?[36:23 - 40:09] Building a web of metrics for growth[40:12 - 42:06] The relationship between metrics, SOMA, and OKRs[42:31 - 49:56] Lightning round[50:00 - 55:27] Takeaways[55:29 - 57:59] Three final questions

<p>Metrics are one of the most important and durable primitives in a company. And if data teams are looking to create value FAST… they have to learn how their businesses work.</p><p><br/></p><p>To figure out how you business works, you have to find the right starting point; back to the METRICS.</p><p><br/></p><p>Join Tim, Juan and special guest Abhi Sivasailam from Levers Lab on this week’s episode of Catalog &amp; Cocktails to hear more about the most valuable use cases. </p><p><br/></p><p>Episode highlights: </p><ul><li>[00:09 - 02:45] Introductions &amp; Cheers</li><li>[02:49 - 03:42] Warmup: What are things in life that you wish happened faster?</li><li>[03:49 - 04:57] Abhi's talk about data teams and company growth models</li><li>[04:59 - 07:21] There's too many data people</li><li>[07:24 - 08:57] What are people wasting their time on: variance and standardization</li><li>[09:01 - 11:07] Challenges to understanding the business and growth models</li><li>[11:12 - 13:03] Metrics form the core of well-defined end states</li><li>[13:05 - 15:03] What Abhi thinks of as a metric, and how it ties into the knowledge graph</li><li>[15:04 - 16:37] A plan of attack improves semantic knowledge and avoiding building as you go</li><li>[16:47 - 17:51] The accretive development problem</li><li>[17:53 - 22:48] A core structure of standards with uniqueness layered on top</li><li>[22:53 - 25:30] Focus on specificities instead of repeated work</li><li>[25:46 - 27:24] Not a bridge too far to say the semantic layer doesn't need to be that creative, but more standardized</li><li>[27:32 - 29:09] Abhi thinks automated integration is impossible</li><li>[29:19 - 32:29] Arbitrary uniqueness vs being coupled to an open standard </li><li>[32:31 - 33:40] SOMA</li><li>[33:42 - 36:22] How does SOMA affect a data person's life, and how does a company adopt the standard?</li><li>[36:23 - 40:09] Building a web of metrics for growth</li><li>[40:12 - 42:06] The relationship between metrics, SOMA, and OKRs</li><li>[42:31 - 49:56] Lightning round</li><li>[50:00 - 55:27] Takeaways</li><li>[55:29 - 57:59] Three final questions</li></ul>

SHOW ME THE METRICS w/ Abhi Sivasailam

Takeaways with Abhi Sivasailam 

Summary Data platforms are exemplified by a complex set of connections that are subject to a set of constantly evolving requirements. In order to make this a tractable problem it is necessary to define boundaries for communication between concerns, which brings with it the need to establish interface contracts for communicating across those boundaries. The recent move toward the data mesh as a formalized architecture that builds on this design provides the language that data teams need to make this a more organized effort. In this episode Abhi Sivasailam shares his experience designing and implementing a data mesh solution with his team at Flexport, and the importance of defining and enforcing data contracts that are implemented at those domain boundaries. Announcements  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets &amp; code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription StreamSets DataOps Platform is the world’s first single platform for building smart data pipelines across hybrid and multi-cloud architectures. Build, run, monitor and manage data pipelines confidently with an end-to-end data integration platform that’s built for constant change. Amp up your productivity with an easy-to-navigate interface and 100s of pre-built connectors. And, get pipelines and new hires up and running quickly with powerful, reusable components that work across batch and streaming. Once you’re up and running, your smart data pipelines are resilient to data drift. Those ongoing and unexpected changes in schema, semantics, and infrastructure. Finally, one single pane of glass for operating and monitoring all your data pipelines. The full transparency and control you desire for your data operations. Get started building pipelines in minutes for free at dataengineeringpodcast.com/streamsets. The first 10 listeners of the podcast that subscribe to StreamSets’ Professional Tier, receive 2 months free after their first month. Your host is Tobias Macey and today I’m interviewing Abhi Sivasailam about the different social and technical interfaces available for defining and enforcing data contracts  Interview  Introduction How did you get involved in the area of data management? Can you start by explaining what your working definition of a "data contract" is? What are the goals and purpose of these contracts?   What are the locations and methods of defining a data contract? What kind of information needs to be encoded in a contract definition?   How do you manage enforcement of contracts? manifestations of contracts in data mesh implementation ergonomics (technical and social) of data contracts and how to prevent them from prohibiting productivity What are the most interesting, innovative, or unexpected approaches to data contracts that you have seen? What are the most interesting, unexpected, or challenging lessons that you have learned while working on data contract implementation? When are data contracts the wrong choice?  Contact Info  LinkedIn @_abhisivasailam on Twitter Website  Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Closing Announcements  Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers  Links  Flexport Debezium  Podcast Episode   Data Mesh At Flexport Presentation Data Mesh  Podcast Episode   Column Names As Contracts podcast episode with Emily Riederer dbtplyr  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SA  Support Data Engineering Podcast

<h2>Summary</h2> <p>Data platforms are exemplified by a complex set of connections that are subject to a set of constantly evolving requirements. In order to make this a tractable problem it is necessary to define boundaries for communication between concerns, which brings with it the need to establish interface contracts for communicating across those boundaries. The recent move toward the data mesh as a formalized architecture that builds on this design provides the language that data teams need to make this a more organized effort. In this episode Abhi Sivasailam shares his experience designing and implementing a data mesh solution with his team at Flexport, and the importance of defining and enforcing data contracts that are implemented at those domain boundaries.</p> <h2>Announcements</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!</li> <li>Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets &amp; code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to <a href="https://www.dataengineeringpodcast.com/atlan?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/atlan</a> today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription</li> <li>StreamSets DataOps Platform is the world’s first single platform for building smart data pipelines across hybrid and multi-cloud architectures. Build, run, monitor and manage data pipelines confidently with an end-to-end data integration platform that’s built for constant change. Amp up your productivity with an easy-to-navigate interface and 100s of pre-built connectors. And, get pipelines and new hires up and running quickly with powerful, reusable components that work across batch and streaming. Once you’re up and running, your smart data pipelines are resilient to data drift. Those ongoing and unexpected changes in schema, semantics, and infrastructure. Finally, one single pane of glass for operating and monitoring all your data pipelines. The full transparency and control you desire for your data operations. Get started building pipelines in minutes for free at <a href="https://www.dataengineeringpodcast.com/streamsets?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/streamsets</a>. The first 10 listeners of the podcast that subscribe to StreamSets’ Professional Tier, receive 2 months free after their first month.</li> <li>Your host is Tobias Macey and today I’m interviewing Abhi Sivasailam about the different social and technical interfaces available for defining and enforcing data contracts</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by explaining what your working definition of a "data contract" is?<br><ul> <li>What are the goals and purpose of these contracts?</li> </ul> </li> <li>What are the locations and methods of defining a data contract?<br><ul> <li>What kind of information needs to be encoded in a contract definition?</li> </ul> </li> <li>How do you manage enforcement of contracts?</li> <li>manifestations of contracts in data mesh implementation</li> <li>ergonomics (technical and social) of data contracts and how to prevent them from prohibiting productivity</li> <li>What are the most interesting, innovative, or unexpected approaches to data contracts that you have seen?</li> <li>What are the most interesting, unexpected, or challenging lessons that you have learned while working on data contract implementation?</li> <li>When are data contracts the wrong choice?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/abhi-sivasailam/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LinkedIn</a></li> <li><a href="https://twitter.com/_abhisivasailam?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@_abhisivasailam</a> on Twitter</li> <li><a href="http://www.abhi.fyi/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Website</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Closing Announcements</h2> <ul> <li>Thank you for listening! Don’t forget to check out our other show, <a href="" rel="nofollow" target="_blank">Podcast.__init__</a> to learn about the Python language, its community, and the innovative ways it is being used.</li> <li>Visit the <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">site</a> to subscribe to the show, sign up for the mailing list, and read the show notes.</li> <li>If you’ve learned something or tried out a project from the show then tell us about it! Email <a href="mailto:hosts@dataengineeringpodcast.com">hosts@dataengineeringpodcast.com</a>) with your story.</li> <li>To help other people find the show please leave a review on <a href="https://itunes.apple.com/us/podcast/data-engineering-podcast/id1193040557?mt=2&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">iTunes</a> and tell your friends and co-workers</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.flexport.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flexport</a></li> <li><a href="https://debezium.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Debezium</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/debezium-change-data-capture-episode-114/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Episode</a></li> </ul> </li> <li><a href="https://www.youtube.com/watch?v=-POiudR2_R0&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Mesh At Flexport Presentation</a></li> <li><a href="https://martinfowler.com/articles/data-mesh-principles.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Mesh</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/data-mesh-revisited-episode-250/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Episode</a></li> </ul> </li> <li><a href="https://www.dataengineeringpodcast.com/controlled-vocabulary-with-dbtplyr-episode-252/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Column Names As Contracts</a> podcast episode with Emily Riederer</li> <li><a href="https://github.com/emilyriederer/dbtplyr?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">dbtplyr</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p> <p></p> <p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

The Importance Of Data Contracts As The Interface For Data Integration With Abhi Sivasailam

Data Mesh Radio Patreon - get access to interviews well before they are releasedEpisode list and links to all available episode transcripts (most interviews from #32 on) hereProvided as a free resource by DataStax AstraDB; George Trujillo's contact info: email (george.trujillo@datastax.com) and LinkedInIn this episode, Abhi Sivasailam, Head of Growth and Analytics at unicorn startup Flexport, and Scott deep dive into data contracts. They covered a LOT of ground including:What is a data contract and how does it relate to API contracts and schema contractsWhy data contracts are so crucial to treating data like a product and keeping data usable by consumersAbhi's rules for a viable data contractThe importance of the analytics engineer to overall data usefulness, especially re domain data ownership in data meshWhy of the socio of the socio-technical approach to data ownership is the most crucial aspectThe lack of proper tooling to monitor and execute data contractsHow to minimize disruptive changes to downstream data consumersThe issues with domains sharing their data as it is persisted/stored in the database instead of sharing the context via the domain modelMuch, much moreAbhi's contact info:Twitter: https://twitter.com/_abhisivasailamLinkedIn: https://www.linkedin.com/in/abhi-sivasailam/Abhi's Data Mesh Learning Meetup: https://www.youtube.com/watch?v=-POiudR2_R0Debezium blog post mentioned by Abhi: https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see hereAll music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, and/or nevesfData Mesh Radio is brought to you as a community resource by DataStax. Check out their high-scale, multi-region database offering (w/ lots of great APIs) and use code DAAP500 for a free $500 credit (apply under "add payment"): AstraDB

<p><a href="https://www.patreon.com/datameshradio" rel="noopener noreferrer nofollow" target="_blank">Data Mesh Radio Patreon</a> - get access to interviews well before they are released</p><p>Episode list and links to all available episode transcripts (most interviews from #32 on) <a href="https://docs.google.com/spreadsheets/d/1ZmCIinVgIm0xjIVFpL9jMtCiOlBQ7LbvLmtmb0FKcQc/edit?usp=sharing" rel="noopener noreferrer nofollow" target="_blank">here</a></p><p>Provided as a free resource by DataStax <a href="https://www.datastax.com/products/datastax-astra?utm_source=DataMeshRadio" rel="noopener noreferrer nofollow" target="_blank">AstraDB</a>; George Trujillo's contact info: email (<a href="mailto:george.trujillo@datastax.com" rel="noopener noreferrer">george.trujillo@datastax.com</a>) and <a href="https://www.linkedin.com/in/georgetrujillo/" rel="noopener noreferrer nofollow" target="_blank">LinkedIn</a></p><p>In this episode, Abhi Sivasailam, Head of Growth and Analytics at unicorn startup Flexport, and Scott deep dive into data contracts. They covered a LOT of ground including:</p><ul><li>What is a data contract and how does it relate to API contracts and schema contracts</li><li>Why data contracts are so crucial to treating data like a product and keeping data usable by consumers</li><li>Abhi's rules for a viable data contract</li><li>The importance of the analytics engineer to overall data usefulness, especially re domain data ownership in data mesh</li><li>Why of the socio of the socio-technical approach to data ownership is the most crucial aspect</li><li>The lack of proper tooling to monitor and execute data contracts</li><li>How to minimize disruptive changes to downstream data consumers</li><li>The issues with domains sharing their data as it is persisted/stored in the database instead of sharing the context via the domain model</li><li>Much, much more</li></ul><p>Abhi's contact info:</p><p>Twitter: <a href="https://twitter.com/_abhisivasailam" rel="noopener noreferrer nofollow" target="_blank">https://twitter.com/_abhisivasailam</a></p><p>LinkedIn: <a href="https://www.linkedin.com/in/abhi-sivasailam/" rel="noopener noreferrer nofollow" target="_blank">https://www.linkedin.com/in/abhi-sivasailam/</a></p><p>Abhi's Data Mesh Learning Meetup: <a href="https://www.youtube.com/watch?v=-POiudR2_R0" rel="noopener noreferrer nofollow" target="_blank">https://www.youtube.com/watch?v=-POiudR2_R0</a></p><p>Debezium blog post mentioned by Abhi: <a href="https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/" rel="noopener noreferrer nofollow" target="_blank">https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/</a></p><p>Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at <a href="http://datameshlearning.com" rel="nofollow" target="_blank">datameshlearning.com</a> or on LinkedIn: <a href="https://www.linkedin.com/in/scotthirleman/" rel="noopener noreferrer nofollow" target="_blank">https://www.linkedin.com/in/scotthirleman/</a></p><p>If you want to learn more and/or join the Data Mesh Learning Community, see here: <a href="https://datameshlearning.com/community/" rel="noopener noreferrer nofollow" target="_blank">https://datameshlearning.com/community/</a></p><p>If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see <a href="https://docs.google.com/document/d/1WkXLhSH7mnbjfTChD0uuYeIF5Tj0UBLUP4Jvl20Ym10/edit?usp=sharing" rel="noopener noreferrer nofollow" target="_blank">here</a></p><p>All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): <a href="https://pixabay.com/users/lesfm-22579021/" rel="noopener noreferrer nofollow" target="_blank">Lesfm</a>, <a href="https://pixabay.com/users/mondayhopes-22948862/?tab=audio" rel="noopener noreferrer nofollow" target="_blank">MondayHopes</a>, <a href="https://pixabay.com/users/sergequadrado-24990007/" rel="noopener noreferrer nofollow" target="_blank">SergeQuadrado</a>, and/or <a href="https://pixabay.com/users/nevesf-5724572/" rel="noopener noreferrer nofollow" target="_blank">nevesf</a></p><p>Data Mesh Radio is brought to you as a community resource by DataStax. Check out their high-scale, multi-region database offering (w/ lots of great APIs) and use code DAAP500 for a free $500 credit (apply under "add payment"): <a href="https://www.datastax.com/products/datastax-astra?utm_source=DataMeshRadio" rel="noopener noreferrer nofollow" target="_blank">AstraDB</a></p>