Todd Walter's guest appearances

Todd Walter

Data Engineering Podcast

Industrial IoT Spotlight

ISG Scalable Data Management Series

Christopher Ryan

Don Miller

Matt Harrison

Emily Miller

Peter Bull

Jeff Miller

Alex Moazed

Patrick Hunt

Jonathan Katz

Drew Conway

Kevin Moore

Dan Robinson

Jason Shepherd

Matthew Rocklin

Maxime Beauchemin

OwlTail

Summary As your data needs scale across an organization the need for a carefully considered approach to collection, storage, organization, and access becomes increasingly critical. In this episode Todd Walter shares his considerable experience in data curation to clarify the many aspects that are necessary for a successful platform for your business. Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Todd Walter about data curation and how to architect your data systems to support high quality, maintainable intelligence  Interview  Introduction How did you get involved in the area of data management? How do you define data curation? What are some of the high level concerns that are encapsulated in that effort?   How does the size and maturity of a company affect the ways that they architect and interact with their data systems? Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it? What are some of the common mistakes that are made when designing a data architecture and how do they lead to failure? What has changed in terms of complexity and scope for data architecture and curation since you first started working in this space? As “big data” became more widely discussed the common mantra was to store everything because you never know when you’ll need the data that might get thrown away. As the industry is reaching a greater degree of maturity and more regulations are implemented there has been a shift to being more considerate as to what information gets stored and for how long. What are your views on that evolution and what is your litmus test for determining which data to keep? In terms of infrastructure, what are the components of a modern data architecture and how has that changed over the years? What is your opinion on the relative merits of a data warehouse vs a data lake and are they mutually exclusive?   Once an architecture has been established, how do you allow for continued evolution to prevent stagnation and eventual failure? ETL has long been the default approach for building and enforcing data architecture, but there have been significant shifts in recent years due to the emergence of streaming systems and ELT approaches in new data warehouses. What are your thoughts on the landscape for managing data flows and migration and when to use which approach? What are some of the areas of data architecture and curation that are most often forgotten or ignored? What resources do you recommend for anyone who is interested in learning more about the landscape of data architecture and curation?  Contact Info  LinkedIn  Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Teradata Data Architecture Data Curation Data Warehouse Chief Data Officer ETL (Extract, Transform, Load) Data Lake Metadata Data Lineage  Data Provenance   Strata Conference ELT (Extract, Load, Transform) Map-Reduce Hive Pig Spark Data Governance  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SA Support Data Engineering Podcast

<h2>Summary</h2> <p>As your data needs scale across an organization the need for a carefully considered approach to collection, storage, organization, and access becomes increasingly critical. In this episode Todd Walter shares his considerable experience in data curation to clarify the many aspects that are necessary for a successful platform for your business. Using the metaphor of a museum curator carefully managing the precious resources on display and in the vaults, he discusses the various layers of an enterprise data strategy. This includes modeling the lifecycle of your information as a pipeline from the raw, messy, loosely structured records in your data lake, through a series of transformations and ultimately to your data warehouse. He also explains which layers are useful for the different members of the business, and which pitfalls to look out for along the path to a mature and flexible data platform.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> to get a $20 credit and launch a new server in under a minute.</li> <li>You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at <a href="https://www.dataengineeringpodcast.com/metis-machine?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/metis-machine</a> to learn more about how Metis Machine is operationalizing data science.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Join the community in the new Zulip chat workspace at <a href="https://www.dataengineeringpodcast.com/chat?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/chat</a></li> <li>Your host is Tobias Macey and today I’m interviewing Todd Walter about data curation and how to architect your data systems to support high quality, maintainable intelligence</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>How do you define data curation?<br><ul> <li>What are some of the high level concerns that are encapsulated in that effort?</li> </ul> </li> <li>How does the size and maturity of a company affect the ways that they architect and interact with their data systems?</li> <li>Can you walk through the stages of an ideal lifecycle for data within the context of an organizations uses for it?</li> <li>What are some of the common mistakes that are made when designing a data architecture and how do they lead to failure?</li> <li>What has changed in terms of complexity and scope for data architecture and curation since you first started working in this space?</li> <li>As “big data” became more widely discussed the common mantra was to store everything because you never know when you’ll need the data that might get thrown away. As the industry is reaching a greater degree of maturity and more regulations are implemented there has been a shift to being more considerate as to what information gets stored and for how long. What are your views on that evolution and what is your litmus test for determining which data to keep?</li> <li>In terms of infrastructure, what are the components of a modern data architecture and how has that changed over the years?<br><ul> <li>What is your opinion on the relative merits of a data warehouse vs a data lake and are they mutually exclusive?</li> </ul> </li> <li>Once an architecture has been established, how do you allow for continued evolution to prevent stagnation and eventual failure?</li> <li>ETL has long been the default approach for building and enforcing data architecture, but there have been significant shifts in recent years due to the emergence of streaming systems and ELT approaches in new data warehouses. What are your thoughts on the landscape for managing data flows and migration and when to use which approach?</li> <li>What are some of the areas of data architecture and curation that are most often forgotten or ignored?</li> <li>What resources do you recommend for anyone who is interested in learning more about the landscape of data architecture and curation?</li> </ul> <h2>Contact Info</h2> <ul> <li><a href="https://ca.linkedin.com/in/ToddAWalter?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LinkedIn</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.teradata.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Teradata</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_architecture?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Architecture</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_curation?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Curation</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_warehouse?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Warehouse</a></li> <li><a href="https://en.wikipedia.org/wiki/Chief_data_officer?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Chief Data Officer</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_transform,_load?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ETL (Extract, Transform, Load)</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_lake?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Lake</a></li> <li><a href="https://en.wikipedia.org/wiki/Metadata?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Metadata</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_lineage?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Lineage</a> <ul> <li><a href="https://en.wikipedia.org/wiki/Data_lineage#Data_Provenance?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Provenance</a></li> </ul> </li> <li><a href="https://www.podcastinit.com/oreilly-strata?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Strata Conference</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_load,_transform?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ELT (Extract, Load, Transform)</a></li> <li><a href="https://en.wikipedia.org/wiki/MapReduce?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Map-Reduce</a></li> <li><a href="http://hive.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hive</a></li> <li><a href="http://pig.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pig</a></li> <li><a href="http://spark.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_governance?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Governance</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p> <p></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

A Primer On Enterprise Data Curation with Todd Walter - Episode 49

Paul Didier

The Industrial Internet Consortium’s (IIC) Time Sensitive Networking (TSN) Testbed will be used real-time control and synchronization of high performance machines over a single, standard Ethernet network. Learn more about the IIC: http://www.iiconsortium.org/index.htm

The Industrial Internet Consortium’s (IIC) Time Sensitive Networking (TSN) Testbed will be used real-time control and synchronization of high performance machines over a single, standard Ethernet network. Learn more about the IIC: <a href="http://www.iiconsortium.org/index.htm" rel="nofollow" target="_blank">http://www.iiconsortium.org/index.htm</a>

EP002: Time Sensitive Networks - An Interview with Cisco's Paul Didier and National Instruments' Todd Walter

Teradata Past, Present and Future - Todd Walter, Teradata Corp. 10/2/09