Doug Cutting's guest appearances

Doug Cutting

UC Berkeley School of Information

Analyse Asia with Bernard Leong

Data Engineering Podcast

Gary Vaynerchuk

Jessica Lessin

Robert Greene

Gary Liu

Paul Bradley

Kaiser Kuo

David Clark

Ryan Hoover

Nikita Shamgunov

Christopher Ryan

Jamie Metzl

Mitch Kapor

Scott Livingston

Chris Yeh

Andrew Ng

OwlTail

Julien Le Dem

Summary With the wealth of formats for sending and storing data it can be difficult to determine which one to use. In this episode Doug Cutting, creator of Avro, and Julien Le Dem, creator of Parquet, dig into the different classes of serialization formats, what their strengths are, and how to choose one for your workload. They also discuss the role of Arrow as a mechanism for in-memory data sharing and how hardware evolution will influence the state of the art for data formats. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers This is your host Tobias Macey and today I’m interviewing Julien Le Dem and Doug Cutting about data serialization formats and how to pick the right one for your systems.  Interview  Introduction How did you first get involved in the area of data management? What are the main serialization formats used for data storage and analysis? What are the tradeoffs that are offered by the different formats? How have the different storage and analysis tools influenced the types of storage formats that are available? You’ve each developed a new on-disk data format, Avro and Parquet respectively. What were your motivations for investing that time and effort? Why is it important for data engineers to carefully consider the format in which they transfer their data between systems? What are the switching costs involved in moving from one format to another after you have started using it in a production system?   What are some of the new or upcoming formats that you are each excited about? How do you anticipate the evolving hardware, patterns, and tools for processing data to influence the types of storage formats that maintain or grow their popularity?  Contact Information  Doug: cutting on GitHub Blog @cutting on Twitter   Julien Email @J_ on Twitter Blog julienledem on GitHub    Links  Apache Avro Apache Parquet Apache Arrow Hadoop Apache Pig Xerox Parc Excite Nutch Vertica Dremel White Paper  Twitter Blog on Release of Parquet   CSV XML Hive Impala Presto Spark SQL Brotli ZStandard Apache Drill Trevni Apache Calcite  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SA Support Data Engineering Podcast

<h2>Summary</h2> <p>With the wealth of formats for sending and storing data it can be difficult to determine which one to use. In this episode Doug Cutting, creator of Avro, and Julien Le Dem, creator of Parquet, dig into the different classes of serialization formats, what their strengths are, and how to choose one for your workload. They also discuss the role of Arrow as a mechanism for in-memory data sharing and how hardware evolution will influence the state of the art for data formats.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure</li> <li>When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.</li> <li>Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to <a href="https://www.dataengineeringpodcast.com/gocd?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/gocd</a> to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.</li> <li>You can help support the show by checking out the <a href="https://www.patreon.com/dataengineeringpodcast?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Patreon page</a> which is linked from the site.</li> <li>To help other people find the show you can leave a review on <a href="https://itunes.apple.com/us/podcast/data-engineering-podcast/id1193040557?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">iTunes</a>, or <a href="https://goo.gl/app/playmusic?ibi=com.google.PlayMusic&amp;isi=691797987&amp;ius=googleplaymusic&amp;link=https://play.google.com/music/m/Ivucmwlkrxbgfe5dhj562xgev4u?t%3DData_Engineering_Podcast%26pcampaignid%3DMKT-na-all-co-pr-mu-pod-16&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Google Play Music</a>, and tell your friends and co-workers</li> <li>This is your host Tobias Macey and today I’m interviewing Julien Le Dem and Doug Cutting about data serialization formats and how to pick the right one for your systems.</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you first get involved in the area of data management?</li> <li>What are the main serialization formats used for data storage and analysis?</li> <li>What are the tradeoffs that are offered by the different formats?</li> <li>How have the different storage and analysis tools influenced the types of storage formats that are available?</li> <li>You’ve each developed a new on-disk data format, Avro and Parquet respectively. What were your motivations for investing that time and effort?</li> <li>Why is it important for data engineers to carefully consider the format in which they transfer their data between systems?<br><ul> <li>What are the switching costs involved in moving from one format to another after you have started using it in a production system?</li> </ul> </li> <li>What are some of the new or upcoming formats that you are each excited about?</li> <li>How do you anticipate the evolving hardware, patterns, and tools for processing data to influence the types of storage formats that maintain or grow their popularity?</li> </ul> <h1>Contact Information</h1> <ul> <li>Doug:<br><ul> <li><a href="https://github.com/cutting?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">cutting</a> on GitHub</li> <li><a href="https://cutting.wordpress.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Blog</a></li> <li><a href="https://twitter.com/cutting?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@cutting</a> on Twitter</li> </ul> </li> <li>Julien<br><ul> <li><a href="mailto:julien@apache.org">Email</a></li> <li><a href="https://twitter.com/J_?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@J_</a> on Twitter</li> <li><a href="https://techblug.wordpress.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Blog</a></li> <li><a href="https://github.com/julienledem?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">julienledem</a> on GitHub</li> </ul> </li> </ul> <h1>Links</h1> <ul> <li><a href="http://avro.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Avro</a></li> <li><a href="https://parquet.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Parquet</a></li> <li><a href="http://arrow.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Arrow</a></li> <li><a href="http://hadoop.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hadoop</a></li> <li><a href="http://pig.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Pig</a></li> <li><a href="https://www.parc.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Xerox Parc</a></li> <li><a href="https://en.wikipedia.org/wiki/Excite?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Excite</a></li> <li><a href="http://nutch.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Nutch</a></li> <li><a href="https://www.vertica.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Vertica</a></li> <li><a href="https://research.google.com/pubs/pub36632.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Dremel White Paper</a> <ul> <li><a href="https://blog.twitter.com/engineering/en_us/a/2013/dremel-made-simple-with-parquet.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Twitter Blog on Release of Parquet</a></li> </ul> </li> <li><a href="https://en.wikipedia.org/wiki/Comma-separated_values?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CSV</a></li> <li><a href="https://en.wikipedia.org/wiki/XML?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">XML</a></li> <li><a href="http://hive.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hive</a></li> <li><a href="http://impala.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Impala</a></li> <li><a href="https://prestodb.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Presto</a></li> <li><a href="https://spark.apache.org/sql/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark SQL</a></li> <li><a href="https://github.com/google/brotli?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Brotli</a></li> <li><a href="http://facebook.github.io/zstd/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ZStandard</a></li> <li><a href="http://drill.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Drill</a></li> <li><a href="https://github.com/cutting/trevni?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Trevni</a></li> <li><a href="https://calcite.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Calcite</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p> <p></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Data Serialization Formats with Doug Cutting and Julien Le Dem - Episode 8

Use of new data technologies now pervades our institutions, both private and government. But this data-driven revolution is far from complete. We can still influence where it takes us. I will discuss some of the current challenges we face, both technical and social, and how we might address them.Doug Cutting (@cutting) is the founder of numerous successful open-source projects, including Lucene, Nutch, Avro, and Hadoop. Doug joined Cloudera in 2009 from Yahoo!, where he was a key member of the team that built and deployed a production Hadoop storage and analysis cluster for mission-critical business analytics. Doug holds a bachelor’s degree from Stanford University and sits on the board of the Apache Software Foundation.

Use of new data technologies now pervades our institutions, both private and government. But this data-driven revolution is far from complete. We can still influence where it takes us. I will discuss some of the current challenges we face, both technical and social, and how we might address them.<br><br>Doug Cutting (@cutting) is the founder of numerous successful open-source projects, including Lucene, Nutch, Avro, and Hadoop. Doug joined Cloudera in 2009 from Yahoo!, where he was a key member of the team that built and deployed a production Hadoop storage and analysis cluster for mission-critical business analytics. Doug holds a bachelor’s degree from Stanford University and sits on the board of the Apache Software Foundation.

Challenges for the Data Ecosystem (Doug Cutting, Chief Architect, Cloudera)

Bernard Leong

With Doug Cutting, the chief architect of Cloudera and the co-founder of Apache Hadoop, we discussed his journey from a software engineer to a contributor and creator of several open source software which are widely used today. We also deep dived into Cloudera and Hadoop, and discussed various topics from talent shortage to how Hadoop The post  Episode 79: Hadoop, Cloudera &amp; Big Data with Doug Cutting appeared first on Analyse Asia.

<p>With Doug Cutting, the chief architect of Cloudera and the co-founder of Apache Hadoop, we discussed his journey from a software engineer to a contributor and creator of several open source software which are widely used today. We also deep dived into Cloudera and Hadoop, and discussed various topics from talent shortage to how Hadoop</p> <p>The post <a href="https://analyse.asia/2015/12/05/episode-79-hadoop-cloudera-open-source-with-doug-cutting/" rel="nofollow" target="_blank"> Episode 79: Hadoop, Cloudera &amp; Big Data with Doug Cutting</a> appeared first on <a href="https://analyse.asia" rel="nofollow" target="_blank">Analyse Asia</a>.</p>

Episode 79: Hadoop, Cloudera & Big Data with Doug Cutting - Analyse Asia with Bernard Leong