Sean T. Allen's guest appearances

Sean T. Allen

Data Engineering Podcast

Nikita Shamgunov

Christopher Ryan

Emily Miller

Peter Bull

Patrick Hunt

Jonathan Katz

Kevin Moore

Dan Robinson

Matthew Rocklin

Maxime Beauchemin

Jean Georges Perrin

Todd Walter

Mark Marinelli

Andy LoPresto

Joe Crobak

OwlTail

Summary Data oriented applications that need to operate on large, fast-moving sterams of information can be difficult to build and scale due to the need to manage their state. In this episode Sean T. Allen, VP of engineering for Wallaroo Labs, explains how Wallaroo was designed and built to reduce the cognitive overhead of building this style of project. He explains the motivation for building Wallaroo, how it is implemented, and how you can start using it today. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers Your host is Tobias Macey and today I’m interviewing Sean T. Allen about Wallaroo, a framework for building and operating stateful data applications at scale  Interview  Introduction How did you get involved in the area of data engineering? What is Wallaroo and how did the project get started? What is the Pony language, and what features does it have that make it well suited for the problem area that you are focusing on? Why did you choose to focus first on Python as the language for interacting with Wallaroo and how is that integration implemented? How is Wallaroo architected internally to allow for distributed state management?  Is the state persistent, or is it only maintained long enough to complete the desired computation? If so, what format do you use for long term storage of the data?   What have been the most challenging aspects of building the Wallaroo platform? Which axes of the CAP theorem have you optimized for? For someone who wants to build an application on top of Wallaroo, what is involved in getting started? Once you have a working application, what resources are necessary for deploying to production and what are the scaling factors?  What are the failure modes that users of Wallaroo need to account for in their application or infrastructure?   What are some situations or problem types for which Wallaroo would be the wrong choice? What are some of the most interesting or unexpected uses of Wallaroo that you have seen? What do you have planned for the future of Wallaroo?  Contact Info  IRC Mailing List Wallaroo Labs Twitter Email Personal Twitter  Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Wallaroo Labs Storm Applied Apache Storm Risk Analysis Pony Language Erlang Akka Tail Latency High Performance Computing Python Apache Software Foundation Beyond Distributed Transactions: An Apostate’s View Consistent Hashing Jepsen Lineage Driven Fault Injection Chaos Engineering QCon 2016 Talk Codemesh in London: How did I get here? CAP Theorem CRDT Sync Free Project Basho Wallaroo on GitHub Docker Puppet Chef Ansible SaltStack Kafka TCP Dask Data Engineering Episode About Dask Beowulf Cluster Redis Flink Haskell  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>Data oriented applications that need to operate on large, fast-moving sterams of information can be difficult to build and scale due to the need to manage their state. In this episode Sean T. Allen, VP of engineering for Wallaroo Labs, explains how Wallaroo was designed and built to reduce the cognitive overhead of building this style of project. He explains the motivation for building Wallaroo, how it is implemented, and how you can start using it today.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure</li> <li>When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.</li> <li>Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to <a href="https://www.dataengineeringpodcast.com/gocd?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/gocd</a> to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.</li> <li>You can help support the show by checking out the <a href="https://www.patreon.com/dataengineeringpodcast?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Patreon page</a> which is linked from the site.</li> <li>To help other people find the show you can leave a review on <a href="https://itunes.apple.com/us/podcast/data-engineering-podcast/id1193040557?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">iTunes</a>, or <a href="https://goo.gl/app/playmusic?ibi=com.google.PlayMusic&amp;isi=691797987&amp;ius=googleplaymusic&amp;link=https://play.google.com/music/m/Ivucmwlkrxbgfe5dhj562xgev4u?t%3DData_Engineering_Podcast%26pcampaignid%3DMKT-na-all-co-pr-mu-pod-16&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Google Play Music</a>, and tell your friends and co-workers</li> <li>Your host is Tobias Macey and today I’m interviewing Sean T. Allen about Wallaroo, a framework for building and operating stateful data applications at scale</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data engineering?</li> <li>What is Wallaroo and how did the project get started?</li> <li>What is the Pony language, and what features does it have that make it well suited for the problem area that you are focusing on?</li> <li>Why did you choose to focus first on Python as the language for interacting with Wallaroo and how is that integration implemented?</li> <li>How is Wallaroo architected internally to allow for distributed state management? <ul> <li>Is the state persistent, or is it only maintained long enough to complete the desired computation?</li> <li>If so, what format do you use for long term storage of the data?</li> </ul> <p></p></li> <li>What have been the most challenging aspects of building the Wallaroo platform?</li> <li>Which axes of the CAP theorem have you optimized for?</li> <li>For someone who wants to build an application on top of Wallaroo, what is involved in getting started?</li> <li>Once you have a working application, what resources are necessary for deploying to production and what are the scaling factors?<p></p> <ul> <li>What are the failure modes that users of Wallaroo need to account for in their application or infrastructure?</li> </ul> <p></p></li> <li>What are some situations or problem types for which Wallaroo would be the wrong choice?</li> <li>What are some of the most interesting or unexpected uses of Wallaroo that you have seen?</li> <li>What do you have planned for the future of Wallaroo?</li> </ul><p></p> <h2>Contact Info</h2> <ul> <li><a href="https://webchat.freenode.net/?channels=#wallaroo&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">IRC</a></li> <li><a href="https://groups.io/g/wallaroo/topics?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Mailing List</a></li> <li><a href="https://www.twitter.com/wallaroolabs?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Wallaroo Labs Twitter</a></li> <li><a href="sean@wallaroolabs.com" rel="nofollow" target="_blank">Email</a></li> <li><a href="https://www.twitter.com/seantallen?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Personal Twitter</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.wallaroolabs.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Wallaroo Labs</a></li> <li><a href="http://amzn.to/2l4flAa?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Storm Applied</a></li> <li><a href="http://storm.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Storm</a></li> <li><a href="https://en.wikipedia.org/wiki/Risk_analysis?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Risk Analysis</a></li> <li><a href="https://www.ponylang.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pony Language</a></li> <li><a href="https://www.erlang.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Erlang</a></li> <li><a href="https://akka.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Akka</a></li> <li><a href="http://blog.virtualstoragezone.com/what-the-heck-is-tail-latency-anyways/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Tail Latency</a></li> <li><a href="https://insidehpc.com/hpc-basic-training/what-is-hpc/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">High Performance Computing</a></li> <li><a href="https://www.python.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Python</a></li> <li><a href="http://apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Software Foundation</a></li> <li><a href="https://pdfs.semanticscholar.org/c20b/aa16cb57ff4979569871d15294fa720bbc23.pdf?_ga=2.169444321.1349048119.1514172024-709975131.1514172024&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Beyond Distributed Transactions: An Apostate’s View</a></li> <li><a href="https://en.wikipedia.org/wiki/Consistent_hashing?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Consistent Hashing</a></li> <li><a href="https://jepsen.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Jepsen</a></li> <li><a href="https://people.eecs.berkeley.edu/~palvaro/molly.pdf?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Lineage Driven Fault Injection</a></li> <li><a href="http://principlesofchaos.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Chaos Engineering</a></li> <li><a href="https://www.infoq.com/presentations/trust-distributed-systems?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">QCon 2016 Talk</a></li> <li><a href="https://www.youtube.com/watch?v=6MsPDtpe2tg&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Codemesh in London: How did I get here?</a></li> <li><a href="https://en.wikipedia.org/wiki/CAP_theorem?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CAP Theorem</a></li> <li><a href="https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CRDT</a></li> <li><a href="https://syncfree.lip6.fr/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Sync Free Project</a></li> <li><a href="https://github.com/basho?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Basho</a></li> <li><a href="https://github.com/wallarooLabs/wallaroo?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Wallaroo on GitHub</a></li> <li><a href="https://www.docker.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Docker</a></li> <li><a href="https://puppet.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Puppet</a></li> <li><a href="https://www.chef.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Chef</a></li> <li><a href="https://www.ansible.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Ansible</a></li> <li><a href="https://saltstack.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">SaltStack</a></li> <li><a href="https://kafka.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka</a></li> <li><a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">TCP</a></li> <li><a href="https://dask.pydata.org/en/latest/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Dask</a></li> <li><a href="https://www.dataengineeringpodcast.com/episode-2-dask-with-matthew-rocklin/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Data Engineering Episode About Dask</a></li> <li><a href="https://en.wikipedia.org/wiki/Beowulf_cluster?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Beowulf Cluster</a></li> <li><a href="https://redis.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Redis</a></li> <li><a href="http://flink.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flink</a></li> <li><a href="https://www.haskell.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Haskell</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Wallaroo with Sean T. Allen - Episode 12