Fabian Hueske's guest appearances

Fabian Hueske

Data Engineering Podcast

Nikita Shamgunov

Christopher Ryan

Emily Miller

Peter Bull

Patrick Hunt

Jonathan Katz

Kevin Moore

Dan Robinson

Matthew Rocklin

Maxime Beauchemin

Jean Georges Perrin

Todd Walter

Mark Marinelli

Andy LoPresto

Joe Crobak

OwlTail

Summary Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. In this episode Fabian Hueske, one of the original authors, explains how Flink is architected, how it is being used to power some of the world’s largest businesses, where it sits in the lanscape of stream processing tools, and how you can start using it today. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Fabian Hueske, co-author of the upcoming O’Reilly book Stream Processing With Apache Flink, about his work on Apache Flink, the stateful streaming engine  Interview  Introduction How did you get involved in the area of data management? Can you start by describing what Flink is and how the project got started? What are some of the primary ways that Flink is used? How does Flink compare to other streaming engines such as Spark, Kafka, Pulsar, and Storm?  What are some use cases that Flink is uniquely qualified to handle?   Where does Flink fit into the current data landscape? How is Flink architected?  How has that architecture evolved? Are there any aspects of the current design that you would do differently if you started over today?   How does scaling work in a Flink deployment?  What are the scaling limits? What are some of the failure modes that users should be aware of?   How is the statefulness of a cluster managed?  What are the mechanisms for managing conflicts? What are the limiting factors for the volume of state that can be practically handled in a cluster and for a given purpose? Can state be shared across processes or tasks within a Flink cluster?   What are the comparative challenges of working with bounded vs unbounded streams of data? How do you handle out of order events in Flink, especially as the delay for a given event increases? For someone who is using Flink in their environment, what are the primary means of interacting with and developing on top of it? What are some of the most challenging or complicated aspects of building and maintaining Flink? What are some of the most interesting or unexpected ways that you have seen Flink used? What are some of the improvements or new features that are planned for the future of Flink? What are some features or use cases that you are explicitly not planning to support? For people who participate in the training sessions that you offer through Data Artisans, what are some of the concepts that they are challenged by?  What do they find most interesting or exciting?    Contact Info  LinkedIn @fhueske on Twitter fhueske on GitHub  Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Flink Data Artisans IBM DB2 Technische Universität Berlin Hadoop Relational Database Google Cloud Dataflow Spark Cascading Java RocksDB Flink Checkpoints Flink Savepoints Kafka Pulsar Storm Scala LINQ (Language INtegrated Query) SQL Backpressure Watermarks HDFS S3 Avro JSON Hive Metastore Dell EMC Pravega  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>Modern applications and data platforms aspire to process events and data in real time at scale and with low latency. Apache Flink is a true stream processing engine with an impressive set of capabilities for stateful computation at scale. In this episode Fabian Hueske, one of the original authors, explains how Flink is architected, how it is being used to power some of the world’s largest businesses, where it sits in the lanscape of stream processing tools, and how you can start using it today.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> today to get a $20 credit and launch a new server in under a minute.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Join the community in the new Zulip chat workspace at <a href="https://www.dataengineeringpodcast.com/chat?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/chat</a></li> <li>Your host is Tobias Macey and today I’m interviewing Fabian Hueske, co-author of the upcoming O’Reilly book <a href="http://shop.oreilly.com/product/0636920057321.do?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Stream Processing With Apache Flink</a>, about his work on Apache Flink, the stateful streaming engine</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by describing what Flink is and how the project got started?</li> <li>What are some of the primary ways that Flink is used?</li> <li>How does Flink compare to other streaming engines such as Spark, Kafka, Pulsar, and Storm? <ul> <li>What are some use cases that Flink is uniquely qualified to handle?</li> </ul> <p></p></li> <li>Where does Flink fit into the current data landscape?</li> <li>How is Flink architected?<p></p> <ul> <li>How has that architecture evolved?</li> <li>Are there any aspects of the current design that you would do differently if you started over today?</li> </ul> <p></p></li> <li>How does scaling work in a Flink deployment?<p></p> <ul> <li>What are the scaling limits?</li> <li>What are some of the failure modes that users should be aware of?</li> </ul> <p></p></li> <li>How is the statefulness of a cluster managed?<p></p> <ul> <li>What are the mechanisms for managing conflicts?</li> <li>What are the limiting factors for the volume of state that can be practically handled in a cluster and for a given purpose?</li> <li>Can state be shared across processes or tasks within a Flink cluster?</li> </ul> <p></p></li> <li>What are the comparative challenges of working with bounded vs unbounded streams of data?</li> <li>How do you handle out of order events in Flink, especially as the delay for a given event increases?</li> <li>For someone who is using Flink in their environment, what are the primary means of interacting with and developing on top of it?</li> <li>What are some of the most challenging or complicated aspects of building and maintaining Flink?</li> <li>What are some of the most interesting or unexpected ways that you have seen Flink used?</li> <li>What are some of the improvements or new features that are planned for the future of Flink?</li> <li>What are some features or use cases that you are explicitly not planning to support?</li> <li>For people who participate in the training sessions that you offer through Data Artisans, what are some of the concepts that they are challenged by?<p></p> <ul> <li>What do they find most interesting or exciting?</li> </ul> <p></p></li> </ul><p></p> <h2>Contact Info</h2> <ul> <li><a href="https://www.linkedin.com/in/fhueske/?originalSubdomain=de&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LinkedIn</a></li> <li><a href="https://twitter.com/fhueske?lang=en&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@fhueske</a> on Twitter</li> <li><a href="https://github.com/fhueske?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">fhueske</a> on GitHub</li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://flink.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flink</a></li> <li><a href="https://data-artisans.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Artisans</a></li> <li><a href="https://en.wikipedia.org/wiki/IBM?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">IBM</a></li> <li><a href="https://www.ibm.com/analytics/us/en/db2/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DB2</a></li> <li><a href="https://www.tu-berlin.de/menue/home/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Technische Universität Berlin</a></li> <li><a href="http://hadoop.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hadoop</a></li> <li><a href="https://en.wikipedia.org/wiki/Relational_database?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Relational Database</a></li> <li><a href="https://cloud.google.com/dataflow/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Google Cloud Dataflow</a></li> <li><a href="https://spark.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark</a></li> <li><a href="https://www.cascading.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Cascading</a></li> <li><a href="https://en.wikipedia.org/wiki/Java_(programming_language)?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Java</a></li> <li><a href="https://rocksdb.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">RocksDB</a></li> <li><a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/checkpoints.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flink Checkpoints</a></li> <li><a href="https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/savepoints.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flink Savepoints</a></li> <li><a href="https://kafka.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka</a></li> <li><a href="https://pulsar.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pulsar</a></li> <li><a href="http://storm.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Storm</a></li> <li><a href="https://www.scala-lang.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Scala</a></li> <li><a href="https://en.wikipedia.org/wiki/Language_Integrated_Query?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LINQ (Language INtegrated Query)</a></li> <li><a href="https://en.wikipedia.org/wiki/SQL?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">SQL</a></li> <li><a href="https://data-artisans.com/blog/how-flink-handles-backpressure?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Backpressure</a></li> <li><a href="https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html#watermarks-in-parallel-streams?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Watermarks</a></li> <li><a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">HDFS</a></li> <li><a href="https://aws.amazon.com/s3/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">S3</a></li> <li><a href="https://avro.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Avro</a></li> <li><a href="https://www.json.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">JSON</a></li> <li><a href="https://cwiki.apache.org/confluence/display/Hive/Design#Design-Metastore?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hive Metastore</a></li> <li><a href="https://www.dellemc.com/en-us/index.htm?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Dell EMC</a></li> <li><a href="http://www.pravega.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pravega</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Stateful, Distributed Stream Processing on Flink with Fabian Hueske - Episode 57