Apache Kafka® 3.5 is here with the capability of previewing migrations between ZooKeeper clusters to KRaft mode. Follow along as Danica Fine highlights key release updates.Kafka Core:KIP-833 provides an updated timeline for KRaft.KIP-866 now is preview and allows migration from an existing ZooKeeper cluster to KRaft mode.KIP-900 introduces a way to bootstrap the KRaft controllers with SCRAM credentials.KIP-903 prevents a data loss scenario by preventing replicas with stale broker epochs from joining the ISR list. KIP-915 streamlines the process of downgrading Kafka's transaction and group coordinators by introducing tagged fields.Kafka Connect:KIP-710 provides the option to use a REST API for internal server communication that can be enabled by setting `dedicated.mode.enable.internal.rest` equal to true. KIP-875 offers support for native offset management in Kafka Connect. Connect cluster administrators can now read offsets for both source and sink connectors. This KIP adds a new STOPPED state for connectors, enabling users to shut down connectors and maintain connector configurations without utilizing resources.KIP-894 makes `IncrementalAlterConfigs` API available for use in MirrorMaker 2 (MM2), adding a new use.incremental.alter.config configuration which takes values “requested,” “never,” and “required.”KIP-911 adds a new source tag for metrics generated by the `MirrorSourceConnector` to help monitor mirroring deployments.Kafka Streams:KIP-339 improves Kafka Streams' error-handling capabilities by addressing serialization errors that occur before message production and extending the interface for custom error handling. KIP-889 introduces versioned state stores in Kafka Streams for temporal join semantics in stream-to-table joins. KIP-904 simplifies table aggregation in Kafka by proposing a change in serialization format to enable one-step aggregation and reduce noise from events with old and new keys/values. KIP-914 modifies how versioned state stores are used in Kafka Streams. Versioned state stores may impact different DSL processors in varying ways, see the documentation for details.Kafka Client:KIP-881 is now complete and introduces new client-side assignor logic for rack-aware consumer balancing for Kafka Consumers. KIP-887 adds the `EnvVarConfigProvider` implementation to Kafka so custom configurations stored in environment variables can be injected into the system by providing the map returned by `System.getEnv()`.KIP 641 introduces the `RecordReader` interface to Kafka's clients module, replacing the deprecated MessageReader Scala trait. EPISODE LINKSSee release notes for Apache Kafka 3.5Read the blog to learn moreDownload and get started with Apache Kafka 3.5Watch the video version of this podcast

Apache Kafka® 3.5 is here with the capability of previewing migrations between ZooKeeper clusters to KRaft mode. Follow along as Danica Fine highlights key release updates. Kafka Core:<ul><li>KIP-833 provides an updated timeline for KRaft.</li><li>KIP-866 now is preview and allows migration from an existing ZooKeeper cluster to KRaft mode.</li><li>KIP-900 introduces a way to bootstrap the KRaft controllers with SCRAM credentials.</li><li>KIP-903 prevents a data loss scenario by preventing replicas with stale broker epochs from joining the ISR list. </li><li>KIP-915 streamlines the process of downgrading Kafka's transaction and group coordinators by introducing tagged fields.</li></ul> Kafka Connect:<ul><li>KIP-710 provides the option to use a REST API for internal server communication that can be enabled by setting `dedicated.mode.enable.internal.rest` equal to true. </li><li>KIP-875 offers support for native offset management in Kafka Connect. Connect cluster administrators can now read offsets for both source and sink connectors. This KIP adds a new STOPPED state for connectors, enabling users to shut down connectors and maintain connector configurations without utilizing resources.</li><li>KIP-894 makes `IncrementalAlterConfigs` API available for use in MirrorMaker 2 (MM2), adding a new use.incremental.alter.config configuration which takes values “requested,” “never,” and “required.”</li><li>KIP-911 adds a new source tag for metrics generated by the `MirrorSourceConnector` to help monitor mirroring deployments.</li></ul> Kafka Streams:<ul><li>KIP-339 improves Kafka Streams' error-handling capabilities by addressing serialization errors that occur before message production and extending the interface for custom error handling. </li><li>KIP-889 introduces versioned state stores in Kafka Streams for temporal join semantics in stream-to-table joins. </li><li>KIP-904 simplifies table aggregation in Kafka by proposing a change in serialization format to enable one-step aggregation and reduce noise from events with old and new keys/values. </li><li>KIP-914 modifies how versioned state stores are used in Kafka Streams. Versioned state stores may impact different DSL processors in varying ways, see the documentation for details.</li></ul> Kafka Client:<ul><li>KIP-881 is now complete and introduces new client-side assignor logic for rack-aware consumer balancing for Kafka Consumers. </li><li>KIP-887 adds the `EnvVarConfigProvider` implementation to Kafka so custom configurations stored in environment variables can be injected into the system by providing the map returned by `System.getEnv()`.</li><li>KIP 641 introduces the `RecordReader` interface to Kafka's clients module, replacing the deprecated MessageReader Scala trait. </li></ul> EPISODE LINKS<ul><li>See <a href="https://archive.apache.org/dist/kafka/3.5.0/RELEASE_NOTES.html" rel="nofollow" target="_blank">release notes</a> for Apache Kafka 3.5</li><li>Read the <a href="https://cnfl.io/apache-kafka-3-5-blog" rel="nofollow" target="_blank">blog</a> to learn more</li><li><a href="https://kafka.apache.org/downloads" rel="nofollow" target="_blank">Download</a> and <a href="https://kafka.apache.org/documentation/#quickstart" rel="nofollow" target="_blank">get started</a> with Apache Kafka 3.5</li><li><a href="https://youtu.be/BVxDFL5iTx8" rel="nofollow" target="_blank">Watch the video</a> version of this podcast</li></ul>

Apache Kafka 3.5 - Kafka Core, Connect, Streams, & Client Updates

After recording 64 episodes and featuring 58 amazing guests, the Streaming Audio podcast series has amassed over 130,000 plays on YouTube in the last year. We're extremely proud of these achievements and feel that it's time to take a well-deserved break. Streaming Audio will be taking a vacation! We want to express our gratitude to you, our valued listeners, for spending 10,000 hours with us on this incredible journey.Rest assured, we will be back with more episodes! In the meantime, feel free to revisit some of our previous episodes. For instance, you can listen to Anna McDonald share her stories about the worst Apache Kafka® bugs she’s ever seen, or listen to Jun Rao offer his expert advice on running Kafka in production. And who could forget the charming backstory behind Mitch Seymour's Kafka storybook, Gently Down the Stream?These memorable episodes brought us joy, and we're thrilled to have shared them with you. As we reflect on our accomplishments with pride, we also look forward to an exciting future. Until we meet again, happy listening!EPISODE LINKSTop 6 Worst Apache Kafka JIRA BugsRunning Apache Kafka in ProductionLearn How Stream-Processing Works The Simplest Way PossibleWatch the video version of this podcastStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)  

After recording 64 episodes and featuring 58 amazing guests, the Streaming Audio podcast series has amassed over 130,000 plays on YouTube in the last year. We're extremely proud of these achievements and feel that it's time to take a well-deserved break. Streaming Audio will be taking a vacation! We want to express our gratitude to you, our valued listeners, for spending 10,000 hours with us on this incredible journey. Rest assured, we will be back with more episodes! In the meantime, feel free to revisit some of our previous episodes. For instance, you can listen to Anna McDonald share her stories about the worst Apache Kafka® bugs she’s ever seen, or listen to Jun Rao offer his expert advice on running Kafka in production. And who could forget the charming backstory behind Mitch Seymour's Kafka storybook, Gently Down the Stream? These memorable episodes brought us joy, and we're thrilled to have shared them with you. As we reflect on our accomplishments with pride, we also look forward to an exciting future. Until we meet again, happy listening! EPISODE LINKS<ul><li><a href="https://developer.confluent.io/podcast/top-6-worst-apache-kafka-jira-bugs/" rel="nofollow" target="_blank">Top 6 Worst Apache Kafka JIRA Bugs</a></li><li><a href="https://developer.confluent.io/podcast/running-apache-kafka-in-production/" rel="nofollow" target="_blank">Running Apache Kafka in Production</a></li><li><a href="https://developer.confluent.io/podcast/learn-how-stream-processing-works-the-simplest-way-possible/" rel="nofollow" target="_blank">Learn How Stream-Processing Works The Simplest Way Possible</a></li><li><a href="https://youtu.be/yFlvWRwRTT8" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)  </li></ul>

A Special Announcement from Streaming Audio

Have you ever struggled with managing data long term, especially as the schema changes over time? In order to manage and leverage data across an organization, it’s essential to have well-defined guidelines and standards in place around data quality, enforcement, and data transfer. To get started, Abraham Leal (Customer Success Technical Architect, Confluent) suggests that organizations associate their Apache Kafka® data with a data contract (schema). A data contract is an agreement between a service provider and data consumers. It defines the management and intended usage of data within an organization. In this episode, Abraham talks to Kris about how to use data contracts and schema enforcement to ensure long-term data management.When an organization sends and stores critical and valuable data in Kafka, more often than not it would like to leverage that data in various valuable ways for multiple business units. Kafka is particularly suited for this use case, but it can be problematic later on if the governance rules aren’t established up front.With schema registry, evolution is easy due to its robust security guarantees. When managing data pipelines, you can also use GitOps automation features for an extra control layer. It allows you to be creative with topic versioning, upcasting/downcasting the data collected, and adding quality assurance steps at the end of each run to ensure your project remains reliable.Abraham explains that Protobuf and Avro are the best formats to use rather than XML or JSON because they are built to handle schema evolution. In addition, they have a much lower overhead per-record, so you can save bandwidth and data storage costs by adopting them.There’s so much more to consider, but if you are thinking about implementing or integrating with your data quality team, Abraham suggests that you use schema registry heavily from the beginning.If you have more questions, Kris invites you to join the conversation. You can also watch the KOR Financial Current talk Abraham mentions or take Danica Fine’s free course on how to use schema registry on Confluent Developer.EPISODE LINKSOS projectKOR Financial Current TalkThe Key Concepts of Schema RegistrySchema Evolution and CompatibilitySchema Registry Made Simple by Confluent Cloud ft. Magesh NandakumarKris Jenkins’ TwitterWatch the video version of this podcastStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

Have you ever struggled with managing data long term, especially as the schema changes over time? In order to manage and leverage data across an organization, it’s essential to have well-defined guidelines and standards in place around data quality, enforcement, and data transfer. To get started, Abraham Leal (Customer Success Technical Architect, Confluent) suggests that organizations associate their Apache Kafka® data with a data contract (schema). A data contract is an agreement between a service provider and data consumers. It defines the management and intended usage of data within an organization. In this episode, Abraham talks to Kris about how to use data contracts and schema enforcement to ensure long-term data management. When an organization sends and stores critical and valuable data in Kafka, more often than not it would like to leverage that data in various valuable ways for multiple business units. Kafka is particularly suited for this use case, but it can be problematic later on if the governance rules aren’t established up front. With schema registry, evolution is easy due to its robust security guarantees. When managing data pipelines, you can also use GitOps automation features for an extra control layer. It allows you to be creative with topic versioning, upcasting/downcasting the data collected, and adding quality assurance steps at the end of each run to ensure your project remains reliable. Abraham explains that Protobuf and Avro are the best formats to use rather than XML or JSON because they are built to handle schema evolution. In addition, they have a much lower overhead per-record, so you can save bandwidth and data storage costs by adopting them. There’s so much more to consider, but if you are thinking about implementing or integrating with your data quality team, Abraham suggests that you use schema registry heavily from the beginning. If you have more questions, Kris invites you to join the conversation. You can also watch the KOR Financial Current talk Abraham mentions or take Danica Fine’s free course on how to use schema registry on Confluent Developer. EPISODE LINKS<ul><li><a href="https://github.com/abraham-leal/ccloud-schema-exporter" rel="nofollow" target="_blank">OS project</a></li><li><a href="https://www.confluent.io/events/current-2022/evolving-schemas-without-schema-evolution/" rel="nofollow" target="_blank">KOR Financial Current Talk</a></li><li><a href="https://developer.confluent.io/learn-kafka/schema-registry/key-concepts/" rel="nofollow" target="_blank">The Key Concepts of Schema Registry</a></li><li><a href="https://docs.confluent.io/platform/current/schema-registry/avro.html#schema-evolution-and-compatibility" rel="nofollow" target="_blank">Schema Evolution and Compatibility</a></li><li><a href="https://developer.confluent.io/podcast/schema-registry-made-simple-by-confluent-cloud-ft-magesh-nandakumar/" rel="nofollow" target="_blank">Schema Registry Made Simple by Confluent Cloud ft. Magesh Nandakumar</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://youtu.be/ZIJB8cs-cJU" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)</li></ul>

How to use Data Contracts for Long-Term Schema Management

Can you use Apache Kafka® and Python together? What’s the current state of Python support? And what are the best options to get started? In this episode, Dave Klein joins Kris to talk about all things Kafka and Python: the libraries, the tools, and the pros &amp; cons. He also talks about the new course he just launched to support Python programmers entering the event-streaming world.Dave has been an active member of the Kafka community for many years and noticed that there were a lot of Kafka resources for Java but few for Python. So he decided to create a course to help people get started using Python and Kafka together.Historically, Java has had the most documentation, and people have often missed how good the Python support is for Kafka users. Python and Kafka are an ideal fit for machine learning applications and data engineering in general. Yet there are a lot of use cases for building, streaming, and machine learning pipelines. In fact, someone conducted a survey to find out what languages were most popular in the Kafka community and Python came in second after Java. That’s how Dave got the idea to create a course for newbies.In this course, Dave combines video lectures with code-heavy exercises to give developers a taste of what the code looks like, how to structure it, a preview of the shape of the code, and the structure of the classes and the functions so you can get hands-on practice using the library. He also covers building a producer and a consumer and using the admin client. And, of course, there is a module that covers working with the schemas supported by the Kafka library.Dave explains that Python opens up a world of opportunity and is ripe for expansion. So if you are ready to dive in, head over to developer.confluent.io to learn more about Dave’s course.EPISODE LINKSBlog: Getting Started with Python for Apache KafkaCourse: Introduction to Apache Kafka for Python DevelopersStep-by-step guide: Building a Python client application for KafkaCoding in MotionBuilding and Designing Events and Event Streams with Apache KafkaWatch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

Can you use Apache Kafka® and Python together? What’s the current state of Python support? And what are the best options to get started? In this episode, Dave Klein joins Kris to talk about all things Kafka and Python: the libraries, the tools, and the pros &amp; cons. He also talks about the new course he just launched to support Python programmers entering the event-streaming world. Dave has been an active member of the Kafka community for many years and noticed that there were a lot of Kafka resources for Java but few for Python. So he decided to create a course to help people get started using Python and Kafka together. Historically, Java has had the most documentation, and people have often missed how good the Python support is for Kafka users. Python and Kafka are an ideal fit for machine learning applications and data engineering in general. Yet there are a lot of use cases for building, streaming, and machine learning pipelines. In fact, someone conducted a survey to find out what languages were most popular in the Kafka community and Python came in second after Java. That’s how Dave got the idea to create a course for newbies. In this course, Dave combines video lectures with code-heavy exercises to give developers a taste of what the code looks like, how to structure it, a preview of the shape of the code, and the structure of the classes and the functions so you can get hands-on practice using the library. He also covers building a producer and a consumer and using the admin client. And, of course, there is a module that covers working with the schemas supported by the Kafka library. Dave explains that Python opens up a world of opportunity and is ripe for expansion. So if you are ready to dive in, head over to <a href="http://developer.confluent.io" rel="nofollow" target="_blank">developer.confluent.io</a> to learn more about Dave’s course. EPISODE LINKS<ul><li>Blog: <a href="https://www.confluent.io/blog/kafka-python-developer-guide/" rel="nofollow" target="_blank">Getting Started with Python for Apache Kafka</a></li><li>Course: <a href="https://developer.confluent.io/learn-kafka/kafka-python/intro/" rel="nofollow" target="_blank">Introduction to Apache Kafka for Python Developers</a></li><li>Step-by-step guide: <a href="https://developer.confluent.io/get-started/python/" rel="nofollow" target="_blank">Building a Python client application for Kafka</a></li><li><a href="https://developer.confluent.io/coding-in-motion-sound-and-vision-build-a-data-streaming-app-with-javascript/" rel="nofollow" target="_blank">Coding in Motion</a></li><li><a href="https://developer.confluent.io/podcast/building-and-designing-events-and-event-streams-with-apache-kafka/" rel="nofollow" target="_blank">Building and Designing Events and Event Streams with Apache Kafka</a></li><li><a href="https://youtu.be/GkpyIh5aWfU" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)</li></ul>

How to use Python with Apache Kafka

In this episode, Kris interviews Doron Porat, Director of Infrastructure at Yotpo, and Liran Yogev, Director of Engineering at ZipRecruiter (formerly at Yotpo), about their experiences and strategies in dealing with data modeling at scale.Yotpo has a vast and active data lake, comprising thousands of datasets that are processed by different engines, primarily Apache Spark™. They wanted to provide users with self-service tools for generating and utilizing data with maximum flexibility, but encountered difficulties, including poor standardization, low data reusability, limited data lineage, and unreliable datasets.The team realized that Yotpo's modeling layer, which defines the structure and relationships of the data, needed to be separated from the execution layer, which defines and processes operations on the data.This separation would give programmers better visibility into data pipelines across all execution engines, storage methods, and formats, as well as more governance control for exploration and automation.To address these issues, they developed YODA, an internal tool that combines excellent developer experience, DBT, Databricks, Airflow, Looker and more, with a strong CI/CD and orchestration layer.Yotpo is a B2B, SaaS e-commerce marketing platform that provides businesses with the necessary tools for accurate customer analytics, remarketing, support messaging, and more.ZipRecruiter is a job site that utilizes AI matching to help businesses find the right candidates for their open roles.EPISODE LINKSCurrent 2022 Talk: Next Gen Data Modeling in the Open Data PlatformData Mesh 101Data Mesh Architecture: A Modern Distributed Data ModelWatch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

In this episode, Kris interviews Doron Porat, Director of Infrastructure at Yotpo, and Liran Yogev, Director of Engineering at ZipRecruiter (formerly at Yotpo), about their experiences and strategies in dealing with data modeling at scale. Yotpo has a vast and active data lake, comprising thousands of datasets that are processed by different engines, primarily Apache Spark™. They wanted to provide users with self-service tools for generating and utilizing data with maximum flexibility, but encountered difficulties, including poor standardization, low data reusability, limited data lineage, and unreliable datasets. The team realized that Yotpo's modeling layer, which defines the structure and relationships of the data, needed to be separated from the execution layer, which defines and processes operations on the data. This separation would give programmers better visibility into data pipelines across all execution engines, storage methods, and formats, as well as more governance control for exploration and automation. To address these issues, they developed YODA, an internal tool that combines excellent developer experience, DBT, Databricks, Airflow, Looker and more, with a strong CI/CD and orchestration layer. Yotpo is a B2B, SaaS e-commerce marketing platform that provides businesses with the necessary tools for accurate customer analytics, remarketing, support messaging, and more. ZipRecruiter is a job site that utilizes AI matching to help businesses find the right candidates for their open roles. EPISODE LINKS<ul><li>Current 2022 Talk: <a href="https://www.confluent.io/events/current-2022/next-gen-data-modeling-in-the-open-data-platform/" rel="nofollow" target="_blank">Next Gen Data Modeling in the Open Data Platform</a></li><li><a href="https://developer.confluent.io/learn-kafka/data-mesh/intro/" rel="nofollow" target="_blank">Data Mesh 101</a></li><li><a href="https://developer.confluent.io/podcast/data-mesh-architecture-a-modern-distributed-data-model/" rel="nofollow" target="_blank">Data Mesh Architecture: A Modern Distributed Data Model</a></li><li><a href="https://youtu.be/xlO3mRKrM_I" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)</li></ul>

Next-Gen Data Modeling, Integrity, and Governance with YODA

Migrating Apache Kafka® clusters can be challenging, especially when moving large amounts of data while minimizing downtime. Michael Dunn (Solutions Architect, Confluent) has worked in the data space for many years, designing and managing systems to support high-volume applications. He has helped many organizations strategize, design, and implement successful Kafka cluster migrations between different environments. In this episode, Michael shares some tips about Kafka cluster migration with Kris, including the pros and cons of the different tools he recommends.Michael explains that there are many reasons why companies migrate their Kafka clusters. For example, they may want to modernize their platforms, move to a self-hosted cloud server, or consolidate clusters. He tells Kris that creating a plan and selecting the right tool before getting started is critical for reducing downtime and minimizing migration risks.The good news is that a few tools can facilitate moving large amounts of data, topics, schemas, applications, connectors, and everything else from one Apache Kafka cluster to another.Kafka MirrorMaker/MirrorMaker2 (MM2) is a stand-alone tool for copying data between two Kafka clusters. It uses source and sink connectors to replicate topics from a source cluster into the destination cluster.Confluent Replicator allows you to replicate data from one Kafka cluster to another. Replicator is similar to MM2, but the difference is that it’s been battle-tested.Cluster Linking is a powerful tool offered by Confluent that allows you to mirror topics from an Apache Kafka 2.4/Confluent Platform 5.4 source cluster to a Confluent Platform 7+ cluster in a read-only state, and is available as a fully-managed service in Confluent Cloud.At the end of the day, Michael stresses that coupled with a well-thought-out strategy and the right tool, Kafka cluster migration can be relatively painless. Following his advice, you should be able to keep your system healthy and stable before and after the migration is complete.EPISODE LINKSMirrorMaker 2ReplicatorCluster LinkingSchema MigrationMulti-Cluster Apache Kafka with Cluster LinkingWatch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

Migrating Apache Kafka® clusters can be challenging, especially when moving large amounts of data while minimizing downtime. Michael Dunn (Solutions Architect, Confluent) has worked in the data space for many years, designing and managing systems to support high-volume applications. He has helped many organizations strategize, design, and implement successful Kafka cluster migrations between different environments. In this episode, Michael shares some tips about Kafka cluster migration with Kris, including the pros and cons of the different tools he recommends. Michael explains that there are many reasons why companies migrate their Kafka clusters. For example, they may want to modernize their platforms, move to a self-hosted cloud server, or consolidate clusters. He tells Kris that creating a plan and selecting the right tool before getting started is critical for reducing downtime and minimizing migration risks. The good news is that a few tools can facilitate moving large amounts of data, topics, schemas, applications, connectors, and everything else from one Apache Kafka cluster to another. Kafka MirrorMaker/MirrorMaker2 (MM2) is a stand-alone tool for copying data between two Kafka clusters. It uses source and sink connectors to replicate topics from a source cluster into the destination cluster. Confluent Replicator allows you to replicate data from one Kafka cluster to another. Replicator is similar to MM2, but the difference is that it’s been battle-tested. Cluster Linking is a powerful tool offered by Confluent that allows you to mirror topics from an Apache Kafka 2.4/Confluent Platform 5.4 source cluster to a Confluent Platform 7+ cluster in a read-only state, and is available as a fully-managed service in Confluent Cloud. At the end of the day, Michael stresses that coupled with a well-thought-out strategy and the right tool, Kafka cluster migration can be relatively painless. Following his advice, you should be able to keep your system healthy and stable before and after the migration is complete. EPISODE LINKS<ul><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0" rel="nofollow" target="_blank">MirrorMaker 2</a></li><li><a href="https://docs.confluent.io/platform/current/multi-dc-deployments/replicator/index.html#replicator-overview" rel="nofollow" target="_blank">Replicator</a></li><li><a href="https://docs.confluent.io/platform/current/multi-dc-deployments/cluster-linking/index.html" rel="nofollow" target="_blank">Cluster Linking</a></li><li><a href="https://docs.confluent.io/platform/current/schema-registry/installation/migrate.html#migrate-schemas" rel="nofollow" target="_blank">Schema Migration</a></li><li><a href="https://developer.confluent.io/podcast/multi-cluster-apache-kafka-with-cluster-linking-ft-nikhil-bhatia/" rel="nofollow" target="_blank">Multi-Cluster Apache Kafka with Cluster Linking</a></li><li><a href="https://youtu.be/oqRiagSnYfQ" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)</li></ul>

Migrate Your Kafka Cluster with Minimal Downtime

dbt is known as being part of the Modern Data Stack for ELT processes. Being in the MDS, dbt Labs believes in having the best of breed for every part of the stack. Oftentimes folks are using an EL tool like Fivetran to pull data from the database into the warehouse, then using dbt to manage the transformations in the warehouse. Analysts can then build dashboards on top of that data, or execute tests.It’s possible for an analyst to adapt this process for use with a microservice application using Apache Kafka® and the same method to pull batch data out of each and every database; however, in this episode, Amy Chen (Partner Engineering Manager, dbt Labs) tells Kris about a better way forward for analysts willing to adopt the streaming mindset: Reusable pipelines using dbt models that immediately pull events into the warehouse and materialize as materialized views by default.dbt Labs is the company that makes and maintains dbt. dbt Core is the open-source data transformation framework that allows data teams to operate with software engineering’s best practices. dbt Cloud is the fastest and most reliable way to deploy dbt. Inside the world of event streaming, there is a push to expand data access beyond the programmers writing the code, and towards everyone involved in the business. Over at dbt Labs they’re attempting something of the reverse— to get data analysts to adopt the best practices of software engineers, and more recently, of streaming programmers. They’re improving the process of building data pipelines while empowering businesses to bring more contributors into the analytics process, with an easy to deploy, easy to maintain platform. It offers version control to analysts who traditionally don’t have access to git, along with the ability to easily automate testing, all in the same place.In this episode, Kris and Amy explore:How to revolutionize testing for analysts with two of dbt’s core functionalitiesWhat streaming in a batch-based analytics world should look likeWhat can be done to improve workflowsHow to democratize access to data for everyone in the businessEPISODE LINKSLearn more about dbt labsAn Analytics Engineer’s Guide to StreamingPanel discussion: If Streaming Is the Answer, Why Are We Still Doing Batch?All Current 2022 sessions and slidesWatch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)   

dbt is known as being part of the Modern Data Stack for ELT processes. Being in the MDS, dbt Labs believes in having the best of breed for every part of the stack. Oftentimes folks are using an EL tool like Fivetran to pull data from the database into the warehouse, then using dbt to manage the transformations in the warehouse. Analysts can then build dashboards on top of that data, or execute tests. It’s possible for an analyst to adapt this process for use with a microservice application using Apache Kafka® and the same method to pull batch data out of each and every database; however, in this episode, Amy Chen (Partner Engineering Manager, dbt Labs) tells Kris about a better way forward for analysts willing to adopt the streaming mindset: Reusable pipelines using dbt models that immediately pull events into the warehouse and materialize as materialized views by default.dbt Labs is the company that makes and maintains dbt. dbt Core is the open-source data transformation framework that allows data teams to operate with software engineering’s best practices. dbt Cloud is the fastest and most reliable way to deploy dbt.  Inside the world of event streaming, there is a push to expand data access beyond the programmers writing the code, and towards everyone involved in the business. Over at dbt Labs they’re attempting something of the reverse— to get data analysts to adopt the best practices of software engineers, and more recently, of streaming programmers. They’re improving the process of building data pipelines while empowering businesses to bring more contributors into the analytics process, with an easy to deploy, easy to maintain platform. It offers version control to analysts who traditionally don’t have access to git, along with the ability to easily automate testing, all in the same place. In this episode, Kris and Amy explore:<ul><li>How to revolutionize testing for analysts with two of dbt’s core functionalities</li><li>What streaming in a batch-based analytics world should look like</li><li>What can be done to improve workflows</li><li>How to democratize access to data for everyone in the business</li></ul>EPISODE LINKS<ul><li>Learn more about <a href="https://www.getdbt.com/" rel="nofollow" target="_blank">dbt labs</a></li><li><a href="https://www.confluent.io/events/current-2022/an-analytics-engineers-guide-to-streaming/" rel="nofollow" target="_blank">An Analytics Engineer’s Guide to Streaming</a></li><li>Panel discussion: <a href="https://developer.confluent.io/podcast/if-streaming-is-the-answer-why-are-we-still-doing-batch/" rel="nofollow" target="_blank">If Streaming Is the Answer, Why Are We Still Doing Batch?</a></li><li><a href="https://www.confluent.io/events/current-2022/" rel="nofollow" target="_blank">All Current 2022 sessions and slides</a></li><li><a href="https://youtu.be/jmFBCQFkX7o" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)   </li></ul>

Real-Time Data Transformation and Analytics with dbt Labs

What’s the next big thing in the future of streaming data? In this episode, Greg DeMichillie (VP of Product and Solutions Marketing, Confluent) talks to Kris about the future of stream processing in environments where the value of data lies in their ability to intercept and interpret data.Greg explains that organizations typically focus on the infrastructure containers themselves, and not on the thousands of data connections that form within. When they finally realize that they don't have a way to manage the complexity of these connections, a new problem arises: how do they approach managing such complexity? That’s where Confluent and Apache Kafka® come into play - they offer a consistent way to organize this seemingly endless web of data so they don't have to face the daunting task of figuring out how to connect their shopping portals or jump through hoops trying different ETL tools on various systems.As more companies seek ways to manage this data, they are asking some basic questions:How to do it?Do best practices exist?How can we get help?The next question for companies who have already adopted Kafka is a bit more complex: "What about my partners?” For example, companies with inventory management systems use supply chain systems to track product creation and shipping. As a result, they need to decide which emails to update, if they need to write custom REST APIs to sit in front of Kafka topics, etc. Advanced use cases like this raise additional questions about data governance, security, data policy, and PII, forcing companies to think differently about data.Greg predicts this is the next big frontier as more companies adopt Kafka internally. And because they will have to think less about where the data is stored and more about how data moves, they will have to solve problems to make managing all that data easier. If you're an enthusiast of real-time data streaming, Greg invites you to attend the Kafka Summit (London) in May and Current (Austin, TX) for a deeper dive into the world of Apache Kafka-related topics now and beyond.EPISODE LINKSWhat’s Ahead of the Future of Data Streaming?If Streaming Is the Answer, Why Are We Still Doing Batch?All Current 2022 sessions and slidesKafka Summit London 2023Current 2023Watch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)   

What’s the next big thing in the future of streaming data? In this episode, Greg DeMichillie (VP of Product and Solutions Marketing, Confluent) talks to Kris about the future of stream processing in environments where the value of data lies in their ability to intercept and interpret data. Greg explains that organizations typically focus on the infrastructure containers themselves, and not on the thousands of data connections that form within. When they finally realize that they don't have a way to manage the complexity of these connections, a new problem arises: how do they approach managing such complexity? That’s where Confluent and Apache Kafka® come into play - they offer a consistent way to organize this seemingly endless web of data so they don't have to face the daunting task of figuring out how to connect their shopping portals or jump through hoops trying different ETL tools on various systems. As more companies seek ways to manage this data, they are asking some basic questions:<ul><li>How to do it?</li><li>Do best practices exist?</li><li>How can we get help?</li></ul>The next question for companies who have already adopted Kafka is a bit more complex: "What about my partners?” For example, companies with inventory management systems use supply chain systems to track product creation and shipping. As a result, they need to decide which emails to update, if they need to write custom REST APIs to sit in front of Kafka topics, etc. Advanced use cases like this raise additional questions about data governance, security, data policy, and PII, forcing companies to think differently about data. Greg predicts this is the next big frontier as more companies adopt Kafka internally. And because they will have to think less about where the data is stored and more about how data moves, they will have to solve problems to make managing all that data easier. If you're an enthusiast of real-time data streaming, Greg invites you to attend the Kafka Summit (London) in May and Current (Austin, TX) for a deeper dive into the world of Apache Kafka-related topics now and beyond. EPISODE LINKS<ul><li><a href="https://devops.com/whats-ahead-for-the-future-of-data-streaming/" rel="nofollow" target="_blank">What’s Ahead of the Future of Data Streaming?</a></li><li><a href="https://developer.confluent.io/podcast/if-streaming-is-the-answer-why-are-we-still-doing-batch/" rel="nofollow" target="_blank">If Streaming Is the Answer, Why Are We Still Doing Batch?</a></li><li><a href="https://www.confluent.io/events/current-2022/" rel="nofollow" target="_blank">All Current 2022 sessions and slides</a></li><li><a href="https://www.kafka-summit.org/events/kafka-summit-london-2023/about" rel="nofollow" target="_blank">Kafka Summit London 2023</a></li><li><a href="https://www.confluent.io/events/current/" rel="nofollow" target="_blank">Current 2023</a></li><li><a href="https://youtu.be/63yN90CXnRI" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)   </li></ul>

What is the Future of Streaming Data?

What can online gaming teach us about making large-scale event management more collaborative in real-time? Ben Gamble (Developer Relations Manager, Aiven)  has come to the world of real-time event streaming from an usual source: the video games industry. And if you stop to think about it, modern online games are complex, distributed real-time data systems with decades of innovative techniques to teach us.In this episode, Ben talks with Kris about integrating gaming concepts with Apache Kafka®. Using Kafka’s state management stream processing, Ben has built systems that can handle real-time event processing at a massive scale, including interesting approaches to conflict resolution and collaboration.Building latency into a system is one way to mask data processing time. Ben says that you can efficiently hide latency issues and prioritize performance improvements by setting an initial target and then optimizing from there. If you measure before optimizing, you can add an extra layer to manage user expectations better. Tricks like adding a visual progress bar give the appearance of progress but actually hide latency and improve the overall user experience.To effectively handle challenging activities, like resolving conflicts and atomic edits, Ben suggests “slicing” (or nano batching) to break down tasks into small, related chunks. Slicing allows each task to be evaluated separately, thus producing timely outcomes that resolve potential background conflicts without the user knowing.Ben also explains how he uses pooling to make collaboration seamless. Pooling is a process that links open requests with potential matches. Similar to booking seats on an airplane, seats are assigned when requests are made. As these types of connections are handled through a Kafka event stream, the initial open requests are eventually fulfilled when seats become available.According to Ben, real-world tools that facilitate collaboration (such as Google Docs and Slack) work similarly. Just like multi-player gaming systems, multiple users can comment or chat in real-time and users perceive instant responses because of the techniques ported over from the gaming world.As Ben sees it, the proliferation of these types of concepts across disciplines will also benefit a more significant number of collaborative systems. Despite being long established for gamers, these patterns can be implemented in more business applications to improve the user experience significantly.EPISODE LINKSGoing Multiplayer With Kafka—Current 2022Building a Dependable Real-Time Betting App with Confluent Cloud and AblyEvent Streaming PatternsWatch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)   

What can online gaming teach us about making large-scale event management more collaborative in real-time? Ben Gamble (Developer Relations Manager, Aiven)  has come to the world of real-time event streaming from an usual source: the video games industry. And if you stop to think about it, modern online games are complex, distributed real-time data systems with decades of innovative techniques to teach us. In this episode, Ben talks with Kris about integrating gaming concepts with Apache Kafka®. Using Kafka’s state management stream processing, Ben has built systems that can handle real-time event processing at a massive scale, including interesting approaches to conflict resolution and collaboration. Building latency into a system is one way to mask data processing time. Ben says that you can efficiently hide latency issues and prioritize performance improvements by setting an initial target and then optimizing from there. If you measure before optimizing, you can add an extra layer to manage user expectations better. Tricks like adding a visual progress bar give the appearance of progress but actually hide latency and improve the overall user experience. To effectively handle challenging activities, like resolving conflicts and atomic edits, Ben suggests “slicing” (or nano batching) to break down tasks into small, related chunks. Slicing allows each task to be evaluated separately, thus producing timely outcomes that resolve potential background conflicts without the user knowing. Ben also explains how he uses pooling to make collaboration seamless. Pooling is a process that links open requests with potential matches. Similar to booking seats on an airplane, seats are assigned when requests are made. As these types of connections are handled through a Kafka event stream, the initial open requests are eventually fulfilled when seats become available. According to Ben, real-world tools that facilitate collaboration (such as Google Docs and Slack) work similarly. Just like multi-player gaming systems, multiple users can comment or chat in real-time and users perceive instant responses because of the techniques ported over from the gaming world. As Ben sees it, the proliferation of these types of concepts across disciplines will also benefit a more significant number of collaborative systems. Despite being long established for gamers, these patterns can be implemented in more business applications to improve the user experience significantly. EPISODE LINKS<ul><li><a href="https://www.confluent.io/events/current-2022/going-multiplayer-with-kafka/" rel="nofollow" target="_blank">Going Multiplayer With Kafka</a>—Current 2022</li><li><a href="https://www.confluent.io/blog/real-time-betting-platform-with-confluent-cloud-and-ably/" rel="nofollow" target="_blank">Building a Dependable Real-Time Betting App with Confluent Cloud and Ably</a></li><li><a href="https://developer.confluent.io/patterns/" rel="nofollow" target="_blank">Event Streaming Patterns</a></li><li><a href="https://youtu.be/-s9HQS1gIAk" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)   </li></ul>

What can Apache Kafka Developers learn from Online Gaming?

Apache Kafka® 3.4 is released! In this special episode, Danica Fine (Senior Developer Advocate, Confluent), shares highlights of the Apache Kafka 3.4 release. This release introduces new KIPs in Kafka Core, Kafka Streams, and Kafka Connect.In Kafka Core:KIP-792 expands the metadata each group member passes to the group leader in its JoinGroup subscription to include the highest stable generation that consumer was a part of. KIP-830 includes a new configuration setting that allows you to disable the JMX reporter for environments where it’s not being used. KIP-854 introduces changes to clean up producer IDs more efficiently, to avoid excess memory usage. It introduces a new timeout parameter that affects the expiry of producer IDs and updates the old parameter to only affect the expiry of transaction IDs.KIP-866 (early access) provides a bridge to migrate between existing Zookeeper clusters to new KRaft mode clusters, enabling the migration of existing metadata from Zookeeper to KRaft. KIP-876 adds a new property that defines the maximum amount of time that the server will wait to generate a snapshot; the default is 1 hour.KIP-881, an extension of KIP-392, makes it so that consumers can now be rack-aware when it comes to partition assignments and consumer rebalancing. In Kafka Streams:KIP-770 updates some Kafka Streams configs and metrics related to the record cache size.KIP-837 allows users to multicast result records to every partition of downstream sink topics and adds functionality for users to choose to drop result records without sending.And finally, for Kafka Connect:KIP-787 allows users to run MirrorMaker2 with custom implementations for the Kafka resource manager and makes it easier to integrate with your ecosystem.Tune in to learn more about the Apache Kafka 3.4 release!EPISODE LINKSSee release notes for Apache Kafka 3.4Read the blog to learn moreDownload Apache Kafka 3.4 and get startedWatch the video version of this podcastJoin the Community 

Apache Kafka® 3.4 is released! In this special episode, Danica Fine (Senior Developer Advocate, Confluent), shares highlights of the Apache Kafka 3.4 release. This release introduces new KIPs in Kafka Core, Kafka Streams, and Kafka Connect. In Kafka Core:<ul><li>KIP-792 expands the metadata each group member passes to the group leader in its JoinGroup subscription to include the highest stable generation that consumer was a part of. </li><li>KIP-830 includes a new configuration setting that allows you to disable the JMX reporter for environments where it’s not being used. </li><li>KIP-854 introduces changes to clean up producer IDs more efficiently, to avoid excess memory usage. It introduces a new timeout parameter that affects the expiry of producer IDs and updates the old parameter to only affect the expiry of transaction IDs.</li><li>KIP-866 (early access) provides a bridge to migrate between existing Zookeeper clusters to new KRaft mode clusters, enabling the migration of existing metadata from Zookeeper to KRaft. </li><li>KIP-876 adds a new property that defines the maximum amount of time that the server will wait to generate a snapshot; the default is 1 hour.</li><li>KIP-881, an extension of KIP-392, makes it so that consumers can now be rack-aware when it comes to partition assignments and consumer rebalancing. </li></ul>In Kafka Streams:<ul><li>KIP-770 updates some Kafka Streams configs and metrics related to the record cache size.</li><li>KIP-837 allows users to multicast result records to every partition of downstream sink topics and adds functionality for users to choose to drop result records without sending.</li></ul>And finally, for Kafka Connect:<ul><li>KIP-787 allows users to run MirrorMaker2 with custom implementations for the Kafka resource manager and makes it easier to integrate with your ecosystem.</li></ul>Tune in to learn more about the Apache Kafka 3.4 release! EPISODE LINKS<ul><li><a href="https://archive.apache.org/dist/kafka/3.4.0/RELEASE_NOTES.html" rel="nofollow" target="_blank">See release notes for Apache Kafka 3.4</a></li><li><a href="https://www.confluent.io/blog/apache-kafka-3-4-0-new-features-and-updates/" rel="nofollow" target="_blank">Read the blog to learn more</a></li><li><a href="https://kafka.apache.org/downloads" rel="nofollow" target="_blank">Download Apache Kafka 3.4</a> and <a href="https://kafka.apache.org/documentation/#quickstart" rel="nofollow" target="_blank">get started</a></li><li><a href="https://youtu.be/E9pqZ4QUeVI" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Community </a></li></ul>

Apache Kafka 3.4 - New Features & Improvements

Streaming Audio features all things Apache Kafka®, Confluent, real-time data, and the cloud. We cover frequently asked questions, best practices, and use cases from the Kafka community—from Kafka connectors and distributed systems, to data mesh, data integration, modern data architectures, and data mesh built with Confluent and cloud Kafka as a service. Join our hosts as they stream through a series of interviews, stories, and use cases with guests from the data streaming industry. Apache®️, Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

This podcast is a must. Excellent content, relevant content, and very professional.

if you use Apache Kafka...

Approachable and informative

Streaming Audio features all things Apache Kafka®, Confluent, real-time data, and the cloud. We cover frequently asked questions, best practices, and use cases from the Kafka community—from Kafka connectors and distributed systems, to data mesh, data integration, modern data architectures, and data mesh built with Confluent and cloud Kafka as a service. Join our hosts as they stream through a series of interviews, stories, and use cases with guests from the data streaming industry. Apache®️, Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.

Streaming Audio: Apache Kafka® & Real-Time Data

Ramesh Sringeri

In today’s episode of Streaming Audio, Tim Berglund sits down with Senior Applications Developer of Mobile Solutions Ramesh Sringeri to discuss Apache Kafka®—specifically two Kafka use cases that Children’s Healthcare of Atlanta is working on.First, they discuss achieving near-real-time streams of data to support meaningful intracranial pressure prediction and managing intracranial pressure (ICP) in a timely manner to help the care team achieve better outcomes with traumatic brain injuries.Children’s Healthcare of Atlanta is in the process of building machine learning models for predicting ICP values 30 and 60 minutes in the future. This will help the care team better prepare for handling potential adverse conditions, where elevated ICP values could lead to undesirable outcomes. The Children’s team is using Kafka, KSQL, and Kafka Streams programs to build a pipeline in which they can test their machine learning models.Ramesh also shares about the work they’re doing to mitigate alarm fatigue for care providers. According to him, the current generation of monitoring devices are not equipped to set up multiple alarm conditions, and sometimes a combination of measures need to cross thresholds to be of concern. Children’s is able to leverage stream processing and KSQL to set up multiple conditions, reducing the number of meaningless alarms conditions that might condition care providers to ignore them.One of the best parts of it all—with Kafka and KSQL, the Children’s team has been able to quickly build data processing pipelines and address business use cases without having to write a lot of code.EPISODE LINKSJoin the Confluent Community SlackFor more, you can check out ksqlDB, the successor to KSQL.

In today’s episode of Streaming Audio, Tim Berglund sits down with Senior Applications Developer of Mobile Solutions Ramesh Sringeri to discuss Apache Kafka®—specifically two Kafka use cases that Children’s Healthcare of Atlanta is working on. First, they discuss achieving near-real-time streams of data to support meaningful intracranial pressure prediction and managing intracranial pressure (ICP) in a timely manner to help the care team achieve better outcomes with traumatic brain injuries. Children’s Healthcare of Atlanta is in the process of building machine learning models for predicting ICP values 30 and 60 minutes in the future. This will help the care team better prepare for handling potential adverse conditions, where elevated ICP values could lead to undesirable outcomes. The Children’s team is using Kafka, KSQL, and Kafka Streams programs to build a pipeline in which they can test their machine learning models. Ramesh also shares about the work they’re doing to mitigate alarm fatigue for care providers. According to him, the current generation of monitoring devices are not equipped to set up multiple alarm conditions, and sometimes a combination of measures need to cross thresholds to be of concern. Children’s is able to leverage stream processing and KSQL to set up multiple conditions, reducing the number of meaningless alarms conditions that might condition care providers to ignore them. One of the best parts of it all—with Kafka and KSQL, the Children’s team has been able to quickly build data processing pipelines and address business use cases without having to write a lot of code. EPISODE LINKS<ul><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li></ul>For more, you can check out <a href="https://ksqldb.io" rel="nofollow" target="_blank">ksqlDB</a>, the successor to KSQL.

Helping Healthcare with Apache Kafka and KSQL ft. Ramesh Sringeri

Bill Bejeck

Tim Berglund interviews Bill Bejeck about the Kafka Streams API and his new book, Kafka Streams in Action. 

Tim Berglund interviews Bill Bejeck about the Kafka Streams API and his new book, <a href="https://www.manning.com/books/kafka-streams-in-action" rel="nofollow" target="_blank">Kafka Streams in Action</a>.

Kafka Streams in Action with Bill Bejeck

Magesh Nandakumar

Tim Berglund and Magesh Nandakumar (Software Engineer, Confluent) discuss why schemas matter for building systems on Apache Kafka®, and how Confluent Schema Registry helps with the problem. They talk about how Schema Registry works, how you can collaborate around schema change through `avsc` files, and what it means for this to be available in Confluent Cloud today.EPISODE LINKSSchema Registry 101Schema ManagementMigrate Schemas to Confluent CloudSchemas, Contracts, and CompatibilityFully managed Apache Kafka as a service! Try free.

Tim Berglund and Magesh Nandakumar (Software Engineer, Confluent) discuss why schemas matter for building systems on Apache Kafka®, and how Confluent Schema Registry helps with the problem. They talk about how Schema Registry works, how you can collaborate around schema change through `avsc` files, and what it means for this to be available in Confluent Cloud today. EPISODE LINKS<ul><li><a href="https://developer.confluent.io/learn-kafka/schema-registry/key-concepts/" rel="nofollow" target="_blank">Schema Registry 101</a></li><li><a href="https://docs.confluent.io/current/schema-registry/index.html" rel="nofollow" target="_blank">Schema Management</a></li><li><a href="https://docs.confluent.io/current/schema-registry/installation/migrate.html#" rel="nofollow" target="_blank">Migrate Schemas to Confluent Cloud</a></li><li><a href="https://www.confluent.io/blog/schemas-contracts-compatibility" rel="nofollow" target="_blank">Schemas, Contracts, and Compatibility</a></li><li><a href="https://cnfl.io/confluent-cloud-free50-podcast" rel="nofollow" target="_blank">Fully managed Apache Kafka as a service! Try free.</a> </li></ul>

Schema Registry Made Simple by Confluent Cloud ft. Magesh Nandakumar

Stewart Bryson

Migrating from traditional ETL tools to an event streaming platform is a process that Stewart Bryson (CEO and founder, Red Pill Analytics), is no stranger to. In this episode, he dispels misconceptions around what “streaming ETL” means, and explains why event streaming and event-driven architectures compel us to rethink old approaches:Not all data is corporate data anymoreNot all data is relational data anymoreThe cost of storing data is now negligibleSupporting modern, distributed event streaming platforms, and the shift of focus from on-premises to the cloud introduces new use cases that focus primarily on building new systems and rebuilding existing ones. From Kafka Connect and stack applications to the importance of tables, events, and logs, Stewart also discusses Gradle and how it’s being used at Red Pill Analytics. EPISODE LINKSDeploying Kafka Streams and KSQL with Gradle – Part 1: Overview and MotivationDeploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL ImplementationsDeploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams Join the Confluent Community SlackFully managed Apache Kafka as a service! Try free.

Migrating from traditional ETL tools to an event streaming platform is a process that Stewart Bryson (CEO and founder, Red Pill Analytics), is no stranger to. In this episode, he dispels misconceptions around what “streaming ETL” means, and explains why event streaming and event-driven architectures compel us to rethink old approaches:<ul><li>Not all data is corporate data anymore</li><li>Not all data is relational data anymore</li><li>The cost of storing data is now negligible</li></ul>Supporting modern, distributed event streaming platforms, and the shift of focus from on-premises to the cloud introduces new use cases that focus primarily on building new systems and rebuilding existing ones. From Kafka Connect and stack applications to the importance of tables, events, and logs, Stewart also discusses Gradle and how it’s being used at Red Pill Analytics. EPISODE LINKS<ul><li><a href="https://www.confluent.io/blog/deploying-kafka-streams-and-ksql-with-gradle-part-1-overview-and-motivation" rel="nofollow" target="_blank">Deploying Kafka Streams and KSQL with Gradle – Part 1: Overview and Motivation</a></li><li><a href="https://www.confluent.io/blog/deploying-kafka-streams-and-ksql-with-gradle-part-2-managing-ksql-implementations" rel="nofollow" target="_blank">Deploying Kafka Streams and KSQL with Gradle – Part 2: Managing KSQL Implementations</a></li><li><a href="https://www.confluent.io/blog/deploying-kafka-streams-and-ksql-with-gradle-part-3-ksql-user-defined-functions-and-kafka-streams" rel="nofollow" target="_blank">Deploying Kafka Streams and KSQL with Gradle – Part 3: KSQL User-Defined Functions and Kafka Streams</a> </li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://cnfl.io/confluent-cloud-free50-podcast" rel="nofollow" target="_blank">Fully managed Apache Kafka as a service! Try free.</a></li></ul>

ETL and Event Streaming Explained ft. Stewart Bryson

Ben Stopford

Microservices are pretty ubiquitous these days. Really “SOA done right,” they reimagine the services pattern in the context of the world we live in today, nearly two decades since the first big service-oriented systems hit production. But what have we learned in this time? There are plenty of war stories. System designers have explored different architectural patterns—REST, events and databases of all types. In this podcast, Tim Berglund and Ben Stopford explore the event-driven paradigm and how it relates to the microservice architectures we build today. Ben dives deep into coupling, evolution and challenges of our increasingly data-oriented culture. He also talks about the future, where data are events and events are data, and touches on real-time architectures that retain the decoupling properties needed to be pluggable, and to evolve. Powerful stuff.EPISODE LINKSDesigning Event-Driven Systems Building a Microservices Ecosystem with Kafka Streams and KSQL

Microservices are pretty ubiquitous these days. Really “SOA done right,” they reimagine the services pattern in the context of the world we live in today, nearly two decades since the first big service-oriented systems hit production. But what have we learned in this time? There are plenty of war stories. System designers have explored different architectural patterns—REST, events and databases of all types.  In this podcast, Tim Berglund and Ben Stopford explore the event-driven paradigm and how it relates to the microservice architectures we build today. Ben dives deep into coupling, evolution and challenges of our increasingly data-oriented culture. He also talks about the future, where data are events and events are data, and touches on real-time architectures that retain the decoupling properties needed to be pluggable, and to evolve. Powerful stuff. EPISODE LINKS<ul><li><a href="https://www.confluent.io/designing-event-driven-systems" rel="nofollow" target="_blank">Designing Event-Driven Systems</a> </li><li><a href="https://www.confluent.io/blog/building-a-microservices-ecosystem-with-kafka-streams-and-ksql/" rel="nofollow" target="_blank">Building a Microservices Ecosystem with Kafka Streams and KSQL</a></li></ul>

Ben Stopford on Microservices and Event Streaming

It’s a problem endemic to the tech world that we are always focused on what’s coming next, that we often forget to look at where we’ve been. Chris Riccomini, who was there at LinkedIn when Apache Kafka® was born, tells us how Kafka and the stream processing framework Samza came about, and also what he’s doing these days at WePay—building systems that use Kafka as a primary datastore.EPISODE LINKSWhen It Absolutely, Positively, Has to be There: Reliability Guarantees in KafkaSo, You Want to Build a Kafka Connector? Source Edition.Kafka is Your Escape Hatch

It’s a problem endemic to the tech world that we are always focused on what’s coming next, that we often forget to look at where we’ve been. Chris Riccomini, who was there at LinkedIn when Apache Kafka® was born, tells us how Kafka and the stream processing framework Samza came about, and also what he’s doing these days at WePay—building systems that use Kafka as a primary datastore. EPISODE LINKS<ul><li><a href="https://www.confluent.io/resources/kafka-summit-2016/absolutely-positively-reliability-guarantees-kafka/" rel="nofollow" target="_blank">When It Absolutely, Positively, Has to be There: Reliability Guarantees in Kafka</a></li><li><a href="https://riccomini.name/build-kafka-connector-source" rel="nofollow" target="_blank">So, You Want to Build a Kafka Connector? Source Edition.</a></li><li><a href="https://riccomini.name/kafka-escape-hatch" rel="nofollow" target="_blank">Kafka is Your Escape Hatch</a></li></ul>

Chris Riccomini on the History of Apache Kafka and Stream Processing

Dani Traphagen

Helping users be successful when it comes to using Apache Kafka® is a large part of Dani Traphagen’s role as a senior systems engineer at Confluent. Whether she’s advising companies on implementing parts of Kafka or rebuilding their systems entirely from the ground up, Dani is passionate about event-driven architecture and the way streaming data provides real-time insights on business activity. She explains the concept of a stream, topic, key, and stream-table duality, and how each of these pieces relate to one another. When it comes to data modeling, Dani covers importance business requirements, including the need for a domain model, practicing domain-driven design principles, and bounded context. She also discusses the attributes of data modeling: time, source, key, header, metadata, and payload, in addition to exploring the significance of data governance and lineage and performing joins.EPISODE LINKSConvert from table to stream and stream to table Distributed, Real-Time Joins and Aggregations on User Activity Events Using Kafka StreamsKSQL in Action: Real-Time Streaming ETL from Oracle Transactional DataKSQL in Action: Enriching CSV Events with Data from RDBMS into AWSJourney to Event Driven – Part 4: Four Pillars of Event Streaming MicroservicesJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.

Helping users be successful when it comes to using Apache Kafka® is a large part of Dani Traphagen’s role as a senior systems engineer at Confluent. Whether she’s advising companies on implementing parts of Kafka or rebuilding their systems entirely from the ground up, Dani is passionate about event-driven architecture and the way streaming data provides real-time insights on business activity. She explains the concept of a stream, topic, key, and stream-table duality, and how each of these pieces relate to one another. When it comes to data modeling, Dani covers importance business requirements, including the need for a domain model, practicing domain-driven design principles, and bounded context. She also discusses the attributes of data modeling: time, source, key, header, metadata, and payload, in addition to exploring the significance of data governance and lineage and performing joins.EPISODE LINKS<ul><li><a href="https://www.michael-noll.com/blog/2018/04/05/of-stream-and-tables-in-kafka-and-stream-processing-part1/" rel="nofollow" target="_blank">Convert from table to stream and stream to table</a> </li><li><a href="https://www.confluent.io/blog/distributed-real-time-joins-and-aggregations-on-user-activity-events-using-kafka-streams/" rel="nofollow" target="_blank">Distributed, Real-Time Joins and Aggregations on User Activity Events Using Kafka Streams</a></li><li><a href="https://www.confluent.io/blog/ksql-in-action-real-time-streaming-etl-from-oracle-transactional-data" rel="nofollow" target="_blank">KSQL in Action: Real-Time Streaming ETL from Oracle Transactional Data</a></li><li><a href="https://www.confluent.io/blog/ksql-in-action-enriching-csv-events-with-data-from-rdbms-into-AWS/" rel="nofollow" target="_blank">KSQL in Action: Enriching CSV Events with Data from RDBMS into AWS</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices" rel="nofollow" target="_blank">Journey to Event Driven – Part 4: Four Pillars of Event Streaming Microservices</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://cnfl.io/confluent-cloud-free50-podcast" rel="nofollow" target="_blank">Fully managed Apache Kafka as a service! Try free.</a></li></ul>

Data Modeling for Apache Kafka – Streams, Topics & More with Dani Traphagen

Rachel Pedreschi

As the head of global field engineering and community at Imply, Rachel Pedreschi is passionate about engaging both externally with customers and internally with departments all across the board, from sales to engineering. Rachel’s involvement in the open source community focuses primarily on Apache Druid, a real-time, high-performance datastore that provides fast, sub-second analytics and complements another powerful open source project as well: Apache Kafka®. Together, Kafka and Druid provide real-time event streaming and high-performance streaming analytics with powerful visualizations.EPISODE LINKSHow To Use Kafka and Druid to Tame Your Router DataETL and Event Streaming Explained ft. Stewart BrysonWho is Abraham Wald?How Not to Be Wrong: The Power of Mathematical ThinkingJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.

As the head of global field engineering and community at Imply, Rachel Pedreschi is passionate about engaging both externally with customers and internally with departments all across the board, from sales to engineering. Rachel’s involvement in the open source community focuses primarily on Apache Druid, a real-time, high-performance datastore that provides fast, sub-second analytics and complements another powerful open source project as well: Apache Kafka®. Together, Kafka and Druid provide real-time event streaming and high-performance streaming analytics with powerful visualizations. EPISODE LINKS<ul><li><a href="https://www.confluent.io/kafka-summit-lon19/how-to-use-kafka-and-druid-tame-router-data/?utm_source=buzzsprout&amp;utm_medium=podcast&amp;utm_campaign=ch.apache-kafka-and-apache-druid-the-perfect-pair_type.community_content.analytics" rel="nofollow" target="_blank">How To Use Kafka and Druid to Tame Your Router Data</a></li><li><a href="https://podcasts.apple.com/us/podcast/etl-and-event-streaming-explained-ft-stewart-bryson/id1401509765?i=1000456066183" rel="nofollow" target="_blank">ETL and Event Streaming Explained ft. Stewart Bryson</a></li><li><a href="https://en.wikipedia.org/wiki/Abraham_Wald" rel="nofollow" target="_blank">Who is Abraham Wald?</a></li><li><a href="https://www.amazon.com/How-Not-Be-Wrong-Mathematical/dp/0143127535/ref=asc_df_0143127535/?tag=hyprod-20&amp;linkCode=df0&amp;hvadid=366282405567&amp;hvpos=1o2&amp;hvnetw=g&amp;hvrand=4079597632967155652&amp;hvpone=&amp;hvptwo=&amp;hvqmt=&amp;hvdev=c&amp;hvdvcmdl=&amp;hvlocint=&amp;hvlocphy=9031926&amp;hvtargid=pla-433373525480&amp;psc=1&amp;tag=&amp;ref=&amp;adgrpid=78795691600&amp;hvpone=&amp;hvptwo=&amp;hvadid=366282405567&amp;hvpos=1o2&amp;hvnetw=g&amp;hvrand=4079597632967155652&amp;hvqmt=&amp;hvdev=c&amp;hvdvcmdl=&amp;hvlocint=&amp;hvlocphy=9031926&amp;hvtargid=pla-433373525480" rel="nofollow" target="_blank">How Not to Be Wrong: The Power of Mathematical Thinking</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree/" rel="nofollow" target="_blank">Fully managed Apache Kafka as a service! Try free.</a></li></ul>

Apache Kafka and Apache Druid – The Perfect Pair ft. Rachel Pedreschi

Guido Schmutz

One way to put Apache Kafka into action is through geofencing and tracking the location data of objects, barges, and cars in real time. Guido Schmutz (Principal Consultant, Trivadis) shares about one such use case involving a German steel company and the development project he worked on for them, which he featured in a talk at Berlin Buzzwords. EPISODE LINKSLocation Analytics – Real-Time Geofencing Using Kafka (Video) Location Analytics – Real-Time Geofencing Using Kafka (Slides) Join the Confluent Community SlackGet 30% off Kafka Summit London registration with the code KSL20Audio

One way to put Apache Kafka into action is through geofencing and tracking the location data of objects, barges, and cars in real time. Guido Schmutz (Principal Consultant, Trivadis) shares about one such use case involving a German steel company and the development project he worked on for them, which he featured in a talk at Berlin Buzzwords. EPISODE LINKS<ul><li><a href="https://www.youtube.com/watch?v=f7fXTzd7u_Y" rel="nofollow" target="_blank">Location Analytics – Real-Time Geofencing Using Kafka (Video)</a> </li><li><a href="https://www.slideshare.net/gschmutz/location-analytics-realtime-geofencing-using-apache-kafka-203115651" rel="nofollow" target="_blank">Location Analytics – Real-Time Geofencing Using Kafka (Slides)</a> </li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://kafka-summit.org/" rel="nofollow" target="_blank">Get 30% off Kafka Summit London registration with the code KSL20Audio</a></li></ul>

Location Data and Geofencing with Apache Kafka ft. Guido Schmutz

Jason Gustafson

Ever wonder what it’s like to be a distributed systems engineer at Confluent? Core Kafka Engineer Jason Gustafson dives into the challenges of working on distributed systems, particularly when it comes to a unique system like Apache Kafka®. He also discusses ways in which Confluent is working with the community to solve active problems and what it takes to be a distributed systems engineer.As always, Confluent is looking for engineers who are interested in distributed systems, and you don’t have to have 10 years of experience to do it!EPISODE LINKSKIP-392: Allow consumers to fetch from closest replicaKafka Improvement ProposalsHow to contributeHow Confluent's Engineering Team is Building the Infrastructure for Real-Time Event StreamingJoin us in Confluent Community Slack

Ever wonder what it’s like to be a distributed systems engineer at Confluent? Core Kafka Engineer Jason Gustafson dives into the challenges of working on distributed systems, particularly when it comes to a unique system like Apache Kafka®. He also discusses ways in which Confluent is working with the community to solve active problems and what it takes to be a distributed systems engineer. As always, Confluent is looking for engineers who are interested in distributed systems, and you don’t have to have 10 years of experience to do it! EPISODE LINKS<ul><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica" rel="nofollow" target="_blank">KIP-392: Allow consumers to fetch from closest replica</a></li><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals" rel="nofollow" target="_blank">Kafka Improvement Proposals</a></li><li><a href="https://kafka.apache.org/contributing" rel="nofollow" target="_blank">How to contribute</a></li><li><a href="https://medium.com/job-portraits/outsized-impact-how-confluents-engineering-team-is-building-the-infrastructure-for-real-time-f751c74cc500" rel="nofollow" target="_blank">How Confluent's Engineering Team is Building the Infrastructure for Real-Time Event Streaming</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join us in Confluent Community Slack</a></li></ul>

Distributed Systems Engineering with Apache Kafka ft. Jason Gustafson

Neil Avery

Neil Avery explores the intersection between FaaS and event streaming applications before taking a quick detour back in time to understand how we've gotten to this point in event-driven applications. He'll explain the pros and cons of FaaS, and cover how in its current state cold starts and latency concerns need to be part of the bigger picture when building streaming applications. Finally, Neil shares five rules that will help you understand how FaaS fits with the event streaming application.EPISODE LINKSJourney to Event Driven – Part 1: Why Event-First Thinking Changes EverythingJourney to Event Driven – Part 2: Programming Models for the Event-Driven ArchitectureJourney to Event Driven – Part 3: The Affinity Between Events, Streams and ServerlessJourney to Event Driven – Part 4: Four Pillars of Event Streaming Microservices

Neil Avery explores the intersection between FaaS and event streaming applications before taking a quick detour back in time to understand how we've gotten to this point in event-driven applications. He'll explain the pros and cons of FaaS, and cover how in its current state cold starts and latency concerns need to be part of the bigger picture when building streaming applications. Finally, Neil shares five rules that will help you understand how FaaS fits with the event streaming application. EPISODE LINKS<ul><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-1-why-event-first-thinking-changes-everything" rel="nofollow" target="_blank">Journey to Event Driven – Part 1: Why Event-First Thinking Changes Everything</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-2-programming-models-event-driven-architecture" rel="nofollow" target="_blank">Journey to Event Driven – Part 2: Programming Models for the Event-Driven Architecture</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-3-affinity-between-events-streams-serverless" rel="nofollow" target="_blank">Journey to Event Driven – Part 3: The Affinity Between Events, Streams and Serverless</a></li><li><a href="https://www.confluent.io/blog/journey-to-event-driven-part-4-four-pillars-of-event-streaming-microservices" rel="nofollow" target="_blank">Journey to Event Driven – Part 4: Four Pillars of Event Streaming Microservices</a></li></ul>

The Future of Serverless and Streaming with Neil Avery

Colin McCabe

Colin McCabe shares about what it’s like being a distributed systems engineer on the Core Kafka team at Confluent, where he has worked previously, and how that led to his interest in Apache Kafka®. As an active member of the Apache open source community, he describes that the community is a place that both welcomes newcomers and fosters different ideas that help make the product the best that it can be for everyone.Being a distributed systems engineer versus a full stack engineer comes with its own unique challenges. Colin offers some advice for those interested in working with Kafka and what the interview process is like at Confluent. It’s not all about what you know, but rather how you collaborate and contribute to the team, and how you get to the answer. Part of finding the answer is getting involved with Apache projects themselves by engaging with others and helping with bug fixes as much as possible, because it’ll help you gain a better grasp on a technology that is ever-changing.EPISODE LINKSKIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason GustafsonJoin the Confluent Community SlackFully managed Apache Kafka as a service! Try free.

Colin McCabe shares about what it’s like being a distributed systems engineer on the Core Kafka team at Confluent, where he has worked previously, and how that led to his interest in Apache Kafka®. As an active member of the Apache open source community, he describes that the community is a place that both welcomes newcomers and fosters different ideas that help make the product the best that it can be for everyone.Being a distributed systems engineer versus a full stack engineer comes with its own unique challenges. Colin offers some advice for those interested in working with Kafka and what the interview process is like at Confluent. It’s not all about what you know, but rather how you collaborate and contribute to the team, and how you get to the answer. Part of finding the answer is getting involved with Apache projects themselves by engaging with others and helping with bug fixes as much as possible, because it’ll help you gain a better grasp on a technology that is ever-changing.EPISODE LINKS<ul><li><a href="https://podcasts.apple.com/us/podcast/distributed-systems-engineering-apache-kafka-ft-jason/id1401509765?i=1000443415616" rel="nofollow" target="_blank">KIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason Gustafson</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://www.confluent.io/confluent-cloud/?utm_source=confluent&amp;utm_medium=podcast&amp;utm_campaign=free50" rel="nofollow" target="_blank">Fully managed Apache Kafka as a service! Try free.</a></li></ul>

Distributed Systems Engineering with Apache Kafka ft. Colin McCabe

Dylan Scott

Author Dylan Scott tells all about his upcoming Manning title Kafka in Action, which shares how Apache Kafka® can be used by beginners who are just starting out their own projects and dispels common Hadoop-related myths, as Kafka has grown to become a powerful event streaming platform beyond big data ecosystems alone. To get 40% off Manning products, use the following code: podcon19EPISODE LINKSJoin us in Confluent Community Slack

Author Dylan Scott tells all about his upcoming Manning title <a href="https://www.manning.com/books/kafka-in-action" rel="nofollow" target="_blank">Kafka in Action</a>, which shares how Apache Kafka® can be used by beginners who are just starting out their own projects and dispels common Hadoop-related myths, as Kafka has grown to become a powerful event streaming platform beyond big data ecosystems alone. To get 40% off Manning products, use the following code: podcon19 EPISODE LINKS<ul><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join us in Confluent Community Slack</a></li></ul>

Kafka in Action with Dylan Scott

Viktor Gamov

There’s something about YAML and the word “Docker” that doesn’t quite sit well with Viktor Gamov (Developer Advocate, Confluent). But Kafka Streams on Kubernetes is a phrase that does.Kubernetes is an open source platform that allows teams to deploy, manage, and automate containerized services and workloads. Running Kafka Streams on Kubernetes simplifies operations and gets your environment allocated faster.Viktor describes what that process looks like and how Jib helps build, test, and deploy Kafka Streams applications on Kubernetes for an improved DevOps experience. He also shares about some exciting projects he’s currently working on. EPISODE LINKSInstalling Apache Kafka® with Ansible ft. Viktor Gamov and Justin ManchesterContainerized Apache Kafka on KubernetesKubernetes 101 | Confluent Operator (1/3)Installation | Confluent Operator (2/3)Confluent Operator vs. Open Source Helm Charts (3/3)Streams Must Flow: Developing Fault-Tolerant Stream Processing Applications with Kafka Streams and KubernetesKafka TutorialsJoin the Confluent Community SlackLearn about Kafka at Confluent Developer

There’s something about YAML and the word “Docker” that doesn’t quite sit well with Viktor Gamov (Developer Advocate, Confluent). But Kafka Streams on Kubernetes is a phrase that does.Kubernetes is an open source platform that allows teams to deploy, manage, and automate containerized services and workloads. Running Kafka Streams on Kubernetes simplifies operations and gets your environment allocated faster.Viktor describes what that process looks like and how Jib helps build, test, and deploy Kafka Streams applications on Kubernetes for an improved DevOps experience. He also shares about some exciting projects he’s currently working on. EPISODE LINKS<ul><li><a href="https://confluent.buzzsprout.com/186154/2088030-installing-apache-kafka-with-ansible-ft-viktor-gamov-and-justin-manchester" rel="nofollow" target="_blank">Installing Apache Kafka® with Ansible ft. Viktor Gamov and Justin Manchester</a></li><li><a href="https://confluent.buzzsprout.com/186154/987419-containerized-kafka-on-kubernetes-with-viktor-gamov" rel="nofollow" target="_blank">Containerized Apache Kafka on Kubernetes</a></li><li><a href="https://www.youtube.com/watch?v=JiDiC5MI7hw" rel="nofollow" target="_blank">Kubernetes 101 | Confluent Operator (1/3)</a></li><li><a href="https://www.youtube.com/watch?v=9HaKP6HBz3s" rel="nofollow" target="_blank">Installation | Confluent Operator (2/3)</a></li><li><a href="https://www.youtube.com/watch?v=lzFuEuqOSNM" rel="nofollow" target="_blank">Confluent Operator vs. Open Source Helm Charts (3/3)</a></li><li><a href="https://speaking.gamov.io/Sorhje/streams-must-flow-developing-fault-tolerant-stream-processing-application-with-kafka-streams-and-kubernetes" rel="nofollow" target="_blank">Streams Must Flow: Developing Fault-Tolerant Stream Processing Applications with Kafka Streams and Kubernetes</a></li><li><a href="https://kafka-tutorials.confluent.io/?utm_source=buzzsprout&amp;utm_medium=podcast&amp;utm_campaign=ch.how-to-run-kafka-streams-applications-on-kubernetes_type.community_content.kubernetes" rel="nofollow" target="_blank">Kafka Tutorials</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://developer.confluent.io/?utm_source=buzzsprout&amp;utm_medium=podcast&amp;utm_campaign=ch.episode-links_type.community_content.developer" rel="nofollow" target="_blank">Learn about Kafka at Confluent Developer</a></li></ul>

How to Run Kafka Streams on Kubernetes ft. Viktor Gamov

Gunnar Morling

Friends don’t let friends do dual writes! Gunnar Morling (Software Engineer, Red Hat) joins us on the podcast to share a little bit about what Debezium is, how it works, and which databases it supports. In addition to covering the various use cases and benefits from change data capture (CDC) in the context of microservices—touching on the outbox pattern in particular, Gunnar walks us through the advantages of log-based CDC as implemented through Debezium over polling-based approaches, why you’d want to avoid dual writes to multiple resources, and engaging with members from the community to work collaboratively on Debezium.EPISODE LINKSJoin us in Confluent Community Slack

Friends don’t let friends do dual writes! Gunnar Morling (Software Engineer, Red Hat) joins us on the podcast to share a little bit about what Debezium is, how it works, and which databases it supports.  In addition to covering the various use cases and benefits from change data capture (CDC) in the context of microservices—touching on the outbox pattern in particular, Gunnar walks us through the advantages of log-based CDC as implemented through Debezium over polling-based approaches, why you’d want to avoid dual writes to multiple resources, and engaging with members from the community to work collaboratively on Debezium. EPISODE LINKS<ul><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join us in Confluent Community Slack</a></li></ul>

Change Data Capture with Debezium ft. Gunnar Morling

Hojjat Jafarpour

KSQL 5.0 now supports stream-stream, stream-table and table-table joins. Tim Berglund interviews Hojjat Jafarpour about all three join types, how they work, what their limitations are and the new kinds of operations they unlock.For more, you can check out ksqlDB, the successor to KSQL.

KSQL 5.0 now supports stream-stream, stream-table and table-table joins. Tim Berglund interviews Hojjat Jafarpour about all three join types, how they work, what their limitations are and the new kinds of operations they unlock. For more, you can check out <a href="https://ksqldb.io" rel="nofollow" target="_blank">ksqlDB</a>, the successor to KSQL.

Joins in KSQL 5.0 with Hojjat Jafarpour

Tim Berglund and Gwen Shapira have a discussion with Koelli Mungee (Customer Operations Lead, Confluent) and cover the latest Apache Kafka upgrades, cloud APIs, and data durability. This is episode #3 of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube, and elsewhere.EPISODE LINKSWatch the video version of this podcastFully managed Apache Kafka as a service! Try free.

Tim Berglund and Gwen Shapira have a discussion with Koelli Mungee (Customer Operations Lead, Confluent) and cover the latest Apache Kafka upgrades, cloud APIs, and data durability. This is episode #3 of "Ask Confluent," a segment in which Gwen Shapira and guests respond to a handful of questions and comments from Twitter, YouTube, and elsewhere. EPISODE LINKS<ul><li><a href="https://youtu.be/_mzRNryhhAo" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://cnfl.io/confluent-cloud-free50-podcast" rel="nofollow" target="_blank">Fully managed Apache Kafka as a service! Try free.</a></li></ul>

Ask Confluent #3: Kafka Upgrades, Cloud APIs and Data Durability

Lena Hall

Lena Hall joins Tim Berglund in the studio to talk about Apache Kafka®, the various ways to run Kafka on Microsoft Azure, Kafka on Kubernetes (K8s), and some exciting events that are happening in the Kafka world. Lena shares about serving double duty as both a senior software engineer and senior cloud developer advocate for Azure Engineering, including her unique roles and responsibilities, and how she balances engineering with advocacy. From writing tech articles to her experience with fuzzing and presence on YouTube, Lena is a strong community supporter and believes in the importance of staying rooted in the world of code as an advocate, because it helps you better understand common challenges and gives you insight as an engineer trying to fix them. It’s important to ask what's good about it and how it can be improved.They also discuss Kubernetes, the benefits of running Kafka on Kubernetes, why it’s popular, and using systems that can integrate with it. With Confluent Operator, it’s faster to spin up new environments, as well as easier to support a larger number of clusters in addition to scaling and configuration changes. As the Kafka ecosystem continues to grow and progress, one of the most notable updates of all is KIP-500 involving ZooKeeper. EPISODE LINKSApache Kafka on Kubernetes – Could You? Should You?Ask Confluent #1: Kubernetes, Confluent Operator, Kafka and KSQLFuzzing, Confluent Operator, StatefulSets, and Azure Kubernetes Service (AKS)Distributed Datastores on Kubernetes (on StatefulSets) at GOTO ChicagoRunning a Distributed Database on Kubernetes on AzureKIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum | PodcastProject Springfield, the Microsoft Project that Lena worked on in regard to KIP-500What is Apache Kafka in Azure HDInsightUse Azure Event Hubs from Apache Kafka applicationsApache Kafka based event streaming platform optimized for Azure StackConfluent Platform on Azure MarketplaceEventually Perfect Distributed Systems: Blog | O'Reilly Velocity KeynoteJoin the Confluent Community SlackFully managed Apache

Lena Hall joins Tim Berglund in the studio to talk about Apache Kafka®, the various ways to run Kafka on Microsoft Azure, Kafka on Kubernetes (K8s), and some exciting events that are happening in the Kafka world.  Lena shares about serving double duty as both a senior software engineer and senior cloud developer advocate for Azure Engineering, including her unique roles and responsibilities, and how she balances engineering with advocacy. From writing tech articles to her experience with fuzzing and presence on YouTube, Lena is a strong community supporter and believes in the importance of staying rooted in the world of code as an advocate, because it helps you better understand common challenges and gives you insight as an engineer trying to fix them. It’s important to ask what's good about it and how it can be improved.They also discuss Kubernetes, the benefits of running Kafka on Kubernetes, why it’s popular, and using systems that can integrate with it. With Confluent Operator, it’s faster to spin up new environments, as well as easier to support a larger number of clusters in addition to scaling and configuration changes. As the Kafka ecosystem continues to grow and progress, one of the most notable updates of all is KIP-500 involving ZooKeeper. EPISODE LINKS<ul><li><a href="https://www.confluent.io/blog/apache-kafka-kubernetes-could-you-should-you" rel="nofollow" target="_blank">Apache Kafka on Kubernetes – Could You? Should You?</a></li><li><a href="https://podcasts.apple.com/ke/podcast/ask-confluent-1-kubernetes-confluent-operator-kafka/id1401509765?i=1000414247823" rel="nofollow" target="_blank">Ask Confluent #1: Kubernetes, Confluent Operator, Kafka and KSQL</a></li><li><a href="https://en.wikipedia.org/wiki/Fuzzing" rel="nofollow" target="_blank">Fuzzing</a>, <a href="https://www.confluent.io/confluent-operator" rel="nofollow" target="_blank">Confluent Operator</a>, <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow" target="_blank">StatefulSets</a>, and <a href="https://azure.microsoft.com/en-us/services/kubernetes-service/" rel="nofollow" target="_blank">Azure Kubernetes Service (AKS)</a></li><li><a href="https://www.youtube.com/watch?v=SoaLsshJA8s&amp;list=PLKKXkUmO94-ZrLGgbXEfhSgBPL6APlrSs" rel="nofollow" target="_blank">Distributed Datastores on Kubernetes (on StatefulSets) at GOTO Chicago</a></li><li><a href="https://lenadroid.github.io/posts/stateful-sets-kubernetes-azure.html" rel="nofollow" target="_blank">Running a Distributed Database on Kubernetes on Azure</a></li><li><a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum" rel="nofollow" target="_blank">KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum</a> | <a href="https://podcasts.apple.com/us/podcast/kip-500-apache-kafka-without-zookeeper-ft-colin-mccabe/id1401509765?i=1000450193125" rel="nofollow" target="_blank">Podcast</a></li><li><a href="https://blogs.microsoft.com/ai/microsoft-previews-project-springfield-cloud-based-bug-detector/" rel="nofollow" target="_blank">Project Springfield, the Microsoft Project that Lena worked on in regard to KIP-500</a></li><li><a href="https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-introduction" rel="nofollow" target="_blank">What is Apache Kafka in Azure HDInsight</a></li><li><a href="https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview" rel="nofollow" target="_blank">Use Azure Event Hubs from Apache Kafka applications</a></li><li><a href="https://www.confluent.io/confluent-platform-azure-stack/" rel="nofollow" target="_blank">Apache Kafka based event streaming platform optimized for Azure Stack</a></li><li><a href="https://azuremarketplace.microsoft.com/en-en/marketplace/apps/confluentinc.confluent-enterprise?tab=Overview" rel="nofollow" target="_blank">Confluent Platform on Azure Marketplace</a></li><li>Eventually Perfect Distributed Systems: <a href="https://lenadroid.github.io/posts/OReillyVelocity/keynote.html" rel="nofollow" target="_blank">Blog</a> | <a href="https://www.youtube.com/watch?v=6rine64tmOM" rel="nofollow" target="_blank">O'Reilly Velocity Keynote</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://www.confluent.io/confluent-cloud/?utm_source=confluent&amp;utm_medium=podcast&amp;utm_campaign=free50" rel="nofollow" target="_blank">Fully managed Apache</a></li></ul>

Apache Kafka on Kubernetes, Microsoft Azure, and ZooKeeper with Lena Hall

Chris Mattmann

TensorFlow is an open source machine learning platform that can be used with Apache Kafka® for deep learning. Chris Mattmann, author of Machine Learning with TensorFlow, introduces us to TensorFlow as a Google technology that teaches computers how to think and make connections like humans do. For example, when there is a signifier that the mind processes, out comes a label to the object in front of you. TensorFlow is Google's version of wrangling various technologies to help group them together and work smoothly as large amounts of data flow through. Chris also breaks down neural networks, how technology simulates cerebral processes that take place when our visual cortex receives a new image, plus a use case that involves Apache Kafka and event streaming to achieve TensorFlow's goals.EPISODE LINKSAsk Confluent #13: Machine Learning with Kai Waehner Join the Confluent Community SlackGet 40% off Machine Learning with TensorFlow using the code podcon19

TensorFlow is an open source machine learning platform that can be used with Apache Kafka® for deep learning. Chris Mattmann, author of Machine Learning with TensorFlow, introduces us to TensorFlow as a Google technology that teaches computers how to think and make connections like humans do. For example, when there is a signifier that the mind processes, out comes a label to the object in front of you. TensorFlow is Google's version of wrangling various technologies to help group them together and work smoothly as large amounts of data flow through.  Chris also breaks down neural networks, how technology simulates cerebral processes that take place when our visual cortex receives a new image, plus a use case that involves Apache Kafka and event streaming to achieve TensorFlow's goals.EPISODE LINKS<ul><li><a href="https://confluent.buzzsprout.com/186154/1081406-ask-confluent-machine-learning-with-kai-waehner" rel="nofollow" target="_blank">Ask Confluent #13: Machine Learning with Kai Waehner</a> </li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li><li><a href="https://www.manning.com/books/machine-learning-with-tensorflow-second-edition?query=Machine%20Learning%20with%20TensorFlow" rel="nofollow" target="_blank">Get 40% off Machine Learning with TensorFlow using the code podcon19</a></li></ul>

Machine Learning with TensorFlow and Apache Kafka ft. Chris Mattmann

KSQL has a solid library of built-in functions, but no library is ever good enough. What if you want to write your own? We’ll learn how today with Hojjat Jafarpour, a software engineer on the KSQL team at Confluent.For more, you can check out ksqlDB, the successor to KSQL.

KSQL has a solid library of built-in functions, but no library is ever good enough. What if you want to write your own? We’ll learn how today with Hojjat Jafarpour, a software engineer on the KSQL team at Confluent. For more, you can check out <a href="https://ksqldb.io" rel="nofollow" target="_blank">ksqlDB</a>, the successor to KSQL.

UDFs and UDAFs in KSQL 5.0 with Hojjat Jafarpour

The best episodes for anyone looking to learn more about topics such as happiness, fulfillment, depression and psychology/mental health in general. With interviews with industry experts, personal stories of individuals, this topic has lessons applicable to everyone. Add episodes you're interested into your queue below!

Best Episodes on Mental Health, Emotions & Thoughts

The best podcasts on investing. For anyone interested in the stories in the investing industry, whether it's an interview with Ray Dalio or Ed Thorp to the writer of The Big Short. This topic includes the most fascinating stories from all the best investors. Add episodes you're interested into your queue below!