Jun Rao's guest appearances

Jun Rao

Software Engineering Radio - the podcast for professional software developers

Streaming Audio: Apache Kafka® & Real-Time Data

The Data Wranglers

Jason Bell

Sam Ritchie

Kent Beck

Cornelia Davis

Anna McDonald

Kelsey Hightower

Lena Hall

Jay Smith

Matt Simpson

Colin McCabe

Sam Newman

Josh Long

Gwen Shapira

Paul Rayner

Dylan Scott

OwlTail

You already know Apache Kafka® is a distributed event streaming system for setting your data in motion, but how does its internal architecture work? No one can explain Kafka’s internal architecture better than Jun Rao, one of its original creators and Co-Founder of Confluent. Jun has an in-depth understanding of Kafka that few others can claim—and he shares that with us in this episode, and in his new Kafka Internals course on Confluent Developer. One of Jun's goals in publishing the Kafka Internals course was to cover the evolution of Kafka since its initial launch. In line with that goal, he discusses the history of Kafka development, including the original thinking behind some of its design decisions, as well as how its features have been improved to better meet its key goals of durability, scalability, and real-time data. With respect to its initial design, Jun relates how Kafka was conceived from the ground up as a distributed system, with compute and storage always maintained as separate entities, so that they could scale independently. Additionally, he shares that Kafka was deliberately made for high throughput since many of the popular messaging systems at the time of its invention were single node, but his team needed to process large volumes of non-transactional data, such as application metrics, various logs, click streams, and IoT information.As regards the evolution of its features, in addition to others, Jun explains these two topics at great length:Consumer rebalancing protocol: The original "stop the world" approach to Kafka's consumer rebalancing—although revolutionary at the time of its launch, was eventually improved upon to take a more incremental approach.Cluster metadata: Moving from the external ZooKeeper to the built-in KRaft protocol allows for better scaling by a factor of ten. according to Jun, and it also means you only need to worry about running a single binary.The Kafka Internals course consists of eleven concise modules, each dense with detail—covering Kafka fundamentals in technical depth. The course also pairs with four hands-on exercise modules led by Senior Developer Advocate Danica Fine. EPISODE LINKSKafka Internals courseHow Apache Kafka Works: An Introduction to Kafka’s InternalsCoding in Motion Workshop: Build a Streaming AppWatch the video version of this podcastKris Jenkins’ TwitterStreaming Audio Playlist Join the Confluent CommunityLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Intro to Event-Driven Microservices with ConfluentUse PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)   

You already know Apache Kafka® is a distributed event streaming system for setting your data in motion, but how does its internal architecture work? No one can explain Kafka’s internal architecture better than Jun Rao, one of its original creators and Co-Founder of Confluent. Jun has an in-depth understanding of Kafka that few others can claim—and he shares that with us in this episode, and in his new Kafka Internals course on Confluent Developer. One of Jun's goals in publishing the Kafka Internals course was to cover the evolution of Kafka since its initial launch. In line with that goal, he discusses the history of Kafka development, including the original thinking behind some of its design decisions, as well as how its features have been improved to better meet its key goals of durability, scalability, and real-time data. With respect to its initial design, Jun relates how Kafka was conceived from the ground up as a distributed system, with compute and storage always maintained as separate entities, so that they could scale independently. Additionally, he shares that Kafka was deliberately made for high throughput since many of the popular messaging systems at the time of its invention were single node, but his team needed to process large volumes of non-transactional data, such as application metrics, various logs, click streams, and IoT information.As regards the evolution of its features, in addition to others, Jun explains these two topics at great length:<ul><li>Consumer rebalancing protocol: The original "stop the world" approach to Kafka's consumer rebalancing—although revolutionary at the time of its launch, was eventually improved upon to take a more incremental approach.</li><li>Cluster metadata: Moving from the external ZooKeeper to the built-in KRaft protocol allows for better scaling by a factor of ten. according to Jun, and it also means you only need to worry about running a single binary.</li></ul>The Kafka Internals course consists of eleven concise modules, each dense with detail—covering Kafka fundamentals in technical depth. The course also pairs with four hands-on exercise modules led by Senior Developer Advocate Danica Fine. EPISODE LINKS<ul><li><a href="https://developer.confluent.io/learn-kafka/architecture/get-started/" rel="nofollow" target="_blank">Kafka Internals course</a></li><li><a href="https://www.confluent.io/blog/apache-kafka-architecture-and-internals-by-jun-rao/" rel="nofollow" target="_blank">How Apache Kafka Works: An Introduction to Kafka’s Internals</a></li><li><a href="https://www.youtube.com/watch?v=P-9svLRyaSw" rel="nofollow" target="_blank">Coding in Motion Workshop: Build a Streaming App</a></li><li><a href="https://www.youtube.com/watch?v=hB_bBLsnJKo&amp;list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE&amp;index=1" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://twitter.com/krisajenkins" rel="nofollow" target="_blank">Kris Jenkins’ Twitter</a></li><li><a href="https://www.youtube.com/playlist?list=PLa7VYi0yPIH1B0i7mhzVi78TIkKSd-0vE" rel="nofollow" target="_blank">Streaming Audio Playlist </a></li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/resources/online-talk/demo-intro-to-event-driven-microservices-with-confluent/" rel="nofollow" target="_blank">Live demo: Intro to Event-Driven Microservices with Confluent</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use PODCAST100 to get an additional $100 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)   </li></ul>

Optimizing Apache Kafka's Internals with Its Co-Creator Jun Rao

Jun Rao (Co-founder, Confluent) explains what relational databases and distributed databases are, how they work, and major differences between the two. He also delves into important lessons he’s learned along the way through the transition from the relational world to the distributed world. To be successful at a place like Confluent, he outlines three fundamental traits that a distributed systems engineer must possess, emphasizing the importance of curiosity and knowledge, care in code development, and being open-minded and collaborative. You may even find that sometimes, the people with the best answers to your problems aren't even at your company! Originally from China, Jun moved to the U.S. for his Ph.D. and eventually landed in IBM research labs. He worked there for over 10 years before moving to LinkedIn, where Apache Kafka® was initially being developed and implemented. EPISODE LINKSGet 30% off Kafka Summit London registration with the code KSL20AudioJoin the Confluent Community Slack

Jun Rao (Co-founder, Confluent) explains what relational databases and distributed databases are, how they work, and major differences between the two. He also delves into important lessons he’s learned along the way through the transition from the relational world to the distributed world. To be successful at a place like Confluent, he outlines three fundamental traits that a distributed systems engineer must possess, emphasizing the importance of curiosity and knowledge, care in code development, and being open-minded and collaborative. You may even find that sometimes, the people with the best answers to your problems aren't even at your company! Originally from China, Jun moved to the U.S. for his Ph.D. and eventually landed in IBM research labs. He worked there for over 10 years before moving to LinkedIn, where Apache Kafka® was initially being developed and implemented. EPISODE LINKS<ul><li><a href="https://kafka-summit.org/" rel="nofollow" target="_blank">Get 30% off Kafka Summit London registration with the code KSL20Audio</a></li><li><a href="https://cnfl.io/confluent-community-slack" rel="nofollow" target="_blank">Join the Confluent Community Slack</a></li></ul>

Distributed Systems Engineering with Apache Kafka ft. Jun Rao