Matteo Merli's guest appearances

Matteo Merli

Data Engineering Podcast

Nikita Shamgunov

Christopher Ryan

Emily Miller

Peter Bull

Patrick Hunt

Jonathan Katz

Kevin Moore

Dan Robinson

Matthew Rocklin

Maxime Beauchemin

Jean Georges Perrin

Todd Walter

Mark Marinelli

Andy LoPresto

Joe Crobak

OwlTail

Rajan Dhabalia

Summary One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence. This week Rajan Dhabalia and Matteo Merli discuss the work they have done on Pulsar, which supports both options, in addition to being globally scalable and fast. They explain how Pulsar is architected, how to scale it, and how it fits into your existing infrastructure. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch. You can help support the show by checking out the Patreon page which is linked from the site. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers A few announcements:  There is still time to register for the O’Reilly Strata Conference in San Jose, CA March 5th-8th. Use the link dataengineeringpodcast.com/strata-san-jose to register and save 20% The O’Reilly AI Conference is also coming up. Happening April 29th to the 30th in New York it will give you a solid understanding of the latest breakthroughs and best practices in AI for business. Go to dataengineeringpodcast.com/aicon-new-york to register and save 20% If you work with data or want to learn more about how the projects you have heard about on the show get used in the real world then join me at the Open Data Science Conference in Boston from May 1st through the 4th. It has become one of the largest events for data scientists, data engineers, and data driven businesses to get together and learn how to be more effective. To save 60% off your tickets go to dataengineeringpodcast.com/odsc-east-2018 and register.   Your host is Tobias Macey and today I’m interviewing Rajan Dhabalia and Matteo Merli about Pulsar, a distributed open source pub-sub messaging system  Interview  Introduction How did you get involved in the area of data management? Can you start by explaining what Pulsar is and what the original inspiration for the project was? What have been some of the most challenging aspects of building and promoting Pulsar? For someone who wants to run Pulsar, what are the infrastructure and network requirements that they should be considering and what is involved in deploying the various components? What are the scaling factors for Pulsar and what aspects of deployment and administration should users pay special attention to? What projects or services do you consider to be competitors to Pulsar and what makes it stand out in comparison? The documentation mentions that there is an API layer that provides drop-in compatibility with Kafka. Does that extend to also supporting some of the plugins that have developed on top of Kafka? One of the popular aspects of Kafka is the persistence of the message log, so I’m curious how Pulsar manages long-term storage and reprocessing of messages that have already been acknowledged? When is Pulsar the wrong tool to use? What are some of the improvements or new features that you have planned for the future of Pulsar?  Contact Info  Matteo  merlimat on GitHub @merlimat on Twitter   Rajan  @dhabaliaraj on Twitter rhabalia on GitHub    Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Pulsar Publish-Subscribe Yahoo Streamlio ActiveMQ Kafka Bookkeeper SLA (Service Level Agreement) Write-Ahead Log Ansible Zookeeper Pulsar Deployment Instructions RabbitMQ Confluent Schema Registry  Podcast Interview   Kafka Connect Wallaroo  Podcast Interview   Kinesis Athenz  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>One of the critical components for modern data infrastructure is a scalable and reliable messaging system. Publish-subscribe systems have been popular for many years, and recently stream oriented systems such as Kafka have been rising in prominence. This week Rajan Dhabalia and Matteo Merli discuss the work they have done on Pulsar, which supports both options, in addition to being globally scalable and fast. They explain how Pulsar is architected, how to scale it, and how it fits into your existing infrastructure.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure</li> <li>When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.</li> <li>You can help support the show by checking out the <a href="https://www.patreon.com/dataengineeringpodcast?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Patreon page</a> which is linked from the site.</li> <li>To help other people find the show you can leave a review on <a href="https://itunes.apple.com/us/podcast/data-engineering-podcast/id1193040557?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">iTunes</a>, or <a href="https://goo.gl/app/playmusic?ibi=com.google.PlayMusic&amp;isi=691797987&amp;ius=googleplaymusic&amp;link=https://play.google.com/music/m/Ivucmwlkrxbgfe5dhj562xgev4u?t%3DData_Engineering_Podcast%26pcampaignid%3DMKT-na-all-co-pr-mu-pod-16&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Google Play Music</a>, and tell your friends and co-workers</li> <li>A few announcements: <ul> <li>There is still time to register for the O’Reilly Strata Conference in San Jose, CA March 5th-8th. Use the link <a href="https://www.dataengineeringpodcast.com/strata-san-jose?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/strata-san-jose</a> to register and save 20%</li> <li>The O’Reilly AI Conference is also coming up. Happening April 29th to the 30th in New York it will give you a solid understanding of the latest breakthroughs and best practices in AI for business. Go to <a href="https://www.dataengineeringpodcast.com/aicon-new-york?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/aicon-new-york</a> to register and save 20%</li> <li>If you work with data or want to learn more about how the projects you have heard about on the show get used in the real world then join me at the Open Data Science Conference in Boston from May 1st through the 4th. It has become one of the largest events for data scientists, data engineers, and data driven businesses to get together and learn how to be more effective. To save 60% off your tickets go to <a href="https://www.dataengineeringpodcast.com/odsc-east-2018?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/odsc-east-2018</a> and register.</li> </ul> <p></p></li> <li>Your host is Tobias Macey and today I’m interviewing Rajan Dhabalia and Matteo Merli about Pulsar, a distributed open source pub-sub messaging system</li> </ul><p></p> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by explaining what Pulsar is and what the original inspiration for the project was?</li> <li>What have been some of the most challenging aspects of building and promoting Pulsar?</li> <li>For someone who wants to run Pulsar, what are the infrastructure and network requirements that they should be considering and what is involved in deploying the various components?</li> <li>What are the scaling factors for Pulsar and what aspects of deployment and administration should users pay special attention to?</li> <li>What projects or services do you consider to be competitors to Pulsar and what makes it stand out in comparison?</li> <li>The documentation mentions that there is an API layer that provides drop-in compatibility with Kafka. Does that extend to also supporting some of the plugins that have developed on top of Kafka?</li> <li>One of the popular aspects of Kafka is the persistence of the message log, so I’m curious how Pulsar manages long-term storage and reprocessing of messages that have already been acknowledged?</li> <li>When is Pulsar the wrong tool to use?</li> <li>What are some of the improvements or new features that you have planned for the future of Pulsar?</li> </ul> <h2>Contact Info</h2> <ul> <li>Matteo <ul> <li><a href="https://github.com/merlimat?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">merlimat</a> on GitHub</li> <li><a href="https://twitter.com/merlimat?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@merlimat</a> on Twitter</li> </ul> <p></p></li> <li>Rajan<p></p> <ul> <li><a href="https://twitter.com/dhabaliaraj?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@dhabaliaraj</a> on Twitter</li> <li><a href="https://github.com/rdhabalia?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">rhabalia</a> on GitHub</li> </ul> <p></p></li> </ul><p></p> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://pulsar.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pulsar</a></li> <li><a href="https://en.wikipedia.org/wiki/Publish–subscribe_pattern?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Publish-Subscribe</a></li> <li><a href="https://www.yahoo.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Yahoo</a></li> <li><a href="https://streaml.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Streamlio</a></li> <li><a href="https://streaml.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ActiveMQ</a></li> <li><a href="https://kafka.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka</a></li> <li><a href="https://bookkeeper.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Bookkeeper</a></li> <li><a href="https://en.wikipedia.org/wiki/Service-level_agreement?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">SLA (Service Level Agreement)</a></li> <li><a href="https://en.wikipedia.org/wiki/Write-ahead_logging?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Write-Ahead Log</a></li> <li><a href="https://www.ansible.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Ansible</a></li> <li><a href="https://zookeeper.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Zookeeper</a></li> <li><a href="https://pulsar.apache.org/docs/v1.21.0-incubating/deployment/cluster/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pulsar Deployment Instructions</a></li> <li><a href="http://www.rabbitmq.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">RabbitMQ</a></li> <li><a href="https://github.com/confluentinc/schema-registry?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Confluent Schema Registry</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/confluent-schema-registry-with-ewen-cheslack-postava-episode-10/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="https://www.confluent.io/product/connectors/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka Connect</a></li> <li><a href="https://www.wallaroolabs.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Wallaroo</a><p></p> <ul> <li><a href="https://www.dataengineeringpodcast.com/wallaroo-with-sean-t-allen-episode-12/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="https://aws.amazon.com/kinesis/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kinesis</a></li> <li><a href="https://github.com/yahoo/athenz?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Athenz</a></li> </ul><p></p> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Pulsar: Fast And Scalable Messaging with Rajan Dhabalia and Matteo Merli - Episode 17