Ryan Crawcour's guest appearances

Ryan Crawcour

.NET Rocks!

SQL Down Under

Streaming Audio: Apache Kafka® & Real-Time Data

David Campbell

Jason Bell

Sam Ritchie

Jessica Moss

Kent Beck

Bob Ward

Cornelia Davis

Kevin Kline

Jim Gray

Anna McDonald

Kelsey Hightower

Lena Hall

Jay Smith

Matt Simpson

Colin McCabe

OwlTail

When building solutions for customers in Microsoft Azure, it is not uncommon to come across customers who are deeply entrenched in the Apache Kafka® ecosystem and want to continue expanding within it. Thus, figuring out how to connect Azure first-party services to this ecosystem is of the utmost importance.Ryan CrawCour is a Microsoft engineer who has been working on all things data and analytics for the past 10+ years, including building out services like Azure Cosmos DB, which is used by millions of people around the globe. More recently, Ryan has taken a customer-facing role where he gets to help customers build the best solutions possible using Microsoft Azure’s cloud platform and development tools. In one case, Ryan helped a customer leverage their existing Kafka investments and persist event messages in a durable managed database system in Azure. They chose Azure Cosmos DB, a fully managed, distributed, modern NoSQL database service as their preferred database, but the question remained as to how they would feed events from their Kafka infrastructure into Azure Cosmos DB, as well as how they could get changes from their database system back into their Kafka topics. Although integration is in his blood, Ryan confesses that he is relatively new to the world of Kafka and has learned to adjust to what he finds in his customers’ environments. Oftentimes this is Kafka, and for many good reasons, customers don’t want to change this core part of their solution infrastructure. This has led him to embrace Kafka and the ecosystem around it, enabling him to better serve customers. He’s been closely tracking the development and progress of Kafka Connect. To him, it is the natural step from Kafka as a messaging infrastructure to Kafka as a key pillar in an integration scenario. Kafka Connect can be thought of as a piece of middleware that can be used to connect a variety of systems to Kafka in a bidirectional manner. This means getting data from Kafka into your downstream systems, often databases, and also taking changes that occur in these systems and publishing them back to Kafka where other systems can then react. One day, a customer asked him how to connect Azure Cosmos DB to Kafka. There wasn’t a connector at the time, so he helped build two with the Confluent team: a sink connector, where data flows from Kafka topics into Azure Cosmos DB, as well as a source connector, where Azure Cosmos DB is the source of data pushing changes that occur in the database into Kafka topics.EPISODE LINKSIntegrating Azure and Confluent: Ingesting Data to Azure Cosmos DB through Apache Kafka Download the Azure Cosmos DB Connector (Source and Sink) Join the Confluent CommunityGitHub: Kafka Connect for Azure Cosmos DBWatch the video version of this podcastLearn more with Kafka tutorials, resources, and guides at Confluent DeveloperLive demo: Kafka streaming in 10 minutes on Confluent CloudUse 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)

When building solutions for customers in Microsoft Azure, it is not uncommon to come across customers who are deeply entrenched in the Apache Kafka® ecosystem and want to continue expanding within it. Thus, figuring out how to connect Azure first-party services to this ecosystem is of the utmost importance.Ryan CrawCour is a Microsoft engineer who has been working on all things data and analytics for the past 10+ years, including building out services like Azure Cosmos DB, which is used by millions of people around the globe. More recently, Ryan has taken a customer-facing role where he gets to help customers build the best solutions possible using Microsoft Azure’s cloud platform and development tools. In one case, Ryan helped a customer leverage their existing Kafka investments and persist event messages in a durable managed database system in Azure. They chose Azure Cosmos DB, a fully managed, distributed, modern NoSQL database service as their preferred database, but the question remained as to how they would feed events from their Kafka infrastructure into Azure Cosmos DB, as well as how they could get changes from their database system back into their Kafka topics. Although integration is in his blood, Ryan confesses that he is relatively new to the world of Kafka and has learned to adjust to what he finds in his customers’ environments. Oftentimes this is Kafka, and for many good reasons, customers don’t want to change this core part of their solution infrastructure. This has led him to embrace Kafka and the ecosystem around it, enabling him to better serve customers. He’s been closely tracking the development and progress of Kafka Connect. To him, it is the natural step from Kafka as a messaging infrastructure to Kafka as a key pillar in an integration scenario. Kafka Connect can be thought of as a piece of middleware that can be used to connect a variety of systems to Kafka in a bidirectional manner. This means getting data from Kafka into your downstream systems, often databases, and also taking changes that occur in these systems and publishing them back to Kafka where other systems can then react. One day, a customer asked him how to connect Azure Cosmos DB to Kafka. There wasn’t a connector at the time, so he helped build two with the Confluent team: a sink connector, where data flows from Kafka topics into Azure Cosmos DB, as well as a source connector, where Azure Cosmos DB is the source of data pushing changes that occur in the database into Kafka topics.EPISODE LINKS<ul><li><a href="https://www.confluent.io/blog/integrate-azure-and-confluent-to-stream-data-into-cosmos-db/" rel="nofollow" target="_blank">Integrating Azure and Confluent: Ingesting Data to Azure Cosmos DB through Apache Kafka</a> </li><li><a href="https://www.confluent.io/hub/microsoftcorporation/kafka-connect-cosmos" rel="nofollow" target="_blank">Download the Azure Cosmos DB Connector (Source and Sink)</a> </li><li><a href="https://www.confluent.io/community/ask-the-community/" rel="nofollow" target="_blank">Join the Confluent Community</a></li><li><a href="https://github.com/microsoft/kafka-connect-cosmosdb" rel="nofollow" target="_blank">GitHub: Kafka Connect for Azure Cosmos DB</a></li><li><a href="https://www.youtube.com/watch?v=aL0SCCre_0w" rel="nofollow" target="_blank">Watch the video version of this podcast</a></li><li><a href="https://developer.confluent.io/" rel="nofollow" target="_blank">Learn more with Kafka tutorials, resources, and guides at Confluent Developer</a></li><li><a href="https://www.confluent.io/online-talks/confluent-cloud-demo/" rel="nofollow" target="_blank">Live demo: Kafka streaming in 10 minutes on Confluent Cloud</a></li><li><a href="https://www.confluent.io/confluent-cloud/tryfree" rel="nofollow" target="_blank">Use 60PDCAST to get an additional $60 of free Confluent Cloud usage</a> (<a href="https://www.confluent.io/confluent-cloud-promo-disclaimer/" rel="nofollow" target="_blank">details</a>)</li></ul>

Connecting Azure Cosmos DB with Apache Kafka - Better Together ft. Ryan CrawCour

SDU Show 64 features Microsoft Azure DocumentDB team member Ryan Crawcour discussing what SQL Server DBAs and developers need to know about DocumentDB

SDU Show 64 with guest Ryan Crawcour