Yoni Iny's guest appearances

Yoni Iny

Data Engineering Podcast

Nikita Shamgunov

Christopher Ryan

Emily Miller

Peter Bull

Patrick Hunt

Jonathan Katz

Kevin Moore

Dan Robinson

Matthew Rocklin

Maxime Beauchemin

Jean Georges Perrin

Todd Walter

Mark Marinelli

Andy LoPresto

Joe Crobak

OwlTail

Summary A data lake can be a highly valuable resource, as long as it is well built and well managed. Unfortunately, that can be a complex and time-consuming effort, requiring specialized knowledge and diverting resources from your primary business. In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Yoni Iny about Upsolver, a data lake platform that lets developers integrate and analyze streaming data with ease  Interview  Introduction How did you get involved in the area of data management? Can you start by describing what Upsolver is and how it got started?  What are your goals for the platform?   There are a lot of opinions on both sides of the data lake argument. When is it the right choice for a data platform?  What are the shortcomings of a data lake architecture?   How is Upsolver architected?  How has that architecture changed over time? How do you manage schema validation for incoming data? What would you do differently if you were to start over today?   What are the biggest challenges at each of the major stages of the data lake? What is the workflow for a user of Upsolver and how does it compare to a self-managed data lake? When is Upsolver the wrong choice for an organization considering implementation of a data platform? Is there a particular scale or level of data maturity for an organization at which they would be better served by moving management of their data lake in house? What features or improvements do you have planned for the future of Upsolver?  Contact Info  Yoni  yoniiny on GitHub LinkedIn   Upsolver  Website @upsolver on Twitter LinkedIn Facebook    Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Upsolver Data Lake Israeli Army Data Warehouse Data Engineering Podcast Episode About Data Curation Three Vs Kafka Spark Presto Drill Spot Instances Object Storage Cassandra Redis Latency Avro Parquet ORC Data Engineering Podcast Episode About Data Serialization Formats SSTables Run Length Encoding CSV (Comma Separated Values) Protocol Buffers Kinesis ETL DevOps Prometheus Cloudwatch DataDog InfluxDB SQL Pandas Confluent KSQL  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>A data lake can be a highly valuable resource, as long as it is well built and well managed. Unfortunately, that can be a complex and time-consuming effort, requiring specialized knowledge and diverting resources from your primary business. In this episode Yoni Iny, CTO of Upsolver, discusses the various components that are necessary for a successful data lake project, how the Upsolver platform is architected, and how modern data lakes can benefit your organization.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> to get a $20 credit and launch a new server in under a minute.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Join the community in the new Zulip chat workspace at <a href="https://www.dataengineeringpodcast.com/chat?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/chat</a></li> <li>Your host is Tobias Macey and today I’m interviewing Yoni Iny about Upsolver, a data lake platform that lets developers integrate and analyze streaming data with ease</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by describing what Upsolver is and how it got started? <ul> <li>What are your goals for the platform?</li> </ul> <p></p></li> <li>There are a lot of opinions on both sides of the data lake argument. When is it the right choice for a data platform?<p></p> <ul> <li>What are the shortcomings of a data lake architecture?</li> </ul> <p></p></li> <li>How is Upsolver architected?<p></p> <ul> <li>How has that architecture changed over time?</li> <li>How do you manage schema validation for incoming data?</li> <li>What would you do differently if you were to start over today?</li> </ul> <p></p></li> <li>What are the biggest challenges at each of the major stages of the data lake?</li> <li>What is the workflow for a user of Upsolver and how does it compare to a self-managed data lake?</li> <li>When is Upsolver the wrong choice for an organization considering implementation of a data platform?</li> <li>Is there a particular scale or level of data maturity for an organization at which they would be better served by moving management of their data lake in house?</li> <li>What features or improvements do you have planned for the future of Upsolver?</li> </ul><p></p> <h2>Contact Info</h2> <ul> <li>Yoni <ul> <li><a href="https://github.com/yoniiny?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">yoniiny</a> on GitHub</li> <li><a href="https://www.linkedin.com/in/yoni-iny-7496173/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LinkedIn</a></li> </ul> <p></p></li> <li>Upsolver<p></p> <ul> <li><a href="https://www.upsolver.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Website</a></li> <li><a href="https://twitter.com/upsolver/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@upsolver</a> on Twitter</li> <li><a href="https://www.linkedin.com/company/upsolver/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LinkedIn</a></li> <li><a href="https://www.facebook.com/upsolver/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Facebook</a></li> </ul> <p></p></li> </ul><p></p> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.upsolver.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Upsolver</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_lake?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Lake</a></li> <li><a href="https://en.wikipedia.org/wiki/Israel_Defense_Forces?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Israeli Army</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_warehouse?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Warehouse</a></li> <li><a href="https://www.dataengineeringpodcast.com/a-primer-on-enterprise-data-curation-with-todd-walter-episode-49/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Data Engineering Podcast Episode About Data Curation</a></li> <li><a href="https://www.zdnet.com/article/volume-velocity-and-variety-understanding-the-three-vs-of-big-data/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Three Vs</a></li> <li><a href="https://kafka.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka</a></li> <li><a href="https://spark.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark</a></li> <li><a href="https://prestodb.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Presto</a></li> <li><a href="https://drill.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Drill</a></li> <li><a href="https://aws.amazon.com/ec2/spot/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spot Instances</a></li> <li><a href="https://en.wikipedia.org/wiki/Object_storage?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Object Storage</a></li> <li><a href="https://cassandra.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Cassandra</a></li> <li><a href="https://redis.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Redis</a></li> <li><a href="https://en.wikipedia.org/wiki/Latency_(engineering)?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Latency</a></li> <li><a href="https://avro.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Avro</a></li> <li><a href="https://parquet.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Parquet</a></li> <li><a href="https://orc.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ORC</a></li> <li><a href="https://www.dataengineeringpodcast.com/data-serialization-with-doug-cutting-and-julien-le-dem-episode-8/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Data Engineering Podcast Episode About Data Serialization Formats</a></li> <li><a href="http://distributeddatastore.blogspot.com/2013/08/cassandra-sstable-storage-format.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">SSTables</a></li> <li><a href="https://en.wikipedia.org/wiki/Run-length_encoding?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Run Length Encoding</a></li> <li><a href="https://en.wikipedia.org/wiki/Comma-separated_values?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CSV (Comma Separated Values)</a></li> <li><a href="https://en.wikipedia.org/wiki/Protocol_Buffers?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Protocol Buffers</a></li> <li><a href="https://aws.amazon.com/kinesis/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kinesis</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_transform,_load?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ETL</a></li> <li><a href="https://en.wikipedia.org/wiki/DevOps?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DevOps</a></li> <li><a href="https://prometheus.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Prometheus</a></li> <li><a href="https://aws.amazon.com/cloudwatch/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Cloudwatch</a></li> <li><a href="https://www.datadoghq.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DataDog</a></li> <li><a href="https://www.influxdata.com/time-series-platform/influxdb/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">InfluxDB</a></li> <li><a href="https://en.wikipedia.org/wiki/SQL?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">SQL</a></li> <li><a href="https://pandas.pydata.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pandas</a></li> <li><a href="https://www.confluent.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Confluent</a></li> <li><a href="https://www.confluent.io/product/ksql/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">KSQL</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

How Upsolver Is Building A Data Lake Platform In The Cloud with Yoni Iny - Episode 56