Jean Georges Perrin's guest appearances

Jean Georges Perrin

Data Engineering Podcast

Chinchilla Squeaks

Nikita Shamgunov

Christopher Ryan

Julien Lemoine

Emily Miller

Peter Bull

Matt Biilmann

Cornelia Davis

Patrick Hunt

Martin Wimpress

Jonathan Katz

Robert Reeves

Kolton Andrus

Daniel Foster

Kevin Moore

Dan Robinson

OwlTail

In this episode I speak with Jean Georges Perrin about his forthcoming book "Spark in Action", where we cover Apache Spark, writing books, and why many are wrong about IBM. I also cover Twine, Open Source and Alexa and the UK's health service.https://www.manning.com/books/spark-in-action-second-editionDiscount code for any Manning books for you all: podchinc19And 5 free eBook codes for Jean's book if it caught your interest! spchc-3424 spchc-552D spchc-445B spchc-3087 spchc-F63D They are already activated and will last for two months. --- Send in a voice message: https://anchor.fm/chinchillasqueaks/message This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit chinchillasqueaks.substack.com

<p>In this episode I speak with Jean Georges Perrin about his forthcoming book "Spark in Action", where we cover Apache Spark, writing books, and why many are wrong about IBM. I also cover Twine, Open Source and Alexa and the UK's health service.</p><p><a href="https://www.manning.com/books/spark-in-action-second-edition" rel="nofollow" target="_blank">https://www.manning.com/books/spark-in-action-second-edition</a></p><p>Discount code for any Manning books for you all: <strong>podchinc19</strong></p><p>And 5 free eBook codes for Jean's book if it caught your interest!</p><ul><br/> <li>spchc-3424</li> <li>spchc-552D</li> <li>spchc-445B</li> <li>spchc-3087</li> <li>spchc-F63D</li></ul><p> They are already activated and will last for two months. </p>--- <br/><br/>Send in a voice message: <a href="https://anchor.fm/chinchillasqueaks/message" rel="nofollow" target="_blank">https://anchor.fm/chinchillasqueaks/message</a> <br/><br/>This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://chinchillasqueaks.substack.com?utm_medium=podcast&amp;utm_campaign=CTA_1" rel="nofollow" target="_blank">chinchillasqueaks.substack.com</a>

Apache Spark with Jean Georges Perrin, Twine, Alexa and the NHS

Summary Apache Spark is a popular and widely used tool for a variety of data oriented projects. With the large array of capabilities, and the complexity of the underlying system, it can be difficult to understand how to get started using it. Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. In this episode he helps to make sense of what Spark is, how it works, and the various ways that you can use it. He also discusses what you need to know to get it deployed and keep it running in a production environment and how it fits into the overall data ecosystem. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Jean Georges Perrin, author of the upcoming Manning book Spark In Action 2nd Edition, about the ways that Spark is used and how it fits into the data landscape  Interview  Introduction How did you get involved in the area of data management? Can you start by explaining what Spark is?  What are some of the main use cases for Spark? What are some of the problems that Spark is uniquely suited to address? Who uses Spark?   What are the tools offered to Spark users? How does it compare to some of the other streaming frameworks such as Flink, Kafka, or Storm? For someone building on top of Spark what are the main software design paradigms?  How does the design of an application change as you go from a local development environment to a production cluster?   Once your application is written, what is involved in deploying it to a production environment? What are some of the most useful strategies that you have seen for improving the efficiency and performance of a processing pipeline? What are some of the edge cases and architectural considerations that engineers should be considering as they begin to scale their deployments? What are some of the common ways that Spark is deployed, in terms of the cluster topology and the supporting technologies? What are the limitations of the Spark programming model?  What are the cases where Spark is the wrong choice?   What was your motivation for writing a book about Spark?  Who is the target audience?   What have been some of the most interesting or useful lessons that you have learned in the process of writing a book about Spark? What advice do you have for anyone who is considering or currently using Spark?  Contact Info  @jgperrin on Twitter Blog  Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Book Discount  Use the code poddataeng18 to get 40% off of all of Manning’s products at manning.com  Links  Apache Spark Spark In Action Book code examples in GitHub Informix International Informix Users Group MySQL Microsoft SQL Server ETL (Extract, Transform, Load) Spark SQL and Spark In Action‘s chapter 11 Spark ML and Spark In Action‘s chapter 18 Spark Streaming (structured) and Spark In Action‘s chapter 10 Spark GraphX Hadoop Jupyter  Podcast Interview   Zeppelin Databricks IBM Watson Studio Kafka Flink  Podcast Episode   AWS Kinesis Yarn HDFS Hive Scala PySpark DAG Spark Catalyst Spark Tungsten Spark UDF AWS EMR Mesos DC/OS Kubernetes Dataframes  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>Apache Spark is a popular and widely used tool for a variety of data oriented projects. With the large array of capabilities, and the complexity of the underlying system, it can be difficult to understand how to get started using it. Jean George Perrin has been so impressed by the versatility of Spark that he is writing a book for data engineers to hit the ground running. In this episode he helps to make sense of what Spark is, how it works, and the various ways that you can use it. He also discusses what you need to know to get it deployed and keep it running in a production environment and how it fits into the overall data ecosystem.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> today to get a $20 credit and launch a new server in under a minute.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Join the community in the new Zulip chat workspace at <a href="https://www.dataengineeringpodcast.com/chat?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/chat</a></li> <li>Your host is Tobias Macey and today I’m interviewing Jean Georges Perrin, author of the upcoming Manning book Spark In Action 2nd Edition, about the ways that Spark is used and how it fits into the data landscape</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by explaining what Spark is? <ul> <li>What are some of the main use cases for Spark?</li> <li>What are some of the problems that Spark is uniquely suited to address?</li> <li>Who uses Spark?</li> </ul> <p></p></li> <li>What are the tools offered to Spark users?</li> <li>How does it compare to some of the other streaming frameworks such as Flink, Kafka, or Storm?</li> <li>For someone building on top of Spark what are the main software design paradigms?<p></p> <ul> <li>How does the design of an application change as you go from a local development environment to a production cluster?</li> </ul> <p></p></li> <li>Once your application is written, what is involved in deploying it to a production environment?</li> <li>What are some of the most useful strategies that you have seen for improving the efficiency and performance of a processing pipeline?</li> <li>What are some of the edge cases and architectural considerations that engineers should be considering as they begin to scale their deployments?</li> <li>What are some of the common ways that Spark is deployed, in terms of the cluster topology and the supporting technologies?</li> <li>What are the limitations of the Spark programming model?<p></p> <ul> <li>What are the cases where Spark is the wrong choice?</li> </ul> <p></p></li> <li>What was your motivation for writing a book about Spark?<p></p> <ul> <li>Who is the target audience?</li> </ul> <p></p></li> <li>What have been some of the most interesting or useful lessons that you have learned in the process of writing a book about Spark?</li> <li>What advice do you have for anyone who is considering or currently using Spark?</li> </ul><p></p> <h2>Contact Info</h2> <ul> <li><a href="https://twitter.com/jgperrin?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@jgperrin</a> on Twitter</li> <li><a href="http://jgp.net?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Blog</a></li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Book Discount</h2> <ul> <li>Use the code poddataeng18 to get 40% off of all of Manning’s products at <a href="https://www.manning.com?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">manning.com</a></li> </ul> <h2>Links</h2> <ul> <li><a href="https://spark.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Spark</a></li> <li><a href="https://www.manning.com/books/spark-in-action-second-edition?a_aid=jgp&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark In Action</a></li> <li><a href="https://github.com/search?o=desc&amp;q=user%3Ajgperrin+sparkwithjava&amp;s=stars&amp;type=Repositories&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Book code examples in GitHub</a></li> <li><a href="https://www.ibm.com/analytics/informix?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Informix</a></li> <li><a href="http://iiug.org?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">International Informix Users Group</a></li> <li><a href="https://www.mysql.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">MySQL</a></li> <li><a href="https://www.microsoft.com/en-us/sql-server/default.aspx?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Microsoft SQL Server</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_transform,_load?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ETL (Extract, Transform, Load)</a></li> <li><a href="https://spark.apache.org/sql/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark SQL</a> and <a href="https://www.manning.com/books/spark-in-action-second-edition?a_aid=jgp&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark In Action</a>‘s chapter 11</li> <li><a href="https://spark.apache.org/mllib/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark ML</a> and <a href="https://www.manning.com/books/spark-in-action-second-edition?a_aid=jgp&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark In Action</a>‘s chapter 18</li> <li><a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark Streaming (structured)</a> and <a href="https://www.manning.com/books/spark-in-action-second-edition?a_aid=jgp&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark In Action</a>‘s chapter 10</li> <li><a href="https://spark.apache.org/graphx/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark GraphX</a></li> <li><a href="http://hadoop.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hadoop</a></li> <li><a href="https://jupyter.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Jupyter</a> <ul> <li><a href="https://www.podcastinit.com/episode-10-brian-granger-and-fernando-perez-of-the-ipython-project/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="http://zeppelin.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Zeppelin</a></li> <li><a href="https://databricks.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Databricks</a></li> <li><a href="https://www.ibm.com/cloud/watson-studio?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">IBM Watson Studio</a></li> <li><a href="https://kafka.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka</a></li> <li><a href="https://flink.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flink</a><p></p> <ul> <li><a href="https://www.dataengineeringpodcast.com/apache-flink-with-fabian-hueske-episode-57/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Episode</a></li> </ul> <p></p></li> <li><a href="https://aws.amazon.com/kinesis/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">AWS Kinesis</a></li> <li><a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Yarn</a></li> <li><a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">HDFS</a></li> <li><a href="https://hive.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Hive</a></li> <li><a href="https://www.scala-lang.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Scala</a></li> <li><a href="https://spark.apache.org/docs/latest/api/python/index.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">PySpark</a></li> <li><a href="https://livebook.manning.com/#!/book/spark-in-action-second-edition/chapter-4?a_aid=jgp&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DAG</a></li> <li><a href="https://databricks.com/glossary/catalyst-optimizer?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark Catalyst</a></li> <li><a href="https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark Tungsten</a></li> <li><a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-udfs.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark UDF</a></li> <li><a href="https://aws.amazon.com/emr/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">AWS EMR</a></li> <li><a href="http://mesos.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Mesos</a></li> <li><a href="https://dcos.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DC/OS</a></li> <li><a href="https://kubernetes.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kubernetes</a></li> <li><a href="https://livebook.manning.com/#!/book/spark-in-action-second-edition/chapter-3?a_aid=jgp&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Dataframes</a></li> </ul><p></p> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Putting Apache Spark Into Action with Jean Georges Perrin - Episode 60