James Meickle's guest appearances

James Meickle

Data Engineering Podcast

Nikita Shamgunov

Christopher Ryan

Emily Miller

Peter Bull

Patrick Hunt

Jonathan Katz

Kevin Moore

Dan Robinson

Matthew Rocklin

Maxime Beauchemin

Jean Georges Perrin

Todd Walter

Mark Marinelli

Andy LoPresto

Joe Crobak

OwlTail

Summary The theory behind how a tool is supposed to work and the realities of putting it into practice are often at odds with each other. Learning  the pitfalls and best practices from someone who has gained that knowledge the hard way can save you from wasted time and frustration. In this episode James Meickle discusses his recent experience building a new installation of Airflow. He points out the strengths, design flaws, and areas of improvement for the framework. He also describes the design patterns and workflows that his team has built to allow them to use Airflow as the basis of their data science platform. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing James Meickle about his experiences building a new Airflow installation  Interview  Introduction How did you get involved in the area of data management? What was your initial project requirement?  What tooling did you consider in addition to Airflow? What aspects of the Airflow platform led you to choose it as your implementation target?   Can you describe your current deployment architecture?  How many engineers are involved in writing tasks for your Airflow installation?   What resources were the most helpful while learning about Airflow design patterns?  How have you architected your DAGs for deployment and extensibility?   What kinds of tests and automation have you put in place to support the ongoing stability of your deployment? What are some of the dead-ends or other pitfalls that you encountered during the course of this project? What aspects of Airflow have you found to be lacking that you would like to see improved? What did you wish someone had told you before you started work on your Airflow installation?  If you were to start over would you make the same choice? If Airflow wasn’t available what would be your second choice?   What are your next steps for improvements and fixes?  Contact Info  @eronarn on Twitter Website eronarn on GitHub  Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Quantopian Harvard Brain Science Initiative DevOps Days Boston Google Maps API Cron ETL (Extract, Transform, Load) Azkaban Luigi AWS Glue Airflow Pachyderm  Podcast Interview   AirBnB Python YAML Ansible REST (Representational State Transfer) SAML (Security Assertion Markup Language) RBAC (Role-Based Access Control) Maxime Beauchemin  Medium Blog   Celery Dask  Podcast Interview   PostgreSQL  Podcast Interview   Redis Cloudformation Jupyter Notebook Qubole Astronomer  Podcast Interview   Gunicorn Kubernetes Airflow Improvement Proposals Python Enhancement Proposals (PEP)  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>The theory behind how a tool is supposed to work and the realities of putting it into practice are often at odds with each other. Learning  the pitfalls and best practices from someone who has gained that knowledge the hard way can save you from wasted time and frustration. In this episode James Meickle discusses his recent experience building a new installation of Airflow. He points out the strengths, design flaws, and areas of improvement for the framework. He also describes the design patterns and workflows that his team has built to allow them to use Airflow as the basis of their data science platform.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> to get a $20 credit and launch a new server in under a minute.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Join the community in the new Zulip chat workspace at <a href="https://www.dataengineeringpodcast.com/chat?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/chat</a></li> <li>Your host is Tobias Macey and today I’m interviewing James Meickle about his experiences building a new Airflow installation</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>What was your initial project requirement? <ul> <li>What tooling did you consider in addition to Airflow?</li> <li>What aspects of the Airflow platform led you to choose it as your implementation target?</li> </ul> <p></p></li> <li>Can you describe your current deployment architecture?<p></p> <ul> <li>How many engineers are involved in writing tasks for your Airflow installation?</li> </ul> <p></p></li> <li>What resources were the most helpful while learning about Airflow design patterns?<p></p> <ul> <li>How have you architected your DAGs for deployment and extensibility?</li> </ul> <p></p></li> <li>What kinds of tests and automation have you put in place to support the ongoing stability of your deployment?</li> <li>What are some of the dead-ends or other pitfalls that you encountered during the course of this project?</li> <li>What aspects of Airflow have you found to be lacking that you would like to see improved?</li> <li>What did you wish someone had told you before you started work on your Airflow installation?<p></p> <ul> <li>If you were to start over would you make the same choice?</li> <li>If Airflow wasn’t available what would be your second choice?</li> </ul> <p></p></li> <li>What are your next steps for improvements and fixes?</li> </ul><p></p> <h2>Contact Info</h2> <ul> <li><a href="https://github.com/Eronarn?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@eronarn</a> on Twitter</li> <li><a href="http://permadeath.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Website</a></li> <li><a href="https://github.com/eronarn?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">eronarn</a> on GitHub</li> </ul> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://www.quantopian.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Quantopian</a></li> <li><a href="http://brain.harvard.edu/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Harvard Brain Science Initiative</a></li> <li><a href="https://www.devopsdays.org/events/2018-boston/welcome/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DevOps Days Boston</a></li> <li><a href="https://cloud.google.com/maps-platform/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Google Maps API</a></li> <li><a href="https://en.wikipedia.org/wiki/Cron?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Cron</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_transform,_load?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ETL (Extract, Transform, Load)</a></li> <li><a href="https://azkaban.github.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Azkaban</a></li> <li><a href="http://luigi.readthedocs.io/en/stable/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Luigi</a></li> <li><a href="https://aws.amazon.com/glue/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">AWS Glue</a></li> <li><a href="http://airflow.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Airflow</a></li> <li><a href="http://pachyderm.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Pachyderm</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/epsiode-1-pachyderm-with-daniel-whitenack/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="https://www.airbnb.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">AirBnB</a></li> <li><a href="https://www.python.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Python</a></li> <li><a href="http://yaml.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">YAML</a></li> <li><a href="https://www.ansible.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Ansible</a></li> <li><a href="https://en.wikipedia.org/wiki/Representational_state_transfer?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">REST (Representational State Transfer)</a></li> <li><a href="https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">SAML (Security Assertion Markup Language)</a></li> <li><a href="https://en.wikipedia.org/wiki/Role-based_access_control?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">RBAC (Role-Based Access Control)</a></li> <li><a href="https://www.linkedin.com/in/maximebeauchemin?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Maxime Beauchemin</a><p></p> <ul> <li><a href="https://medium.com/@maximebeauchemin?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Medium Blog</a></li> </ul> <p></p></li> <li><a href="http://www.celeryproject.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Celery</a></li> <li><a href="https://dask.pydata.org/en/latest/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Dask</a><p></p> <ul> <li><a href="https://www.dataengineeringpodcast.com/episode-2-dask-with-matthew-rocklin/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="https://www.postgresql.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">PostgreSQL</a><p></p> <ul> <li><a href="https://www.dataengineeringpodcast.com/postgresql-with-jonathan-katz-episode-42/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="https://redis.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Redis</a></li> <li><a href="https://aws.amazon.com/cloudformation/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Cloudformation</a></li> <li><a href="http://jupyter.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Jupyter Notebook</a></li> <li><a href="https://www.qubole.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Qubole</a></li> <li><a href="https://www.astronomer.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Astronomer</a><p></p> <ul> <li><a href="https://www.dataengineeringpodcast.com/astronomer-with-ry-walker-episode-6/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="http://gunicorn.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Gunicorn</a></li> <li><a href="https://kubernetes.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kubernetes</a></li> <li><a href="https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Airflow Improvement Proposals</a></li> <li><a href="https://www.python.org/dev/peps/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Python Enhancement Proposals (PEP)</a></li> </ul><p></p> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Putting Airflow Into Production With James Meickle - Episode 43