Andy LoPresto's guest appearances

Andy LoPresto

Data Engineering Podcast

Nikita Shamgunov

Christopher Ryan

Emily Miller

Peter Bull

Patrick Hunt

Jonathan Katz

Kevin Moore

Dan Robinson

Matthew Rocklin

Maxime Beauchemin

Jean Georges Perrin

Todd Walter

Mark Marinelli

Joe Crobak

Will McGinnis

OwlTail

Summary Data integration and routing is a constantly evolving problem and one that is fraught with edge cases and complicated requirements. The Apache NiFi project models this problem as a collection of data flows that are created through a self-service graphical interface. This framework provides a flexible platform for building a wide variety of integrations that can be managed and scaled easily to fit your particular needs. In this episode project members Kevin Doran and Andy LoPresto discuss the ways that NiFi can be used, how to start using it in your environment, and plans for future development. They also explained how it fits in the broad landscape of data tools, the interesting and challenging aspects of the project, and how to build new extensions. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $20 credit and launch a new server in under a minute. Are you struggling to keep up with customer request and letting errors slip into production?  Want to try some of the innovative ideas in this podcast but don’t have time? DataKitchen’s DataOps software allows your team to quickly iterate and deploy pipelines of code, models, and data sets while improving quality. Unlike a patchwork of manual operations, DataKitchen makes your team shine by providing an end to end DataOps solution with minimal programming that uses the tools you love. Join the DataOps movement and sign up for the newsletter at datakitchen.io/de today. After that learn more about why you should be doing DataOps by listening to the Head Chef in the Data Kitchen at dataengineeringpodcast.com/datakitchen Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Your host is Tobias Macey and today I’m interviewing Kevin Doran and Andy LoPresto about Apache NiFi  Interview  Introduction How did you get involved in the area of data management? Can you start by explaining what NiFi is? What is the motivation for building a GUI as the primary interface for the tool when the current trend is to represent everything as code? How did you get involved with the project?  Where does it sit in the broader landscape of data tools?   Does the data that is processed by NiFi flow through the servers that it is running on (á la Spark/Flink/Kafka), or does it orchestrate actions on other systems (á la Airflow/Oozie)?  How do you manage versioning and backup of data flows, as well as promoting them between environments?   One of the advertised features is tracking provenance for data flows that are managed by NiFi. How is that data collected and managed?  What types of reporting are available across this information?   What are some of the use cases or requirements that lend themselves well to being solved by NiFi?  When is NiFi the wrong choice?   What is involved in deploying and scaling a NiFi installation?  What are some of the system/network parameters that should be considered? What are the scaling limitations?   What have you found to be some of the most interesting, unexpected, and/or challenging aspects of building and maintaining the NiFi project and community? What do you have planned for the future of NiFi?  Contact Info  Kevin Doran  @kevdoran on Twitter Email   Andy LoPresto  @yolopey on Twitter Email    Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  NiFi HortonWorks DataFlow HortonWorks Apache Software Foundation Apple CSV XML JSON Perl Python Internet Scale Asset Management Documentum DataFlow NSA (National Security Agency) 24 (TV Show) Technology Transfer Program Agile Software Development Waterfall Spark Flink Kafka Oozie Luigi Airflow FluentD ETL (Extract, Transform, and Load) ESB (Enterprise Service Bus) MiNiFi Java C++ Provenance Kubernetes Apache Atlas Data Governance Kibana K-Nearest Neighbors DevOps DSL (Domain Specific Language) NiFi Registry Artifact Repository Nexus NiFi CLI Maven Archetype IoT Docker Backpressure NiFi Wiki TLS (Transport Layer Security) Mozilla TLS Observatory NiFi Flow Design System Data Lineage GDPR (General Data Protection Regulation)  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>Data integration and routing is a constantly evolving problem and one that is fraught with edge cases and complicated requirements. The Apache NiFi project models this problem as a collection of data flows that are created through a self-service graphical interface. This framework provides a flexible platform for building a wide variety of integrations that can be managed and scaled easily to fit your particular needs. In this episode project members Kevin Doran and Andy LoPresto discuss the ways that NiFi can be used, how to start using it in your environment, and plans for future development. They also explained how it fits in the broad landscape of data tools, the interesting and challenging aspects of the project, and how to build new extensions.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> to get a $20 credit and launch a new server in under a minute.</li> <li>Are you struggling to keep up with customer request and letting errors slip into production?  Want to try some of the innovative ideas in this podcast but don’t have time? DataKitchen’s DataOps software allows your team to quickly iterate and deploy pipelines of code, models, and data sets while improving quality. Unlike a patchwork of manual operations, DataKitchen makes your team shine by providing an end to end DataOps solution with minimal programming that uses the tools you love. Join the DataOps movement and sign up for the newsletter at <a href="https://datakitchen.io/de?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">datakitchen.io/de</a> today. After that learn more about why you should be doing DataOps by listening to the Head Chef in the Data Kitchen at <a href="https://www.dataengineeringpodcast.com/datakitchen-dataops-with-chris-bergh-episode-26/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/datakitchen</a></li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Your host is Tobias Macey and today I’m interviewing Kevin Doran and Andy LoPresto about Apache NiFi</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by explaining what NiFi is?</li> <li>What is the motivation for building a GUI as the primary interface for the tool when the current trend is to represent everything as code?</li> <li>How did you get involved with the project? <ul> <li>Where does it sit in the broader landscape of data tools?</li> </ul> <p></p></li> <li>Does the data that is processed by NiFi flow through the servers that it is running on (á la Spark/Flink/Kafka), or does it orchestrate actions on other systems (á la Airflow/Oozie)?<p></p> <ul> <li>How do you manage versioning and backup of data flows, as well as promoting them between environments?</li> </ul> <p></p></li> <li>One of the advertised features is tracking provenance for data flows that are managed by NiFi. How is that data collected and managed?<p></p> <ul> <li>What types of reporting are available across this information?</li> </ul> <p></p></li> <li>What are some of the use cases or requirements that lend themselves well to being solved by NiFi?<p></p> <ul> <li>When is NiFi the wrong choice?</li> </ul> <p></p></li> <li>What is involved in deploying and scaling a NiFi installation?<p></p> <ul> <li>What are some of the system/network parameters that should be considered?</li> <li>What are the scaling limitations?</li> </ul> <p></p></li> <li>What have you found to be some of the most interesting, unexpected, and/or challenging aspects of building and maintaining the NiFi project and community?</li> <li>What do you have planned for the future of NiFi?</li> </ul><p></p> <h2>Contact Info</h2> <ul> <li>Kevin Doran <ul> <li><a href="https://twitter.com/kevdoran?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@kevdoran</a> on Twitter</li> <li><a href="mailto:kdoran@apache.org">Email</a></li> </ul> <p></p></li> <li>Andy LoPresto<p></p> <ul> <li><a href="https://twitter.com/yolopey?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@yolopey</a> on Twitter</li> <li><a href="mailto:alopresto@apache.org">Email</a></li> </ul> <p></p></li> </ul><p></p> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="http://nifi.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NiFi</a></li> <li><a href="https://hortonworks.com/products/data-platforms/hdf/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">HortonWorks DataFlow</a></li> <li><a href="https://hortonworks.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">HortonWorks</a></li> <li><a href="http://apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Software Foundation</a></li> <li><a href="https://www.apple.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apple</a></li> <li><a href="https://en.wikipedia.org/wiki/Comma-separated_values?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CSV</a></li> <li><a href="https://en.wikipedia.org/wiki/XML?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">XML</a></li> <li><a href="http://json.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">JSON</a></li> <li><a href="https://www.perl.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Perl</a></li> <li><a href="https://www.python.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Python</a></li> <li><a href="https://gist.github.com/acolyer/95ef23802803cb8b4eb5?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Internet Scale</a></li> <li><a href="https://en.wikipedia.org/wiki/Asset_management?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Asset Management</a></li> <li><a href="https://documentum.opentext.com/documentum-platform/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Documentum</a></li> <li><a href="https://en.wikipedia.org/wiki/Dataflow?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DataFlow</a></li> <li><a href="https://www.nsa.gov/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NSA (National Security Agency)</a></li> <li><a href="https://en.wikipedia.org/wiki/24_(TV_series)?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">24 (TV Show)</a></li> <li><a href="https://www.nsa.gov/what-we-do/research/technology-transfer/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Technology Transfer Program</a></li> <li><a href="https://en.wikipedia.org/wiki/Agile_software_development?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Agile Software Development</a></li> <li><a href="https://en.wikipedia.org/wiki/Waterfall_model?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Waterfall</a></li> <li><a href="http://spark.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Spark</a></li> <li><a href="http://flink.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Flink</a></li> <li><a href="http://kafka.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kafka</a></li> <li><a href="http://oozie.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Oozie</a></li> <li><a href="http://luigi.readthedocs.io/en/stable/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Luigi</a></li> <li><a href="http://airflow.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Airflow</a></li> <li><a href="https://www.fluentd.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">FluentD</a></li> <li><a href="https://en.wikipedia.org/wiki/Extract,_transform,_load?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ETL (Extract, Transform, and Load)</a></li> <li><a href="https://en.wikipedia.org/wiki/Enterprise_service_bus?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">ESB (Enterprise Service Bus)</a></li> <li><a href="https://nifi.apache.org/minifi/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">MiNiFi</a></li> <li><a href="https://en.wikipedia.org/wiki/Java_(programming_language)?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Java</a></li> <li><a href="https://en.wikipedia.org/wiki/C%2B%2B?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">C++</a></li> <li><a href="https://en.wikipedia.org/wiki/Provenance?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Provenance</a></li> <li><a href="https://kubernetes.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kubernetes</a></li> <li><a href="http://atlas.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Apache Atlas</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_governance?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Governance</a></li> <li><a href="https://www.elastic.co/products/kibana?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kibana</a></li> <li><a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">K-Nearest Neighbors</a></li> <li><a href="https://en.wikipedia.org/wiki/DevOps?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DevOps</a></li> <li><a href="https://en.wikipedia.org/wiki/Domain-specific_language?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">DSL (Domain Specific Language)</a></li> <li><a href="https://nifi.apache.org/registry.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NiFi Registry</a></li> <li><a href="https://en.wikipedia.org/wiki/Binary_repository_manager?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Artifact Repository</a></li> <li><a href="https://www.sonatype.com/nexus-repository-sonatype?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Nexus</a></li> <li><a href="https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NiFi CLI</a></li> <li><a href="https://maven.apache.org/guides/introduction/introduction-to-archetypes.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Maven Archetype</a></li> <li><a href="https://en.wikipedia.org/wiki/Internet_of_things?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">IoT</a></li> <li><a href="https://www.docker.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Docker</a></li> <li><a href="https://en.wikipedia.org/wiki/Backpressure_routing?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Backpressure</a></li> <li><a href="https://cwiki.apache.org/confluence/display/NIFI?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NiFi Wiki</a></li> <li><a href="https://en.wikipedia.org/wiki/Transport_Layer_Security?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">TLS (Transport Layer Security)</a></li> <li><a href="https://github.com/mozilla/tls-observatory?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Mozilla TLS Observatory</a></li> <li><a href="http://nifi.apache.org/fds.html?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NiFi Flow Design System</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_lineage?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Lineage</a></li> <li><a href="https://en.wikipedia.org/wiki/General_Data_Protection_Regulation?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">GDPR (General Data Protection Regulation)</a></li> </ul> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Building Data Flows In Apache NiFi With Kevin Doran and Andy LoPresto - Episode 39