Thomas Hazel's guest appearances

Thomas Hazel

Data Engineering Podcast

Joel Beasley | Modern CTO

Screaming in the Cloud

Jeff Smith

Jason Warner

Peter Cooper

Pete Cheslock

Rob Zuber

Anurag Gupta

Chris Harris

Ryan Singer

Chris Hill

Jonan Scheffler

Donna Johnson

David King

Andy Budd

Brian Balfour

Nick Caldwell

OwlTail

Today we are talking to Thomas, the Founder and CTO at ChaosSearch.  And we discuss their mind blowing innovation in data storage and search, How constraints can breed creativity, and why data is cheap but information is expensive.All of this, right here, right now on the Modern CTO Podcast!

Today we are talking to Thomas, the Founder and CTO at ChaosSearch.  And we discuss their mind blowing innovation in data storage and search, How constraints can breed creativity, and why data is cheap but information is expensive.<br><br>All of this, right here, right now on the Modern CTO Podcast!

#191 Thomas Hazel CTO & Founder at ChaosSearch

SEARCHing in the CHAOS with Thomas Hazel

Summary Elasticsearch is a powerful tool for storing and analyzing data, but when using it for logs and other time oriented information it can become problematic to keep all of your history. Chaos Search was started to make it easy for you to keep all of your data and make it usable in S3, so that you can have the best of both worlds. In this episode the CTO, Thomas Hazel, and VP of Product, Pete Cheslock, describe how they have built a platform to let you keep all of your history, save money, and reduce your operational overhead. They also explain some of the types of data that you can use with Chaos Search, how to load it into S3, and when you might want to choose it over Amazon Athena for our serverless data analysis. Preamble  Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to dataengineeringpodcast.com/linode to get a $/0 credit and launch a new server in under a minute. You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at dataengineeringpodcast.com/metis-machine to learn more about how Metis Machine is operationalizing data science. Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch. Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Your host is Tobias Macey and today I’m interviewing Pete Cheslock and Thomas Hazel about Chaos Search and their effort to bring historical depth to your Elasticsearch data  Interview  Introduction How did you get involved in the area of data management? Can you start by explaining what you have built at Chaos Search and the problems that you are trying to solve with it?  What types of data are you focused on supporting? What are the challenges inherent to scaling an elasticsearch infrastructure to large volumes of log or metric data?   Is there any need for an Elasticsearch cluster in addition to Chaos Search? For someone who is using Chaos Search, what mechanisms/formats would they use for loading their data into S3? What are the benefits of implementing the Elasticsearch API on top of your data in S3 as opposed to using systems such as Presto or Drill to interact with the same information via SQL? Given that the S3 API has become a de facto standard for many other object storage platforms, what would be involved in running Chaos Search on data stored outside of AWS? What mechanisms do you use to allow for such drastic space savings of indexed data in S3 versus in an Elasticsearch cluster? What is the system architecture that you have built to allow for querying terabytes of data in S3?  What are the biggest contributors to query latency and what have you done to mitigate them?   What are the options for access control when running queries against the data stored in S3? What are some of the most interesting or unexpected uses of Chaos Search and access to large amounts of historical log information that you have seen? What are your plans for the future of Chaos Search?  Contact Info  Pete Cheslock  @petecheslock on Twitter Website   Thomas Hazel  @thomashazel on Twitter LinkedIn    Parting Question  From your perspective, what is the biggest gap in the tooling or technology for data management today?  Links  Chaos Search AWS S3 Cassandra Elasticsearch  Podcast Interview   PostgreSQL Distributed Systems Information Theory Lucene Inverted Index Kibana Logstash NVMe AWS KMS Kinesis FluentD Parquet Athena Presto Drill Backblaze OpenStack Swift Minio EMR DataDog NewRelic Elastic Beats Metricbeat Graphite Snappy Scala Akka Elastalert Tensorflow X-Pack Data Lake  The intro and outro music is from The Hug by The Freak Fandango Orchestra  / CC BY-SASupport Data Engineering Podcast

<h2>Summary</h2> <p>Elasticsearch is a powerful tool for storing and analyzing data, but when using it for logs and other time oriented information it can become problematic to keep all of your history. Chaos Search was started to make it easy for you to keep all of your data and make it usable in S3, so that you can have the best of both worlds. In this episode the CTO, Thomas Hazel, and VP of Product, Pete Cheslock, describe how they have built a platform to let you keep all of your history, save money, and reduce your operational overhead. They also explain some of the types of data that you can use with Chaos Search, how to load it into S3, and when you might want to choose it over Amazon Athena for our serverless data analysis.</p> <h2>Preamble</h2> <ul> <li>Hello and welcome to the Data Engineering Podcast, the show about modern data management</li> <li>When you’re ready to build your next pipeline you’ll need somewhere to deploy it, so check out Linode. With private networking, shared block storage, node balancers, and a 40Gbit network, all controlled by a brand new API you’ve got everything you need to run a bullet-proof data platform. Go to <a href="https://www.dataengineeringpodcast.com/linode?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/linode</a> to get a $/0 credit and launch a new server in under a minute.</li> <li>You work hard to make sure that your data is reliable and accurate, but can you say the same about the deployment of your machine learning models? The Skafos platform from Metis Machine was built to give your data scientists the end-to-end support that they need throughout the machine learning lifecycle. Skafos maximizes interoperability with your existing tools and platforms, and offers real-time insights and the ability to be up and running with cloud-based production scale infrastructure instantaneously. Request a demo at <a href="https://www.dataengineeringpodcast.com/metis-machine?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/metis-machine</a> to learn more about how Metis Machine is operationalizing data science.</li> <li>Go to <a href="https://www.dataengineeringpodcast.com?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com</a> to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.</li> <li>Join the community in the new Zulip chat workspace at <a href="https://www.dataengineeringpodcast.com/chat?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">dataengineeringpodcast.com/chat</a></li> <li>Your host is Tobias Macey and today I’m interviewing Pete Cheslock and Thomas Hazel about Chaos Search and their effort to bring historical depth to your Elasticsearch data</li> </ul> <h2>Interview</h2> <ul> <li>Introduction</li> <li>How did you get involved in the area of data management?</li> <li>Can you start by explaining what you have built at Chaos Search and the problems that you are trying to solve with it? <ul> <li>What types of data are you focused on supporting?</li> <li>What are the challenges inherent to scaling an elasticsearch infrastructure to large volumes of log or metric data?</li> </ul> <p></p></li> <li>Is there any need for an Elasticsearch cluster in addition to Chaos Search?</li> <li>For someone who is using Chaos Search, what mechanisms/formats would they use for loading their data into S3?</li> <li>What are the benefits of implementing the Elasticsearch API on top of your data in S3 as opposed to using systems such as Presto or Drill to interact with the same information via SQL?</li> <li>Given that the S3 API has become a de facto standard for many other object storage platforms, what would be involved in running Chaos Search on data stored outside of AWS?</li> <li>What mechanisms do you use to allow for such drastic space savings of indexed data in S3 versus in an Elasticsearch cluster?</li> <li>What is the system architecture that you have built to allow for querying terabytes of data in S3?<p></p> <ul> <li>What are the biggest contributors to query latency and what have you done to mitigate them?</li> </ul> <p></p></li> <li>What are the options for access control when running queries against the data stored in S3?</li> <li>What are some of the most interesting or unexpected uses of Chaos Search and access to large amounts of historical log information that you have seen?</li> <li>What are your plans for the future of Chaos Search?</li> </ul><p></p> <h2>Contact Info</h2> <ul> <li>Pete Cheslock <ul> <li><a href="https://twitter.com/petecheslock?lang=en&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@petecheslock</a> on Twitter</li> <li><a href="https://pete.wtf/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Website</a></li> </ul> <p></p></li> <li>Thomas Hazel<p></p> <ul> <li><a href="https://twitter.com/thomashazel?lang=en&amp;utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">@thomashazel</a> on Twitter</li> <li><a href="https://www.linkedin.com/in/thomashazel?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">LinkedIn</a></li> </ul> <p></p></li> </ul><p></p> <h2>Parting Question</h2> <ul> <li>From your perspective, what is the biggest gap in the tooling or technology for data management today?</li> </ul> <h2>Links</h2> <ul> <li><a href="https://chaossearch.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Chaos Search</a></li> <li><a href="https://aws.amazon.com/s3/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">AWS S3</a></li> <li><a href="http://cassandra.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Cassandra</a></li> <li><a href="https://www.elastic.co/products/elasticsearch?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Elasticsearch</a> <ul> <li><a href="https://www.dataengineeringpodcast.com/elastic-stack-with-philipp-krenn-episode-23/?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">Podcast Interview</a></li> </ul> <p></p></li> <li><a href="https://www.postgresql.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">PostgreSQL</a></li> <li><a href="https://en.wikipedia.org/wiki/Distributed_computing?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Distributed Systems</a></li> <li><a href="https://en.wikipedia.org/wiki/Information_theory?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Information Theory</a></li> <li><a href="http://lucene.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Lucene</a></li> <li><a href="https://en.wikipedia.org/wiki/Inverted_index?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Inverted Index</a></li> <li><a href="https://www.elastic.co/products/kibana?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kibana</a></li> <li><a href="https://www.elastic.co/products/logstash?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Logstash</a></li> <li><a href="https://en.wikipedia.org/wiki/NVM_Express?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NVMe</a></li> <li><a href="https://aws.amazon.com/kms/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">AWS KMS</a></li> <li><a href="https://aws.amazon.com/kinesis/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Kinesis</a></li> <li><a href="https://www.fluentd.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">FluentD</a></li> <li><a href="http://parquet.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Parquet</a></li> <li><a href="https://aws.amazon.com/athena/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Athena</a></li> <li><a href="https://prestodb.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Presto</a></li> <li><a href="http://drill.apache.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Drill</a></li> <li><a href="https://www.backblaze.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Backblaze</a></li> <li><a href="https://wiki.openstack.org/wiki/Swift?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">OpenStack Swift</a></li> <li><a href="https://www.minio.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Minio</a></li> <li><a href="https://aws.amazon.com/emr/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">EMR</a></li> <li><a href="https://www.dataengineeringpodcast.com/datadog?utm_source=rss&amp;utm_medium=rss" rel="nofollow" target="_blank">DataDog</a></li> <li><a href="https://newrelic.com/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">NewRelic</a></li> <li><a href="https://www.elastic.co/products/beats?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Elastic Beats</a></li> <li><a href="https://www.elastic.co/products/beats/metricbeat?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Metricbeat</a></li> <li><a href="https://graphiteapp.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Graphite</a></li> <li><a href="https://github.com/google/snappy?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Snappy</a></li> <li><a href="https://www.scala-lang.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Scala</a></li> <li><a href="https://akka.io/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Akka</a></li> <li><a href="https://github.com/Yelp/elastalert?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Elastalert</a></li> <li><a href="https://www.tensorflow.org/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Tensorflow</a></li> <li><a href="https://www.elastic.co/products/stack?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">X-Pack</a></li> <li><a href="https://en.wikipedia.org/wiki/Data_lake?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">Data Lake</a></li> </ul><p></p> <p>The intro and outro music is from <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Hug</a> by <a href="http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">The Freak Fandango Orchestra</a>  / <a href="http://creativecommons.org/licenses/by-sa/3.0/?utm_source=rss&amp;utm_medium=rss" rel="noopener nofollow" target="_blank">CC BY-SA</a></p><p><a href="https://dataengineering.supercast.com/" rel="payment nofollow" target="_blank">Support Data Engineering Podcast</a></p>

Keep Your Data And Query It Too Using Chaos Search with Thomas Hazel and Pete Cheslock - Episode 47