Carl Steinbach's guest appearances

Carl Steinbach

Software Engineering Daily

Data Archives - Software Engineering Daily

Software Daily

MLOps.community 

Joey Parsons

Zhenxiao Luo

Alan Murphy

Vinoth Chandar

Sam Shah

Kevin Xu

Wenbin Fang

Kalyan Ramanathan

Joshua Haas

John Whaley

Michael Armbrust

Matthew Rocklin

Allison King

Sujith Nair

David Potere

OwlTail

LinkedIn is a social network with petabytes of data. In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs.LinkedIn has been developing its data infrastructure since the early days of the Hadoop ecosystem. LinkedIn started using Hadoop in 2008, and in the last 11 years, the company has adopted streaming frameworks, distributed databases, and newer execution runtimes like Apache Spark.With the popularization of machine learning, there are more applications for data engineering than ever before. But the tooling around data engineering means that it is still hard for developers to find data sets, clean their data, and build reliable models. Carl Steinbach is an engineer at LinkedIn working on tools for data engineering. In today’s episode, Carl discusses the data platform inside LinkedIn, and the strategies that the company has developed around storing and computing large amounts of data. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

LinkedIn is a social network with petabytes of data. In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs.LinkedIn has been developing its data infrastructure since the early days of the Hadoop ecosystem. LinkedIn started using Hadoop in 2008, and in the last 11 years, the company has adopted streaming frameworks, distributed databases, and newer execution runtimes like Apache Spark.With the popularization of machine learning, there are more applications for data engineering than ever before. But the tooling around data engineering means that it is still hard for developers to find data sets, clean their data, and build reliable models. Carl Steinbach is an engineer at LinkedIn working on tools for data engineering. In today’s episode, Carl discusses the data platform inside LinkedIn, and the strategies that the company has developed around storing and computing large amounts of data. Full disclosure: LinkedIn is a sponsor of Software Engineering Daily.

LinkedIn Data Platform with Carl Steinbach

LinkedIn is a social network with petabytes of data.  In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs. LinkedIn has been developing its data infrastructure since the early days of the Hadoop ecosystem. LinkedIn started using Hadoop in 2008, and in the last 11 years, the company has adopted streaming frameworks, distributed databases, and newer execution runtimes like Apache Spark. With the popularization of machine learning, there are more applications for data engineering than ever before. But the tooling around data engineering means that it is still hard for developers to find data sets, clean their data, and build reliable models.  Carl Steinbach is an engineer at LinkedIn working on tools for data engineering. In today’s episode, Carl discusses the data platform inside LinkedIn, and the strategies that the company has developed around storing and computing large amounts of data.  Full disclosure: LinkedIn is a sponsor of Software Engineering Daily. Check out our active projects:  We are hiring a head of growth. If you like Software Engineering Daily and consider yourself competent in sales, marketing, and strategy, send me an email: jeff@softwareengineeringdaily.com FindCollabs is a place to build open source software. The SEDaily app for iOS and Android includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. Subscribe for ad-free episodes.  The post LinkedIn Data Platform with Carl Steinbach appeared first on Software Engineering Daily.

LinkedIn is a social network with petabytes of data.  In order to store that data, LinkedIn distributes and replicates that data across a large cluster of machines running the Hadoop Distributed File System. In order to run calculations across its large data set, LinkedIn needs to split the computation up using MapReduce-style jobs. LinkedIn has been developing its data infrastructure since the early days of the Hadoop ecosystem. LinkedIn started using Hadoop in 2008, and in the last 11 years, the company has adopted streaming frameworks, distributed databases, and newer execution runtimes like Apache Spark. With the popularization of machine learning, there are more applications for data engineering than ever before. But the tooling around data engineering means that it is still hard for developers to find data sets, clean their data, and build reliable models.  Carl Steinbach is an engineer at LinkedIn working on tools for data engineering. In today’s episode, Carl discusses the data platform inside LinkedIn, and the strategies that the company has developed around storing and computing large amounts of data.  Full disclosure: LinkedIn is a sponsor of Software Engineering Daily. <h2>Check out our active projects:</h2> <ul> <li>We are hiring a <a href="https://www.softwaredaily.com/jobs/5c3a6cf0f7fb250029d873a9" rel="nofollow" target="_blank">head of growth</a>. If you like Software Engineering Daily and consider yourself competent in sales, marketing, and strategy, send me an email: jeff@softwareengineeringdaily.com</li> <li><a href="https://findcollabs.com/" rel="nofollow" target="_blank">FindCollabs</a> is a place to build open source software.</li> <li><a href="https://itunes.apple.com/us/app/software-engineering-daily/id1253734426?mt=8" rel="nofollow" target="_blank">The SEDaily app for iOS</a> and <a href="https://play.google.com/store/apps/details?id=com.koalatea.sedaily" rel="nofollow" target="_blank">Android</a> includes all 1000 of our old episodes, as well as related links, greatest hits, and topics. <a href="https://softwareengineeringdaily.com/subscribe" rel="nofollow" target="_blank">Subscribe</a> for ad-free episodes.</li> </ul> The post <a href="https://softwareengineeringdaily.com/2019/10/23/linkedin-data-platform-with-carl-steinbach/" rel="nofollow" target="_blank">LinkedIn Data Platform with Carl Steinbach</a> appeared first on <a href="https://softwareengineeringdaily.com" rel="nofollow" target="_blank">Software Engineering Daily</a>.