Matt Gershoff's guest appearances

Matt Gershoff

The Analytics Power Hour

Technical Marketing Handbook

Adam Greco

Simo Ahava

Tim Harford

Nancy Duarte

Emily Oster

Lea Pica

Hilary Mason

Cassie Kozyrkov

Els Aerts

Peter Fader

Steve Mulder

Joe Sutherland

John Wallace

Chad Sanderson

Tim Hwang

OwlTail

After the “drought” of the last three episodes, you can finally relax as it’s not just me doing a solo episode. And what an amazing guest I managed to persuade to join me! Matt Gershoff of Conductrics is a good friend and an eloquent speaker. Above all, he’s so incredibly knowledgeable about experimentation and about statistical inference in general. All the work we do with data is a subset of statistical inference. We collect data to draw conclusions, which we hope to extrapolate to the experiences and intuitions that we haven’t (yet) measured against. Often this is done through means of passive analysis, which is more related to the traditional “analytics” work done in organizations. Data is collected, and it is mined for insight and for feeding into new hypothesis generation. The other approach is that of experimentation, where a business question is first formulated, then data is collected specifically for the needs of answering this question, and finally the results are analyzed on the basis of whether there’s (sufficient) evidence to suggest that an answer was found to the question or not. Matt walks us through the ins and outs of techniques such as A/B testing, the frequentist and Bayesian approaches, and how it’s really up to organizations and the individuals within to make the right calls with how they approach data. No tool or service delivers automatic gratification. Having said that, I would be remiss to not point out how amazing Conductrics is as an experimentation platform. It promotes transparency and open-source approaches over black boxes and magically produced results. It’s very developer-friendly, being designed and developed with powerful APIs in mind first and foremost, although it does provide a very intuitive user interface for running more traditional A/B test setups with. We didn’t have enough time to discuss the Conductrics approach, which is why I’m comfortable in sharing this little unsolicited (and definitely not paid-for) shout out to what is my favorite experimentation platform out there! Anyway, listen to the podcast to find out what trams in Helsinki have to do with understanding the p-value in the frequentist approach.. Listen to the episode using the player or find it in your favorite podcast service. Topics  00:00:00 – Introduction 00:08:07 – Matt shares his thoughts on experimentation and the scientific method. 00:12:47 – What are A/B tests used for? 00:17:54 – How do you know what type of data to collect, and how much of it, for any given test? 00:23:21 – Should the analytics tools we use guide the users to drawing better and more statistically sound conclusions? 00:31:07 – The analyst should be aware of the circumstances in which the data was collected. 00:35:19 – How to deal with analysts or stakeholders who misinterpret or refuse to accept the results of a test? 00:41:44 – The problem with analyzing the data without having formulated the correct questions before data collection. 00:44:57 – By analyzing the results in different ways, could you change an inconclusive result to a conclusive one? 00:46:50 – Using experiment results for discovery and new hypothesis generation. 00:48:14 – What is the p-value? 00:54:04 – What is the Bayesian approach? 00:56:41 – How important is it to account for errors (e.g. false positives/negatives) in test design? 01:00:16 – If you could make everyone in digital organizations adopt a skill or know-how to help the organization be more sensitive to data and statistical inference, what would that skill be? 01:04:22 – How can people follow what Matt is writing about and sharing online? 01:05:23 – Outro  Notes and references  Conductrics Neyman-Pearson lemma T-test Randomized Control Trial (RCT) Frequentist and Bayesian methods Nobel Prize in Economic Sciences 2021 Deborah Mayo: Statistical Inference As Severe Testing HiPPO: Highest Paid Person’s Opinion Benjamini-Hochberg Procedure Prior probability in Bayesian statistical inference Minimum detectable effect Matt Gershoff (Twitter) Conductrics Blog  The post TMH #9: Experimentation with Matt Gershoff appeared first on Simmer.

After the “drought” of the last three episodes, you can finally relax as it’s not just me doing a solo episode. And what an amazing guest I managed to persuade to join me! Matt Gershoff of <a href="https://conductrics.com/" rel="nofollow" target="_blank">Conductrics</a> is a good friend and an eloquent speaker. Above all, he’s so incredibly knowledgeable about experimentation and about statistical inference in general. All the work we do with data is a subset of statistical inference. We collect data to draw conclusions, which we hope to extrapolate to the experiences and intuitions that we haven’t (yet) measured against. Often this is done through means of passive analysis, which is more related to the traditional “analytics” work done in organizations. Data is collected, and it is mined for insight and for feeding into new hypothesis generation. The other approach is that of experimentation, where a business question is first formulated, then data is collected specifically for the needs of answering this question, and finally the results are analyzed on the basis of whether there’s (sufficient) evidence to suggest that an answer was found to the question or not. Matt walks us through the ins and outs of techniques such as A/B testing, the frequentist and Bayesian approaches, and how it’s really up to organizations and the individuals within to make the right calls with how they approach data. No tool or service delivers automatic gratification. Having said that, I would be remiss to not point out how amazing Conductrics is as an experimentation platform. It promotes transparency and open-source approaches over black boxes and magically produced results. It’s very developer-friendly, being designed and developed with powerful APIs in mind first and foremost, although it does provide a very intuitive user interface for running more traditional A/B test setups with. We didn’t have enough time to discuss the Conductrics approach, which is why I’m comfortable in sharing this little unsolicited (and definitely not paid-for) shout out to what is my favorite experimentation platform out there! Anyway, listen to the podcast to find out what trams in Helsinki have to do with understanding the p-value in the frequentist approach.. Listen to the episode using the player or find it in your <a href="https://www.teamsimmer.com/tmh-podcast/" rel="nofollow" target="_blank">favorite podcast service</a>. <h2>Topics</h2> <ul> <li>00:00:00 – Introduction</li> <li>00:08:07 – Matt shares his thoughts on experimentation and the scientific method.</li> <li>00:12:47 – What are A/B tests used for?</li> <li>00:17:54 – How do you know what type of data to collect, and how much of it, for any given test?</li> <li>00:23:21 – Should the analytics tools we use guide the users to drawing better and more statistically sound conclusions?</li> <li>00:31:07 – The analyst should be aware of the circumstances in which the data was collected.</li> <li>00:35:19 – How to deal with analysts or stakeholders who misinterpret or refuse to accept the results of a test?</li> <li>00:41:44 – The problem with analyzing the data without having formulated the correct questions before data collection.</li> <li>00:44:57 – By analyzing the results in different ways, could you change an inconclusive result to a conclusive one?</li> <li>00:46:50 – Using experiment results for discovery and new hypothesis generation.</li> <li>00:48:14 – What is the p-value?</li> <li>00:54:04 – What is the Bayesian approach?</li> <li>00:56:41 – How important is it to account for errors (e.g. false positives/negatives) in test design?</li> <li>01:00:16 – If you could make everyone in digital organizations adopt a skill or know-how to help the organization be more sensitive to data and statistical inference, what would that skill be?</li> <li>01:04:22 – How can people follow what Matt is writing about and sharing online?</li> <li>01:05:23 – Outro</li> </ul> <h2>Notes and references</h2> <ul> <li><a href="https://conductrics.com/" rel="nofollow" target="_blank">Conductrics </a></li> <li><a href="https://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma" rel="nofollow" target="_blank">Neyman-Pearson lemma </a></li> <li><a href="https://en.wikipedia.org/wiki/Student%27s_t-test" rel="nofollow" target="_blank">T-test </a></li> <li><a href="https://en.wikipedia.org/wiki/Randomized_controlled_trial" rel="nofollow" target="_blank">Randomized Control Trial (RCT) </a></li> <li><a href="https://cxl.com/blog/bayesian-frequentist-ab-testing/" rel="nofollow" target="_blank">Frequentist and Bayesian methods </a></li> <li><a href="https://www.nytimes.com/2021/10/11/business/nobel-economics-prize-david-card-joshua-angrist-guido-imbens.html" rel="nofollow" target="_blank">Nobel Prize in Economic Sciences 2021 </a></li> <li><a href="https://www.amazon.com/Statistical-Inference-Severe-Testing-Statistics/dp/1107664640" rel="nofollow" target="_blank">Deborah Mayo: Statistical Inference As Severe Testing </a></li> <li>HiPPO: Highest Paid Person’s Opinion </li> <li><a href="https://www.statisticshowto.com/benjamini-hochberg-procedure/" rel="nofollow" target="_blank">Benjamini-Hochberg Procedure </a></li> <li><a href="https://en.wikipedia.org/wiki/Prior_probability" rel="nofollow" target="_blank">Prior probability in Bayesian statistical inference </a></li> <li><a href="https://www.analytics-toolkit.com/glossary/minimum-detectable-effect/" rel="nofollow" target="_blank">Minimum detectable effect </a></li> <li><a href="https://www.twitter.com/mgershoff/" rel="nofollow" target="_blank">Matt Gershoff (Twitter) </a></li> <li><a href="https://conductrics.com/blog/" rel="nofollow" target="_blank">Conductrics Blog </a></li> </ul> The post <a href="https://www.teamsimmer.com/2021/10/19/experimentation-with-matt-gershoff/" rel="nofollow" target="_blank">TMH #9: Experimentation with Matt Gershoff</a> appeared first on <a href="https://www.teamsimmer.com" rel="nofollow" target="_blank">Simmer</a>.

TMH #9: Experimentation with Matt Gershoff

Let's pretend your goal as an analyst is to eloquently and accurately explain reinforcement learning. Now, let's pretend that you get to try that explanation again and again, and we'll give you an electric shock every time you state something inaccurately and a cookie every time you say something right. Well, you're an analyst, so you're now wondering if this is some clever play on words about cookies. As it happened, we didn't give Matt Gershoff from Conductrics any cookies of any kind in his return to the show. Instead, we gave him a lifetime's supply of opportunities to say, "Well, no, it's not really like that," which is a special kind of nourishment for the exceptionally smart and patient! In other words, the gang walked through a range of analogies and examples about machine learning, reinforcement learning, and AI with Matt, and no electric sheep were harmed in the process.  For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the  show page.

Let's pretend your goal as an analyst is to eloquently and accurately explain reinforcement learning. Now, let's pretend that you get to try that explanation again and again, and we'll give you an electric shock every time you state something inaccurately and a cookie every time you say something right. Well, you're an analyst, so you're now wondering if this is some clever play on words about cookies. As it happened, we didn't give <a href="https://twitter.com/mgershoff" rel="noopener nofollow" target="_blank">Matt Gershoff</a> from <a href="https://conductrics.com/" rel="noopener nofollow" target="_blank">Conductrics</a> any cookies of any kind in his return to the show. Instead, we gave him a lifetime's supply of opportunities to say, "Well, no, it's not really like that," which is a special kind of nourishment for the exceptionally smart and patient! In other words, the gang walked through a range of analogies and examples about machine learning, reinforcement learning, and AI with Matt, and no electric sheep were harmed in the process.  For complete show notes, including links to items mentioned in this episode and a transcript of the show, visit the <a href="http://www.analyticshour.io/2017/09/12/071-reinforcement-learning-matt-gershoff/" rel="nofollow" target="_blank"> show page</a>.

#071: Reinforcement Learning with Matt Gershoff