Marco Tulio Ribeiro's guest appearances

Marco Tulio Ribeiro

NLP Highlights

Stanford MLSys Seminar

Ellie Pavlick

Ines Montani

Noah Smith

Michael Collins

Brendan O'Connor

Cody Coleman

Victor Sanh

Sam Bowman

Nathan Schneider

Matt Gardner

Chris Dyer

John Hewitt

Dan Roth

Leon Derczynski

Beidi Chen

OwlTail

Marco Tulio Ribeiro on "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" We will present CheckList, a task-agnostic methodology and tool for testing NLP models inspired by principles of behavioral testing in software engineering. We will show a lot of fun bugs we discovered with CheckList, both in commercial models (Microsoft, Amazon, Google) and research models (BERT, RoBERTA for sentiment analysis, QQP, SQuAD). We'll also present comparisons between CheckList and the status quo, in a case study at Microsoft and a user study with researchers and engineers. We show that CheckList is a really helpful process and tool for testing and finding bugs in NLP models, both for practitioners and researchers.

Marco Tulio Ribeiro on "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" We will present CheckList, a task-agnostic methodology and tool for testing NLP models inspired by principles of behavioral testing in software engineering. We will show a lot of fun bugs we discovered with CheckList, both in commercial models (Microsoft, Amazon, Google) and research models (BERT, RoBERTA for sentiment analysis, QQP, SQuAD). We'll also present comparisons between CheckList and the status quo, in a case study at Microsoft and a user study with researchers and engineers. We show that CheckList is a really helpful process and tool for testing and finding bugs in NLP models, both for practitioners and researchers.

10/15/20 #1 Marco Tulio Ribeiro - Beyond Accuracy: Behavioral Testing of NLP Models with CheckList

We invited Marco Tulio Ribeiro, a Senior Researcher at Microsoft, to talk about evaluating NLP models using behavioral testing, a framework borrowed from Software Engineering. Marco describes three kinds of black-box tests the check whether NLP models satisfy certain necessary conditions. While breaking the standard IID assumption, this framework presents a way to evaluate whether NLP systems are ready for real-world use. We also discuss what capabilities can be tested using this framework, how one can come up with good tests, and the need for an evolving set of behavioral tests for NLP systems.Marco’s homepage: https://homes.cs.washington.edu/~marcotcr/

We invited Marco Tulio Ribeiro, a Senior Researcher at Microsoft, to talk about evaluating NLP models using behavioral testing, a framework borrowed from Software Engineering. Marco describes three kinds of black-box tests the check whether NLP models satisfy certain necessary conditions. While breaking the standard IID assumption, this framework presents a way to evaluate whether NLP systems are ready for real-world use. We also discuss what capabilities can be tested using this framework, how one can come up with good tests, and the need for an evolving set of behavioral tests for NLP systems. Marco’s homepage: <a href="https://homes.cs.washington.edu/~marcotcr/" rel="nofollow" target="_blank">https://homes.cs.washington.edu/~marcotcr/</a>

114 - Behavioral Testing of NLP Models, with Marco Tulio Ribeiro