Yuri Grinshteyn's guest appearances

Yuri Grinshteyn

Google Cloud Platform Podcast

Page it to the Limit

Data on Kubernetes Community

Nathen Harvey

Liz Fong-Jones

Eric Anderson

John Bohannon

Manuel Lima

Margaret Mitchell

Jessie Frazelle

Joanna Smith

Juliet Hougland

Michelle Casbon

Jesse Houston

Mark Chmarny

Ville Aikas

Dale Markowitz

Sherol Chen

OwlTail

https://go.dok.community/slack https://dok.community/  ABSTRACT OF THE TALK  Join this session to learn:  - why reliability is a key feature of your service - why SLOs are the best way to measure it and set reliability targets - how error budgets increase your velocity - what an error budget burn rate is - how to set up alert using burn rates to optimize your response and operational load  BIO  Yuri works with Google Cloud Platform customers to help them design, architect, build, and operate reliable applications and services and advocates for SRE principles and practices. He occasionally writes code in Node.js, Python, and Go and likes to play video games on the train ride to work.

<a href="https://go.dok.community/slack" rel="nofollow ugc" target="_blank">https://go.dok.community/slack</a> <a href="https://dok.community/" rel="nofollow ugc" target="_blank">https://dok.community/</a> ABSTRACT OF THE TALK Join this session to learn: - why reliability is a key feature of your service - why SLOs are the best way to measure it and set reliability targets - how error budgets increase your velocity - what an error budget burn rate is - how to set up alert using burn rates to optimize your response and operational load BIO Yuri works with Google Cloud Platform customers to help them design, architect, build, and operate reliable applications and services and advocates for SRE principles and practices. He occasionally writes code in Node.js, Python, and Go and likes to play video games on the train ride to work.

DoK Talks #81- How to Alert on SLOs using Error Budget Burn Rate // Yuri Grinshteyn

Steve McGhee

Our old pal Mark Mirchandani is back this week, joining Stephanie Wong and our guests Steve McGhee and Yuri Grinshteyn to talk about Site Reliability Engineering. SRE is Google’s way of helping companies of all sizes create consistent, predictable, and functional projects. It helps clients approach operations from a software engineering stand point so that growing systems can be managed efficiently. We talk about the challenges of implementing best SRE practices and how companies can overcome these. Though the benefits of SRE are many, it can be difficult for clients to grasp. Steve and Yuri tell us the process they go through with customers to help them set realistic goals and work to make reliable, scalable projects with little downtime. By starting small and taking wins early, Steve says clients reap the rewards of SRE and are encouraged to push forward. Yuri’s customer-centric approach encourages companies to prioritize alerts that affect the user experience, thus limiting inbox mayhem and keeping customers happy. Alerts based on symptoms, Steve says, help accomplish this goal. Later, Yuri and Steve describe the best ways for companies to get started with SRE. Realistic goals and specific detailed plans can make the journey less bumpy for clients, and Google’s SRE team can help. Steve McGhee Steve was an SRE at Google for about 10 years, then left to help a company build reliable systems on the Cloud. Now he’s back at Google, helping more companies do that. Yuri Grinshteyn Yuri works with Google Cloud Platform customers to help them design, architect, build, and operate reliable applications and services. He also advocates for SRE principles and practices on YouTube and elsewhere. Cool things of the week  Fresh updates: Google Cloud 2021 Summits  blog Why you need to explain machine learning models  blog  GCP Podcast Episode 260: Responsible AI with Craig Wiley and Tracy Frey  podcast GCP Podcast Episode 249: ML Lifecycle with Dale Markowitz and Craig Wiley  podcast GCP Podcast Episode 214: AI in Healthcare with Dale Markowitz  podcast    Interview  Site Reliability Engineering site Reliability Architecture Framework site Site Reliability Engineering: Measuring and Managing Reliability on Coursera  site Developing a Google SRE Culture on Coursera  site How Lowe’s meets customer demand with Google SRE practices  blog GCP Podcast Episode 68: The Home Depot with William Bonnell  podcast GCP Podcast Episode 213: The Art of SLOs with Alex Bramley  podcast GCP Podcast Episode 127: SRE vs Devops with Liz Fong-Jones and Seth Vargo  podcast GCP Podcast Episode 72: Customer Reliability Engineering with Luke Stone  podcast GCP Podcast Episode 38: Site Reliability Engineering with Paul Newson  podcast GCP Podcast Episode 59: SRE II with Paul Newson  podcast  What’s something cool you’re working on? Yuri has been working on Engineering for Reliability. Stephanie has been working on her new series  What’s New in Networking.

Our old pal <a href="https://twitter.com/markmirch" rel="nofollow" target="_blank">Mark Mirchandani</a> is back this week, joining <a href="https://twitter.com/stephr_wong" rel="nofollow" target="_blank">Stephanie Wong</a> and our guests <a href="https://twitter.com/stevemcghee" rel="nofollow" target="_blank">Steve McGhee</a> and <a href="https://twitter.com/YuriGrinshteyn" rel="nofollow" target="_blank">Yuri Grinshteyn</a> to talk about Site Reliability Engineering. SRE is Google’s way of helping companies of all sizes create consistent, predictable, and functional projects. It helps clients approach operations from a software engineering stand point so that growing systems can be managed efficiently. We talk about the challenges of implementing best SRE practices and how companies can overcome these. Though the benefits of SRE are many, it can be difficult for clients to grasp. Steve and Yuri tell us the process they go through with customers to help them set realistic goals and work to make reliable, scalable projects with little downtime. By starting small and taking wins early, Steve says clients reap the rewards of SRE and are encouraged to push forward. Yuri’s customer-centric approach encourages companies to prioritize alerts that affect the user experience, thus limiting inbox mayhem and keeping customers happy. Alerts based on symptoms, Steve says, help accomplish this goal. Later, Yuri and Steve describe the best ways for companies to get started with SRE. Realistic goals and specific detailed plans can make the journey less bumpy for clients, and Google’s SRE team can help. Steve McGhee <a href="https://twitter.com/stevemcghee" rel="nofollow" target="_blank">Steve</a> was an SRE at Google for about 10 years, then left to help a company build reliable systems on the Cloud. Now he’s back at Google, helping more companies do that. Yuri Grinshteyn <a href="https://twitter.com/YuriGrinshteyn" rel="nofollow" target="_blank">Yuri</a> works with Google Cloud Platform customers to help them design, architect, build, and operate reliable applications and services. He also advocates for SRE principles and practices on YouTube and elsewhere. Cool things of the week <ul> <li>Fresh updates: Google Cloud 2021 Summits <a href="https://cloud.google.com/blog/topics/events/news-updates-on-the-google-cloud-summit-digital-event-series-2021" rel="nofollow" target="_blank"> blog</a></li> <li>Why you need to explain machine learning models <a href="https://cloud.google.com/blog/products/ai-machine-learning/why-you-need-to-explain-machine-learning-models" rel="nofollow" target="_blank"> blog</a> <ul> <li>GCP Podcast Episode 260: Responsible AI with Craig Wiley and Tracy Frey <a href="https://www.gcppodcast.com/post/episode-260-responsible-ai-with-craig-wiley-and-tracy-frey/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 249: ML Lifecycle with Dale Markowitz and Craig Wiley <a href="https://www.gcppodcast.com/post/episode-249-ml-lifecycle-with-dale-markowitz-and-craig-wiley/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 214: AI in Healthcare with Dale Markowitz <a href="https://www.gcppodcast.com/post/episode-214-ai-in-healthcare-with-dale-markowitz/" rel="nofollow" target="_blank"> podcast</a></li> </ul> </li> </ul> Interview <ul> <li>Site Reliability Engineering <a href="https://sre.google" rel="nofollow" target="_blank">site</a></li> <li>Reliability Architecture Framework <a href="https://cloud.google.com/architecture/framework/reliability" rel="nofollow" target="_blank">site</a></li> <li>Site Reliability Engineering: Measuring and Managing Reliability on Coursera <a href="https://www.coursera.org/programs/google-specialization?collectionId=¤tTab=CATALOG&amp;productId=zMg9YpBtEeiWbhI9enrBdA&amp;productType=course&amp;showMiniModal=true" rel="nofollow" target="_blank"> site</a></li> <li>Developing a Google SRE Culture on Coursera <a href="https://www.coursera.org/programs/google-specialization?collectionId=¤tTab=CATALOG&amp;productId=JXTF3ajdEeq3YxIGYItKdQ&amp;productType=course&amp;showMiniModal=true" rel="nofollow" target="_blank"> site</a></li> <li>How Lowe’s meets customer demand with Google SRE practices <a href="https://cloud.google.com/blog/products/devops-sre/how-lowes-leverages-google-sre-practices" rel="nofollow" target="_blank"> blog</a></li> <li>GCP Podcast Episode 68: The Home Depot with William Bonnell <a href="https://www.gcppodcast.com/post/episode-68-the-home-depot-with-william-bonnell/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 213: The Art of SLOs with Alex Bramley <a href="https://www.gcppodcast.com/post/episode-213-the-art-of-slos-with-alex-bramley/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 127: SRE vs Devops with Liz Fong-Jones and Seth Vargo <a href="https://www.gcppodcast.com/post/episode-127-sre-vs-devops-with-liz-fong-jones-and-seth-vargo/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 72: Customer Reliability Engineering with Luke Stone <a href="https://www.gcppodcast.com/post/episode-72-customer-reliability-engineering-with-luke-stone/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 38: Site Reliability Engineering with Paul Newson <a href="https://www.gcppodcast.com/post/episode-38-site-reliability-engineering-with-paul-newson/" rel="nofollow" target="_blank"> podcast</a></li> <li>GCP Podcast Episode 59: SRE II with Paul Newson <a href="https://www.gcppodcast.com/post/episode-59-sre-ii-with-paul-newson/" rel="nofollow" target="_blank"> podcast</a></li> </ul> What’s something cool you’re working on? Yuri has been working on <a href="https://www.youtube.com/watch?v=U53wC2A75Is" rel="nofollow" target="_blank">Engineering for Reliability</a>. Stephanie has been working on her new series <a href="https://www.youtube.com/watch?v=lyLMyQ7vJUI&amp;list=PLIivdWyY5sqK_yw5KHsGVYd--ZCIoUwEM&amp;index=2" rel="nofollow" target="_blank"> What’s New in Networking</a>.

SRE III with Steve McGhee and Yuri Grinshteyn

The History of Alerting Yuri discusses the history of alerting and why alerting should be on things that impact the customer experience. Yuri: “It’s one of the soapboxes that I find myself on when talking to customers. It so often happens that customers will look for help with alerting; ‘How will I know if I am having an issue with my infrastructure, how will I know if I have high memory consumption?’ You should never alert on things in your infrastructure, you should only alert on things that impact your customer experience.” Yuri and Julie discuss common mistakes customers make dating back to the beginning of alerting and how products like PagerDuty have changed the way alerting should be done. A little bit about PagerDuty Yuri and Julie discuss where PagerDuty came from and where it is today. Yuri: “The fundamental problem that you are trying to solve, which is like hey I get 1,000 alerts an hour and I don’t know which ones are important. That’s not the problem we should be buying PagerDuty to solve, that’s the problem we should be addressing at the root.” Julie and Yuri continue to discuss the issues with email filtering and essentially “training people to ignore alerts” Service Level Indicator and Service Level Objective Usage Yuri talks about how it’s about the service owner and who is ultimately accountable for reliability of that system and what user happiness means in this context. He continues to discuss SLI’s and SLO’s. Yuri: “We use SLI’s as a proxy for user happiness.” Yuri and Julie discuss setting up alerting with SLI’s and SLO’s in mind along with alerts needing to be humanly actionable, with a little bit of error budgets sprinkled in. Yuri: “Things that are not directly contributing to or impacting user happiness, those should be created as tickets in a ticketing system… there is no need to wake someone in the middle of the night.” Continued discussion around when alerting should wake someone up in the middle of the night. Julie: “Making sure every alert that wakes a human up is humanly actionable sounds great but isn’t always easy, and it comes down to fine tuning. Do you have recommendations?” Yuri: “People often feel that if they don’t have an alert for it, it’s not actually happening..” Decrypting the Metrics Julie: “Let’s go deeper on how do we really dig deep into what the customer experience means when you are looking at service level indicators and service level objectives” Yuri: “You have to have a good understanding of what are people actually trying to do, and then some way of quantifying.” The discussion continues on what metrics we use to quantify customer success and performance. Yuri: “The closer we are able to collect this information to the customer, the more accurate it is going to be.” Technical Debt Shifting topics to how technical debt is expressed as a gap in knowledge, and how people treat their systems as a black box. Yuri: “You’ll hear the term ‘black box monitoring’ because they don’t actually know how it works.” Continued discussion on how technical debt manifests itself in monitoring. Service Ownership Yuri discusses what service ownership means to him and how it is really “engineer empowerment”, and what that means to reliability. Additional Resources  PagerDuty Home Page Episode transcribed by Rev

<h3>The History of Alerting</h3> Yuri discusses the history of alerting and why alerting should be on things that impact the customer experience. Yuri: “It’s one of the soapboxes that I find myself on when talking to customers. It so often happens that customers will look for help with alerting; ‘How will I know if I am having an issue with my infrastructure, how will I know if I have high memory consumption?’ You should never alert on things in your infrastructure, you should only alert on things that impact your customer experience.” Yuri and Julie discuss common mistakes customers make dating back to the beginning of alerting and how products like PagerDuty have changed the way alerting should be done. <h3>A little bit about PagerDuty</h3> Yuri and Julie discuss where PagerDuty came from and where it is today. Yuri: “The fundamental problem that you are trying to solve, which is like hey I get 1,000 alerts an hour and I don’t know which ones are important. That’s not the problem we should be buying PagerDuty to solve, that’s the problem we should be addressing at the root.” Julie and Yuri continue to discuss the issues with email filtering and essentially “training people to ignore alerts” <h3>Service Level Indicator and Service Level Objective Usage</h3> Yuri talks about how it’s about the service owner and who is ultimately accountable for reliability of that system and what user happiness means in this context. He continues to discuss SLI’s and SLO’s. Yuri: “We use SLI’s as a proxy for user happiness.” Yuri and Julie discuss setting up alerting with SLI’s and SLO’s in mind along with alerts needing to be humanly actionable, with a little bit of error budgets sprinkled in. Yuri: “Things that are not directly contributing to or impacting user happiness, those should be created as tickets in a ticketing system… there is no need to wake someone in the middle of the night.” Continued discussion around when alerting should wake someone up in the middle of the night. Julie: “Making sure every alert that wakes a human up is humanly actionable sounds great but isn’t always easy, and it comes down to fine tuning. Do you have recommendations?” Yuri: “People often feel that if they don’t have an alert for it, it’s not actually happening..” <h3>Decrypting the Metrics</h3> Julie: “Let’s go deeper on how do we really dig deep into what the customer experience means when you are looking at service level indicators and service level objectives” Yuri: “You have to have a good understanding of what are people actually trying to do, and then some way of quantifying.” The discussion continues on what metrics we use to quantify customer success and performance. Yuri: “The closer we are able to collect this information to the customer, the more accurate it is going to be.” <h3>Technical Debt</h3> Shifting topics to how technical debt is expressed as a gap in knowledge, and how people treat their systems as a black box. Yuri: “You’ll hear the term ‘black box monitoring’ because they don’t actually know how it works.” Continued discussion on how technical debt manifests itself in monitoring. <h3>Service Ownership</h3> Yuri discusses what service ownership means to him and how it is really “engineer empowerment”, and what that means to reliability. <h3>Additional Resources</h3> <ul> <li><a href="https://www.pagerduty.com/" rel="nofollow" target="_blank">PagerDuty Home Page</a></li> <li>Episode transcribed by <a href="https://www.rev.com" rel="nofollow" target="_blank">Rev</a></li></ul>

Smart Alerting and Service Level Objectives With Yuri Grinshteyn