AWS Morning Brief for the week of April 24, 2023 with Corey Quinn. Links: If you're around tomorrow night (Tuesday) at 6PM, I'll be at The Ramp in SF; let me buy you a drink.  Announcing Dev Environment dashboard for Amazon CodeCatalyst (Preview)  Amazon DynamoDB now supports up to 50 concurrent table restores Amazon EC2 supports Ubuntu Pro operating system in a subscription-included model  Amazon EFS now supports up to 10 GiB/s of throughput  Increased visibility of your carbon emissions data with AWS Customer Carbon Footprint Tool  Python 3.10 runtime now available in AWS Lambda  Understanding techniques to reduce AWS Lambda costs in serverless applications  Use Amazon DynamoDB global tables in DynamoDB Shell   Announcing General Availability of Amazon CodeCatalyst 

AWS Morning Brief for the week of April 24, 2023 with Corey Quinn. Links:<ul> <li>If you're around tomorrow night (Tuesday) at 6PM, I'll be at <a href="https://rampsf.com/" rel="nofollow" target="_blank">The Ramp in SF</a>; let me buy you a drink.</li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-codecatalyst-preview-dev-environment-dashboard/" rel="nofollow" target="_blank">Announcing Dev Environment dashboard for Amazon CodeCatalyst (Preview)</a> </li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-dynamodb-concurrent-table-restores/" rel="nofollow" target="_blank">Amazon DynamoDB now supports up to 50 concurrent table restores</a></li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-ec2-ubuntu-pro-subscription-model/" rel="nofollow" target="_blank">Amazon EC2 supports Ubuntu Pro operating system in a subscription-included model</a></li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-efs-10-gibs-throughput/" rel="nofollow" target="_blank">Amazon EFS now supports up to 10 GiB/s of throughput</a> </li> <li><a href="https://aws.amazon.com/blogs/aws-cloud-financial-management/increased-visibility-of-your-carbon-emissions-data-with-aws-customer-carbon-footprint-tool/" rel="nofollow" target="_blank">Increased visibility of your carbon emissions data with AWS Customer Carbon Footprint Tool</a></li> <li> <a href="https://aws.amazon.com/blogs/compute/python-3-10-runtime-now-available-in-aws-lambda/" rel="nofollow" target="_blank">Python 3.10 runtime now available in AWS Lambda</a> </li> <li><a href="https://aws.amazon.com/blogs/compute/understanding-techniques-to-reduce-aws-lambda-costs-in-serverless-applications/" rel="nofollow" target="_blank">Understanding techniques to reduce AWS Lambda costs in serverless applications</a></li> <li> <a href="https://aws.amazon.com/blogs/database/use-amazon-dynamodb-global-tables-in-dynamodb-shell/" rel="nofollow" target="_blank">Use Amazon DynamoDB global tables in DynamoDB Shell</a> </li> <li> <a href="https://aws.amazon.com/blogs/devops/announcing-general-availability-of-amazon-codecatalyst/" rel="nofollow" target="_blank">Announcing General Availability of Amazon CodeCatalyst</a> </li></ul>

RSA Beckons to Sell You a Firewall

Last week in security news: Creating an AWS Backup Account, Azure had another cross-tenant access vulnerability, Security Hub Hurts My Self-Esteem, and more!Links: Corey hosted a partner panel at AWS Container Day at KubeCon  This post on using OIDC to secure your CI/CD pipelines mirrors what I did with GitHub actions a year or so ago. Teri Radichel has a piece on Creating an AWS Backup Account  Slack is conducting an absolute masterclass in how to screw up messaging to your target audience. Azure had another cross-tenant access vulnerability  Security Hub Hurts My Self-Esteem AWS Security Profile: Matt Luttrell, Principal Solutions Architect for AWS Identity Tool of the Week: iamlive 

Last week in security news: Creating an AWS Backup Account, Azure had another cross-tenant access vulnerability, Security Hub Hurts My Self-Esteem, and more!Links:<ul> <li>Corey hosted a partner panel at <a href="http://snark.cloud/containerday" rel="nofollow" target="_blank">AWS Container Day</a> at KubeCon </li> <li>This post on <a href="https://jxel.dev/posts/using-oidc-to-secure-your-ci/cd-pipelines-no-more-long-lived-access-keys/" rel="nofollow" target="_blank">using OIDC</a> to secure your CI/CD pipelines mirrors what I did with GitHub actions a year or so ago.</li> <li>Teri Radichel has a piece on <a href="https://medium.com/cloud-security/creating-an-aws-backup-account-ea0d3c8e25b8" rel="nofollow" target="_blank">Creating an AWS Backup Account</a> </li> <li>Slack is conducting an <a href="https://twitter.com/QuinnyPig/status/1645921959841267713?s=20" rel="nofollow" target="_blank">absolute masterclass</a> in how to screw up messaging to your target audience.</li> <li>Azure had another <a href="https://www.cloudvulndb.org/data_gateway_rce" rel="nofollow" target="_blank">cross-tenant access vulnerability</a> </li> <li><a href="https://www.youtube.com/playlist?list=PLbQEBX5XUDb-zy3W_C9JVI_teDgEPKL2X" rel="nofollow" target="_blank">Security Hub Hurts My Self-Esteem</a></li> <li><a href="https://aws.amazon.com/blogs/security/aws-security-profile-matt-luttrell-principal-solutions-architect-for-aws-identity/" rel="nofollow" target="_blank">AWS Security Profile: Matt Luttrell, Principal Solutions Architect for AWS Identity</a></li> <li>Tool of the Week: <a href="https://github.com/iann0036/iamlive" rel="nofollow" target="_blank">iamlive</a> </li></ul>

Screwing Up the Messaging and Also the RSA Dates

AWS Morning Brief for the week of April 17, 2023 with Corey Quinn. This week is RSA in San Francisco; I'll be haunting the expo hall at some point, so if you're in town say hi.Links: The Last Week in AWS Job Board continues to thrive; thanks for your ongoing support. Amazon Chime SDK updates Service Level Agreement Amazon CodeWhisperer is now generally available Amazon Connect now enables agents to handle voice calls, chats, and tasks concurrently  Amazon EC2 Serial Console is now available on EC2 bare metal instances  Amazon RDS for MySQL now supports up to 15 read replicas for RDS Multi-AZ deployment option with two readable standby database instances  AWS Graviton2-based Amazon EC2 instances are available in additional regions  AWS Ground Station now supports Wideband Digital Intermediate Frequency  AWS Lambda adds support for Node.js 18 in the AWS GovCloud (US) Regions   Introducing AWS Lambda response streaming   Understanding Amazon DynamoDB latency  Announcing New Tools for Building with Generative AI on AWS  AWS Now Supports Credentials-fetcher for gMSA on Amazon Linux 2023   AWS investment in South Africa results in economic ripple effect   New Global AWS Data Processing Addendum   15 cool things we found inside the Spheres, Amazon’s urban rainforest in downtown Seattle 

AWS Morning Brief for the week of April 17, 2023 with Corey Quinn. This week is RSA in San Francisco; I'll be haunting the expo hall at some point, so if you're in town say hi.Links:<ul> <li>The <a href="https://jobs.lastweekinaws.com/" rel="nofollow" target="_blank">Last Week in AWS Job Board</a> continues to thrive; thanks for your ongoing support.</li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-chime-sdk-service-level-agreement/" rel="nofollow" target="_blank">Amazon Chime SDK updates Service Level Agreement</a></li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-codewhisperer-generally-available/" rel="nofollow" target="_blank">Amazon CodeWhisperer is now generally available</a></li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-connect-agents-calls-chats-tasks/" rel="nofollow" target="_blank">Amazon Connect now enables agents to handle voice calls, chats, and tasks concurrently</a></li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-ec2-serial-console-bare-metal-instances/?" rel="nofollow" target="_blank">Amazon EC2 Serial Console is now available on EC2 bare metal instances</a> </li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-rds-mysql-15-read-replicas-multi-az-two-readable-instances/" rel="nofollow" target="_blank">Amazon RDS for MySQL now supports up to 15 read replicas for RDS Multi-AZ deployment option with two readable standby database instances</a></li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/04/aws-graviton2-based-amazon-ec2-instances-additional-regions/" rel="nofollow" target="_blank">AWS Graviton2-based Amazon EC2 instances are available in additional regions</a> </li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/aws-ground-station-wideband-digital-intermediate-frequency/" rel="nofollow" target="_blank">AWS Ground Station now supports Wideband Digital Intermediate Frequency</a></li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/04/aws-lambda-node-js-18-govcloud-regions/" rel="nofollow" target="_blank">AWS Lambda adds support for Node.js 18 in the AWS GovCloud (US) Regions</a> </li> <li> <a href="https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/" rel="nofollow" target="_blank">Introducing AWS Lambda response streaming</a> </li> <li> <a href="https://aws.amazon.com/blogs/database/understanding-amazon-dynamodb-latency/" rel="nofollow" target="_blank">Understanding Amazon DynamoDB latency</a> </li> <li><a href="https://aws.amazon.com/blogs/machine-learning/announcing-new-tools-for-building-with-generative-ai-on-aws/" rel="nofollow" target="_blank">Announcing New Tools for Building with Generative AI on AWS</a></li> <li> <a href="https://aws.amazon.com/blogs/opensource/aws-now-supports-credentials-fetcher-for-gmsa-on-amazon-linux-2023/" rel="nofollow" target="_blank">AWS Now Supports Credentials-fetcher for gMSA on Amazon Linux 2023</a> </li> <li> <a href="https://aws.amazon.com/blogs/publicsector/aws-investment-south-africa-results-economic-ripple-effect/" rel="nofollow" target="_blank">AWS investment in South Africa results in economic ripple effect</a> </li> <li> <a href="https://aws.amazon.com/blogs/security/new-global-aws-data-processing-addendum/" rel="nofollow" target="_blank">New Global AWS Data Processing Addendum</a> </li> <li> <a href="https://www.aboutamazon.com/news/workplace/amazon-spheres-seattle-inside-tour" rel="nofollow" target="_blank">15 cool things we found inside the Spheres, Amazon’s urban rainforest in downtown Seattle</a> </li></ul>

Barest Metal Instances

Last week in security news: Logging strategies for security incident response, A Department of Energy report shows some rather serious gaps in security monitoring, A dedicated repository of winners of the S3 Bucket Negligence Awards, and more!Links: Zoom took an outage and the message was clearly AWS generated. Root cause? Misconfigured SCP. A Department of Energy report shows some rather serious gaps in the security monitoring of their cloud environments. Logging strategies for security incident response Reduce triage time for security investigations with Amazon Detective visualizations and export data TLS inspection configuration for encrypted traffic and AWS Network Firewall A dedicated repository of winners of the S3 Bucket Negligence Awards.

Last week in security news: Logging strategies for security incident response, A Department of Energy report shows some rather serious gaps in security monitoring, A dedicated repository of winners of the S3 Bucket Negligence Awards, and more!Links:<ul> <li>Zoom took an outage and the message was clearly AWS generated. Root cause? <a href="https://twitter.com/arkadiyt/status/1642983032314560513?s=20" rel="nofollow" target="_blank">Misconfigured SCP</a>.</li> <li>A <a href="https://www.energy.gov/ig/articles/audit-doe-oig-23-18" rel="nofollow" target="_blank">Department of Energy report</a> shows some rather serious gaps in the security monitoring of their cloud environments.</li> <li><a href="https://aws.amazon.com/blogs/security/logging-strategies-for-security-incident-response/" rel="nofollow" target="_blank">Logging strategies for security incident response</a></li> <li><a href="https://aws.amazon.com/blogs/security/reduce-triage-time-for-security-investigations-with-detective-visualizations-and-export-data/" rel="nofollow" target="_blank">Reduce triage time for security investigations with Amazon Detective visualizations and export data</a></li> <li><a href="https://aws.amazon.com/blogs/security/tls-inspection-configuration-for-encrypted-traffic-and-aws-network-firewall/" rel="nofollow" target="_blank">TLS inspection configuration for encrypted traffic and AWS Network Firewall</a></li> <li>A <a href="https://github.com/nagwww/s3-leaks" rel="nofollow" target="_blank">dedicated repository</a> of winners of the S3 Bucket Negligence Awards.</li></ul>

"A Quiet Week" He Says, Tempting Fate

AWS Morning Brief Extras edition for the week of April 12, 2023.Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/localstack-why-local-development-for-cloud-workloads-makes-senseNever miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts Buy our merch https://store.lastweekinaws.comWhat's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

AWS Morning Brief Extras edition for the week of April 12, 2023. Want to give your ears a break and read this as an article? You’re looking for this link.<a href="https://www.lastweekinaws.com/blog/localstack-why-local-development-for-cloud-workloads-makes-sense" rel="nofollow" target="_blank">https://www.lastweekinaws.com/blog/localstack-why-local-development-for-cloud-workloads-makes-sense</a> Never miss an episode<ul> <li><a href="https://lastweekinaws.com/" rel="nofollow" target="_blank">Join the Last Week in AWS newsletter</a></li> <li>Subscribe wherever you get your podcasts</li> </ul> Help the show<ul> <li><a href="https://lastweekinaws.com/review" rel="nofollow" target="_blank">Leave a review</a></li> <li><a href="https://lastweekinaws.com/feedback/" rel="nofollow" target="_blank">Share your feedback</a></li> <li>Subscribe wherever you get your podcasts</li> </ul> Buy our merch <ul><li><a href="https://store.lastweekinaws.com" rel="nofollow" target="_blank">https://store.lastweekinaws.com</a></li></ul> What's Corey up to?<ul> <li><a href="https://twitter.com/quinnypig/" rel="nofollow" target="_blank">Follow Corey on Twitter (@quinnypig)</a></li> <li><a href="https://duckbillgroup.com/clients/" rel="nofollow" target="_blank">See our recent work at the Duckbill Group</a></li> <li><a href="https://www.duckbillgroup.com/services/" rel="nofollow" target="_blank">Apply to work with Corey and the Duckbill Group to help lower your AWS bill</a></li> </ul>

LocalStack: Why Local Development for Cloud Workloads Makes Sense

AWS Morning Brief for the week of April 10, 2023 with Corey Quinn. Links: Console Toolbar is now generally available for AWS CloudShell Announcing CSV Export for AWS Resource Explorer Search Results Announcing Utilization Notifications for EC2 On-Demand Capacity Everything you need to know about AWS Billing Conductor’s new pricing model How to use Amazon CloudWatch to monitor Amazon DynamoDB table size and item count metrics Implement resource counters with Amazon DynamoDB AWS Organizations, moving an organization member account to another organization: Part 3  Build secure multi-account multi-VPC connectivity for your applications with Amazon VPC Lattice  Higher education cloud financial planning: A former CFO’s perspective How the Think Big for Small Business program helps small businesses win big contracts Amazon started passing out Small Business labels to giant companies. Perfect imperfections: how AWS is innovating on diamond materials for quantum communication with Element Six

AWS Morning Brief for the week of April 10, 2023 with Corey Quinn. Links:<ul> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/03/console-toolbar-generally-available-aws-cloudshell/" rel="nofollow" target="_blank">Console Toolbar is now generally available for AWS CloudShell</a></li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/csv-export-aws-resource-explorer-search-results/" rel="nofollow" target="_blank">Announcing CSV Export for AWS Resource Explorer Search Results</a></li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/04/utilization-notifications-ec2-on-demand-capacity-reservations/" rel="nofollow" target="_blank">Announcing Utilization Notifications for EC2 On-Demand Capacity</a></li> <li><a href="https://aws.amazon.com/blogs/aws-cloud-financial-management/ad-everything-you-need-to-know-about-aws-billing-conductors-new-pricing-model/" rel="nofollow" target="_blank">Everything you need to know about AWS Billing Conductor’s new pricing model</a></li> <li><a href="https://aws.amazon.com/blogs/database/how-to-use-amazon-cloudwatch-to-monitor-amazon-dynamodb-table-size-and-item-count-metrics/" rel="nofollow" target="_blank">How to use Amazon CloudWatch to monitor Amazon DynamoDB table size and item count metrics</a></li> <li><a href="https://aws.amazon.com/blogs/database/implement-resource-counters-with-amazon-dynamodb/" rel="nofollow" target="_blank">Implement resource counters with Amazon DynamoDB</a></li> <li><a href="https://aws.amazon.com/blogs/mt/aws-organizations-moving-an-organization-member-account-to-another-organization-part-3/" rel="nofollow" target="_blank">AWS Organizations, moving an organization member account to another organization: Part 3</a></li> <li> <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/build-secure-multi-account-multi-vpc-connectivity-for-your-applications-with-amazon-vpc-lattice/" rel="nofollow" target="_blank">Build secure multi-account multi-VPC connectivity for your applications with Amazon VPC Lattice</a> </li> <li><a href="https://aws.amazon.com/blogs/publicsector/higher-education-cloud-financial-planning-former-cfos-perspective/" rel="nofollow" target="_blank">Higher education cloud financial planning: A former CFO’s perspective</a></li> <li><a href="https://aws.amazon.com/blogs/publicsector/think-big-for-small-business-program-helps-small-businesses-win-big-contracts/" rel="nofollow" target="_blank">How the Think Big for Small Business program helps small businesses win big contracts</a></li> <li>Amazon started passing out <a href="https://www.theinformation.com/articles/amazon-hands-out-small-business-label-to-huge-sellers?rc=ez8msn" rel="nofollow" target="_blank">Small Business labels to giant companies</a>.</li> <li><a href="https://aws.amazon.com/blogs/quantum-computing/perfect-imperfections-how-aws-is-innovating-on-diamond-materials-for-quantum-communication-with-element-six/" rel="nofollow" target="_blank">Perfect imperfections: how AWS is innovating on diamond materials for quantum communication with Element Six</a></li></ul>

Your Network Bill is Now Diamonds

Last week in security news: Gain insights and knowledge at AWS re:Inforce 2023, InvalidClientTokenId, a repository of AWS customer breaches, and more!Links: If you're in New York City proper, I hope to see you tonight at 7PM at Vol de Nuit   We're hiring an Account Exec to handle media sales for this very podcast. Should you be the person who refers the successful candidate, we'll give you a $3K USD referral fee. Nick Frichette has found an undocumented Amplify API and used it to leak AWS Account IDs. Friend of the newsletter Chris Farris has started an AWS security consulting practice.  Gain insights and knowledge at AWS re:Inforce 2023  How to use Amazon GuardDuty and AWS WAF v2 to automatically block suspicious hosts  InvalidClientTokenId: The security token included in the request is invalid error Someone is curating this repository of AWS customer breaches.

Last week in security news: Gain insights and knowledge at AWS re:Inforce 2023, InvalidClientTokenId, a repository of AWS customer breaches, and more!Links:<ul> <li>If you're in New York City proper, I hope to see you tonight at 7PM at <a href="https://voldenuitbar.com/" rel="nofollow" target="_blank">Vol de Nuit</a> </li> <li> <a href="https://jobs.lastweekinaws.com/job/870429-account-executive-sponsorship-sales-the-duckbill-group" rel="nofollow" target="_blank">We're hiring an Account Exec</a> to handle media sales for this very podcast. Should you be the person who refers the successful candidate, we'll give you a $3K USD referral fee.</li> <li>Nick Frichette has found an <a href="https://frichetten.com/blog/undocumented-amplify-api-leak-account-id/" rel="nofollow" target="_blank">undocumented Amplify API</a> and used it to leak AWS Account IDs.</li> <li>Friend of the newsletter Chris Farris has <a href="https://www.chrisfarris.com/post/primeharbor/" rel="nofollow" target="_blank">started an AWS security consulting practice</a>.</li> <li> <a href="https://aws.amazon.com/blogs/security/gain-insights-and-knowledge-at-aws-reinforce-2023/" rel="nofollow" target="_blank">Gain insights and knowledge at AWS re:Inforce 2023</a> </li> <li><a href="https://aws.amazon.com/blogs/security/how-to-use-amazon-guardduty-and-aws-waf-v2-to-automatically-block-suspicious-hosts/" rel="nofollow" target="_blank">How to use Amazon GuardDuty and AWS WAF v2 to automatically block suspicious hosts</a></li> <li> <a href="https://blog.leapp.cloud/how-to-fix-aws-invalidclienttokenid-the-security-token-included-in-the-request-is-invalid" rel="nofollow" target="_blank">InvalidClientTokenId: The security token included in the request is invalid</a> error</li> <li>Someone is curating this repository of <a href="https://github.com/ramimac/aws-customer-security-incidents" rel="nofollow" target="_blank">AWS customer breaches</a>.</li></ul>

A Repository of AWS Customer Breaches

AWS Morning Brief for the week of April 3, 2023 with Corey Quinn. Links:  Amazon Kendra launches Featured Results   AWS Chatbot now supports search of AWS resources and AWS content   AWS Copilot adds support for full customization with AWS CDK or YAML overrides  AWS re:Post now includes AWS Knowledge Center articles New Cost Explorer users now get Cost Anomaly Detection by default Introducing Data on EKS – Modernize Data Workloads on Amazon EKS  Friend microservices using Amazon DynamoDB and event filtering 

AWS Morning Brief for the week of April 3, 2023 with Corey Quinn. Links:<ul> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-kendra-featured-results/" rel="nofollow" target="_blank">Amazon Kendra launches Featured Results</a> </li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/03/aws-chatbot-search-resources-content/" rel="nofollow" target="_blank">AWS Chatbot now supports search of AWS resources and AWS content</a> </li> <li> <a href="https://aws.amazon.com/about-aws/whats-new/2023/03/aws-copilot-customization-cdk-yaml-overrides/" rel="nofollow" target="_blank">AWS Copilot adds support for full customization with AWS CDK or YAML overrides</a> </li> <li><a href="https://aws.amazon.com/about-aws/whats-new/2023/03/aws-re-post-includes-knowledge-center-articles/" rel="nofollow" target="_blank">AWS re:Post now includes AWS Knowledge Center articles</a></li> <li><a href="https://aws.amazon.com/blogs/aws-cloud-financial-management/new-aws-cost-explorer-users-can-now-automatically-detect-cost-anomalies/" rel="nofollow" target="_blank">New Cost Explorer users now get Cost Anomaly Detection by default</a></li> <li><a href="https://aws.amazon.com/blogs/containers/introducing-data-on-eks-modernize-data-workloads-on-amazon-eks/" rel="nofollow" target="_blank">Introducing Data on EKS – Modernize Data Workloads on Amazon EKS</a></li> <li> <a href="https://aws.amazon.com/blogs/database/friend-microservices-using-amazon-dynamodb-and-event-filtering/" rel="nofollow" target="_blank">Friend microservices using Amazon DynamoDB and event filtering</a> </li></ul>

Friendship Started with Microservices

Last week in security news: Github accidentally published its RSA host keys for SSH, Automate IAM credential reports for large AWS Organizations, The Tool of the Week, and more!Links: Sad news; infosec luminary Kelly ‘Aloria’ Lum has regrettably passed away. Automate IAM credential reports for large AWS Organizations Github accidentally published its RSA host keys for SSH. How to use Amazon Macie to reduce the cost of discovering sensitive data Use backups to recover from security incidents Tool of the Week: Chekov 

Last week in security news: Github accidentally published its RSA host keys for SSH, Automate IAM credential reports for large AWS Organizations, The Tool of the Week, and more!Links:<ul> <li>Sad news; infosec luminary <a href="https://techcrunch.com/2023/03/22/kelly-aloria-lum-passes-away-at-41-obituary/" rel="nofollow" target="_blank">Kelly ‘Aloria’ Lum</a> has regrettably passed away.</li> <li><a href="https://aws.amazon.com/blogs/infrastructure-and-automation/automate-iam-credential-reports-for-large-aws-organizations/" rel="nofollow" target="_blank">Automate IAM credential reports for large AWS Organizations</a></li> <li><a href="https://www.theregister.com/2023/03/24/github_changes_its_ssh_host/" rel="nofollow" target="_blank">Github accidentally published its RSA host keys for SSH.</a></li> <li><a href="https://aws.amazon.com/blogs/security/how-to-use-amazon-macie-to-reduce-the-cost-of-discovering-sensitive-data/" rel="nofollow" target="_blank">How to use Amazon Macie to reduce the cost of discovering sensitive data</a></li> <li><a href="https://aws.amazon.com/blogs/security/use-backups-to-recover-from-security-incidents/" rel="nofollow" target="_blank">Use backups to recover from security incidents</a></li> <li>Tool of the Week: <a href="https://www.checkov.io/" rel="nofollow" target="_blank">Chekov</a> </li></ul>

 GitHub's Bad Key Week

AWS Morning Brief Extras edition for the week of March 29, 2023.Want to give your ears a break and read this as an article? You’re looking for this link.https://www.lastweekinaws.com/blog/s3-as-an-eternal-serviceNever miss an episode Join the Last Week in AWS newsletter Subscribe wherever you get your podcasts Help the show Leave a review Share your feedback Subscribe wherever you get your podcasts Buy our merch https://store.lastweekinaws.comWhat's Corey up to? Follow Corey on Twitter (@quinnypig) See our recent work at the Duckbill Group Apply to work with Corey and the Duckbill Group to help lower your AWS bill 

AWS Morning Brief Extras edition for the week of March 29, 2023. Want to give your ears a break and read this as an article? You’re looking for this link.<a href="https://www.lastweekinaws.com/blog/s3-as-an-eternal-service" rel="nofollow" target="_blank">https://www.lastweekinaws.com/blog/s3-as-an-eternal-service</a>Never miss an episode<ul> <li><a href="https://lastweekinaws.com/" rel="nofollow" target="_blank">Join the Last Week in AWS newsletter</a></li> <li>Subscribe wherever you get your podcasts</li> </ul> Help the show<ul> <li><a href="https://lastweekinaws.com/review" rel="nofollow" target="_blank">Leave a review</a></li> <li><a href="https://lastweekinaws.com/feedback/" rel="nofollow" target="_blank">Share your feedback</a></li> <li>Subscribe wherever you get your podcasts</li> </ul> Buy our merch <ul><li><a href="https://store.lastweekinaws.com" rel="nofollow" target="_blank">https://store.lastweekinaws.com</a></li></ul> What's Corey up to?<ul> <li><a href="https://twitter.com/quinnypig/" rel="nofollow" target="_blank">Follow Corey on Twitter (@quinnypig)</a></li> <li><a href="https://duckbillgroup.com/clients/" rel="nofollow" target="_blank">See our recent work at the Duckbill Group</a></li> <li><a href="https://www.duckbillgroup.com/services/" rel="nofollow" target="_blank">Apply to work with Corey and the Duckbill Group to help lower your AWS bill</a></li> </ul>

S3 as an Eternal Service

The latest in AWS news, sprinkled with snark. Posts about AWS come out over sixty times a day. We filter through it all to find the hidden gems, the community contributions--the stuff worth hearing about! Then we summarize it with snark and share it with you--minus the nonsense.

I spent a week looking for the “Last Week in A. W.S” until I realized that this podcast was incorrectly named. 10/10 would search again.

Corey cannot spell

If you’re looking for AWS updates without being your usual dry tech podcast. Then this podcast is for you. 

I truly enjoy Corey AWS insight and humor.

Failed comedian tries Cloud

AWS Morning Brief

AWS Morning Brief for the week of February 17, 2020.

EBS Gets Overly Multi-Attached

AWS Morning Brief for the week of March 9, 2020.

Nothing’s Certain but Death and Distinguished Engineers

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Hello and welcome to our Networking in the Cloud, mini series sponsored by ThousandEyes. That's right. There may be just one of you, but there are a thousand eyes on a more serious note. ThousandEyes has sponsored their cloud performance benchmarking report for 2019 at the end of last year. Talking about what it looks like when you race various cloud providers. They looked at all the big cloud providers and determined what does performance look like from an end user perspective? What does the user experience look like among and between different cloud providers? To get your copy of this report, you can visit snark.cloud/realclouds. Why real clouds? Well, because they raced AWS, Azure, GCP, IBM Cloud and Alibaba, all of which are real clouds.They did not include Oracle Cloud because once again they are real clouds. Check out your copy of the report at snark.cloud/realclouds. It's interesting that that report focuses on the end user experience because as this mini series begins to wind down, we're talking today about the last mile and its impact on what perceived cloud performance looks like. And I will admit that even having given this entire mini series and having a bit of a network engineering background, once upon a time, I still wind up in a fun world of always defaulting to blaming my local crappy ISP.Now today, my local ISP is amazing. I use Sonic in San Francisco. I get Symmetric Gigabit. It's the exact opposite of Comcast who was my last provider until Sonic came to my neighborhood and it was fun that day because I looked up and down the block and saw no fewer than six Sonic trucks ripping Comcast out by the short and curlies. Which let's not kid ourselves, is something we all wish we could do and I was the happiest boy in town the day I got to do it. Now, the hard part is figuring out that yes, it is in fact a local ISP problem because it isn't always. This is also fortuitous because I spent the last month or so fixing my own local internet situation and today I'd like to tell you a little bit more about that as well as how and why.Originally when I first moved into my roughly, we'll call it 2,800 square foot house, it's spread across three stories, I wound up getting EEROs, that's E-E-R-O. They're a mesh network set up that was acquired by Amazon after I'd purchased them. These are generation one. The wireless environment in San Francisco is challenging and in certain parts of my house, the reception as a result, wound up being a steaming bowl of horse crap. The big challenge was figuring out that, that's what the problem was. With weird dropouts and handoff issues, it was interesting. This one area that caused immediate improvement was not having these things talk to each other wirelessly as most full mesh systems will do, but instead making sure that they were cabled up appropriately to a switch, the central patch panel and then hooked them in. Now you have to be careful with switches because a lot of stuff won't do anything approaching full throughput because that can get expensive and a lot of consumer gear is crap.This was a managed HP pro curved device back in the days that HP made networking equipment. That was great. And it's still crap, but it is crap that works at full line rate. So there's that. Next I wound up figuring that ... all right, it's time to take this seriously. So I did some research and talked to people I know who are actually good at things, instead of sounding on the internet like they're good at things. And I figured the next step was to buy some Ubiquiti Networks style stuff. Great. We go ahead and trot some of that out. It's an enterprise gear. It's full mesh. I of course now have a guest wifi that you have to pay for to use the hotspot. It's called Toss a coin to your wifi for an SS ID because of course it is. I have problems. And it's fun and I can play these stupid games, but suddenly every weird internet problem I had in my house started getting better as a result.And it's astonishing how that changed my perception of various third party services. None of whom, by the way, had anything to do with my actual problem. But there were still some perceptual differences. And this impacts the cloud in a number of subtle ways and that's what I want to talk about today. So one of the biggest impacts is DNS. And I don't mean that in the sense of big cloud provider DNS, we've already talked about how DNS works in a previous episode. But rather what resolver you wind up using yourself. One of the things that I did as a part of this upgrade, is I rolled out a distribution of Linux called Pi-hole, which sounds incredibly insulting as applied to people as in, you know what, you should shut? Your Pi-hole. However, it's designed to run on top of Raspberry Pi and provide a DNS server that creatively blocks ads.And that's super neat. I liked the idea of just blocking ad servers, but you have to trust whatever you're using for a DNS resolver because of a few specific use cases that I stumbled over as I went down this process. One, it turns out that having access to every website you'd care to visit as far as a list of things you've been doing, is not really the most privacy conscious thing in the universe. Now, for some reason, the internet collectively decided, you know who we trust with all the things that we look at on the internet and have no worries about giving that information to? That's right. Freaking Google. So eight dot eight dot eight dot eight, was a famously to remember open resolver and it works super well. It's quick. It returns everything. The problem is, is that Google's primary business model is very clearly surveillance and I don't do anything particularly interesting.If you look at my DNS history, you're going to find a lot of things that you'd think you could use to blackmail me, but it turns out you actually can't because I talk about them on podcasts. That's right. I use Route 53 as a database. What of it? And it's all very strange just as far as even without anything to hide, I still feel this sense of pervasive creepiness at the idea that a giant company can look at this. Can look at my previous browsing history. So blocking things like that are of interest to me. So okay, instead, if I run Pi-hole that acts as my own resolver but then it winds up passing queries on to an upstream provider. I mean I could run my own, but that has other latency concerns and DNS latency when you're making requests is super indicative because the entire internet has gone collectively dumb. And decided to display a simple static webpage, You need to make 30 distinct DNS request in series and wait for them all to come back and other ridiculous nonsense that is the modern web today.What makes this extra special is I figured out, okay, I'm not going to go with Google or CloudFlare because it has other problematic aspects to its business that we need not go into here, but okay, we'll pick Level 3 as another example. And Level 3 was recently sold to CenturyLink and they are, to be polite the devil because they break DNS in horrifying and obscene ways. Namely, whenever something isn't found, it returns a result to its own personalized search engine, which means that anything that you have that is depending upon certain ... an X domain or no resolution available results, suddenly has a result returned. Well, you can set a cookie in your browser to wind up avoiding that behavior so we don't see what the problem is.The problem jack hole is that I can't set cookies in a DNS system that is querying it from a Daymond perspective. A whole bunch of email block lists software runs based upon DNS results and when you suddenly have results returning for everything, they either block nothing or they block everything to taste. And that winds up being obnoxious and difficult. Now, I'll talk to you a little bit more about how this impacts the cloud in just a second. But first I would like to thank once again ThousandEyes for making this entire ridiculous rant possible. In addition to their cloud performance benchmark report, ThousandEyes winds up giving companies insight into what's going on, on the broader internet routing issues, provider failures upstream, different companies having different problems. It more or less is a real time traffic meets weather map for the internet. This helps companies who use them wind up getting a better perspective of what the current end user experience is and begin routing around provider failures, yelling at providers, et cetera, Ideally before those errors become evident to customers.To learn more, visit ThousandEyes.com and tell them Cory sent you. In fact, they may very well say something like, "Wow, you heard about us from Corey and you still looked us up. What a Testament to how awesome our product is." My thanks again to ThousandEyes for putting up with my ridiculous nonsense to sponsor this ridiculous podcast. Now I hate running hardware mostly because I have a knack for breaking things, so when I ran the Pi-hole, sorry, I still can't take that name seriously. I didn't want to run it at an actual Raspberry or anything locally myself that I could accidentally spill water into. If you take a look at my previous history with laptops and liquids, you can understand why I have that perspective. So instead I decided I wanted to just run it in an EC2 instance because "Hey, those credits aren't going to spend themselves."This is odd because normally when I'm trying to build something in the cloud, I put it in either Oregon or Virginia. For this thing, I parked it in the more expensive capacity constrained region known as US West One because it is close to Northern California where I personally call home. The latency was far lower, which means for every round trip to talk to a DNS server over a VPN because I'm not an irresponsible moron, it takes less time. Which leads to a faster perceived user experience. Now, am I saying that automatically having a closer DNS server is going to lead to a faster user experience? Not necessarily. And depending entirely upon the profile of what application you're interacting with, you're going to see mixed results, but generally speaking, a faster responding DNS server is going to have a positive overall experience on what it's like to be using online web applications for whoever you happen to be and whatever you tend to be accessing.So having a resolver that is closer to me was important, not important enough for me to actually have a Raspberry PI or something like that here that I have to feed, maintain, watch get compromised, turn into an attack platform, and destroy my television slash scale which are currently busy monitoring a DDoS attack against the entire internet infrastructure because everything is terrible. Now this is an optimization that can lead to a better experience on the end user side, but ... and this is where I think a lot of cloud providers get things hilariously wrong, it's usually not latency from a DNS server or latency between where your customer is and the resource that they're speaking to. It's that most applications, be they web app or otherwise are designed like complete crap. They have a series of different connections to different hosts that need to all be contacted, resolved correctly, and transfer data to render a page correctly.One of the best examples we saw of this was when GDPR first came out and rather than a justice for folks who are admitting from Europe, a bunch of websites just dropped all the tracking, all the garbage, and you could see for a while, sites like the Los Angeles Times would load in 20k and fractions of a second instead of the 30 megabyte monstrosities we see now. If you take a look at any given webpage that loads today, it's enormous and it's crappy and having faster DNS resolution times is helpful, sure. But because you're straining raw sewage with your teeth when dealing with the internet, it's worse than just showing you ads. It's tracking behavior, it's malware shoved under ad networks. If you take a look at any given website property that I have with last week and AWS with this podcast, et cetera, yes, I sell ads.That's normal. What I don't do is track the living hell out of my customers. If you wind up visiting the website from a browser that has significant ad blocking turned on like I do, you don't get a degraded experience. You don't see a whole host of things getting destroyed and it's still not perfect. I still think we're pulling CSS from third party hosted sites, which is monstrous and obnoxious to me, but I'm not a web designer. I pay other people for that problem. In conclusion, yes, improving your local environment and your crappy ISP to a better ISP is going to lead to better outcomes. But there's so much else that can be done before getting to that point. And I think collectively, we missed the boat a lot when we talk about how performance is a cloud providers ultimate responsibility, yada, yada, yada. No, it's ours.Stop designing things like crap and start treating customers with a modicum of respect and I suspect that things will become a lot better for folks who are creating interesting content and sharing useful things. Sure, the rent seekers are going to have different problems, but I don't really care what happens to them. I don't mean to be unkind, but there we have it. In conclusion, the last mile matters a lot, partially because it is the experience filter through which everything we do on the internet matters to customers and partially because so many of these things treat it completely like garbage. That's it for this episode. I am Corey Quinn. This is our Networking in the Cloud mini series of the AWS morning brief. For comments, feedback, et cetera, you can find me on Twitter @Quinnypig. Q-U-I, double N-Y pig. Or make sure to leave a five star review for this episode regardless of what you actually thought of it and tell me exactly what my problem is in the comments. Thanks again to ThousandEyes for their continuing and ongoing support of this ridiculous podcast.Announcer: This has been a HumblePod production. Stay Humble.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at <a href="https://www.lastweekinaws.com/" rel="nofollow" target="_blank">The Duckbill Group</a>, the author of the weekly <a href="https://www.lastweekinaws.com/newsletter/" rel="nofollow" target="_blank">Last Week in AWS</a> newsletter, and the host of two podcasts: <a href="https://www.screaminginthecloud.com" rel="nofollow" target="_blank">Screaming in the Cloud</a> and, you guessed it, <a href="https://www.lastweekinaws.com/podcast/aws-morning-brief/" rel="nofollow" target="_blank">AWS Morning Brief</a>, which you’re about to listen to.TranscriptCorey: Hello and welcome to our Networking in the Cloud, mini series sponsored by <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a>. That's right. There may be just one of you, but there are a thousand eyes on a more serious note. <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> has sponsored their cloud performance benchmarking report for 2019 at the end of last year. Talking about what it looks like when you race various cloud providers. They looked at all the big cloud providers and determined what does performance look like from an end user perspective? What does the user experience look like among and between different cloud providers? To get your copy of this report, you can visit <a href="https://snark.cloud/realclouds" rel="nofollow" target="_blank">snark.cloud/realclouds</a>. Why real clouds? Well, because they raced AWS, Azure, GCP, IBM Cloud and Alibaba, all of which are real clouds. They did not include Oracle Cloud because once again they are real clouds. Check out your copy of the report at snark.cloud/realclouds. It's interesting that that report focuses on the end user experience because as this mini series begins to wind down, we're talking today about the last mile and its impact on what perceived cloud performance looks like. And I will admit that even having given this entire mini series and having a bit of a network engineering background, once upon a time, I still wind up in a fun world of always defaulting to blaming my local crappy ISP. Now today, my local ISP is amazing. I use Sonic in San Francisco. I get Symmetric Gigabit. It's the exact opposite of Comcast who was my last provider until Sonic came to my neighborhood and it was fun that day because I looked up and down the block and saw no fewer than six Sonic trucks ripping Comcast out by the short and curlies. Which let's not kid ourselves, is something we all wish we could do and I was the happiest boy in town the day I got to do it. Now, the hard part is figuring out that yes, it is in fact a local ISP problem because it isn't always. This is also fortuitous because I spent the last month or so fixing my own local internet situation and today I'd like to tell you a little bit more about that as well as how and why. Originally when I first moved into my roughly, we'll call it 2,800 square foot house, it's spread across three stories, I wound up getting EEROs, that's E-E-R-O. They're a mesh network set up that was acquired by Amazon after I'd purchased them. These are generation one. The wireless environment in San Francisco is challenging and in certain parts of my house, the reception as a result, wound up being a steaming bowl of horse crap. The big challenge was figuring out that, that's what the problem was. With weird dropouts and handoff issues, it was interesting. This one area that caused immediate improvement was not having these things talk to each other wirelessly as most full mesh systems will do, but instead making sure that they were cabled up appropriately to a switch, the central patch panel and then hooked them in. Now you have to be careful with switches because a lot of stuff won't do anything approaching full throughput because that can get expensive and a lot of consumer gear is crap. This was a managed HP pro curved device back in the days that HP made networking equipment. That was great. And it's still crap, but it is crap that works at full line rate. So there's that. Next I wound up figuring that ... all right, it's time to take this seriously. So I did some research and talked to people I know who are actually good at things, instead of sounding on the internet like they're good at things. And I figured the next step was to buy some Ubiquiti Networks style stuff. Great. We go ahead and trot some of that out. It's an enterprise gear. It's full mesh. I of course now have a guest wifi that you have to pay for to use the hotspot. It's called Toss a coin to your wifi for an SS ID because of course it is. I have problems. And it's fun and I can play these stupid games, but suddenly every weird internet problem I had in my house started getting better as a result. And it's astonishing how that changed my perception of various third party services. None of whom, by the way, had anything to do with my actual problem. But there were still some perceptual differences. And this impacts the cloud in a number of subtle ways and that's what I want to talk about today. So one of the biggest impacts is DNS. And I don't mean that in the sense of big cloud provider DNS, we've already talked about how DNS works in a previous episode. But rather what resolver you wind up using yourself. One of the things that I did as a part of this upgrade, is I rolled out a distribution of Linux called Pi-hole, which sounds incredibly insulting as applied to people as in, you know what, you should shut? Your Pi-hole. However, it's designed to run on top of Raspberry Pi and provide a DNS server that creatively blocks ads. And that's super neat. I liked the idea of just blocking ad servers, but you have to trust whatever you're using for a DNS resolver because of a few specific use cases that I stumbled over as I went down this process. One, it turns out that having access to every website you'd care to visit as far as a list of things you've been doing, is not really the most privacy conscious thing in the universe. Now, for some reason, the internet collectively decided, you know who we trust with all the things that we look at on the internet and have no worries about giving that information to? That's right. Freaking Google. So eight dot eight dot eight dot eight, was a famously to remember open resolver and it works super well. It's quick. It returns everything. The problem is, is that Google's primary business model is very clearly surveillance and I don't do anything particularly interesting. If you look at my DNS history, you're going to find a lot of things that you'd think you could use to blackmail me, but it turns out you actually can't because I talk about them on podcasts. That's right. I use Route 53 as a database. What of it? And it's all very strange just as far as even without anything to hide, I still feel this sense of pervasive creepiness at the idea that a giant company can look at this. Can look at my previous browsing history. So blocking things like that are of interest to me. So okay, instead, if I run Pi-hole that acts as my own resolver but then it winds up passing queries on to an upstream provider. I mean I could run my own, but that has other latency concerns and DNS latency when you're making requests is super indicative because the entire internet has gone collectively dumb. And decided to display a simple static webpage, You need to make 30 distinct DNS request in series and wait for them all to come back and other ridiculous nonsense that is the modern web today. What makes this extra special is I figured out, okay, I'm not going to go with Google or CloudFlare because it has other problematic aspects to its business that we need not go into here, but okay, we'll pick Level 3 as another example. And Level 3 was recently sold to CenturyLink and they are, to be polite the devil because they break DNS in horrifying and obscene ways. Namely, whenever something isn't found, it returns a result to its own personalized search engine, which means that anything that you have that is depending upon certain ... an X domain or no resolution available results, suddenly has a result returned. Well, you can set a cookie in your browser to wind up avoiding that behavior so we don't see what the problem is. The problem jack hole is that I can't set cookies in a DNS system that is querying it from a Daymond perspective. A whole bunch of email block lists software runs based upon DNS results and when you suddenly have results returning for everything, they either block nothing or they block everything to taste. And that winds up being obnoxious and difficult. Now, I'll talk to you a little bit more about how this impacts the cloud in just a second.  But first I would like to thank once again <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> for making this entire ridiculous rant possible. In addition to their cloud performance benchmark report, <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> winds up giving companies insight into what's going on, on the broader internet routing issues, provider failures upstream, different companies having different problems. It more or less is a real time traffic meets weather map for the internet. This helps companies who use them wind up getting a better perspective of what the current end user experience is and begin routing around provider failures, yelling at providers, et cetera, Ideally before those errors become evident to customers. To learn more, visit <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes.com</a> and tell them Cory sent you. In fact, they may very well say something like, "Wow, you heard about us from Corey and you still looked us up. What a Testament to how awesome our product is." My thanks again to <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> for putting up with my ridiculous nonsense to sponsor this ridiculous podcast. Now I hate running hardware mostly because I have a knack for breaking things, so when I ran the Pi-hole, sorry, I still can't take that name seriously. I didn't want to run it at an actual Raspberry or anything locally myself that I could accidentally spill water into. If you take a look at my previous history with laptops and liquids, you can understand why I have that perspective. So instead I decided I wanted to just run it in an EC2 instance because "Hey, those credits aren't going to spend themselves." This is odd because normally when I'm trying to build something in the cloud, I put it in either Oregon or Virginia. For this thing, I parked it in the more expensive capacity constrained region known as US West One because it is close to Northern California where I personally call home. The latency was far lower, which means for every round trip to talk to a DNS server over a VPN because I'm not an irresponsible moron, it takes less time. Which leads to a faster perceived user experience. Now, am I saying that automatically having a closer DNS server is going to lead to a faster user experience? Not necessarily. And depending entirely upon the profile of what application you're interacting with, you're going to see mixed results, but generally speaking, a faster responding DNS server is going to have a positive overall experience on what it's like to be using online web applications for whoever you happen to be and whatever you tend to be accessing. So having a resolver that is closer to me was important, not important enough for me to actually have a Raspberry PI or something like that here that I have to feed, maintain, watch get compromised, turn into an attack platform, and destroy my television slash scale which are currently busy monitoring a DDoS attack against the entire internet infrastructure because everything is terrible. Now this is an optimization that can lead to a better experience on the end user side, but ... and this is where I think a lot of cloud providers get things hilariously wrong, it's usually not latency from a DNS server or latency between where your customer is and the resource that they're speaking to. It's that most applications, be they web app or otherwise are designed like complete crap. They have a series of different connections to different hosts that need to all be contacted, resolved correctly, and transfer data to render a page correctly. One of the best examples we saw of this was when GDPR first came out and rather than a justice for folks who are admitting from Europe, a bunch of websites just dropped all the tracking, all the garbage, and you could see for a while, sites like the Los Angeles Times would load in 20k and fractions of a second instead of the 30 megabyte monstrosities we see now. If you take a look at any given webpage that loads today, it's enormous and it's crappy and having faster DNS resolution times is helpful, sure. But because you're straining raw sewage with your teeth when dealing with the internet, it's worse than just showing you ads. It's tracking behavior, it's malware shoved under ad networks. If you take a look at any given website property that I have with last week and AWS with this podcast, et cetera, yes, I sell ads. That's normal. What I don't do is track the living hell out of my customers. If you wind up visiting the website from a browser that has significant ad blocking turned on like I do, you don't get a degraded experience. You don't see a whole host of things getting destroyed and it's still not perfect. I still think we're pulling CSS from third party hosted sites, which is monstrous and obnoxious to me, but I'm not a web designer. I pay other people for that problem. In conclusion, yes, improving your local environment and your crappy ISP to a better ISP is going to lead to better outcomes. But there's so much else that can be done before getting to that point. And I think collectively, we missed the boat a lot when we talk about how performance is a cloud providers ultimate responsibility, yada, yada, yada. No, it's ours. Stop designing things like crap and start treating customers with a modicum of respect and I suspect that things will become a lot better for folks who are creating interesting content and sharing useful things. Sure, the rent seekers are going to have different problems, but I don't really care what happens to them. I don't mean to be unkind, but there we have it. In conclusion, the last mile matters a lot, partially because it is the experience filter through which everything we do on the internet matters to customers and partially because so many of these things treat it completely like garbage. That's it for this episode. I am Corey Quinn. This is our Networking in the Cloud mini series of the AWS morning brief. For comments, feedback, et cetera, you can find me on Twitter <a href="https://twitter.com/QuinnyPig" rel="nofollow" target="_blank">@Quinnypig</a>. Q-U-I, double N-Y pig. Or make sure to leave a five star review for this episode regardless of what you actually thought of it and tell me exactly what my problem is in the comments. Thanks again to <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> for their continuing and ongoing support of this ridiculous podcast. Announcer: This has been a <a href="http://humblepod.com/" rel="nofollow" target="_blank">HumblePod</a> production. Stay Humble.

Networking in the Cloud Fundamentals: Cloud and the Last Mile

AWS Morning Brief for the week of December 16th, 2019.

AWS Dep-Ric-ates Treasured Offering

AWS Morning Brief for the week of January 13th, 2020.

Your Database Will Explode in Sixty Seconds

AWS Morning Brief for the week of March 16, 2020.

The Saddest Kubernetes Hanukkah

AWS Morning Brief for the week of December 2nd, 2019.

Wherever You May Rome

AWS Morning Brief for the week of December 30th, 2019.

Listener Mailbag

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Hello, and welcome back to our Networking in the Cloud mini series sponsored by ThousandEyes. That's right. ThousandEyes is state-of-the-cloud Performance Benchmark Report is now available for your perusal. It's really providing a lot of baseline that we're taking all of the miniseries information from. It pointed us in a bunch of interesting directions and helps us tell stories that are actually, for a change, backed by data rather than pure sarcasm. To get your copy, visit snark.cloud/realclouds because it only covers real cloud providers. Thanks again to ThousandEyes for their ridiculous support of this shockingly informative podcast mini series.It's a basic fact of cloud that things break all the time. I've been joking for a while that a big competitive advantage that Microsoft brings to this space is that they have 40 years of experience apologizing for software failures, except that's not really a joke. It's true. There's something to be said for the idea of apologizing to both technical and business people about real or perceived failures being its own skillset, and they have a lot more experience than anyone in this space.There are two schools of thought around how to avoid having to apologize for service or component failures to your customers. The first is to build super expensive but super durable things, and you can kind of get away with this in typical data center environments right up until you can't, and then it turns out that your SAN just exploded. You're really not diversifying with most SANs. You're just putting all of your eggs in a really expensive basket, and of course, if you're still with power or networking outage, nothing can talk to the SAN, and you're back to square one.The other approach is to come at it with a perspective of building in redundancy to everything and eliminating single points of failure. That's usually the better path in the cloud. You don't ever want to have a single point of failure if you can reasonably avoid it, so going with multiple everythings starts to make sense to a point. Going with a full on multi-cloud story is a whole separate kettle of nonsense we'll get to another time. But you realize at some point you will have single points of failure and you're not going to be able to solve for that. We still only have one planet going around one sun for example. If either of those things explode, well, computers aren't really anyone's concern anymore. However, betting the entire farm on one EC2 instance is generally something you'll want to avoid if at all possible.In the world of AWS, there aren't data centers in the way that you or I would contextualize them. Instead, they have constructs known as availability zones and those composed to form a different construct called regions. Presumably, other cloud providers have similar constructs over in non-AWS land, but we're focusing on AWS as implementation in this series, again, because they have a giant multi-year head start over every other cloud provider, and even that manifests in those other cloud providers comparing what they've built and how they operate to AWS. If that upsets you and you work at one of those other cloud providers, well, you should have tried harder. Let's dive in to a discussion of data centers, availability zones, and regions today.Take an empty warehouse and shove it full of server racks. Congratulations. You have built the bare minimum requirement for a data center at its most basic layer. Your primary constraint and why it's a lot harder than it sounds is power, and to a lesser extent, cooling. Computers aren't just crunching numbers, they're also throwing off waste heat. You've got to think an awful lot about how to keep that heat out of the data center.At some point, you can't shove more capacity into that warehouse-style building just because you can't cool it if it's all running at the same time. If your data center's particularly robust, meaning you didn't cheap out on it, you're going to have different power distribution substations that feed the building from different lines that enter the building at different corners. You're going to see similar things with cooling as well, multiply redundant cooling systems.One of the big challenges, of course, when dealing with this physical infrastructure is validating that what it says on the diagram is what's actually there in the physical environment. That can be a trickier thing to explore than you would hope. Also, if you have a whole bunch of systems sitting in that warehouse and you take a power outage, well, you have to plan for this thing known as inrush current.Normally, it's steady state. Computers generally draw a known quantity of power. But when you first turn them on, if you've ever dealt with data center servers, the first thing they do is they power up everything to self-test. They sound like a jet fighter taking off as all the fans spin up. If you're not careful, and all these things turn on at once, you'll see a giant power spike that winds up causing issues, blowing breakers, maxing out consumption, so having a staggered start becomes a concern as well. Having spent too much time in data centers, I am painfully familiar with this problem of how you safely and sanely recover from site-wide events, but that's a bit out of scope, thankfully, because in the cloud, this is less of a problem.Let's talk about the internet and getting connectivity to these things. This is the Networking in the Cloud podcast after all. You're ideally going to have multiple providers running fiber lines to that data center hoping to avoid fiber's natural predator, the noble backhoe. Now, ideally, all those fiber lines are going over different paths, but again, hard thing to prove, so doing your homework's important, but here's something folks don't always consider: If you have a hundred gigabit ethernet links to each computer, which is not cheap, but doable, and then you have 20 servers in a rack, each rack theoretically needs to be able to speak at least two terabit at all times to each other server in each other rack, and most of them can't do that. They wind up having bottle-necking issues.As a result, when you have high-traffic applications speaking between systems, you need to make sure that they're aware of something known as rack affinity. In other words, are there bottlenecks between these systems, and how do you minimize those to make sure the crosstalk works responsibly? There are a lot of dragons in here, but let's hand-wave past all of it because we're talking about cloud here. The point of this is that there's an awful lot of nuance to running data centers, and AWS and other large cloud providers do a better job of it than you do. That's not me insulting your data center staff. That's just a fact. They have the scale and the staff and the expertise of running these things operationally that very few other companies are going to be able to touch.Sure, if you're Facebook, you probably have some expertise in this as well, and a lot of this won't apply to you, but you know that already. If you're wondering whether what I'm talking about here applies to your environment, unless you know for a fact it doesn't, it does. Assume that. This impacts your approach in the cloud, the networking, durability, and the concept of blast radius, and the forms that AWS gives us that wrap these concepts for us are twofold, and I want to cover them today: regions and availability zones.An availability zone, or A-Zee, or A-Zed if you're not in the United States, is effectively a set of data centers, and yes, that's plural. It's not just different racks with different power buses in the same room. AWS tries to guarantee that there is no shared power, network, or control plane between availability zones, but you can expect some issues to impact an entire availability zone as a result. Ergo, if you're building something important, you're going to want it to be in at least multiple availability zones.To that end, here's a fun fact that trips up nearly everyone the first time they see it. If you have an AWS account, you might see that there's an outage in a particular availability zone, us-west-2a. Meanwhile, in my account, I see an outage in us-west-2c. Who's right in that scenario? Well, we both are because availability zone names aren't consistent between AWS accounts.Relatively recently, about a year ago as of this recording, they announced a zone ID that is consistent between accounts, but people still don't talk about it in those terms. They're still talking with the old style region us-west-2 followed by a letter for the availability zone. You still have to disambiguate those back to zone IDs with an extra step. That also doesn't solve the problem for you because note that even with indirect issues that you're seeing, they can still impact other availability zones, even with a completely separate control plane because if you have things that are running in two availability zones for your application, and one of those availability zones drops off the internet for a while, you're suddenly seeing twice the load in the availability zone that's still working. You're also probably not the only customer that has planned for this and has built out in multiple availability zones, so other folks are going to be seeing the exact same type of behavior.As a result, failures will then cascade and manifest as slow performance in the good availability zones, and it's super hard to plan for. It's also super hard to detect, which brings us back to our sponsor, ThousandEyes. ThousandEyes provides a global observer perspective on what's going on internet-wide with a bunch of different providers. It helps answer the question when one of these incidents hits of "is it my code, is it the last deployment that we did, or is there something global that's causing this problem?"ThousandEyes provides that global observer perspective that helps you figure out immediately was it your code or was it something infrastructure-based because if it's not your code and it is infrastructure-based, suddenly, you can stop looking at everything you just shipped to production and instead look at mitigating this according to an established DR plan. Thanks again to ThousandEyes for sponsoring this. To learn more, visit thousandeyes.com, and tell them Corey sent you. They seem to like me for some reason that we really can't tell.You finally build something that's in multiple availability zones, but as mentioned, that cascade effect can be a challenge, so this is where we get into the idea of multiple regions. A region is two or more availability zones, usually three or more, but there are some legacy stories behind that, and those are separated by very large distances. In the United States, for example, there's one in Oregon, one in Ohio, one in Virginia, and one, sort of, in Northern California. The challenge, of course, is building applications that work spanning multiple regions, and there are a couple of issues with this. If you're trying to tilt at the windmill of multi-cloud, I would strongly encourage you to start by going multiple region in a single provider first.This removes a lot of the finicky bits of multi-cloud, like the services work differently, and if you pick the right regions, you'll have a one-to-one affinity between all of the different services, and that's awesome. Once you wind up getting multiple regions online, then you start to see a lot of the challenges with this approach. Different things experience latency very differently.There's also data transfer cost to consider. Whether data is traversing between regions or between availability zones, it does incur an additional cost, so anything with a high replication factor is going to be of some concern. We'll talk specifically about data transfer costs in a future episode. Additionally, if you're in a single provider and going multiple regions, they do have dedicated links between their regions that usually wind up providing better performance and faster speeds than you're going to see traversing the general internet, but see the previous episode on global accelerator to figure out a little bit more about some of the caveats there.One thing to also consider is that because AWS does have a severed control plane that does not extend to multiple regions, there are two things that this impacts. The first is that we have never yet seen a networking event that traverses more than one region. The counterpoint is that not all services are available in all regions, so make sure that you wind up selecting appropriate regions based upon their region service availability table.Further, you're also going to want to make sure that the pricing aligns with it. The region in Northern California, for example, doesn't have nearly as many availability zones as the rest, and everything in that region does tend to cost more, so pay attention to that. There are two other regions that were announced, or region-like things that were announced recently at AWS re:Invent.The first is the local zone, which is a different type of availability zone. They only have one so far. It is generally available in preview, which means that words no longer have the same meaning anymore, and it's an extension into Los Angeles of the region based in Oregon, us-west-2. This enables companies and other organizations in Los Angeles to have lower latency access to AWS resources when effectively tens of milliseconds or less matter for certain workloads.It's fascinating, but it does suffer from a lack of durability that you're going to see in a fully baked region, so use it if you have to, but if you can avoid it, you're potentially saving yourself some ops burden down the road. They've also taken their outposts, which are fundamentally just racks full of AWS equipment that you can now rent and put in your facility, and done some partnerships with cell companies. In the United States, they've started with Verizon, and they're exploring 5G and calling this AWS wavelength. This is relevant if and only if you're looking at building 5G type applications in partnership with Verizon. Most folks aren't, so it's not going to be super relevant, but it is a type of global infrastructure to pay attention to.Fundamentally, understanding the differences between regions and availability zones in AWS, or their equivalent in other providers, is going to be critical for planning for your DR type of tests. It's super unfortunate to wind up testing your DR plan and finding everything works, and then testing it during an actual... and then using it during an actual outage and discovering everything's slow and the provisioning takes forever because everyone has the same plan that you do. Take some time, make sure that you understand these region and availability zone concepts when you're building out your infrastructure plan, and ideally, everything goes a lot more smoothly for you.That's all I've got to say on this particular topic. If you have questions, please feel free to ask them. On Twitter, I'm QuinnyPig. That's Q-U-I, double N, Y Pig, and I'll do my best to either answer them myself or point you to someone smart who can answer them more authoritatively. If you've enjoyed this podcast, please leave a five-star review in Apple Podcasts. If you've hated this podcast, please leave a five-star review in Apple Podcasts and a funny comment, so I have something to laugh at while crying. I'm cloud economist Corey Quinn, and I'll talk to you more next week.Announcer: This is been a HumblePod production. Stay humble.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at <a href="https://www.lastweekinaws.com/" rel="nofollow" target="_blank">The Duckbill Group</a>, the author of the weekly <a href="https://www.lastweekinaws.com/newsletter/" rel="nofollow" target="_blank">Last Week in AWS</a> newsletter, and the host of two podcasts: <a href="https://www.screaminginthecloud.com" rel="nofollow" target="_blank">Screaming in the Cloud</a> and, you guessed it, <a href="https://www.lastweekinaws.com/podcast/aws-morning-brief/" rel="nofollow" target="_blank">AWS Morning Brief</a>, which you’re about to listen to.Transcript Corey: Hello, and welcome back to our Networking in the Cloud mini series sponsored by ThousandEyes. That's right. <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> is state-of-the-cloud Performance Benchmark Report is now available for your perusal. It's really providing a lot of baseline that we're taking all of the miniseries information from. It pointed us in a bunch of interesting directions and helps us tell stories that are actually, for a change, backed by data rather than pure sarcasm. To get your copy, visit <a href="https://snark.cloud/realclouds" rel="nofollow" target="_blank">snark.cloud/realclouds</a> because it only covers real cloud providers. Thanks again to ThousandEyes for their ridiculous support of this shockingly informative podcast mini series. It's a basic fact of cloud that things break all the time. I've been joking for a while that a big competitive advantage that Microsoft brings to this space is that they have 40 years of experience apologizing for software failures, except that's not really a joke. It's true. There's something to be said for the idea of apologizing to both technical and business people about real or perceived failures being its own skillset, and they have a lot more experience than anyone in this space. There are two schools of thought around how to avoid having to apologize for service or component failures to your customers. The first is to build super expensive but super durable things, and you can kind of get away with this in typical data center environments right up until you can't, and then it turns out that your SAN just exploded. You're really not diversifying with most SANs. You're just putting all of your eggs in a really expensive basket, and of course, if you're still with power or networking outage, nothing can talk to the SAN, and you're back to square one. The other approach is to come at it with a perspective of building in redundancy to everything and eliminating single points of failure. That's usually the better path in the cloud. You don't ever want to have a single point of failure if you can reasonably avoid it, so going with multiple everythings starts to make sense to a point. Going with a full on multi-cloud story is a whole separate kettle of nonsense we'll get to another time. But you realize at some point you will have single points of failure and you're not going to be able to solve for that. We still only have one planet going around one sun for example. If either of those things explode, well, computers aren't really anyone's concern anymore. However, betting the entire farm on one EC2 instance is generally something you'll want to avoid if at all possible. In the world of AWS, there aren't data centers in the way that you or I would contextualize them. Instead, they have constructs known as availability zones and those composed to form a different construct called regions. Presumably, other cloud providers have similar constructs over in non-AWS land, but we're focusing on AWS as implementation in this series, again, because they have a giant multi-year head start over every other cloud provider, and even that manifests in those other cloud providers comparing what they've built and how they operate to AWS. If that upsets you and you work at one of those other cloud providers, well, you should have tried harder. Let's dive in to a discussion of data centers, availability zones, and regions today. Take an empty warehouse and shove it full of server racks. Congratulations. You have built the bare minimum requirement for a data center at its most basic layer. Your primary constraint and why it's a lot harder than it sounds is power, and to a lesser extent, cooling. Computers aren't just crunching numbers, they're also throwing off waste heat. You've got to think an awful lot about how to keep that heat out of the data center. At some point, you can't shove more capacity into that warehouse-style building just because you can't cool it if it's all running at the same time. If your data center's particularly robust, meaning you didn't cheap out on it, you're going to have different power distribution substations that feed the building from different lines that enter the building at different corners. You're going to see similar things with cooling as well, multiply redundant cooling systems. One of the big challenges, of course, when dealing with this physical infrastructure is validating that what it says on the diagram is what's actually there in the physical environment. That can be a trickier thing to explore than you would hope. Also, if you have a whole bunch of systems sitting in that warehouse and you take a power outage, well, you have to plan for this thing known as inrush current. Normally, it's steady state. Computers generally draw a known quantity of power. But when you first turn them on, if you've ever dealt with data center servers, the first thing they do is they power up everything to self-test. They sound like a jet fighter taking off as all the fans spin up. If you're not careful, and all these things turn on at once, you'll see a giant power spike that winds up causing issues, blowing breakers, maxing out consumption, so having a staggered start becomes a concern as well. Having spent too much time in data centers, I am painfully familiar with this problem of how you safely and sanely recover from site-wide events, but that's a bit out of scope, thankfully, because in the cloud, this is less of a problem. Let's talk about the internet and getting connectivity to these things. This is the Networking in the Cloud podcast after all. You're ideally going to have multiple providers running fiber lines to that data center hoping to avoid fiber's natural predator, the noble backhoe. Now, ideally, all those fiber lines are going over different paths, but again, hard thing to prove, so doing your homework's important, but here's something folks don't always consider: If you have a hundred gigabit ethernet links to each computer, which is not cheap, but doable, and then you have 20 servers in a rack, each rack theoretically needs to be able to speak at least two terabit at all times to each other server in each other rack, and most of them can't do that. They wind up having bottle-necking issues. As a result, when you have high-traffic applications speaking between systems, you need to make sure that they're aware of something known as rack affinity. In other words, are there bottlenecks between these systems, and how do you minimize those to make sure the crosstalk works responsibly? There are a lot of dragons in here, but let's hand-wave past all of it because we're talking about cloud here. The point of this is that there's an awful lot of nuance to running data centers, and AWS and other large cloud providers do a better job of it than you do. That's not me insulting your data center staff. That's just a fact. They have the scale and the staff and the expertise of running these things operationally that very few other companies are going to be able to touch. Sure, if you're Facebook, you probably have some expertise in this as well, and a lot of this won't apply to you, but you know that already. If you're wondering whether what I'm talking about here applies to your environment, unless you know for a fact it doesn't, it does. Assume that. This impacts your approach in the cloud, the networking, durability, and the concept of blast radius, and the forms that AWS gives us that wrap these concepts for us are twofold, and I want to cover them today: regions and availability zones. An availability zone, or A-Zee, or A-Zed if you're not in the United States, is effectively a set of data centers, and yes, that's plural. It's not just different racks with different power buses in the same room. AWS tries to guarantee that there is no shared power, network, or control plane between availability zones, but you can expect some issues to impact an entire availability zone as a result. Ergo, if you're building something important, you're going to want it to be in at least multiple availability zones. To that end, here's a fun fact that trips up nearly everyone the first time they see it. If you have an AWS account, you might see that there's an outage in a particular availability zone, us-west-2a. Meanwhile, in my account, I see an outage in us-west-2c. Who's right in that scenario? Well, we both are because availability zone names aren't consistent between AWS accounts. Relatively recently, about a year ago as of this recording, they announced a zone ID that is consistent between accounts, but people still don't talk about it in those terms. They're still talking with the old style region us-west-2 followed by a letter for the availability zone. You still have to disambiguate those back to zone IDs with an extra step. That also doesn't solve the problem for you because note that even with indirect issues that you're seeing, they can still impact other availability zones, even with a completely separate control plane because if you have things that are running in two availability zones for your application, and one of those availability zones drops off the internet for a while, you're suddenly seeing twice the load in the availability zone that's still working. You're also probably not the only customer that has planned for this and has built out in multiple availability zones, so other folks are going to be seeing the exact same type of behavior. As a result, failures will then cascade and manifest as slow performance in the good availability zones, and it's super hard to plan for. It's also super hard to detect, which brings us back to our sponsor, <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes.</a> ThousandEyes provides a global observer perspective on what's going on internet-wide with a bunch of different providers. It helps answer the question when one of these incidents hits of "is it my code, is it the last deployment that we did, or is there something global that's causing this problem?" ThousandEyes provides that global observer perspective that helps you figure out immediately was it your code or was it something infrastructure-based because if it's not your code and it is infrastructure-based, suddenly, you can stop looking at everything you just shipped to production and instead look at mitigating this according to an established DR plan. Thanks again to ThousandEyes for sponsoring this. To learn more, visit <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">thousandeyes.com</a>, and tell them Corey sent you. They seem to like me for some reason that we really can't tell. You finally build something that's in multiple availability zones, but as mentioned, that cascade effect can be a challenge, so this is where we get into the idea of multiple regions. A region is two or more availability zones, usually three or more, but there are some legacy stories behind that, and those are separated by very large distances. In the United States, for example, there's one in Oregon, one in Ohio, one in Virginia, and one, sort of, in Northern California. The challenge, of course, is building applications that work spanning multiple regions, and there are a couple of issues with this. If you're trying to tilt at the windmill of multi-cloud, I would strongly encourage you to start by going multiple region in a single provider first. This removes a lot of the finicky bits of multi-cloud, like the services work differently, and if you pick the right regions, you'll have a one-to-one affinity between all of the different services, and that's awesome. Once you wind up getting multiple regions online, then you start to see a lot of the challenges with this approach. Different things experience latency very differently. There's also data transfer cost to consider. Whether data is traversing between regions or between availability zones, it does incur an additional cost, so anything with a high replication factor is going to be of some concern. We'll talk specifically about data transfer costs in a future episode. Additionally, if you're in a single provider and going multiple regions, they do have dedicated links between their regions that usually wind up providing better performance and faster speeds than you're going to see traversing the general internet, but see the previous episode on global accelerator to figure out a little bit more about some of the caveats there. One thing to also consider is that because AWS does have a severed control plane that does not extend to multiple regions, there are two things that this impacts. The first is that we have never yet seen a networking event that traverses more than one region. The counterpoint is that not all services are available in all regions, so make sure that you wind up selecting appropriate regions based upon their region service availability table. Further, you're also going to want to make sure that the pricing aligns with it. The region in Northern California, for example, doesn't have nearly as many availability zones as the rest, and everything in that region does tend to cost more, so pay attention to that. There are two other regions that were announced, or region-like things that were announced recently at AWS re:Invent. The first is the local zone, which is a different type of availability zone. They only have one so far. It is generally available in preview, which means that words no longer have the same meaning anymore, and it's an extension into Los Angeles of the region based in Oregon, us-west-2. This enables companies and other organizations in Los Angeles to have lower latency access to AWS resources when effectively tens of milliseconds or less matter for certain workloads. It's fascinating, but it does suffer from a lack of durability that you're going to see in a fully baked region, so use it if you have to, but if you can avoid it, you're potentially saving yourself some ops burden down the road. They've also taken their outposts, which are fundamentally just racks full of AWS equipment that you can now rent and put in your facility, and done some partnerships with cell companies. In the United States, they've started with Verizon, and they're exploring 5G and calling this AWS wavelength. This is relevant if and only if you're looking at building 5G type applications in partnership with Verizon. Most folks aren't, so it's not going to be super relevant, but it is a type of global infrastructure to pay attention to. Fundamentally, understanding the differences between regions and availability zones in AWS, or their equivalent in other providers, is going to be critical for planning for your DR type of tests. It's super unfortunate to wind up testing your DR plan and finding everything works, and then testing it during an actual... and then using it during an actual outage and discovering everything's slow and the provisioning takes forever because everyone has the same plan that you do. Take some time, make sure that you understand these region and availability zone concepts when you're building out your infrastructure plan, and ideally, everything goes a lot more smoothly for you. That's all I've got to say on this particular topic. If you have questions, please feel free to ask them. On Twitter, I'm QuinnyPig. That's Q-U-I, double N, Y Pig, and I'll do my best to either answer them myself or point you to someone smart who can answer them more authoritatively. If you've enjoyed this podcast, please leave a five-star review in Apple Podcasts. If you've hated this podcast, please leave a five-star review in Apple Podcasts and a funny comment, so I have something to laugh at while crying. I'm cloud economist Corey Quinn, and I'll talk to you more next week. Announcer: This is been a HumblePod production. Stay humble.

Networking in the Cloud Fundamentals: Regions and Availability Zones in AWS

AWS Morning Brief for Tuesday, December 10th, 2019.

reInvent Wrap-up, Part 2

AWS Morning Brief for Friday, December 13th, 2019

reInvent Wrap-up, Part 4

Amazon Lex

AWS Morning Brief for the week of January 6th, 2020.

Burning Amazon Lex to CD-ROM

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Knock knock. Who's there? A DDOS attack. A DDOS a... Knock. Knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock.Welcome to what we're calling Networking in the Cloud, episodes six, How Things Break in the Cloud, sponsored by ThousandEyes. ThousandEyes recently launched their state of the cloud performance benchmark report that effectively lets you compare and contrast performance and other aspects between the five large cloud providers, AWS, Azure, GCP, Alibaba and IBM cloud. Oracle cloud was not invited because we are talking about real clouds here. You can get your copy of this report at snark.cloud/realclouds. and they compare and contrast an awful lot of interesting things. One thing that we're not going to compare and contrast though, because of my own personal beliefs, is the outages of different cloud providers.Making people in companies, by the way, companies are composed of people, making them feel crappy about their downtime is mean, first off. Secondly, if companies are shamed for outages, it in turn makes it far likelier that they won't disclose having suffered an outage. And when companies talk about their outages in constructive blameless ways, there are incredibly valuable lessons that we all can learn from it. So let's dive into this a bit.If there's one thing that computers do well, better than almost anything else, it's break. And this is, and I'm not being sarcastic when I say this, a significant edge that Microsoft has when they come to cloud. They have 40 some odd years of experience in apologizing for software failures. That's not trying to be insulting to Microsoft, it's what computers do, they break. And being able to explain that intelligently to business stakeholders is incredibly important. They're masters at that. They also have a 20 year head start on everyone else in the space. What makes this interesting and useful is that in the cloud, computers break differently than people would expect them to in a non-cloud environment.Once upon a time when you were running servers and data centers, if you see everything suddenly go offline, you have some options. You can call the data center directly to see if someone cut the fiber, in case you were unaware of fiber optic cables' sole natural predator in the food chain is the mighty backhoe. So maybe something backhoed out some fiber lines, maybe the power is dead to the data center, maybe the entire thing exploded, burst into flames and burned to the ground, but you can call people. In the cloud, it doesn't work that way. Here in the cloud, instead you check Twitter because it's 3:00 AM and Nagios is the original call of duty or PagerDuty calls you, because you didn't need that sleep anyway, telling you there is something amiss with your site. So when a large bond provider takes an outage, and you're hanging out on Twitter at two in the morning, you can see DevOps Twitter come to life in the middle of the night, as they chatter back and forth.And incidentally, if that's you, understand a nuance of AWS availability zone naming. When people say things like us-east-1a is having a problem and someone else says, "No, I just see us-east-1c is having a problem," you're probably talking about the same availability zone. Those letters change, non deterministically, between accounts. You can pull zone IDs, and those are consistent. But by and large, that was originally to avoid having problems like everyone picking A, as humans tend to do or C, getting the reputation as the crappy one.So why would you check Twitter to figure out if your cloud provider's having a massive outage? Well, because honestly, the AWS status page is completely full of lies and gaslights you. It is as green as the healthiest Christmas tree you can imagine, even when things are exploding for a disturbingly long period of time. If you visit the website, stop.lying.cloud, you'll find a Lambda and Edge function that I've put there that cuts out some of the croft, but it's not perfect. And the reason behind this, after I gave them a bit too much crap one day and I got a phone call that started with, "Now you listen here," it turns out that there are humans in the loop, and they need to validate that there is in fact a systemic issue at AWS and what that issue might be, and then finally come up with a way to report that in a way that ideally doesn't get people sued and manually update the status page. Meanwhile, your site's on fire. So that is a trailing function, not a leading function.Alternately, you could always check ThousandEyes. That's right, this episode is sponsored by ThousandEyes. In addition to the report we mentioned earlier, you can think of them as Google Maps of the internet without the creepy privacy overreach issues. Just like you wouldn't necessarily want to commute during rush hour without checking where traffic is going to be and which route was faster, businesses rely on ThousandEyes to see the end to end paths their applications and services are taking in real time to identify where the slow downs are, where the outages are and what's causing problems. They use ThousandEyes to see what's breaking where and then importantly, ThousandEyes shares that data directly with the offending service providers. Not just to hold them accountable, but also to get them to fix the issue fast. Ideally, before it impacts users. But on this episode, it already has.So let's say that you don't have the good sense to pay for ThousandEyes or you're not on Twitter, for whatever reason, watching people flail around helplessly trying to figure out what's going on. Instead, you're now trying desperately to figure out whether this issue is the last deploy your team did or if it's a global problem. And the first thing people try to do in the event of an issue is, "Oh crap, what did we just change? undo it." And often that is a knee jerk response that can make things worse if it's not actually your code that caused the problem. Worse, it can eat up precious time at the beginning of an outage. If you knew that it was a single availability zone or an entire AWS region that was having a problem, you could instead be working to fail over to a different location instead of wasting valuable incident retime checking Twitter or looking over your last 200 commits.Part of the problem, and the reason this is the way that it is, is that unlike rusting computers in your data center currently being savaged by raccoons, things in the cloud break differently. You don't have the same diagnostic tools, you don't have the same level of visibility into what the hardware is doing, and the behaviors themselves are radically different. I have a half dozen tips and tricks on how to monitor whether or not your data center's experiencing a problem remotely, but they don't work in the cloud because you're not allowed to break into us-east-1 and install your own hardware. Believe me, I've tried. I still have the scars to prove it. Instead, you have to deal with this problem of behaviors looking differently.For example, sometimes you can talk to one set of servers but the other is completely non-responsive to you, but those two server sets can still talk to one another intermittently. So you wind up with each one of them thinking at times they're the only ones there or you can talk to both of them but they can't talk to each other. There are different kinds of failures and they all look slightly different. Occasionally, it looks like slow API responses. Latencies are increasing. Well, that's an awfully nice way to say that suddenly your database doesn't. It often can look like a certain subset of systems that seem slow or intermittent. Remember as well that availability zones aren't multiple buildings. It's not just one room with different racks being called different AZs, the way we used to do things badly in crappy data center land. It's super hard to take out 20 square blocks and cause multiple AZ outages at the same time, at least it is with that attitude.So instead of automatically assuming that, "Well, it works for me on this other account, so things are fine," dig deeper into it. Often issues in 1AZ have cascading effects and you see other popular sites on the internet starting to have problems. Maybe it's not just you. The fact that this is sort of state of the art for monitoring these issues is a separate issue. The problem comes in when people haven't changed their thinking to reflect this new cloud reality.There's no better example of this than DR exercises or disaster recovery. Now, most ops folks, and I still sort of count myself as one, have tremendous levels of experience with disasters, planning for disasters, recovering from disasters, and notable cases causing disasters. The problem is is that very often stories about how to handle disasters don't work in the real world. An easy example is if you're running in us-east-1 and your disaster recovery approach is, "Oh, we're just going to spin up the site in us-west-2 in Oregon, great." Now there are problems with that approach, but let's skip over a few of them and get to the interesting ones.First, if you're doing this during a test and you spin up a bunch of east-2 instances or other services, great, that's going to probably work super well for purposes of your test. The challenge of course is that when you're in the middle of an actual disaster, you are not the only person who has that strategy in mind for how they're going to handle the disaster. So suddenly us-west-2 and other regions, I don't mean to pick on Oregon in particular, are going to suffer from inrush issues. Very often that means that the API calls to the control plane in the cloud, wind up becoming impacted, latencies start to increase. There have been scenarios in the past where it takes up to an hour to have instances come online after you've requested them. So if you need to have an active DR site ready to go, you have to pay for the durability for those instances and other services to already be up and running.Secondly, if you're like most jobs, you'll test your DR site every quarter or every year, and you'll find at the first past that, "Oh, it didn't work, it broke immediately." So you go back, you fix the thing, you try it again, and it breaks differently. And after enough of this, you finally beat something together and it works and you call it done. You put the binder on the shelf where no one will ever read it again and everything is just fine until the next commit breaks your DR plan again. And the problem is is that that's in the best of times where there's no actual disaster. Trying to make that work in any reasonable approach during a disaster in the middle of the night, where not everyone's firing on all cylinders, that becomes a problem.I also strongly suggest that you don't approach business continuity planning, or BCP, the way that I did, and it's why I stopped being invited to those meetings. The problem that we ran into was, "Okay, let's pretend for the sake of argument that San Francisco is no longer able to conduct business," to which my immediate response is, "Oh dear heavens, is my family okay?" "Yes, yes, your family's fine, everyone's fine, but magically we can't do any computer work." Okay, I struggled to identify that, but all right, let's pretend I care that much about my job and not about my family. Cool. I understand everyone's family relationships are different and for some folks that works.All right, next step. Simultaneously, us-east-1 is completely unusable. "Okay, so let me get this straight, not only is San Francisco now magically not usable, but also roughly a hundred square miles of Northern Virginia is also completely unusable. And at this point, I'm not hunkering down in a basement cowering, waiting for the end of days because why exactly?" And the response was "Just roll with it, it'll be fine. Now, we need to have a facility outside of the city for you to go to and in a different provider, have all the backups, you can rehydrate this new. And at the end of that project, we're going to be able to do this whenever we need to." At which point I stared at people for the longest time and said, "You get that we sell ads here, right? And furthermore, let's pretend that everything you say is true, us-east-1 is irreparably damaged and I don't want to spend time with my family in a disaster like that because everyone's fine. Why do I still work here rather than going to make extortionate money as a consultant somewhere else who is not prepared nearly as well as we have?" And then I wasn't invited to those meetings anymore.One last angle that people tend to approach this stuff from is the idea of, well ,the service needs an SLA or service level agreement. Some AWS services have them, some do not. But they don't mean what you think they do. Route 53 famously has a 100% SLA. If they don't meet that, first, they owe you some small portion of your route 53 bill which, spoiler, is probably not a large pile of money. Secondly, because they've published, that everything else, including a number of AWS services themselves, almost certainly build to that SLA. So it breaks, they owe you some small pile of money, but when that outage hits, because everything breaks, it's what they do, it impacts your site. No, you can't trust various SLA metrics as statements that services will never go down. You own your own availability. You can't outsource the responsibility of that to third parties, no matter how much you might want to.It may sound like I'm suggesting that things in the cloud always break and that you shouldn't be in the cloud at all if you can't withstand an outage. I strongly disagree. There are reasons to stay with a cloud provider. First, they're going to diagnose and fix the problem with a far larger staff who is far better equipped to handle these issues, then you'll be able to independently in almost every case.Secondly, if there's a massive disruption to a public cloud provider, then you're going to be in good company. The headlines are not going to be about your company's outage, they're going to be about the cloud provider. There's some reputational risks that gets mitigated as a direct result.Finally, if all of that fails and you still go down and everyone makes fun of you for it, well, you can always go for consolation on Twitter.This has been another episode of Networking in the Cloud. I'm cloud economist, Corey Quinn. Thank you for joining us and we'll talk soon. Thanks again to ThousandEyes for their sponsorship of this ridiculous podcast.Announcer: This has been a HumblePod production. Stay humble.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at <a href="https://www.lastweekinaws.com/" rel="nofollow" target="_blank">The Duckbill Group</a>, the author of the weekly <a href="https://www.lastweekinaws.com/newsletter/" rel="nofollow" target="_blank">Last Week in AWS</a> newsletter, and the host of two podcasts: <a href="https://www.screaminginthecloud.com" rel="nofollow" target="_blank">Screaming in the Cloud</a> and, you guessed it, <a href="https://www.lastweekinaws.com/podcast/aws-morning-brief/" rel="nofollow" target="_blank">AWS Morning Brief</a>, which you’re about to listen to. Transcript Corey: Knock knock. Who's there? A DDOS attack. A DDOS a... Knock. Knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock, knock. Welcome to what we're calling Networking in the Cloud, episodes six, How Things Break in the Cloud, sponsored by <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a>. ThousandEyes recently launched their state of the cloud performance benchmark report that effectively lets you compare and contrast performance and other aspects between the five large cloud providers, AWS, Azure, GCP, Alibaba and IBM cloud. Oracle cloud was not invited because we are talking about real clouds here. You can get your copy of this report at <a href="https://snark.cloud/realclouds" rel="nofollow" target="_blank">snark.cloud/realclouds</a>. and they compare and contrast an awful lot of interesting things. One thing that we're not going to compare and contrast though, because of my own personal beliefs, is the outages of different cloud providers. Making people in companies, by the way, companies are composed of people, making them feel crappy about their downtime is mean, first off. Secondly, if companies are shamed for outages, it in turn makes it far likelier that they won't disclose having suffered an outage. And when companies talk about their outages in constructive blameless ways, there are incredibly valuable lessons that we all can learn from it. So let's dive into this a bit. If there's one thing that computers do well, better than almost anything else, it's break. And this is, and I'm not being sarcastic when I say this, a significant edge that Microsoft has when they come to cloud. They have 40 some odd years of experience in apologizing for software failures. That's not trying to be insulting to Microsoft, it's what computers do, they break. And being able to explain that intelligently to business stakeholders is incredibly important. They're masters at that. They also have a 20 year head start on everyone else in the space. What makes this interesting and useful is that in the cloud, computers break differently than people would expect them to in a non-cloud environment. Once upon a time when you were running servers and data centers, if you see everything suddenly go offline, you have some options. You can call the data center directly to see if someone cut the fiber, in case you were unaware of fiber optic cables' sole natural predator in the food chain is the mighty backhoe. So maybe something backhoed out some fiber lines, maybe the power is dead to the data center, maybe the entire thing exploded, burst into flames and burned to the ground, but you can call people. In the cloud, it doesn't work that way. Here in the cloud, instead you check Twitter because it's 3:00 AM and Nagios is the original call of duty or PagerDuty calls you, because you didn't need that sleep anyway, telling you there is something amiss with your site. So when a large bond provider takes an outage, and you're hanging out on Twitter at two in the morning, you can see DevOps Twitter come to life in the middle of the night, as they chatter back and forth. And incidentally, if that's you, understand a nuance of AWS availability zone naming. When people say things like us-east-1a is having a problem and someone else says, "No, I just see us-east-1c is having a problem," you're probably talking about the same availability zone. Those letters change, non deterministically, between accounts. You can pull zone IDs, and those are consistent. But by and large, that was originally to avoid having problems like everyone picking A, as humans tend to do or C, getting the reputation as the crappy one. So why would you check Twitter to figure out if your cloud provider's having a massive outage? Well, because honestly, the AWS status page is completely full of lies and gaslights you. It is as green as the healthiest Christmas tree you can imagine, even when things are exploding for a disturbingly long period of time. If you visit the website, stop.lying.cloud, you'll find a Lambda and Edge function that I've put there that cuts out some of the croft, but it's not perfect. And the reason behind this, after I gave them a bit too much crap one day and I got a phone call that started with, "Now you listen here," it turns out that there are humans in the loop, and they need to validate that there is in fact a systemic issue at AWS and what that issue might be, and then finally come up with a way to report that in a way that ideally doesn't get people sued and manually update the status page. Meanwhile, your site's on fire. So that is a trailing function, not a leading function. Alternately, you could always check <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a>. That's right, this episode is sponsored by ThousandEyes. In addition to the report we mentioned earlier, you can think of them as Google Maps of the internet without the creepy privacy overreach issues. Just like you wouldn't necessarily want to commute during rush hour without checking where traffic is going to be and which route was faster, businesses rely on ThousandEyes to see the end to end paths their applications and services are taking in real time to identify where the slow downs are, where the outages are and what's causing problems. They use ThousandEyes to see what's breaking where and then importantly, ThousandEyes shares that data directly with the offending service providers. Not just to hold them accountable, but also to get them to fix the issue fast. Ideally, before it impacts users. But on this episode, it already has. So let's say that you don't have the good sense to pay for ThousandEyes or you're not on Twitter, for whatever reason, watching people flail around helplessly trying to figure out what's going on. Instead, you're now trying desperately to figure out whether this issue is the last deploy your team did or if it's a global problem. And the first thing people try to do in the event of an issue is, "Oh crap, what did we just change? undo it." And often that is a knee jerk response that can make things worse if it's not actually your code that caused the problem. Worse, it can eat up precious time at the beginning of an outage. If you knew that it was a single availability zone or an entire AWS region that was having a problem, you could instead be working to fail over to a different location instead of wasting valuable incident retime checking Twitter or looking over your last 200 commits. Part of the problem, and the reason this is the way that it is, is that unlike rusting computers in your data center currently being savaged by raccoons, things in the cloud break differently. You don't have the same diagnostic tools, you don't have the same level of visibility into what the hardware is doing, and the behaviors themselves are radically different. I have a half dozen tips and tricks on how to monitor whether or not your data center's experiencing a problem remotely, but they don't work in the cloud because you're not allowed to break into us-east-1 and install your own hardware. Believe me, I've tried. I still have the scars to prove it. Instead, you have to deal with this problem of behaviors looking differently. For example, sometimes you can talk to one set of servers but the other is completely non-responsive to you, but those two server sets can still talk to one another intermittently. So you wind up with each one of them thinking at times they're the only ones there or you can talk to both of them but they can't talk to each other. There are different kinds of failures and they all look slightly different. Occasionally, it looks like slow API responses. Latencies are increasing. Well, that's an awfully nice way to say that suddenly your database doesn't. It often can look like a certain subset of systems that seem slow or intermittent. Remember as well that availability zones aren't multiple buildings. It's not just one room with different racks being called different AZs, the way we used to do things badly in crappy data center land. It's super hard to take out 20 square blocks and cause multiple AZ outages at the same time, at least it is with that attitude. So instead of automatically assuming that, "Well, it works for me on this other account, so things are fine," dig deeper into it. Often issues in 1AZ have cascading effects and you see other popular sites on the internet starting to have problems. Maybe it's not just you. The fact that this is sort of state of the art for monitoring these issues is a separate issue. The problem comes in when people haven't changed their thinking to reflect this new cloud reality. There's no better example of this than DR exercises or disaster recovery. Now, most ops folks, and I still sort of count myself as one, have tremendous levels of experience with disasters, planning for disasters, recovering from disasters, and notable cases causing disasters. The problem is is that very often stories about how to handle disasters don't work in the real world. An easy example is if you're running in us-east-1 and your disaster recovery approach is, "Oh, we're just going to spin up the site in us-west-2 in Oregon, great." Now there are problems with that approach, but let's skip over a few of them and get to the interesting ones. First, if you're doing this during a test and you spin up a bunch of east-2 instances or other services, great, that's going to probably work super well for purposes of your test. The challenge of course is that when you're in the middle of an actual disaster, you are not the only person who has that strategy in mind for how they're going to handle the disaster. So suddenly us-west-2 and other regions, I don't mean to pick on Oregon in particular, are going to suffer from inrush issues. Very often that means that the API calls to the control plane in the cloud, wind up becoming impacted, latencies start to increase. There have been scenarios in the past where it takes up to an hour to have instances come online after you've requested them. So if you need to have an active DR site ready to go, you have to pay for the durability for those instances and other services to already be up and running. Secondly, if you're like most jobs, you'll test your DR site every quarter or every year, and you'll find at the first past that, "Oh, it didn't work, it broke immediately." So you go back, you fix the thing, you try it again, and it breaks differently. And after enough of this, you finally beat something together and it works and you call it done. You put the binder on the shelf where no one will ever read it again and everything is just fine until the next commit breaks your DR plan again. And the problem is is that that's in the best of times where there's no actual disaster. Trying to make that work in any reasonable approach during a disaster in the middle of the night, where not everyone's firing on all cylinders, that becomes a problem. I also strongly suggest that you don't approach business continuity planning, or BCP, the way that I did, and it's why I stopped being invited to those meetings. The problem that we ran into was, "Okay, let's pretend for the sake of argument that San Francisco is no longer able to conduct business," to which my immediate response is, "Oh dear heavens, is my family okay?" "Yes, yes, your family's fine, everyone's fine, but magically we can't do any computer work." Okay, I struggled to identify that, but all right, let's pretend I care that much about my job and not about my family. Cool. I understand everyone's family relationships are different and for some folks that works. All right, next step. Simultaneously, us-east-1 is completely unusable. "Okay, so let me get this straight, not only is San Francisco now magically not usable, but also roughly a hundred square miles of Northern Virginia is also completely unusable. And at this point, I'm not hunkering down in a basement cowering, waiting for the end of days because why exactly?" And the response was "Just roll with it, it'll be fine. Now, we need to have a facility outside of the city for you to go to and in a different provider, have all the backups, you can rehydrate this new. And at the end of that project, we're going to be able to do this whenever we need to." At which point I stared at people for the longest time and said, "You get that we sell ads here, right? And furthermore, let's pretend that everything you say is true, us-east-1 is irreparably damaged and I don't want to spend time with my family in a disaster like that because everyone's fine. Why do I still work here rather than going to make extortionate money as a consultant somewhere else who is not prepared nearly as well as we have?" And then I wasn't invited to those meetings anymore. One last angle that people tend to approach this stuff from is the idea of, well ,the service needs an SLA or service level agreement. Some AWS services have them, some do not. But they don't mean what you think they do. Route 53 famously has a 100% SLA. If they don't meet that, first, they owe you some small portion of your route 53 bill which, spoiler, is probably not a large pile of money. Secondly, because they've published, that everything else, including a number of AWS services themselves, almost certainly build to that SLA. So it breaks, they owe you some small pile of money, but when that outage hits, because everything breaks, it's what they do, it impacts your site. No, you can't trust various SLA metrics as statements that services will never go down. You own your own availability. You can't outsource the responsibility of that to third parties, no matter how much you might want to. It may sound like I'm suggesting that things in the cloud always break and that you shouldn't be in the cloud at all if you can't withstand an outage. I strongly disagree. There are reasons to stay with a cloud provider. First, they're going to diagnose and fix the problem with a far larger staff who is far better equipped to handle these issues, then you'll be able to independently in almost every case. Secondly, if there's a massive disruption to a public cloud provider, then you're going to be in good company. The headlines are not going to be about your company's outage, they're going to be about the cloud provider. There's some reputational risks that gets mitigated as a direct result. Finally, if all of that fails and you still go down and everyone makes fun of you for it, well, you can always go for consolation on Twitter. This has been another episode of Networking in the Cloud. I'm cloud economist, Corey Quinn. Thank you for joining us and we'll talk soon. Thanks again to <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> for their sponsorship of this ridiculous podcast. Announcer: This has been a HumblePod production. Stay humble.

Networking in the Cloud Fundamentals, Part 6

AWS Morning Brief for the week of February 10, 2020.

Polly Brand Voice Want a Platypus?

AWS Morning Brief for the week of July 1st, 2019.

reInforce Meant Learning

AWS Morning Brief for the week of February 3, 2020.

Lies, Damned Lies, and Sponsored Benchmarks

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.Show Notes CHAOSSEARCH.io Twitter: https://twitter.com/QuinnyPig  TranscriptCorey Quinn: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it.On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. CHAOSSEARCH. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at CHAOSSEARCH.io.Historically, many best practices were, in fact, best practices. But over time, the way that we engage with systems changes. The problems that we’re trying to solve for start resembling other problems. And, at some point entire industries shift. So, what you should have been doing five years ago is not necessarily what you should be doing today. Today, I’d like to talk a little bit about not one or two edge case problems, as I have in previous editions of the Whiteboard Confessional, but rather, I want to talk about an overall pattern that’s shifted. And that shift has been surprisingly sudden, yet gradual enough that you may not entirely have noticed. This goes back into, let’s say 2012, 2013, and is in some ways the story of how I learned to speak publicly. So this is indirectly one of the origin stories of me as a podcaster, and continuing to engage my ongoing love affair with the sound of my own voice. I was one of the very early developers behind SaltStack. Salt, for those who are unfamiliar, is a remote execution framework slash configuration management system that let me participate in code development. It turns out that when you have a pattern of merging every random pull request that some jackass winds up submitting, and then immediately submitting a follow up pull request that fixes everything you just merged in, it’s, first, not the most scalable thing in the world, but on balance provides such a wonderful welcoming community, that people become addicted to participating in it. And SaltStack nailed this in the early days. Now, before this, I’d been focusing on configuration management in a variety of different ways. Some of the very early answers for this were CFEngine, which was written by an academic and is exactly what you would expect an academic to write. It feels more theoretical than it does something that you would want to use in production. But okay, Bcfg2 was something else in this space, and the fact that that is its name tells you everything you need to know about how user-friendly that was. And then the world shifted. We saw Puppet and Chef both arise. You can argue which came first, I don’t care enough in 2020 to have that conversation. But they wound up promoting a model of a domain-specific language, in Puppet’s case, versus chef where they decided, “All right, great, we’re gonna build this stuff out in Ruby.” From there, we then saw a further evolution of Ansible and SaltStack, which really round out the top four. Now, all of these systems fundamentally do the same thing, which is how do we wind up making the current state of a given system look like it should? That means, how do you make sure that certain packages are installed across all of your fleet? How do you make sure that the right users exist across your entire fleet? How do you guarantee that there are files in place, that have the right contents? And when the contents of those files change, how do you restart services? Effectively, how do you run arbitrary commands and converge the state of a remote system so it looks like it should? Because trying to manage systems at scale is awful. You heard in a previous week what happened when I tried to run this sort of system by using a Distributed SSH client. Yeah, it turns out that mistakes are huge and hard to fix. This speaks toward the direction of moving into cattle instead of pets when it comes to managing systems. And all of these systems more or less took a different approach to it. And some were more aligned with how I saw the world than others did. So I started speaking about SaltStack back in 2012 and giving conference talks. The secret to giving a good conference talk, of course, is to give a whole bunch of really terrible ones first, and woo boy were these awful. I would put documentation on the slides. I would then read the documentation to people frantically trying to teach folks the ins and outs of a technical system in 45 minutes or less. It was about as engaging as it probably sounds like. Over time, I learned not to do that, but because no one else was speaking about SaltStack I was sort of in a rarefied position of being able to tell a story, and learn to tell stories, about a platform that I was passionate about, as it engaged a larger and larger community. Now, why am I talking about all of this on the Whiteboard Confessional? Excellent question. But first, in the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider CHAOSSEARCH. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to CHAOSSEARCH.io. And my thanks to them for sponsoring this incredibly depressing podcast. The reason I bring up configuration management across the board is not because I want to talk about the pattern of doing terrible things within it, and oh, there are terrible patterns, but rather to talk about how in 2020, it is almost certainly an anti-pattern if you’re using it at all. Let me explain. The idea of configuration management was how you wind up getting a bunch of systems converged to a common state. Well, in the world of immutable infrastructure, in no small part ushered in by Docker, and later other systems like it, suddenly spinning a new container took less than a second, and it was exactly as you’d want it to be. So one-line configuration changes didn’t require redeploying fleets of servers, just iterating forward to the next version of the container. That, in turn, became a much better approach, and then you could get into a world where there wasn’t even an SSH daemon running, so people could not get into these containers in the first place to muck around and break the config, which was a big piece of why you had to have these things running continuously. And over time, as we saw this pattern continued to emerge, the number of systems and use cases for which having something stateful that was managing this became fewer and fewer. Now, with the rise of cloud providers being what they are, you wind up having a slightly different problem, where, if I’m trying to provision a whole bunch of EC2 instances to do things, well, first, I can change the AMI, and we’ve been able to do that for 15 years, but config management was still a great plan back then. Now with things like auto-scaling groups, with containers running on top of these systems, you’re getting into a place where you’re using something like Chef for initial configuration of an instance, maybe, at best, but you’re not going to be running this in a persistent way. where it’s constantly checking in with a central control system anymore. In fact, an awful lot of the things that Chef historically would do are now being handled by user data, where you give a system a script to run on the first time that it boots. Once that script completes successfully, it reports into a load balancer whatnot as being ready to go. But even that is something of an anti-pattern now because historically, we had the problem of, “Okay, I’m going to have my golden image, my AMI that spins up everything that I’d want, then I’m going to hand off to Chef. So you would see these instances being spun up, and then it would take 20 minutes or so for it to run through all the various Chef recipes and pass the health checks. That’s great, but if you need that instance to be ready sooner than that, that’s not really a great pattern. That, in turn, has led to people who are specialized in the configuration management space, feeling the ground beneath their feet start to erode. If you look at any seriously scaled out system, there’s a bit of that in there, but most of the stuff that matters, most of the things break systems at scale are being run through things like Terraform instead, where you’re provisioning exactly what needs to be there up front, letting your cloud provider get the right AMI or baseline image into production, and from there, everything else being handled by things like [etcd 00:10:22], or by which version of a container they’re running to service a particular service. Now, I don’t agree with all of the steps that people have taken down this path, in fact, I make fun of an awful lot on this and other shows, but the problem that we’re seeing that these things solve has not gone away, but it’s become much smaller, which means that, eventually, if you’re specializing in configuration management, as I once did, you’re going to take a look around and you’re going to realize that this is not a growth industry in any meaningful sense for these vendors who generally tend to charge on a per-host basis. Sure, there are existing large scale environments that are continuing to expand their usage of these services, but there isn’t nearly as much net new going on there, which means that we’re starting to see what looks like an industry in decline. One of my common career tips is to imagine that I’m going to start a company tomorrow. And in the next 10 years, it’s going to grow from a tiny startup to a component of the S&amp;P 500. From a tiny company to a giant company. At what point during that evolution over the next 10 years, does that tiny startup become a customer of a particular tool or a particular market segment? If the answer is, it probably doesn’t, then whatever is serving that industry niche is almost certainly not going to survive longer term in its current state. Sure, there’s a giant long tail of existing big e-Enterprise offerings out there that are going to address that in a bunch of interesting and exciting ways, but that’s not something that I would necessarily hang my hat on as a technologist trying to focus on something that is going to be relevant for the next five to 10 years. I, instead, would prefer to focus on something that’s growing. That means today, cloud services? Absolutely. Serverless? Most definitely. Kubernetes? Unfortunately. Docker? Well, sort of. That’s largely not become its own independent skill set anymore. It’s started to slip below the surface of awareness of what people care about, in the same way as almost everything else we talk about on this show eventually will as well, leaving in the realm of a few highly advanced specialists to wind up playing around with them. And that, in short form, is what happened to my once passionate love affair with configuration management, now turning into something I basically don’t even bother to ask people about, because I can relatively safely assume that the answer is not nearly as relevant as it once was, to their technical success. For the Whiteboard Confessional sub-series of the AWS Morning Brief, I’m Cloud Economist Corey Quinn, and I’ll talk to you next week.Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at @quinnypig, and let me know what I should talk about next time.Announcer: This has been a HumblePod production. Stay humble.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at <a href="https://www.lastweekinaws.com/" rel="nofollow" target="_blank">The Duckbill Group</a>, the author of the weekly <a href="https://www.lastweekinaws.com/newsletter/" rel="nofollow" target="_blank">Last Week in AWS</a> newsletter, and the host of two podcasts: <a href="https://www.screaminginthecloud.com/" rel="nofollow" target="_blank">Screaming in the Cloud</a> and, you guessed it, <a href="https://www.lastweekinaws.com/podcast/aws-morning-brief/" rel="nofollow" target="_blank">AWS Morning Brief</a>, which you’re about to listen to. Show Notes<ul> <li><a href="http://CHAOSSEARCH.io" rel="nofollow" target="_blank">CHAOSSEARCH.io</a></li> <li>Twitter: <a href="https://twitter.com/QuinnyPig" rel="nofollow" target="_blank">https://twitter.com/QuinnyPig</a> </li> </ul> Transcript Corey Quinn: Welcome to AWS Morning Brief: Whiteboard Confessional. I’m Cloud Economist Corey Quinn. This weekly show exposes the semi-polite lie that is whiteboard architecture diagrams. You see, a child can draw a whiteboard architecture, but the real world is a mess. We discuss the hilariously bad decisions that make it into shipping products, the unfortunate hacks the real-world forces us to build, and that the best to call your staging environment is “theory”. Because invariably whatever you’ve built works in the theory, but not in production. Let’s get to it. On this show, I talk an awful lot about architectural patterns that are horrifying. Let’s instead talk for a moment about something that isn’t horrifying. <a href="http://chaossearch.io/" rel="nofollow" target="_blank">CHAOSSEARCH</a>. Architecturally, they do things right. They provide a log analytics solution that separates out your storage from your compute. The data lives inside of your S3 buckets, and you can access it using APIs you’ve come to know and tolerate, through a series of containers that live next to that S3 storage. Rather than replicating massive clusters that you have to care and feed for yourself, instead, you now get to focus on just storing data, treating it like you normally would other S3 data and not replicating it, storing it on expensive disks in triplicate, and fundamentally not having to deal with the pains of running other log analytics infrastructure. Check them out today at <a href="http://chaossearch.io/" rel="nofollow" target="_blank">CHAOSSEARCH.io</a>. Historically, many best practices were, in fact, best practices. But over time, the way that we engage with systems changes. The problems that we’re trying to solve for start resembling other problems. And, at some point entire industries shift. So, what you should have been doing five years ago is not necessarily what you should be doing today. Today, I’d like to talk a little bit about not one or two edge case problems, as I have in previous editions of the Whiteboard Confessional, but rather, I want to talk about an overall pattern that’s shifted. And that shift has been surprisingly sudden, yet gradual enough that you may not entirely have noticed. This goes back into, let’s say 2012, 2013, and is in some ways the story of how I learned to speak publicly. So this is indirectly one of the origin stories of me as a podcaster, and continuing to engage my ongoing love affair with the sound of my own voice. I was one of the very early developers behind SaltStack. Salt, for those who are unfamiliar, is a remote execution framework slash configuration management system that let me participate in code development. It turns out that when you have a pattern of merging every random pull request that some jackass winds up submitting, and then immediately submitting a follow up pull request that fixes everything you just merged in, it’s, first, not the most scalable thing in the world, but on balance provides such a wonderful welcoming community, that people become addicted to participating in it. And SaltStack nailed this in the early days.  Now, before this, I’d been focusing on configuration management in a variety of different ways. Some of the very early answers for this were CFEngine, which was written by an academic and is exactly what you would expect an academic to write. It feels more theoretical than it does something that you would want to use in production. But okay, Bcfg2 was something else in this space, and the fact that that is its name tells you everything you need to know about how user-friendly that was. And then the world shifted. We saw Puppet and Chef both arise. You can argue which came first, I don’t care enough in 2020 to have that conversation. But they wound up promoting a model of a domain-specific language, in Puppet’s case, versus chef where they decided, “All right, great, we’re gonna build this stuff out in Ruby.” From there, we then saw a further evolution of Ansible and SaltStack, which really round out the top four. Now, all of these systems fundamentally do the same thing, which is how do we wind up making the current state of a given system look like it should? That means, how do you make sure that certain packages are installed across all of your fleet? How do you make sure that the right users exist across your entire fleet? How do you guarantee that there are files in place, that have the right contents? And when the contents of those files change, how do you restart services? Effectively, how do you run arbitrary commands and converge the state of a remote system so it looks like it should? Because trying to manage systems at scale is awful.  You heard in a previous week what happened when I tried to run this sort of system by using a Distributed SSH client. Yeah, it turns out that mistakes are huge and hard to fix. This speaks toward the direction of moving into cattle instead of pets when it comes to managing systems. And all of these systems more or less took a different approach to it. And some were more aligned with how I saw the world than others did. So I started speaking about SaltStack back in 2012 and giving conference talks. The secret to giving a good conference talk, of course, is to give a whole bunch of really terrible ones first, and woo boy were these awful. I would put documentation on the slides. I would then read the documentation to people frantically trying to teach folks the ins and outs of a technical system in 45 minutes or less. It was about as engaging as it probably sounds like. Over time, I learned not to do that, but because no one else was speaking about SaltStack I was sort of in a rarefied position of being able to tell a story, and learn to tell stories, about a platform that I was passionate about, as it engaged a larger and larger community. Now, why am I talking about all of this on the Whiteboard Confessional? Excellent question.  But first, in the late 19th and early 20th centuries, democracy flourished around the world. This was good for most folks, but terrible for the log analytics industry because there was now a severe shortage of princesses to kidnap for ransom to pay for their ridiculous implementations. It doesn’t have to be that way. Consider <a href="http://chaossearch.io/" rel="nofollow" target="_blank">CHAOSSEARCH</a>. The data lives in your S3 buckets in your AWS accounts, and we know what that costs. You don’t have to deal with running massive piles of infrastructure to be able to query that log data with APIs you’ve come to know and tolerate, and they’re just good people to work with. Reach out to <a href="http://chaossearch.io/" rel="nofollow" target="_blank">CHAOSSEARCH.io</a>. And my thanks to them for sponsoring this incredibly depressing podcast.  The reason I bring up configuration management across the board is not because I want to talk about the pattern of doing terrible things within it, and oh, there are terrible patterns, but rather to talk about how in 2020, it is almost certainly an anti-pattern if you’re using it at all. Let me explain. The idea of configuration management was how you wind up getting a bunch of systems converged to a common state. Well, in the world of immutable infrastructure, in no small part ushered in by Docker, and later other systems like it, suddenly spinning a new container took less than a second, and it was exactly as you’d want it to be. So one-line configuration changes didn’t require redeploying fleets of servers, just iterating forward to the next version of the container. That, in turn, became a much better approach, and then you could get into a world where there wasn’t even an SSH daemon running, so people could not get into these containers in the first place to muck around and break the config, which was a big piece of why you had to have these things running continuously.  And over time, as we saw this pattern continued to emerge, the number of systems and use cases for which having something stateful that was managing this became fewer and fewer. Now, with the rise of cloud providers being what they are, you wind up having a slightly different problem, where, if I’m trying to provision a whole bunch of EC2 instances to do things, well, first, I can change the AMI, and we’ve been able to do that for 15 years, but config management was still a great plan back then. Now with things like auto-scaling groups, with containers running on top of these systems, you’re getting into a place where you’re using something like Chef for initial configuration of an instance, maybe, at best, but you’re not going to be running this in a persistent way. where it’s constantly checking in with a central control system anymore. In fact, an awful lot of the things that Chef historically would do are now being handled by user data, where you give a system a script to run on the first time that it boots. Once that script completes successfully, it reports into a load balancer whatnot as being ready to go. But even that is something of an anti-pattern now because historically, we had the problem of, “Okay, I’m going to have my golden image, my AMI that spins up everything that I’d want, then I’m going to hand off to Chef. So you would see these instances being spun up, and then it would take 20 minutes or so for it to run through all the various Chef recipes and pass the health checks.  That’s great, but if you need that instance to be ready sooner than that, that’s not really a great pattern. That, in turn, has led to people who are specialized in the configuration management space, feeling the ground beneath their feet start to erode. If you look at any seriously scaled out system, there’s a bit of that in there, but most of the stuff that matters, most of the things break systems at scale are being run through things like Terraform instead, where you’re provisioning exactly what needs to be there up front, letting your cloud provider get the right AMI or baseline image into production, and from there, everything else being handled by things like [etcd 00:10:22], or by which version of a container they’re running to service a particular service. Now, I don’t agree with all of the steps that people have taken down this path, in fact, I make fun of an awful lot on this and other shows, but the problem that we’re seeing that these things solve has not gone away, but it’s become much smaller, which means that, eventually, if you’re specializing in configuration management, as I once did, you’re going to take a look around and you’re going to realize that this is not a growth industry in any meaningful sense for these vendors who generally tend to charge on a per-host basis. Sure, there are existing large scale environments that are continuing to expand their usage of these services, but there isn’t nearly as much net new going on there, which means that we’re starting to see what looks like an industry in decline.  One of my common career tips is to imagine that I’m going to start a company tomorrow. And in the next 10 years, it’s going to grow from a tiny startup to a component of the S&amp;P 500. From a tiny company to a giant company. At what point during that evolution over the next 10 years, does that tiny startup become a customer of a particular tool or a particular market segment? If the answer is, it probably doesn’t, then whatever is serving that industry niche is almost certainly not going to survive longer term in its current state. Sure, there’s a giant long tail of existing big e-Enterprise offerings out there that are going to address that in a bunch of interesting and exciting ways, but that’s not something that I would necessarily hang my hat on as a technologist trying to focus on something that is going to be relevant for the next five to 10 years. I, instead, would prefer to focus on something that’s growing. That means today, cloud services? Absolutely. Serverless? Most definitely. Kubernetes? Unfortunately. Docker? Well, sort of. That’s largely not become its own independent skill set anymore. It’s started to slip below the surface of awareness of what people care about, in the same way as almost everything else we talk about on this show eventually will as well, leaving in the realm of a few highly advanced specialists to wind up playing around with them. And that, in short form, is what happened to my once passionate love affair with configuration management, now turning into something I basically don’t even bother to ask people about, because I can relatively safely assume that the answer is not nearly as relevant as it once was, to their technical success.  For the Whiteboard Confessional sub-series of the AWS Morning Brief, I’m Cloud Economist Corey Quinn, and I’ll talk to you next week. Thank you for joining us on Whiteboard Confessional. If you have terrifying ideas, please reach out to me on twitter at <a href="https://twitter.com/QuinnyPig" rel="nofollow" target="_blank">@quinnypig</a>, and let me know what I should talk about next time. Announcer: This has been a <a href="https://www.humblepod.com" rel="nofollow" target="_blank">HumblePod</a> production. Stay humble.

Whiteboard Confessional: Configuration MisManagement

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at The Duckbill Group, the author of the weekly Last Week in AWS newsletter, and the host of two podcasts: Screaming in the Cloud and, you guessed it, AWS Morning Brief, which you’re about to listen to.TranscriptCorey: Welcome to the AWS Morning Brief, specifically our 12-part mini series, Networking In The Cloud, sponsored by ThousandEyes. ThousandEyes recently released their state of the cloud benchmark performance report. They raced five clouds together and gave a comparative view of the networking strengths, weaknesses, and approaches of those various providers. Take a look at what it means for you. There's actionable advice hidden within, as well as incredibly useful comparative data, so you can start comparing apples to oranges instead of apples to baseballs. Check them out and get your copy today at snark.cloud/realclouds. That's snark.cloud/realclouds because Oracle cloud was not invited to participate.Now, one thing that they did not bother to talk about in that report, is how much all of that data transfer across different providers costs. Today I'd like to talk about that, which is a bit of a lie because I'm not here to talk about it at all, I'm here to rant like a freaking lunatic for which I make no apologies whatsoever.This episode is about data transfer pricing in AWS. Because honestly I need to rant about something and this topic is entirely too near and dear to my heart, given that I spend most of my time fixing AWS bills for interesting and various sophisticated clients.Let's begin with a simple question. The answer to which is guaranteed to piss you off like almost nothing else. What does it cost to move a gigabyte of data in AWS? Think about that for a second. The correct answer, of course, is that nobody freaking knows. There is no way to get a deterministic answer to that question without asking a giant boatload of other questions.Let me give you some examples, and before I do, I would like to call out that every number I'm about to mention applies only to us-east-1, because of course different regions in different places have varying costs, that every single one of these numbers is different in other places sometimes, but not always. Why? Because things are awful. I told you I was going to rant. I'm not apologizing for it at this point.Let's begin simply and talk about what it takes to just shove a gigabyte of data into AWS. Now in most cases that's free. Inbound bandwidth is always free to AWS usually, until it passes through with load balancer or does something else but we'll get there. What does it cost to move data between two AWS regions? Great. The answer to that is, two cents per gigabyte in the primary regions, except there's one use case which gets slightly less. And that is moving between us-east-1 and us-east-2. One is in Virginia, two is in Ohio. That is half price at one cent per gigabyte. My working theory behind that is that it's because even data wants to get the hell out of Ohio.Let's take it a step further. Let's say you were in an individual region. What does it cost to move data from 1-AZ to another? The documentation was exquisitely unclear, and I had to do some experiments with spinning up a few instances in otherwise empty AWS accounts, and using DD and Netcat to hurl data across various links to find out the answer and then wait till it showed up on my bill. The answer is it also costs 2 cents per gigabyte, the same cost as region to region. It's one cent per gigabyte out of an AZ and one cent per gigabyte in to an AZ. And that's right, it means you get charged twice. If you move 10 gigabytes, you are charged for 20 gigabytes on that particular metric.This also has the fun ancillary side effect of meaning that moving data between Virginia and Ohio is cheaper to do that cross region transfer than it is to move that same data within an existing region. Oh wait, it gets dumber than that. What do load balancer data transfer fees look like? The correct answer is who the hell knows? On the old classic load balancers, it was 0.8 cents per gigabyte in or out to the internet and there was also an instance fee, but that's not what we're talking about today. Traffic from any existing load balancer today to something inside of an AZ is free unless it crosses an availability zone and then we're back into cross AZ data transfer territory and anything going from an availability zone to a load balancer costs one cent per gigabyte.Now the newer load balancer generations, the ALDs and the NLDS, what does that cost? Nobody freaking knows because data throughput is just one of several dimensions that go into a load balancer capacity unit, which mean that what your data transfer price is going to look like is going to vary wildly because in this particular case, it's not data transfer itself. There's still that as it leaves, but you also have to pay for this as an additional through the load balancer fee, but it's blended into an LCU, so it's not at all obvious at times that that is in fact what you're being billed for.In another episode of this mini series, we talked about global accelerator. Now there's a site to site VPN option, which they had for a while, but at re:Invent last year they announced a accelerated VPN option that leverages a lot of global accelerator technology to let that site to site VPN take advantage significantly of the global accelerator. Now what does that cost? I could not freaking tell you. There are, I am not exaggerating, five distinct billing line items, if you run an accelerated site to site VPN and of course, all of them cost you money. I am not exaggerating. That is the actual state of the world. It is incredibly annoying. It is so annoying that I'm going to have to take a break before I blow a blood vessel to tell you more about ThousandEyes instead.So other than the cloud report, what is ThousandEyes? They effectively act as the global observer that watches the entire internet from a whole bunch of different listening posts around that internet and keeps track in near real time of what's going on, what's being slow, what providers are having issues and giving information directly to your folks on your side to be able to understand, adapt and mitigate those outages and slow downs. It helps immediately get to the point of is this a networking problem globally or is it our last crappy code deploy that broke things? If this sounds like something that might be useful for you or your team, I encourage you to check them out at thousandeyes.com. They're a fantastic company with a fantastic product and best of all their billing makes sense.We're back to ranting again. That's right. My problem with the AWS data transfer pricing is not that it's shitty and complex, but also that it's expensive. Pricing largely has not changed since AWS launched and you're effectively seeing 1998 bandwidth prices as a direct result of this. In data center land, the way this works is you pay for a link between two places and however much traffic you put over it, you're charged at the 95th percentile, so you can have bursts and spikes that exceed that limit, but you're paying effectively a flat rate for whatever your throughput looks like over the course of the billing period. It's not the most straightforward thing in the world, but it's a lot less expensive than you wind up paying for the same thing in the cloud.Somehow AWS has managed to successfully convince an entire generation of companies that bandwidth is a rare, precious, expensive commodity. Unless of course it's bandwidth directly into AWS from the internet, in which case it is of course free, and you can have as much of that as you want. Data checks in, it doesn't check out. This in turn leads to a lot of weird patterns. For example, if you have a mobile app that winds up reporting data to something that lives in an AWS region, rather than having that replicated on your dime, you could theoretically have that mobile app report it to two different regions. It doubles your user's bandwidth on potentially a mobile plan, but it saves you money. How crappy of a dynamic is that?Now there are other services that wind up leveraging aspects of data transfer pricing in obnoxious ways and there are ways around this too. PrivateLink, for example, a link between two different VPCs, in some cases in different accounts, saves you money on the data exodus charge so you don't have to go across the internet to do it. Great, sure, that's right. It drops it down to one cent per gigabyte in each direction, but you're still paying, at scale, a significant amount of money for what is in effect AWS just moving data around its internal network.Direct Connect, the service that links a AWS VPC to your on premises data center also saves you money on that for data out that traverses from AWS to your data center, but in reverse, it costs you more than using the internet because again, ingress is free. You could theoretically have data if you're doing a large copy from your data center into AWS, use the public internet and put it directly into S3 or something like that. Why? Because this entire story is a carnival of bullshit. It's awful. Nobody likes it.Let's talk about CloudFront. AWS's CDN product. It's kind of spendy as far as CDNs go, not horribly so, but what's fun about this is first off, what it costs for data in or out of CloudFront varies depending upon what region that data access is coming from and you don't have fine grain control over any of that. You don't know where your customers are going to come from and in some cases you can pay three times more if a customer accesses your same application with the same traffic pattern from different parts of the world than, if they do it from down the street from your office. And what's even more obnoxious about this is they still have an obnoxious competitive advantage over other CDN offerings because unlike anyone else, they can privilege their own environment and say that traffic from the origin to the CloudFront distribution is free. No one else can do that. So if you have a lot of data that you're pushing through CloudFront that isn't easily cached, that is an advantage that CloudFront has that makes it very expensive to even consider using something else.This ties into a bigger challenge, notice as well that AWS has a large pile of, let's call them substandard offerings in some respects, that are managed service equivalents of things you could build and run yourself, RDS for a bunch of database offerings. Amazon Elasticsearch, Neptune, DocumentDB, and a whole host of other managed services that have presences in multiple availability zones, offer free replication of that data when you use the managed service. So if you can tolerate the obnoxious sharp edges of those managed services and it works well enough for your use case, they have a tremendous leg up over running your own implementation of the open source products that those things ape. Or having a third party vendor that manages that service for you because one way or another, unless you're using the AWS version of it, either the vendor who's managing it for you or you, have to pay that data transfer charge between AZ's. In some cases, both of you do. Only Amazon gets to ride data transfer between AZ's for free.This one isn't particularly a data transfer charge directly, although it looks a lot like one, I speak of the managed NAT Gateway, there's a data processing fee in addition to the instance fee, of four and a half cents per gigabyte. That doesn't sound like a lot until you realize that if you have a web facing application that you put in a private subnet that needs to talk to the outside world, every gigabyte you put through costs four and a half cents, that becomes a massive expense. If you run your own NAT Instances, sure you have more overhead, the hourly charge is about the same as you'll spend for the managed NAT Gateway, but the data processing fee completely vanishes. Remember as well that that data processing fee is in addition to any data transfer fees you would be paying. So you go from four and a half cents per gig, to nothing.Similarly and very tightly related, if you forget to put a free S3 Gateway endpoint in a private subnet that has a managed NAT Gateway in it, every gigabyte of data that you transfer into S3, in turn winds up having to incur that four and a half cent per gigabyte charge. That is the same it costs to store that gigabyte and us-east-1 for just shy of two months. That's enormous. You can make it free, but it's laid there for you as a trap for the unwary.Lastly, if you're thinking that all regions are equivalent, they're not. You can see four to five times the expense, if you go to the region in Sao Paulo. That is not entirely AWS's fault, it's largely due, to my understanding, from telecom monopolies in that part of the world that make bandwidth incredibly expensive for virtually everyone. That's not something that I can fault them with, but it's still irritating.What I can fault them with is that absolutely all of this, everything that I have spoken about today shows up on your bill in a whole mess of places that are incredibly difficult to unpack. Sometimes you're charged for the same data multiple times in different places and heaven forbid you try to figure out what's caused the change, which application workload do you have in that account that is suddenly responsible for a whole lot of AZ crosstalk? The only real answer without efficient tagging and the foresight to have those tags in place before this moment, is looking at VPC flow logs, which you've enabled, right? But those are annoying and confusing and difficult to parse as well.Look, I'm not saying that this is intentional on AWS's part, I'm saying this is just the opposite. I'm saying when it comes to data transfer pricing, it is very clear that no one is effectively minding the store. The folks who build out the networks and the folks who handle pricing and the folks who work on service teams that leverage both of those things, apparently none of those people are allowed to talk to one another. Data transfer is the white space between AWS services. The next time your AWS account manager asks how they can help you with anything, please be sure to yell at them about the data transfer pricing portion of your bill. It's the only way that this apparently will one day get fixed for everyone.I'm cloud economist Corey Quinn. This is the Networking In The Cloud mini series. If you've enjoyed this podcast, please leave a five star review and a comment telling me what you liked. If you didn't like this podcast, please leave a five star review anyway, and a comment telling me what my problem is.Announcer: This has been a HumblePod production. Stay humble.

About Corey QuinnOver the course of my career, I’ve worn many different hats in the tech world: systems administrator, systems engineer, director of technical operations, and director of DevOps, to name a few. Today, I’m a cloud economist at <a href="https://www.lastweekinaws.com/" rel="nofollow" target="_blank">The Duckbill Group</a>, the author of the weekly <a href="https://www.lastweekinaws.com/newsletter/" rel="nofollow" target="_blank">Last Week in AWS</a> newsletter, and the host of two podcasts: <a href="https://www.screaminginthecloud.com" rel="nofollow" target="_blank">Screaming in the Cloud</a> and, you guessed it, <a href="https://www.lastweekinaws.com/podcast/aws-morning-brief/" rel="nofollow" target="_blank">AWS Morning Brief</a>, which you’re about to listen to.Transcript Corey: Welcome to the AWS Morning Brief, specifically our 12-part mini series, Networking In The Cloud, sponsored by ThousandEyes. ThousandEyes recently released their state of the cloud benchmark performance report. They raced five clouds together and gave a comparative view of the networking strengths, weaknesses, and approaches of those various providers. Take a look at what it means for you. There's actionable advice hidden within, as well as incredibly useful comparative data, so you can start comparing apples to oranges instead of apples to baseballs. Check them out and get your copy today at <a href="http://snark.cloud/realclouds" rel="nofollow" target="_blank">snark.cloud/realclouds</a>. That's <a href="http://snark.cloud/realclouds" rel="nofollow" target="_blank">snark.cloud/realclouds</a> because Oracle cloud was not invited to participate. Now, one thing that they did not bother to talk about in that report, is how much all of that data transfer across different providers costs. Today I'd like to talk about that, which is a bit of a lie because I'm not here to talk about it at all, I'm here to rant like a freaking lunatic for which I make no apologies whatsoever. This episode is about data transfer pricing in AWS. Because honestly I need to rant about something and this topic is entirely too near and dear to my heart, given that I spend most of my time fixing AWS bills for interesting and various sophisticated clients. Let's begin with a simple question. The answer to which is guaranteed to piss you off like almost nothing else. What does it cost to move a gigabyte of data in AWS? Think about that for a second. The correct answer, of course, is that nobody freaking knows. There is no way to get a deterministic answer to that question without asking a giant boatload of other questions. Let me give you some examples, and before I do, I would like to call out that every number I'm about to mention applies only to us-east-1, because of course different regions in different places have varying costs, that every single one of these numbers is different in other places sometimes, but not always. Why? Because things are awful. I told you I was going to rant. I'm not apologizing for it at this point. Let's begin simply and talk about what it takes to just shove a gigabyte of data into AWS. Now in most cases that's free. Inbound bandwidth is always free to AWS usually, until it passes through with load balancer or does something else but we'll get there. What does it cost to move data between two AWS regions? Great. The answer to that is, two cents per gigabyte in the primary regions, except there's one use case which gets slightly less. And that is moving between us-east-1 and us-east-2. One is in Virginia, two is in Ohio. That is half price at one cent per gigabyte. My working theory behind that is that it's because even data wants to get the hell out of Ohio. Let's take it a step further. Let's say you were in an individual region. What does it cost to move data from 1-AZ to another? The documentation was exquisitely unclear, and I had to do some experiments with spinning up a few instances in otherwise empty AWS accounts, and using DD and Netcat to hurl data across various links to find out the answer and then wait till it showed up on my bill. The answer is it also costs 2 cents per gigabyte, the same cost as region to region. It's one cent per gigabyte out of an AZ and one cent per gigabyte in to an AZ. And that's right, it means you get charged twice. If you move 10 gigabytes, you are charged for 20 gigabytes on that particular metric. This also has the fun ancillary side effect of meaning that moving data between Virginia and Ohio is cheaper to do that cross region transfer than it is to move that same data within an existing region. Oh wait, it gets dumber than that. What do load balancer data transfer fees look like? The correct answer is who the hell knows? On the old classic load balancers, it was 0.8 cents per gigabyte in or out to the internet and there was also an instance fee, but that's not what we're talking about today. Traffic from any existing load balancer today to something inside of an AZ is free unless it crosses an availability zone and then we're back into cross AZ data transfer territory and anything going from an availability zone to a load balancer costs one cent per gigabyte. Now the newer load balancer generations, the ALDs and the NLDS, what does that cost? Nobody freaking knows because data throughput is just one of several dimensions that go into a load balancer capacity unit, which mean that what your data transfer price is going to look like is going to vary wildly because in this particular case, it's not data transfer itself. There's still that as it leaves, but you also have to pay for this as an additional through the load balancer fee, but it's blended into an LCU, so it's not at all obvious at times that that is in fact what you're being billed for. In another episode of this mini series, we talked about global accelerator. Now there's a site to site VPN option, which they had for a while, but at re:Invent last year they announced a accelerated VPN option that leverages a lot of global accelerator technology to let that site to site VPN take advantage significantly of the global accelerator. Now what does that cost? I could not freaking tell you. There are, I am not exaggerating, five distinct billing line items, if you run an accelerated site to site VPN and of course, all of them cost you money. I am not exaggerating. That is the actual state of the world. It is incredibly annoying. It is so annoying that I'm going to have to take a break before I blow a blood vessel to tell you more about <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a> instead. So other than the cloud report, what is <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">ThousandEyes</a>? They effectively act as the global observer that watches the entire internet from a whole bunch of different listening posts around that internet and keeps track in near real time of what's going on, what's being slow, what providers are having issues and giving information directly to your folks on your side to be able to understand, adapt and mitigate those outages and slow downs. It helps immediately get to the point of is this a networking problem globally or is it our last crappy code deploy that broke things? If this sounds like something that might be useful for you or your team, I encourage you to check them out at <a href="https://www.thousandeyes.com/" rel="nofollow" target="_blank">thousandeyes.com</a>. They're a fantastic company with a fantastic product and best of all their billing makes sense. We're back to ranting again. That's right. My problem with the AWS data transfer pricing is not that it's shitty and complex, but also that it's expensive. Pricing largely has not changed since AWS launched and you're effectively seeing 1998 bandwidth prices as a direct result of this. In data center land, the way this works is you pay for a link between two places and however much traffic you put over it, you're charged at the 95th percentile, so you can have bursts and spikes that exceed that limit, but you're paying effectively a flat rate for whatever your throughput looks like over the course of the billing period. It's not the most straightforward thing in the world, but it's a lot less expensive than you wind up paying for the same thing in the cloud. Somehow AWS has managed to successfully convince an entire generation of companies that bandwidth is a rare, precious, expensive commodity. Unless of course it's bandwidth directly into AWS from the internet, in which case it is of course free, and you can have as much of that as you want. Data checks in, it doesn't check out. This in turn leads to a lot of weird patterns. For example, if you have a mobile app that winds up reporting data to something that lives in an AWS region, rather than having that replicated on your dime, you could theoretically have that mobile app report it to two different regions. It doubles your user's bandwidth on potentially a mobile plan, but it saves you money. How crappy of a dynamic is that? Now there are other services that wind up leveraging aspects of data transfer pricing in obnoxious ways and there are ways around this too. PrivateLink, for example, a link between two different VPCs, in some cases in different accounts, saves you money on the data exodus charge so you don't have to go across the internet to do it. Great, sure, that's right. It drops it down to one cent per gigabyte in each direction, but you're still paying, at scale, a significant amount of money for what is in effect AWS just moving data around its internal network. Direct Connect, the service that links a AWS VPC to your on premises data center also saves you money on that for data out that traverses from AWS to your data center, but in reverse, it costs you more than using the internet because again, ingress is free. You could theoretically have data if you're doing a large copy from your data center into AWS, use the public internet and put it directly into S3 or something like that. Why? Because this entire story is a carnival of bullshit. It's awful. Nobody likes it. Let's talk about CloudFront. AWS's CDN product. It's kind of spendy as far as CDNs go, not horribly so, but what's fun about this is first off, what it costs for data in or out of CloudFront varies depending upon what region that data access is coming from and you don't have fine grain control over any of that. You don't know where your customers are going to come from and in some cases you can pay three times more if a customer accesses your same application with the same traffic pattern from different parts of the world than, if they do it from down the street from your office. And what's even more obnoxious about this is they still have an obnoxious competitive advantage over other CDN offerings because unlike anyone else, they can privilege their own environment and say that traffic from the origin to the CloudFront distribution is free. No one else can do that. So if you have a lot of data that you're pushing through CloudFront that isn't easily cached, that is an advantage that CloudFront has that makes it very expensive to even consider using something else. This ties into a bigger challenge, notice as well that AWS has a large pile of, let's call them substandard offerings in some respects, that are managed service equivalents of things you could build and run yourself, RDS for a bunch of database offerings. Amazon Elasticsearch, Neptune, DocumentDB, and a whole host of other managed services that have presences in multiple availability zones, offer free replication of that data when you use the managed service. So if you can tolerate the obnoxious sharp edges of those managed services and it works well enough for your use case, they have a tremendous leg up over running your own implementation of the open source products that those things ape. Or having a third party vendor that manages that service for you because one way or another, unless you're using the AWS version of it, either the vendor who's managing it for you or you, have to pay that data transfer charge between AZ's. In some cases, both of you do. Only Amazon gets to ride data transfer between AZ's for free. This one isn't particularly a data transfer charge directly, although it looks a lot like one, I speak of the managed NAT Gateway, there's a data processing fee in addition to the instance fee, of four and a half cents per gigabyte. That doesn't sound like a lot until you realize that if you have a web facing application that you put in a private subnet that needs to talk to the outside world, every gigabyte you put through costs four and a half cents, that becomes a massive expense. If you run your own NAT Instances, sure you have more overhead, the hourly charge is about the same as you'll spend for the managed NAT Gateway, but the data processing fee completely vanishes. Remember as well that that data processing fee is in addition to any data transfer fees you would be paying. So you go from four and a half cents per gig, to nothing. Similarly and very tightly related, if you forget to put a free S3 Gateway endpoint in a private subnet that has a managed NAT Gateway in it, every gigabyte of data that you transfer into S3, in turn winds up having to incur that four and a half cent per gigabyte charge. That is the same it costs to store that gigabyte and us-east-1 for just shy of two months. That's enormous. You can make it free, but it's laid there for you as a trap for the unwary. Lastly, if you're thinking that all regions are equivalent, they're not. You can see four to five times the expense, if you go to the region in Sao Paulo. That is not entirely AWS's fault, it's largely due, to my understanding, from telecom monopolies in that part of the world that make bandwidth incredibly expensive for virtually everyone. That's not something that I can fault them with, but it's still irritating. What I can fault them with is that absolutely all of this, everything that I have spoken about today shows up on your bill in a whole mess of places that are incredibly difficult to unpack. Sometimes you're charged for the same data multiple times in different places and heaven forbid you try to figure out what's caused the change, which application workload do you have in that account that is suddenly responsible for a whole lot of AZ crosstalk? The only real answer without efficient tagging and the foresight to have those tags in place before this moment, is looking at VPC flow logs, which you've enabled, right? But those are annoying and confusing and difficult to parse as well. Look, I'm not saying that this is intentional on AWS's part, I'm saying this is just the opposite. I'm saying when it comes to data transfer pricing, it is very clear that no one is effectively minding the store. The folks who build out the networks and the folks who handle pricing and the folks who work on service teams that leverage both of those things, apparently none of those people are allowed to talk to one another. Data transfer is the white space between AWS services. The next time your AWS account manager asks how they can help you with anything, please be sure to yell at them about the data transfer pricing portion of your bill. It's the only way that this apparently will one day get fixed for everyone. I'm cloud economist Corey Quinn. This is the Networking In The Cloud mini series. If you've enjoyed this podcast, please leave a five star review and a comment telling me what you liked. If you didn't like this podcast, please leave a five star review anyway, and a comment telling me what my problem is. Announcer: This has been a HumblePod production. Stay humble.

Networking in the Cloud Fundamentals: Data Transfer Pricing

AWS Morning Brief for Wednesday, December 11th, 2019

reInvent Wrap-up, Part 3

AWS Morning Brief for the week of December 23rd, 2019.

It's a Horrible Lyfebin

The best podcast episodes on Startups. For anyone working in a startup, or anyone interested in the incredible stories behind how people built companies like Instagram, and Facebook. Offering both practical advice and insights, as well as the stories for how many of the biggest startups got to where they are today. Add episodes you're interested into your queue below!

OwlTail

Best Startups Podcasts

The best podcast episodes on romantic relationships. Episodes talk about the 5 love languages, intimacy, and the taboo topics of sexual desires and infidelity. This topic is a deep dive into the science and complexities that face us in our relationships with others. Add episodes you're interested into your queue below!