SRE Weekly - SRE Weekly Issue #279
Articles
This is a presentation by Laura Nolan (with text transcript) all about cascading failure, what causes it, how to avoid it, and how to deal with it when it happens.
I love how succinct this is:
[…] in any system where we design to fail over, so any mechanism at all that redistributes load from a failed component to still working components, we create the potential for a cascading failure to happen.
Laura Nolan — Slack (presented at InfoQ)
It’s so easy to explain an incident by describing how management could have prevented it from investing additional resources.
Lorin goes on to explain the “trap” part: it’s easy to stop investigating an incident too soon and declare the cause “greedy executives”, preventing us from learning more.
Lorin Hochstein
They redesigned one of their caching systems in 2020, and it paid off handsomely during the GameStop saga. This article discusses the redesign and considers what would have happened without it.
Garrett Hoffman — Reddit
The lessons are:
- Do retrospectives for small incidents first.
- Do a retrospective soon after the incident.
- Alert on the user experience.
All great advice, and #1 is an interesting idea I hadn’t heard before.
Robert Ross — FireHydrant
We can’t engineer reliability in a vacuum. This is a great explainer on how SRE siloing happens, the problems it causes, and how to break SRE out of its shell.
JJ Tang — Rootly
This ASRS (Aviation Safety Reporting System) Callback issue has some real-world examples of resilient systems in action.
Nasa Asrs
Facing a common kubernetes node failure modes, Cloudflare uses open source tools (one published by them) to perform automatic restarts.
In the past 30 days, we’ve used the above automatic node remediation process to action 571 nodes. That has saved our humans a considerable amount of time.
Andrew DeMaria — Cloudflare
Outages
|
Older messages
SRE Weekly Issue #278
Monday, July 12, 2021
View on sreweekly.com A message from our sponsor, StackHawk: Learn how our team at StackHawk tests external cookie authentication using Ktor, and check out some of the helper functions we wrote to make
SRE Weekly Issue #277
Monday, July 5, 2021
View on sreweekly.com A message from our sponsor, StackHawk: Planelty saved weeks of work by implementing StackHawk instead of building an internal ZAP service. See how: https://sthwk.com/planetly-
SRE Weekly Issue #276
Monday, June 28, 2021
View on sreweekly.com A message from our sponsor, StackHawk: Get ready for some GraphQL! Tune in this Tuesday, June 29 at 9 AM PT for an automated GraphQL security testing learning lab. Register: http:
SRE Weekly Issue #275
Monday, June 21, 2021
View on sreweekly.com A message from our sponsor, StackHawk: Join ZAP Founder & Project Lead Simon Bennetts on June 30 for a live AMA where he will be answering questions on all things open source
SRE Weekly Issue #274
Monday, June 14, 2021
View on sreweekly.com A message from our sponsor, StackHawk: Join the GraphQL Security Testing Learning Lab on June 29 at 9 AM PT. Learn how to run automated security testing against your GraphQL APIs
You Might Also Like
A deal made in cloud security heaven
Thursday, April 18, 2024
Meta's Llama 3 goes public and hackers hold World-Check data for ransom View this email online in your browser By Christine Hall Thursday, April 18, 2024 Welcome to TechCrunch PM! I'm glad you
💎 Issue 413 - RubyJS-Vite
Thursday, April 18, 2024
This week's Awesome Ruby Newsletter Read this email on the Web The Awesome Ruby Newsletter Issue » 413 Release Date Apr 18, 2024 Your weekly report of the most popular Ruby news, articles and
💻 Issue 406 - Swift for C++ Practitioners, Part 1
Thursday, April 18, 2024
This week's Awesome .NET Weekly Read this email on the Web The Awesome .NET Weekly Issue » 406 Release Date Apr 18, 2024 Your weekly report of the most popular .NET news, articles and projects
💻 Issue 413 - How to implement HLS Video Streaming in a React App
Thursday, April 18, 2024
This week's Awesome Node.js Weekly Read this email on the Web The Awesome Node.js Weekly Issue » 413 Release Date Apr 18, 2024 Your weekly report of the most popular Node.js news, articles and
📱 Issue 407 - Textual Healing: iOS Text Editing Minutiae
Thursday, April 18, 2024
This week's Awesome iOS Weekly Read this email on the Web The Awesome iOS Weekly Issue » 407 Release Date Apr 18, 2024 Your weekly report of the most popular iOS news, articles and projects Popular
💻 Issue 413 - Interview with Senior JavaScript Developer 2024 [video]
Thursday, April 18, 2024
This week's Awesome JavaScript Weekly Read this email on the Web The Awesome JavaScript Weekly Issue » 413 Release Date Apr 18, 2024 Your weekly report of the most popular JavaScript news, articles
💻 Issue 331 - 30+ app ideas with complete source code
Thursday, April 18, 2024
This week's Awesome React Weekly Read this email on the Web The Awesome React Weekly Issue » 331 Release Date Apr 18, 2024 Your weekly report of the most popular React news, articles and projects
💻 Issue 408 - Curl: Hyper, is it worth it?
Thursday, April 18, 2024
This week's Awesome Rust Weekly Read this email on the Web The Awesome Rust Weekly Issue » 408 Release Date Apr 18, 2024 Your weekly report of the most popular Rust news, articles and projects
📱 Issue 410 - Swift for C++ Practitioners, Part 1
Thursday, April 18, 2024
This week's Awesome Swift Weekly Read this email on the Web The Awesome Swift Weekly Issue » 410 Release Date Apr 18, 2024 Your weekly report of the most popular Swift news, articles and projects
🤷🏻♂️ What to Do When Windows Won't Boot — How to Try the Android 15 Beta
Thursday, April 18, 2024
Also: We Tried a Small AI Voice Recorder, and More! How-To Geek Logo April 18, 2024 📩 Get expert reviews, the hottest deals, how-to's, breaking news, and more delivered directly to your inbox by