SRE Weekly Issue #279

View on

A message from our sponsor, StackHawk:

On July 28, ZAP Creator Simon Bennetts is giving a first look at ZAP’s new automation framework. Grab your spot:


This is a presentation by Laura Nolan (with text transcript) all about cascading failure, what causes it, how to avoid it, and how to deal with it when it happens.

I love how succinct this is:

[…] in any system where we design to fail over, so any mechanism at all that redistributes load from a failed component to still working components, we create the potential for a cascading failure to happen.

Laura Nolan — Slack (presented at InfoQ)

It’s so easy to explain an incident by describing how management could have prevented it from investing additional resources.

Lorin goes on to explain the “trap” part: it’s easy to stop investigating an incident too soon and declare the cause “greedy executives”, preventing us from learning more.

Lorin Hochstein

They redesigned one of their caching systems in 2020, and it paid off handsomely during the GameStop saga. This article discusses the redesign and considers what would have happened without it.

Garrett Hoffman — Reddit

The lessons are:

  1. Do retrospectives for small incidents first.
  2. Do a retrospective soon after the incident.
  3. Alert on the user experience.

All great advice, and #1 is an interesting idea I hadn’t heard before.

Robert Ross — FireHydrant

We can’t engineer reliability in a vacuum. This is a great explainer on how SRE siloing happens, the problems it causes, and how to break SRE out of its shell.

JJ Tang — Rootly

This ASRS (Aviation Safety Reporting System) Callback issue has some real-world examples of resilient systems in action.

Nasa Asrs

Facing a common kubernetes node failure modes, Cloudflare uses open source tools (one published by them) to perform automatic restarts.

In the past 30 days, we’ve used the above automatic node remediation process to action 571 nodes. That has saved our humans a considerable amount of time.

Andrew DeMaria — Cloudflare


This email was sent to you
why did I get this?    unsubscribe from this list    update subscription preferences
SRE Weekly · PO Box 253 · South Lancaster, MA 01561-0253 · USA

Older messages

SRE Weekly Issue #278

Monday, July 12, 2021

View on A message from our sponsor, StackHawk: Learn how our team at StackHawk tests external cookie authentication using Ktor, and check out some of the helper functions we wrote to make

SRE Weekly Issue #277

Monday, July 5, 2021

View on A message from our sponsor, StackHawk: Planelty saved weeks of work by implementing StackHawk instead of building an internal ZAP service. See how:

SRE Weekly Issue #276

Monday, June 28, 2021

View on A message from our sponsor, StackHawk: Get ready for some GraphQL! Tune in this Tuesday, June 29 at 9 AM PT for an automated GraphQL security testing learning lab. Register: http:

SRE Weekly Issue #275

Monday, June 21, 2021

View on A message from our sponsor, StackHawk: Join ZAP Founder & Project Lead Simon Bennetts on June 30 for a live AMA where he will be answering questions on all things open source

SRE Weekly Issue #274

Monday, June 14, 2021

View on A message from our sponsor, StackHawk: Join the GraphQL Security Testing Learning Lab on June 29 at 9 AM PT. Learn how to run automated security testing against your GraphQL APIs

JSK Weekly - August 05, 2021

Thursday, August 5, 2021

Happy Work Like A Dog Day everyone! Depending on what kind of dog you have, that could either mean hard working, lazing around all day or just simply playing in the sun all day but which ever it is,

The numbers don't lie | Act now to save on Disrupt passes

Thursday, August 5, 2021

Don't miss out on $100 savings before Friday Your Logo Disrupt Email Header. Event is September 21 to 23 Sign up to network with 10k TechCrunch enthusiasts at Disrupt Great news for budget-

A terminal dashboard for K8s, semantic grep for code, and a GitHub/GitLab alternative

Thursday, August 5, 2021

StackShare Weekly Email not displaying correctly? View it in your browser. StackShare Weekly Digest August 5th, 2021 Sponsored by CircleCI. Let CircleCI focus on CI/CD, so you can build the next big

Infographic | Visualizing the 4,000-Year History of Global Power 💪

Thursday, August 5, 2021

We examine an ambitious timeline that details the power of various civilizations going all the way back to 2000 BC TIMELESS Histomap: Visualizing the 4000 Year History of Global Power We examine an

Issue 175 - Tesla's mobile app redesign

Thursday, August 5, 2021

🎨 View this email in your browser If you are just now finding out about Tesletter, you can subscribe here! If you already know Tesletter and want to support us, check out our Patreon page If you have

 Automate The Planet- Compelling Tuesday

Thursday, August 5, 2021

Compelling Tuesday xUnit Tutorial | Part 7 | Geolocation Testing Using xUnit The last module of my XUnit course. There I discuss what Geolocation testing is and how to write such automated tests. Check

Programmer Weekly - Issue 66

Thursday, August 5, 2021

View this email in your browser Programmer Weekly Welcome to issue 66 of Programmer Weekly. Let's get straight to the links this week. From Our Sponsor Retool: The Fastest Way To Build Internal

Daily Coding Problem: Problem #486 [Medium]

Thursday, August 5, 2021

Daily Coding Problem Good morning! Here's a solution to yesterday's problem. This is your coding interview problem for today. This problem was asked by Pinterest. At a party, there is a single

New Course: Integrate Combine Into an App!

Thursday, August 5, 2021

Hey there! We have a new course for you. 🎥 Integrate Combine Into an App iOS & SWIFT • 27 MIN • INTERMEDIATE Check it out! Learn Combine in iOS by practice—while building a Chuck Norris jokes app!

Python Weekly - Issue 511

Thursday, August 5, 2021

View this email in your browser Python Weekly Welcome to issue 511 of Python Weekly. Let's get straight to the links this week. From Our Sponsor SonarLint Free and Open Source IDE Extension for