SRE Weekly - SRE Weekly Issue #283

We needed a way to deploy our new service seamlessly, and to roll back that deploy should something go wrong. Ultimately many, many, things did go wrong, and every bit of failure tolerance put into the system proved to be worth its weight in gold because none of this was visible to customers.

Geoffrey Plouviez — Cloudflare

I especially like the idea of tailoring retrospective documents to disparate audiences — you may have more than you realize.

Emily Arnott — Blameless

An analysis of two incidents from the venerable John Allspaw.  These are from 2012 back when he was at Etsy, and yet there’s still a ton we can learn now by reading them.

John Allspaw — Etsy

An account of how Gojek responds to production issues, and why the RCA is a critical part of the process.

Sooraj Rajmohan — Gojek

Type carefully… or rather, design resilient systems.

JJ Tang — Rootly

Requiring development teams to fully own their services can lead to siloing and redundancy. Heroku works to ameliorate that by embedding SREs in development teams.

Johnny Boursiquot — Salesforce (presented at QCon)

I’ve shared some articles here suggesting doing away with incident metrics like MTTR entirely. This author says that they are useful, but the numbers must be properly ccontextualized.

Vanessa Huerta Granda — Learning From Incidents

Everything could be fine, or we could failing to report or missing problems altogether — we’re flying blind.

Chris Evans — incident.io