SRE Weekly - SRE Weekly Issue #437

View on sreweekly.com

This week's issue is entirely focused on the CrowdStrike incident: more details on what happened, analysis, and learnings. I'll be back next week with a selection of all of the great stuff you folks have been writing while I've been off on vacation for the past two weeksmy RSS reader is packed with awesomeness!

A message from our sponsor, FireHydrant:

Migrate off of PagerDuty, save money, and then have all of your configuration exported as Terraform modules? We did that. We know one of the hardest parts of leaving a legacy tool is the old configuration, that's why we dedicated time to build the Signals migrator, making it easy to switch.

https://firehydrant.com/blog/speedrun-to-signals-automated-migrations-are-here/

This week, CrowdStrike posted quite a bit more detail about what happened on July 19. The short of it seems to be an argument count mismatch, but as with any incident of this sort, there are multiple contributing factors.

The report also continues the conversation about the use of kernel mode in a product such as this, amounting to a public conversation with Microsoft that is intriguing to watch from the outside.

  CrowdStrike

This article has some interesting details about antitrust regulations(!) related to security vendors running code in kernel mode. There's also an intriguing story of a very similar crash on Linux endpoints running CrowdStrike's Falcon.

Note: this one is from a couple of weeks ago and some of its conjectures don't quite line up with details that have been released in the interim.

  Gergely Orosz

While it mentions the CrowdStrike incident only in vague terms, this article discusses why slowly rolling out updates isn't a universal solution and can bring its own problems.

  Chris Siebenmann

Some thoughts on staged rollouts and the CrowdStrike outage:

The notion we tried to get known far and wide was "nothing goes everywhere at once".

Note that this post was published before CrowdStrike's RCA which subsequently confirmed that their channel file updates were not deployed through staged rollouts.

  rachelbythebay

[...] there may be risks in your system that haven’t manifested as minor outages.

Jumping off from the CrowdStrike incident, this one asks us to look for reliability problems in parts of our infrastructure that we've grown to trust.

  Lorin Hochstein

While CrowdStrike's RCA has quite a bit of technical detail, this post reminds us that we need a lot more context to really understand how an incident came to be.

  Lorin Hochstein

In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code.

I didn't realize that Microsoft is working on eBPF for Windows.

  Brendan Gregg

This post isn't about what Crowdstrike should have done. Instead, I use the resources to provide context and takeaways we can apply to our teams and organizations.

  Bob Walker — Octopus Deploy







This email was sent to you
why did I get this?    unsubscribe from this list    update subscription preferences
SRE Weekly, a production of Tinker Tinker Tinker, LLC · PO Box 253 · South Lancaster, MA 01561-0253 · USA

Older messages

SRE Weekly Issue #436

Monday, August 5, 2024

View on sreweekly.com A message from our sponsor, FireHydrant: Migrate off of PagerDuty, save money, and then have all of your configuration exported as Terraform modules? We did that. We know one of

SRE Weekly Issue #435

Monday, July 29, 2024

View on sreweekly.com A message from our sponsor, FireHydrant: We've gone all out on our new integration with Microsoft Teams. If you're a MS Teams user, FireHydrant now supports the most

SRE Weekly Issue #434

Monday, July 22, 2024

View on sreweekly.com A message from our sponsor, FireHydrant: We've gone all out on our new integration with Microsoft Teams. If you're a MS Teams user, FireHydrant now supports the most

SRE Weekly Issue #433

Monday, July 15, 2024

View on sreweekly.com A message from our sponsor, FireHydrant: We've gone all out on our new integration with Microsoft Teams. If you're a MS Teams user, FireHydrant now supports the most

SRE Weekly Issue #432

Monday, July 8, 2024

View on sreweekly.com A message from our sponsor, FireHydrant: We've gone all out on our new integration with Microsoft Teams. If you're a MS Teams user, FireHydrant now supports the most

You Might Also Like

🔎 How to Search Reddit Like a Pro — 9 Reasons to Always Use Windows With a VPN

Tuesday, November 12, 2024

Also: Tips for Setting Up a Mobile VR Office, and More! How-To Geek Logo November 12, 2024 Did You Know In the 2016 film Doctor Strange, the characters of both Doctor Strange and the villain Dormammu (

Web Scraping Tips, Python 3.13 Performance Boosts, Writing Interpreters & More

Tuesday, November 12, 2024

Introduction to Web Scraping With Python #655 – NOVEMBER 12, 2024 VIEW IN BROWSER The PyCoder's Weekly Logo Introduction to Web Scraping With Python In this video course, you'll learn all about

Daily Coding Problem: Problem #1606 [Easy]

Tuesday, November 12, 2024

Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by PayPal. Given a binary tree, determine whether or not it is height-balanced. A height-

Charted | Breaking Down the U.S. Government's 2024 Fiscal Year 💰

Tuesday, November 12, 2024

Net interest payments cost the US government $882 billion in fiscal year 2024, the third-largest outlay in the final budget. View Online | Subscribe | Download Our App Presented by Hinrich Foundation

Spyglass Dispatch: AI's Independence Race • EU's Bad Meta Ads • AI Chip Shenanigans • Netflix Ads Religion

Tuesday, November 12, 2024

AI's Independence Race • EU's Bad Meta Ads • AI Chip Shenanigans • Netflix Ads Religion The Spyglass Dispatch is a free newsletter sent out daily on weekdays. Feel free to forward it on to

The Big T

Tuesday, November 12, 2024

Top Tech Content sent at Noon! How the world collects web data Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today, November 12, 2024? The HackerNoon

Deadline Extended: 2 Weeks Left to Compete for Over $7000 in the AI-chatbot Writing Contest🔥

Tuesday, November 12, 2024

Great news, newsletterest1 ! The submission deadline for the #ai-chatbot writing contest has been extended! You now have until November 21, 2024, to submit your unique AI chatbot ideas for a chance to

A very demure, very mindful issue

Tuesday, November 12, 2024

Plus a look at memory regions, Go's birthday, and we invent a brand new word. | #​531 — November 12, 2024 Unsub | Web Version Together with Frontend Masters logo Go Weekly Happy Birthday, Go! Go

Visual Capitalist is revealing all of its biggest secrets... 📊

Tuesday, November 12, 2024

You can get in on our newest project if you act now. View Online | Subscribe | Download Our App We're revealing our biggest secrets... The question we get asked the most is: "How does Visual

🔓🐍 Unlock Your Python Potential with Instructor-Led Courses

Tuesday, November 12, 2024

Hey there, If you've been looking for a way to go beyond on-demand tutorials and really master Python, we've got something special for you... For the first time, Real Python is launching an