Welcome to the new week!

Event-driven systems are famous for immutability, and GDPR is famous for data removal. Does that mean that event-driven systems cannot be compliant with privacy rules?

Data governance practices are, unfortunately, not discussed enough in event-driven systems. People treat events more like freehand drawing, sometimes as mesh, but ending too often with a mess instead.

In my latest article, I explained both techniques specific to event-driven systems and tools like Kafka, Marten, and EventStoreDB and put them into broader strategic thinking about privacy.

• How to deal with privacy and GDPR in Event-Driven systems

It’s a follow-up to the guide to GDPR for busy developers I published two weeks ago.

A reminder about tomorrow’s webinar about Leveraging BPMN for Seamless Team Collaboration in Software Development with Mário Bittencourt. We’ll learn how to use it to describe and document our business process to get a common understanding.

It’ll happen on 28th November at 6 PM. See more details.

Speaking about collaboration and modelling, I have two more resources for you:

• Gien Verschatse - Collaborative Software Design

• Nick Tune - Sustainable software development with collaborative domain modeling

They nicely explain the variety of tools we can use and their importance in building a collaboration culture in our teams. That’s critical to gain insights and perspectives from different stakeholders. Of course, it’s tricky, and we should also consider mental fatigue to make it sustainable.

I was mentally fatigued from last week’s OpenAI madness. Let me just put it here as it is, out of the chronicler’s duty…

• The Verge - Sam Altman to return as CEO of OpenAI

Yeah… Let’s move on.

Moving the data on is the key thing in Event Streaming solutions. Having data is one thing, but processing it efficiently, passing it to others, analysing it, and correlating it is a growing challenge. We already have tools that provide the possibility to do it inflight (like Apache Flink), but the number of data to process is growing even faster.

Check what Confluent CEO Jay Kreps thinks about the future of it and how Stipe is doing that on a bigger scale:

• Jay Kreps - Streaming into the Future - The Evolution & Impact of Data Streaming Platforms

• Jeff Chao - Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Capture (slides)

Have a look also at the insights from Claire Caroll on how to get better skills in being a data analyst:

• Claire Carroll - How to be a more Impactful Data Analyst

Automation is a critical part of continuous integration and deployment. But how do you fluently switch automation systems? If we’re running our project long enough, our tooling will need to change. That includes CI/CD, and there being fast and supportive of the process is extremely important.

For big projects such changes are too often postponed, as we don’t want to have our delivery process disturbed. Check two great resources from Spotify.

• CoRecursive Podcast - Platform Takes The Pain with Pia Nilsson

• Spotify - Switching Build Systems, Seamlessly

The first one is super intriguing as Pia Nilsson explains how they transitioned from full team autonomy, which too often meant an isolated island, to a more centralised platform. It’s a great story on transitioning the platform team mentality from playing with CI/CD tools and experimenting to becoming an enabling team for others. I love the term Platform Takes The Pain, it’s a great motto, as too often I saw that platform teams believe they’re the foundational and elitist team. At the same time, they’re there to help others deliver business value.

The seconds show how they switched their build systems for iOS applications to Bazel. It’s a great, not a long, but multilayered case study showing how to tackle such migration and what to consider while doing it. A nice operation on trying to cut the time you’ll have to build systems to the bare minimum. As always, the hybrid approach costs a lot.

Maintaining multiple build systems during a migration can be costly. In our case, we saw differences between the two build systems running in parallel which, at times, caused interesting compilation failures in one of the two build systems. For example, developers would report builds passing locally but failing in CI. Even though we had greatly reduced the time developers spent waiting for their changes to be tested in CI, our platform engineers now spent more time helping feature engineers debug build system issues.(…) We realized that the only way to completely remove this maintenance cost was to move to a single build system as soon as possible. 

If you’re into Platform Teams, check this resource from CNCF:

• CNCF - Platforms White Paper

They released their guidance explaining:

1. Why platforms?

2. What is a platform

3. Attributes of successful platforms

4. Attributes of successful platform teams

5. Challenges when implementing platforms

6. How to measure the success of platforms

7. Capabilities of platforms

And also remember, as Charity Mayors said in:

• Yan Cui - Charity Majors on DevOps culture and team dynamic

It’s not about whether you deploy on Fridays or not; it’s about “owning your own shit”. So, be responsible for what you’re running and make yourself accountable for how you run and what you run on prod.

Mistakes can happen and will happen, but we need to be sure that we have the tools, skills and confidence that we’ll be able to handle that.

A nice inspiration for dealing with failures can be research from other industries, especially with more mature engineering practices. For instance:

• Christine Shea, Chris. W. Johnson - Understanding the Contribution of Degraded Modes of Operation as a Cause of Incidents and Accidents in Air Traffic Management

Check also other links!

Cheers!
Oskar

p.s. I invite you to join the paid version of Architecture Weekly. It already contains the exclusive Discord channel for subscribers (and my GitHub sponsors), monthly webinars, etc. It is a vibrant space for knowledge sharing. Don’t wait to be a part of it!

p.s.2. Ukraine is still under brutal Russian invasion. A lot of Ukrainian people are hurt, without shelter and need help. You can help in various ways, for instance, directly helping refugees, spreading awareness, and putting pressure on your local government or companies. You can also support Ukraine by donating, e.g. to the Ukraine humanitarian organisation, Ambulances for Ukraine or Red Cross.

Architecture

• Oskar Dudycz - How to deal with privacy and GDPR in Event-Driven systems

• OneUptime - How moving from AWS to Bare-Metal saved us $230,000 /yr.

• Spotify - Switching Build Systems, Seamlessly

• Prefixy - How We Built Prefixy: A Scalable Prefix Search Service for Powering Autocomplete

• Krzysztof Przychodzki - Beyond the Code - An Engineer’s Battle Against Knowledge Loss

• Jay Kreps - Streaming into the Future - The Evolution & Impact of Data Streaming Platforms

• Jeff Chao - Squirreling Away $640 Billion: How Stripe Leverages Flink for Change Data Capture (slides)

• Sandi Metz - The Wrong Abstraction

• Claire Carroll - How to be a more Impactful Data Analyst

• Maciej "MJ" Jedrzejewski - Mastering Strategic Domain-Driven Design - 4. Tuning of Subdomains

• Gien Verschatse - Collaborative Software Design

• Nick Tune - Sustainable software development with collaborative domain modeling

DevOps

• CoRecursive Podcast - Platform Takes The Pain with Pia Nilsson

• CNCF - Platforms White Paper

• M. McLean, T. Young, A. Sharma, D. Dyla - OpenTelemetry: What's Next? Logs, Profiles...

• Rouan Wilsenach - Ship / Show / Ask. A modern branching strategy

• Yan Cui - Charity Majors on DevOps culture and team dynamic

• Nitric - Why we chose Pulumi over Terraform

API

• API Evangelist - Where Is This API Gateway Thing Going?

• Stefan Tilkov - Why “API-first” might not be a good idea

Databases

• Hussein Nasser - How Shopify’s engineering improved database writes by 50% with ULID

• The Geek Narrator - Graph Database Internals: @neo4j with Michael Hunger

Frontend

• NgRx - Introducing NgRx Signals, Operators, Performance Improvements, Workshops, and more!

• Vite - Vite 5.0 is out!

Testing

• Microsoft - Consumer-driven Contract Testing

Java

• Viktor Klang - Teaching old Streams new tricks

.NET

• Rich Lander - .NET 8 container workshop

• David Whitney - Lo-Fi Service Discovery in .NET8

• Microsoft - Lesson Learned #449: Unleashing Concurrent Threads for Robust Database Health Checks in C#

• Tim Deschryver - NuGet Central Package Management

Industry

• Phil Fersht, Saurabh Gupta - Services firms are out of runway. They must forget Labor Arbitrage and conform to Technology Arbitrage

• The Verge - Sam Altman to return as CEO of OpenAI

Security

• Tom's Hardware - Chinese hackers steal chip designs from major Dutch semiconductor company — perps lurked for over two years to steal NXP's chipmaking IP: Report

• ArsTechnica - Developers can’t seem to stop exposing credentials in publicly accessible code

Trivia

• Christine Shea, Chris. W. Johnson - Understanding the Contribution of Degraded Modes of Operation as a Cause of Incidents and Accidents in Air Traffic Management