⭐️ BFCM Special Edition ⭐️ 

Okay, so we don’t normally slide in your inbox twice in one month, but it’s a special occasion and we don’t want you to miss the fun! We celebrate our engineers and data scientists year round, but during the most important shopping weekend of the year, they deserve a little extra love. Let’s take a look at what their hard work made possible during Black Friday Cyber Monday (BFCM) weekend:

• 📈 We served 75.98 Million requests per minute (1.27 Million requests per second!) to our commerce platform at peak.

• 🚢 Our MySQL fleet performed over 14 Million queries per second (QPS) at peak, averaging over 8.5 Million QPS throughout BFCM.

• 🎚️ To keep our system responsive under heavy load, we processed over 32 Billion asynchronous tasks using Resque.

• 💫 We delivered over 24 Billion webhooks to support the +8,000 apps in the Shopify Partners ecosystem.

• 📇 At peak BFCM, we indexed 16 Gigabytes per second of data of logs.

• 🚀 Our batch data ingestion service ingested 1.1 Trillion total messages during BFCM. 

• 📈 Our Apache Kafka streaming infrastructure served 20 Million messages per second at peak.

• 🌐 We served 7.29 Petabytes of traffic to shoppers outside of North America!

Using Server Sent Events to Simplify Real-time Streaming at Scale

DATA SCIENCE & ENGINEERING  •  BY BAO NGUYEN

Our data visualization enthusiasts out there might already be familiar with the BFCM live map: a dynamic, real-time visual representation of entrepreneurship and commerce happening across the globe. Bao Nguyen (Senior Staff Data Engineer) wrote this post about how it works, and how we used SSE (Server Sent Events) to improve latency and performance for this year’s live map. 

Life-defining work for you—and entrepreneurs

This is life-defining work that directly impacts people’s lives as much as it transforms your own. This is putting the power of the few in the hands of the many, is a future with more voices rather than fewer, and is creating more choices instead of an elite option. We’re arming the rebels. Want to join us?