BetterDev #172 - The Hitchhiker’s Guide to Compression& PostgreSQL at Scale: Saving Space (Basically) for Free
Better Dev #172 Oct 05, 2020
Hi everyone, Welcome to issue #172.
This week, I want to introduce starship again. I linked to it in issue 119. Recently my zsh prompt is too slow. I want to have all information like git repo, kube context, cli version etc but I also do not want my prompt takes a few seconds to load. Switching to starship makes it run faster. You also deal with nice config language instead of shell :-).
We used compression every day, in tar, gzip, zip files. What is the exact algorithm to power them? Today, much of the relevant work to compression is in an obscure corner of the internet between lengthy PhD thesis papers and hard-to-find gems.
Big-O notation, time complexity, space complexity? But what is the abosolute number to do a thing? How long does it take to access RAM, read SSD, or send network package?
Eventually any large-scale website will have to deal with bot, crawler, who usually just consume your resource without bringing any benefit. How we prevent them? HAProxy may not well-know like Nginx because it only does load balancing, cannot do static file servering like Nginx. But it does its job very well and supports many primitives for advanced load balancing such as rate-limiting, which is helpful when dealing with bots.
Find and fix tricky memory leaks caused by detached windows.
DropBox rewrite and 12,000 loc from jQuery to React to solve an issue of rendering pdf different between 3 places. The lesson is: Don’t fix it in UI, fix what’s behind it. A software engineer’s job is not to write code, it’s to solve problems, by writing code when necessary.
A technique that saved BrainTree payment approximately 10% of disk space with very little effort beyond existing processes. In short, carefully choosing column order when creating a table can eliminate padding that would otherwise be needed.
This talk is designed for PostgreSQL administrators. It covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
The Linux TCP stack is a finite state machine. So when closing a connection, 2 sides need to coordinate so the state can be advanced. TIME-WAIT is when the local server close the connection already, the connection is kept around so that any delayed packets can be matched to the connection and handled appropriately. It also helps the remote side know that the connection is closed, so remote can close as well otherwise it is stuck in LAST-ACK state. And it exit this state in 4 minutes. Needless to say, keeping these around on a busy server is expensive, you need CPU, memory to handle them.
Have you ever receive a message with square or question mark character while it shows up fine on sender side? Font fallback is just complex, filled with quirks and unexpected side effects. Font enthusiast Marcin Wichary decided to dig into the issue and prepare an exhaustive guide to font fallback, its beauty, and all the pitfalls.
Writing DNS server is like hello world for networking to me. DNS is small enough, not massive like HTTP, also relying on UDP, so I think it’s a good exercise to learn UDP and the DNS protocol. The language is Rust, despite that I’m not a Rust programmer, I only toy with Rust and I was able to follow just fine so I think you can do the same and learn a few thing about Rust and DNS at the same time.
PG Extras is a tool that allows you to spot common PostgreSQL pitfalls. Ruby, Rails, Elixir, and NodeJS implementations are currently available. In this blog post, I present a step by step guide on using PG Extras library to spot and resolve common PostgreSQL database performance issues.
Code to read
An interesting project from Microsoft to transform old, low resolution photo into sharp, modern, high resolution image
PythonA proof-of-concept for WebRTC signaling using sound. Works with all devices that have microphone + speakers. Runs in the browser.
Video
An old talks in 2014 explain event loop, one of core concept of JavaScript/NodeJS
Tools
a build automation tool for the post-container era. It allows you to execute all your builds in containers. This makes them self-contained, reproducible, portable and parallel.
Scan, Analyze your website to see what you need to do to make it run faster
A Real-Time Website Privacy Inspector. Open source on Github if you want to see how they peform scanning.
You can view this issue in web browser.
If you have any suggestion/feedback, do tell me by replying to this email. I read them all.
No longer want to receive these emails? Unsubscribe
Older messages
BetterDev #171 - How to remove the first 300 million lines from a 700 GB txt file on a system with 1 TB max disk space?
Monday, September 28, 2020
Better Dev #171 Sep 28, 2020 Hi everyone, Welcome to issue #171. How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB max disk space? The accepted answer is boring
BetterDev #170 - What Gitlab leanred after a year on Kubernetes
Monday, September 21, 2020
Better Dev #170 Sep 21, 2020 Hi everyone, Welcome to issue #170. I have noticed many companies has reduced their head count during Covid-19. So if you are looking for a job now, or want a job soon, you
BetterDev #169 - Who will test the tests themselves and How Gitlab upgraded PostgresSQL
Monday, September 14, 2020
Better Dev #169 Sep 14, 2020 Hi everyone, Welcome to issue #169. Thank you for being a reader to my newsletter. I hope you can help spread the words out by telling your friend about this :-) Mutation
BetterDev #168 - The Life of a Data Byte: Be kind and rewind and Things I Learnt from a Senior Software Enginee
Monday, September 7, 2020
Better Dev #168 Sep 07, 2020 Hi everyone, Welcome to issue #168. Let's get straight to our links this time The Life of a Data Byte: Be kind and rewind A byte of data has been stored in a number of
BetterDev #167 - WebRTC for the Curious and APIs at scale
Tuesday, September 1, 2020
Better Dev #167 Aug 31, 2020 Hi everyone, Welcome to issue #167. Let's get straight to our links this time WebRTC for the Curious This book was created by WebRTC implementers to share their hard-
You Might Also Like
Import AI 399: 1,000 samples to make a reasoning model; DeepSeek proliferation; Apple's self-driving car simulator
Friday, February 14, 2025
What came before the golem? ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Defining Your Paranoia Level: Navigating Change Without the Overkill
Friday, February 14, 2025
We've all been there: trying to learn something new, only to find our old habits holding us back. We discussed today how our gut feelings about solving problems can sometimes be our own worst enemy
5 ways AI can help with taxes 🪄
Friday, February 14, 2025
Remotely control an iPhone; 💸 50+ early Presidents' Day deals -- ZDNET ZDNET Tech Today - US February 10, 2025 5 ways AI can help you with your taxes (and what not to use it for) 5 ways AI can help
Recurring Automations + Secret Updates
Friday, February 14, 2025
Smarter automations, better templates, and hidden updates to explore 👀 ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
The First Provable AI-Proof Game: Introducing Butterfly Wings 4
Friday, February 14, 2025
Top Tech Content sent at Noon! Boost Your Article on HackerNoon for $159.99! Read this email in your browser How are you, @newsletterest1? undefined The Market Today #01 Instagram (Meta) 714.52 -0.32%
GCP Newsletter #437
Friday, February 14, 2025
Welcome to issue #437 February 10th, 2025 News BigQuery Cloud Marketplace Official Blog Partners BigQuery datasets now available on Google Cloud Marketplace - Google Cloud Marketplace now offers
Charted | The 1%'s Share of U.S. Wealth Over Time (1989-2024) 💰
Friday, February 14, 2025
Discover how the share of US wealth held by the top 1% has evolved from 1989 to 2024 in this infographic. View Online | Subscribe | Download Our App Download our app to see thousands of new charts from
The Great Social Media Diaspora & Tapestry is here
Friday, February 14, 2025
Apple introduces new app called 'Apple Invites', The Iconfactory launches Tapestry, beyond the traditional portfolio, and more in this week's issue of Creativerly. Creativerly The Great
Daily Coding Problem: Problem #1689 [Medium]
Friday, February 14, 2025
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Google. Given a linked list, sort it in O(n log n) time and constant space. For example,
📧 Stop Conflating CQRS and MediatR
Friday, February 14, 2025
Stop Conflating CQRS and MediatR Read on: my website / Read time: 4 minutes The .NET Weekly is brought to you by: Step right up to the Generative AI Use Cases Repository! See how MongoDB powers your