BetterDev #172 - The Hitchhiker’s Guide to Compression& PostgreSQL at Scale: Saving Space (Basically) for Free
Better Dev #172 Oct 05, 2020
Hi everyone, Welcome to issue #172.
This week, I want to introduce starship again. I linked to it in issue 119. Recently my zsh prompt is too slow. I want to have all information like git repo, kube context, cli version etc but I also do not want my prompt takes a few seconds to load. Switching to starship makes it run faster. You also deal with nice config language instead of shell :-).
We used compression every day, in tar, gzip, zip files. What is the exact algorithm to power them? Today, much of the relevant work to compression is in an obscure corner of the internet between lengthy PhD thesis papers and hard-to-find gems.
Big-O notation, time complexity, space complexity? But what is the abosolute number to do a thing? How long does it take to access RAM, read SSD, or send network package?
Eventually any large-scale website will have to deal with bot, crawler, who usually just consume your resource without bringing any benefit. How we prevent them? HAProxy may not well-know like Nginx because it only does load balancing, cannot do static file servering like Nginx. But it does its job very well and supports many primitives for advanced load balancing such as rate-limiting, which is helpful when dealing with bots.
Find and fix tricky memory leaks caused by detached windows.
DropBox rewrite and 12,000 loc from jQuery to React to solve an issue of rendering pdf different between 3 places. The lesson is: Don’t fix it in UI, fix what’s behind it. A software engineer’s job is not to write code, it’s to solve problems, by writing code when necessary.
A technique that saved BrainTree payment approximately 10% of disk space with very little effort beyond existing processes. In short, carefully choosing column order when creating a table can eliminate padding that would otherwise be needed.
This talk is designed for PostgreSQL administrators. It covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
The Linux TCP stack is a finite state machine. So when closing a connection, 2 sides need to coordinate so the state can be advanced. TIME-WAIT is when the local server close the connection already, the connection is kept around so that any delayed packets can be matched to the connection and handled appropriately. It also helps the remote side know that the connection is closed, so remote can close as well otherwise it is stuck in LAST-ACK state. And it exit this state in 4 minutes. Needless to say, keeping these around on a busy server is expensive, you need CPU, memory to handle them.
Have you ever receive a message with square or question mark character while it shows up fine on sender side? Font fallback is just complex, filled with quirks and unexpected side effects. Font enthusiast Marcin Wichary decided to dig into the issue and prepare an exhaustive guide to font fallback, its beauty, and all the pitfalls.
Writing DNS server is like hello world for networking to me. DNS is small enough, not massive like HTTP, also relying on UDP, so I think it’s a good exercise to learn UDP and the DNS protocol. The language is Rust, despite that I’m not a Rust programmer, I only toy with Rust and I was able to follow just fine so I think you can do the same and learn a few thing about Rust and DNS at the same time.
PG Extras is a tool that allows you to spot common PostgreSQL pitfalls. Ruby, Rails, Elixir, and NodeJS implementations are currently available. In this blog post, I present a step by step guide on using PG Extras library to spot and resolve common PostgreSQL database performance issues.
Code to read
An interesting project from Microsoft to transform old, low resolution photo into sharp, modern, high resolution image
PythonA proof-of-concept for WebRTC signaling using sound. Works with all devices that have microphone + speakers. Runs in the browser.
Video
An old talks in 2014 explain event loop, one of core concept of JavaScript/NodeJS
Tools
a build automation tool for the post-container era. It allows you to execute all your builds in containers. This makes them self-contained, reproducible, portable and parallel.
Scan, Analyze your website to see what you need to do to make it run faster
A Real-Time Website Privacy Inspector. Open source on Github if you want to see how they peform scanning.
You can view this issue in web browser.
If you have any suggestion/feedback, do tell me by replying to this email. I read them all.
No longer want to receive these emails? Unsubscribe
Older messages
BetterDev #171 - How to remove the first 300 million lines from a 700 GB txt file on a system with 1 TB max disk space?
Monday, September 28, 2020
Better Dev #171 Sep 28, 2020 Hi everyone, Welcome to issue #171. How do I remove the first 300 million lines from a 700 GB txt file on a system with 1 TB max disk space? The accepted answer is boring
BetterDev #170 - What Gitlab leanred after a year on Kubernetes
Monday, September 21, 2020
Better Dev #170 Sep 21, 2020 Hi everyone, Welcome to issue #170. I have noticed many companies has reduced their head count during Covid-19. So if you are looking for a job now, or want a job soon, you
BetterDev #169 - Who will test the tests themselves and How Gitlab upgraded PostgresSQL
Monday, September 14, 2020
Better Dev #169 Sep 14, 2020 Hi everyone, Welcome to issue #169. Thank you for being a reader to my newsletter. I hope you can help spread the words out by telling your friend about this :-) Mutation
BetterDev #168 - The Life of a Data Byte: Be kind and rewind and Things I Learnt from a Senior Software Enginee
Monday, September 7, 2020
Better Dev #168 Sep 07, 2020 Hi everyone, Welcome to issue #168. Let's get straight to our links this time The Life of a Data Byte: Be kind and rewind A byte of data has been stored in a number of
BetterDev #167 - WebRTC for the Curious and APIs at scale
Tuesday, September 1, 2020
Better Dev #167 Aug 31, 2020 Hi everyone, Welcome to issue #167. Let's get straight to our links this time WebRTC for the Curious This book was created by WebRTC implementers to share their hard-
You Might Also Like
Spyglass Dispatch: The Xitter Bail Out • OpenAI Tender • Grokking Grok • Smartphone Sales • Fischer Random Chess • Scott Bessent for Treasury
Wednesday, November 27, 2024
The Xitter Bail Out • OpenAI Tender • Grokking Grok • Smartphone Sales • Fischer Random Chess • Scott Bessent for Treasury The Spyglass Dispatch is a free newsletter sent out daily on weekdays. Feel
The Long Road Home: A Story of Loss, Learning, and Renaissance - PART 4
Wednesday, November 27, 2024
Top Tech Content sent at Noon! How the world collects web data Read this email in your browser How are you, @newsletterest1? 🪐 What's happening in tech today, November 27, 2024? The HackerNoon
Top Tech Deals 🏷️ PS5 Slim, 4K TVs, 10th Gen iPad, and More!
Wednesday, November 27, 2024
The Black Friday madness is here! How-To Geek Logo November 27, 2024 Top Tech Deals: PS5 Slim, 4K TVs, 10th Gen iPad, and More! The Black Friday madness is here! Black Friday sales are here, and we
The 165+ best Black Friday deals
Wednesday, November 27, 2024
Windows Super God Mode; Bluesky starter packs; Tech gifts under $100 -- ZDNET ZDNET Tech Today - US November 27, 2024 Black Friday 2024 live blog Best Black Friday deals 2024: 165+ sales live now
⚙️ Neuralink's new trial
Wednesday, November 27, 2024
Plus: Zoom is becoming an AI-first company
Matrix Botnet Exploits IoT Devices in Widespread DDoS Botnet Campaign
Wednesday, November 27, 2024
THN Daily Updates Newsletter cover The AI Value Playbook ($35.99) FREE for a Limited Time Business leaders are challenged by the speed of AI innovation and how to navigate disruption and uncertainty.
The Sequence Chat: Why are Foundation Models so Hard to Explain and What are we Doing About it?
Wednesday, November 27, 2024
Addressing some of the interpretability challenges of foundation models and the emerging fields of mechanistic interpretability and behavioral probing. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Overcoming Perfectionism: How to Break Free from the Enemy of Progress
Wednesday, November 27, 2024
Discover how perfectionism hinders progress and learn practical strategies to overcome the fear of imperfection, boost productivity, and achieve your goals without getting stuck in the pursuit of
🖤 Laravel Black Friday Deals!
Wednesday, November 27, 2024
The biggest deals of the year Laravel Black Friday Deals View in browser Laravel News Editor Note: We are sending this outside the regular Sunday newsletter schedule because some of these specials end
BetterDev #271 - Memory: The Forgotten History and Why did Windows 95 setup use three operating systems?
Wednesday, November 27, 2024
Better Dev #271 Nov 26, 2024 Hi all, Welcome to thanksgiving issue of BetterDev. Hope everyone had a safe and warm thanksgiving. It's getting so cold these days. If you are in warzone such as