BetterDev #232 - Reducing BigQuery Costs: How We Fixed A $1 Million Query
Better Dev #232 Nov 07, 2022
Hi everyone, welcome to first issue of November. This week, where I live, red leaves start to emerge and my family went outside, enjoy seeing the beautiful color of autumn. Social media had been a tough week with Twitter and Elon, same as finance system and crypto drama. The world looked so depressing, but no matter what, being a programmer, knowing how to code is really a privilege, we have a way to make a living while sitting at a deck doing thing we enjoy. No matter what happen, there will always a demand to make website, application or service and a continous stream of jobs require to run these things. We are thankful for the jobs and opportunities we had, be optimist and always leaning. on ward. have a good week everyone.
This week I shared a small tip to debug CI build failure easiser. If you ever debug some tricky CI failure and say to your self: “If I can SSH into this server or container that running the build, I will be able to figure out immediately”, then this could be very useful. It can also apply anytime we need to SSH back into some private infrastructure.
During the infrastructural exploration of a pipeline my team was building, we discovered a query that could have cost us nearly $1 million USD a month in BigQuery. Below, we’ll detail how we reduced this and share our tips for lowering costs in BigQuery.
Recently, the Chrome developers announced their decision to remove the behind-a-flag support for JPEG XL. Cloudinary argue(to which I agree) JPEG XL can bring these benefits without introducing a regression in other areas, at least in terms of technical strengths. We can only hope that the Chrome developers reverse their decision and help JPEG XL catch up with the old formats in terms of software support, so we can all enjoy the benefits it brings.
Almost web app needs to make call to third party API such as taking a payment, send out an email. By wrapping an adapter around each API, presenting only the features of that API the app needs, we can more easily manage and test the app.
when we run diff
or git diff
, we see a visualization of what were add, insert or delete? what algorithm power that?
If you’ve ever wanted to build a web app that you can control with hand gestures as if by magic, this article is for you. With a couple of APIs and some JavaScript, you can build apps that behave like sorcery.
BTree is the most common index type we used every day. For exact lookup, BTree probably best choice most of time, but if you’re looking for a range query, Brin maybe an option
We’re going to dig into some of the flexibility on architecting your database and terminology within Postgres and the trade-offs for different approaches.
When you tune performance for a system, very quickly you run into stuff like nofile, transaparent huge page which usually either has some high level command to set it, or sometime require you to change a
certain parameters by writing to /proc/sys/dev
. So today we will have a look around it.
This is a step by step guide on how to become a DevOps engineer, with links to relevant learning resources. If you plan to become a DevOps engineer, could be a great learning resource.
Code to read
AWS Lambda deployment requires a bundle of entire of your app and all dependencies, package in a certain structure. Zip it and ship it do exactly that, create a zip bundle for a lambda app in node go or rust and deploy to aws.
JavaScripta library for building concurrent applications in Ruby. Polyphony harnesses the power of Ruby fibers to provide a cooperative, sequential coroutine-based concurrency model. Under the hood, Polyphony uses io_uring or libev to maximize I/O performance.
RubyA Pure-go redis driver use RESP3 protocol. If you curios about writing driver, this is a great example
Goa cross-platform embeddable sandboxing library allowing restrictions to Filesystem and Network operations using native operating system APIs. It should be ued for education purpose only at this point
RustVideo
Tools
The Open Source Shopify Alternative. Medusa is a composable engine that combines an amazing developer experience with endless customizations for merchants to scale
a powerful, flexible, Markdown-based authoring framework to create blog or document site in markdown.
It answers thousands of questions about your infrastructure, and integrates with over 300 resources across cloud accounts, Kubernetes, containers, services, VMs, APIs, and more.
finds vulnerabilities and misconfigurations on all systems in your infrastructure including: public and private cloud environments, Kubernetes clusters, containers, container registries, servers and endpoints, SaaS products, infrastructure as code, APIs, and more.
Lambda function for executing pg_dump and upload the output to s3. Can be useful to take a daily/hourly pg_dump of database
You can view this issue in web browser.
If you have any suggestion/feedback, do tell me by replying to this email. I read them all.
No longer want to receive these emails? Unsubscribe
Older messages
BetterDev #231 - Time is an illusion, Unix time doubly so
Monday, October 31, 2022
Better Dev #231 Oct 31, 2022 Hi, everyone, happy halloween. This issue is a haloween treat :-), it has a lot of short and small articles which you can apply to your daily work and small tips and tricks
BetterDev #230 - Secure your server by limit outgoing connection
Monday, October 24, 2022
Better Dev #230 Oct 24, 2022 Hi, everyone, welcome to our second issue of the come back. I had received many emails with encouragement about my resumt. I really really appreciate that. Thank you so
BetterDev #229 - BetterDev is back
Monday, October 17, 2022
Better Dev #229 Oct 17, 2022 Hi, everyone. It has been almost 6 months with out any betterdev issue :-). It was due to my busy schedule. I finally started to get thing back on track. I will come back
BetterDev #228 - Memory leaks: the forgotten side of web performance
Monday, February 7, 2022
Better Dev #228 Feb 07, 2022 Memory leaks: the forgotten side of web performance I've researched and learned enough about client-side memory leaks to know that most web developers aren't
BetterDev #227 - How to avoid layout shifts caused by web fonts and Scaling Kubernetes to Over 4k Nodes and 200k Pods
Tuesday, February 1, 2022
Better Dev #227 Feb 01, 2022 How to avoid layout shifts caused by web fonts Fonts cause layout shifts when the size of the containing element (eg a or paragraph) changes when the web font is downloaded
You Might Also Like
Matrix Botnet Exploits IoT Devices in Widespread DDoS Botnet Campaign
Wednesday, November 27, 2024
THN Daily Updates Newsletter cover The AI Value Playbook ($35.99) FREE for a Limited Time Business leaders are challenged by the speed of AI innovation and how to navigate disruption and uncertainty.
The Sequence Chat: Why are Foundation Models so Hard to Explain and What are we Doing About it?
Wednesday, November 27, 2024
Addressing some of the interpretability challenges of foundation models and the emerging fields of mechanistic interpretability and behavioral probing. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Overcoming Perfectionism: How to Break Free from the Enemy of Progress
Wednesday, November 27, 2024
Discover how perfectionism hinders progress and learn practical strategies to overcome the fear of imperfection, boost productivity, and achieve your goals without getting stuck in the pursuit of
🖤 Laravel Black Friday Deals!
Wednesday, November 27, 2024
The biggest deals of the year Laravel Black Friday Deals View in browser Laravel News Editor Note: We are sending this outside the regular Sunday newsletter schedule because some of these specials end
BetterDev #271 - Memory: The Forgotten History and Why did Windows 95 setup use three operating systems?
Wednesday, November 27, 2024
Better Dev #271 Nov 26, 2024 Hi all, Welcome to thanksgiving issue of BetterDev. Hope everyone had a safe and warm thanksgiving. It's getting so cold these days. If you are in warzone such as
Mapped | Unemployment Rate By U.S. State in 2024 💼
Tuesday, November 26, 2024
As of October 2024, DC and Nevada tied for the highest unemployment rate in the US at 5.7%. Which states saw the lowest rates? View Online | Subscribe | Download Our App FINAL CHANCE - ENDS TONIGHT!
🔊 7 DIY Tips for Soundproofing a Room — Why I Switched to Xfce for Linux Mint
Tuesday, November 26, 2024
Also: Home Theater Sound Terms Explained, and More! How-To Geek Logo November 26, 2024 Did You Know The shiny layer of a CD doesn't contain the data; the plastic polycarbonate layer does. The shiny
JSK Daily for Nov 26, 2024
Tuesday, November 26, 2024
JSK Daily for Nov 26, 2024 View this email in your browser A community curated daily e-mail of JavaScript news JavaScript Certification Black Friday Offer – Up to 54% Off! Certificates.dev, the trusted
NumPy, Loop Targets, Vector Animation, and More
Tuesday, November 26, 2024
NumPy Practical Examples: Useful Techniques #657 – NOVEMBER 26, 2024 VIEW IN BROWSER The PyCoder's Weekly Logo NumPy Practical Examples: Useful Techniques In this tutorial, you'll learn how to
Your Games Quarterly newsletter has arrived
Tuesday, November 26, 2024
What's new for games in Google Play and Android Email not displaying correctly? View it online November 2024 The First Developer Preview of Android 16 The First Developer Preview of Android 16