BetterDev #232 - Reducing BigQuery Costs: How We Fixed A $1 Million Query
Better Dev #232 Nov 07, 2022
Hi everyone, welcome to first issue of November. This week, where I live, red leaves start to emerge and my family went outside, enjoy seeing the beautiful color of autumn. Social media had been a tough week with Twitter and Elon, same as finance system and crypto drama. The world looked so depressing, but no matter what, being a programmer, knowing how to code is really a privilege, we have a way to make a living while sitting at a deck doing thing we enjoy. No matter what happen, there will always a demand to make website, application or service and a continous stream of jobs require to run these things. We are thankful for the jobs and opportunities we had, be optimist and always leaning. on ward. have a good week everyone.
This week I shared a small tip to debug CI build failure easiser. If you ever debug some tricky CI failure and say to your self: “If I can SSH into this server or container that running the build, I will be able to figure out immediately”, then this could be very useful. It can also apply anytime we need to SSH back into some private infrastructure.
During the infrastructural exploration of a pipeline my team was building, we discovered a query that could have cost us nearly $1 million USD a month in BigQuery. Below, we’ll detail how we reduced this and share our tips for lowering costs in BigQuery.
Recently, the Chrome developers announced their decision to remove the behind-a-flag support for JPEG XL. Cloudinary argue(to which I agree) JPEG XL can bring these benefits without introducing a regression in other areas, at least in terms of technical strengths. We can only hope that the Chrome developers reverse their decision and help JPEG XL catch up with the old formats in terms of software support, so we can all enjoy the benefits it brings.
Almost web app needs to make call to third party API such as taking a payment, send out an email. By wrapping an adapter around each API, presenting only the features of that API the app needs, we can more easily manage and test the app.
when we run diff
or git diff
, we see a visualization of what were add, insert or delete? what algorithm power that?
If you’ve ever wanted to build a web app that you can control with hand gestures as if by magic, this article is for you. With a couple of APIs and some JavaScript, you can build apps that behave like sorcery.
BTree is the most common index type we used every day. For exact lookup, BTree probably best choice most of time, but if you’re looking for a range query, Brin maybe an option
We’re going to dig into some of the flexibility on architecting your database and terminology within Postgres and the trade-offs for different approaches.
When you tune performance for a system, very quickly you run into stuff like nofile, transaparent huge page which usually either has some high level command to set it, or sometime require you to change a
certain parameters by writing to /proc/sys/dev
. So today we will have a look around it.
This is a step by step guide on how to become a DevOps engineer, with links to relevant learning resources. If you plan to become a DevOps engineer, could be a great learning resource.
Code to read
AWS Lambda deployment requires a bundle of entire of your app and all dependencies, package in a certain structure. Zip it and ship it do exactly that, create a zip bundle for a lambda app in node go or rust and deploy to aws.
JavaScripta library for building concurrent applications in Ruby. Polyphony harnesses the power of Ruby fibers to provide a cooperative, sequential coroutine-based concurrency model. Under the hood, Polyphony uses io_uring or libev to maximize I/O performance.
RubyA Pure-go redis driver use RESP3 protocol. If you curios about writing driver, this is a great example
Goa cross-platform embeddable sandboxing library allowing restrictions to Filesystem and Network operations using native operating system APIs. It should be ued for education purpose only at this point
RustVideo
Tools
The Open Source Shopify Alternative. Medusa is a composable engine that combines an amazing developer experience with endless customizations for merchants to scale
a powerful, flexible, Markdown-based authoring framework to create blog or document site in markdown.
It answers thousands of questions about your infrastructure, and integrates with over 300 resources across cloud accounts, Kubernetes, containers, services, VMs, APIs, and more.
finds vulnerabilities and misconfigurations on all systems in your infrastructure including: public and private cloud environments, Kubernetes clusters, containers, container registries, servers and endpoints, SaaS products, infrastructure as code, APIs, and more.
Lambda function for executing pg_dump and upload the output to s3. Can be useful to take a daily/hourly pg_dump of database
You can view this issue in web browser.
If you have any suggestion/feedback, do tell me by replying to this email. I read them all.
No longer want to receive these emails? Unsubscribe
Older messages
BetterDev #231 - Time is an illusion, Unix time doubly so
Monday, October 31, 2022
Better Dev #231 Oct 31, 2022 Hi, everyone, happy halloween. This issue is a haloween treat :-), it has a lot of short and small articles which you can apply to your daily work and small tips and tricks
BetterDev #230 - Secure your server by limit outgoing connection
Monday, October 24, 2022
Better Dev #230 Oct 24, 2022 Hi, everyone, welcome to our second issue of the come back. I had received many emails with encouragement about my resumt. I really really appreciate that. Thank you so
BetterDev #229 - BetterDev is back
Monday, October 17, 2022
Better Dev #229 Oct 17, 2022 Hi, everyone. It has been almost 6 months with out any betterdev issue :-). It was due to my busy schedule. I finally started to get thing back on track. I will come back
BetterDev #228 - Memory leaks: the forgotten side of web performance
Monday, February 7, 2022
Better Dev #228 Feb 07, 2022 Memory leaks: the forgotten side of web performance I've researched and learned enough about client-side memory leaks to know that most web developers aren't
BetterDev #227 - How to avoid layout shifts caused by web fonts and Scaling Kubernetes to Over 4k Nodes and 200k Pods
Tuesday, February 1, 2022
Better Dev #227 Feb 01, 2022 How to avoid layout shifts caused by web fonts Fonts cause layout shifts when the size of the containing element (eg a or paragraph) changes when the web font is downloaded
You Might Also Like
Import AI 399: 1,000 samples to make a reasoning model; DeepSeek proliferation; Apple's self-driving car simulator
Friday, February 14, 2025
What came before the golem? ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Defining Your Paranoia Level: Navigating Change Without the Overkill
Friday, February 14, 2025
We've all been there: trying to learn something new, only to find our old habits holding us back. We discussed today how our gut feelings about solving problems can sometimes be our own worst enemy
5 ways AI can help with taxes 🪄
Friday, February 14, 2025
Remotely control an iPhone; 💸 50+ early Presidents' Day deals -- ZDNET ZDNET Tech Today - US February 10, 2025 5 ways AI can help you with your taxes (and what not to use it for) 5 ways AI can help
Recurring Automations + Secret Updates
Friday, February 14, 2025
Smarter automations, better templates, and hidden updates to explore 👀 ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
The First Provable AI-Proof Game: Introducing Butterfly Wings 4
Friday, February 14, 2025
Top Tech Content sent at Noon! Boost Your Article on HackerNoon for $159.99! Read this email in your browser How are you, @newsletterest1? undefined The Market Today #01 Instagram (Meta) 714.52 -0.32%
GCP Newsletter #437
Friday, February 14, 2025
Welcome to issue #437 February 10th, 2025 News BigQuery Cloud Marketplace Official Blog Partners BigQuery datasets now available on Google Cloud Marketplace - Google Cloud Marketplace now offers
Charted | The 1%'s Share of U.S. Wealth Over Time (1989-2024) 💰
Friday, February 14, 2025
Discover how the share of US wealth held by the top 1% has evolved from 1989 to 2024 in this infographic. View Online | Subscribe | Download Our App Download our app to see thousands of new charts from
The Great Social Media Diaspora & Tapestry is here
Friday, February 14, 2025
Apple introduces new app called 'Apple Invites', The Iconfactory launches Tapestry, beyond the traditional portfolio, and more in this week's issue of Creativerly. Creativerly The Great
Daily Coding Problem: Problem #1689 [Medium]
Friday, February 14, 2025
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Google. Given a linked list, sort it in O(n log n) time and constant space. For example,
📧 Stop Conflating CQRS and MediatR
Friday, February 14, 2025
Stop Conflating CQRS and MediatR Read on: my website / Read time: 4 minutes The .NET Weekly is brought to you by: Step right up to the Generative AI Use Cases Repository! See how MongoDB powers your