Welcome to issue #381 January 15th, 2024

News

Making BigQuery ML feature preprocessing reusable and modular - BigQuery ML's new reusable and modular feature engineering are powerful tools that can make it easier to build and maintain machine learning pipelines and power MLOps.

Databases upgrade made easy with in-place major version upgrades on Cloud SQL - This blog post discusses key advantages as well as best practices to transition to a newer version of MySQL and PostgreSQL by leveraging Cloud SQL’s in-place major version upgrade feature.

Cloud switching just got easier: Removing data transfer fees when moving off Google Cloud - Google Cloud customers who wish to stop using Google Cloud and migrate their data to another cloud provider and/or on premises, can take advantage of free network data transfer to migrate their data out of Google Cloud.

Announcing Humboldt, the first cable route between South America and Asia-Pacific

Document AI Custom Extractor, powered by gen AI, is now Generally Available - Document AI Custom Extractor, powered by generative AI, is Generally Available.

Simplify data loading with new enhancements to BigQuery Data Transfer Service - BigQuery Data Transfer Service (DTS) is a fully managed service that automates the loading of data into BigQuery from a variety of sources.

Driving diversity in AI: a new Google for Startup Accelerator for Women Founders from Europe & Israel

Cloud Deploy gains support for custom target types - Announcing support for custom target types in Preview, extending Cloud Deploy’s capabilities beyond container runtime environments.

Introducing the Dataflow ML Starter project: A practical template to develop and deploy Dataflow ML jobs - Dataflow ML Starter project provides all of the essential scaffolding and boilerplate required to quickly and easily create and launch a Beam pipeline.

---

Give your people seamless, secure access to all the apps they need to do their jobs - from anywhere and on any device - with Cameyo’s Virtual App Delivery (VAD) platform and Google Cloud. Download the solution sheet here to learn more.

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Google Cloud discounts: Five ways Kubernetes experts get more for less - In this blog post, we list the best practices you can follow to create a cost-effective environment on Google Kubernetes Engine (GKE), and an overview of the top various cloud discounts available to GKE users.

How to reduce your Google Cloud Compute Engine Bill by 50% with Committed Use Discounts — Part 1 - Only about 20% of the Google Cloud users I speak with have a Committed Use Discount Strategy.

Tag Terraform Module Launched! - This blog gives a high level overview of why to use Terraform Modules and information about my newly launched terraform-tag-module module.

Guide To Centralised Multi Project Log Monitoring with VPC SC - In this article, we go over how to route aggregated logs in a GCP organisation with VPC SC enabled to a log dedicated project.

GKE with Artifact Registry - Create a private Container Registry with Artifact Registry.

App Development, Serverless, Databases, DevOps

Troubleshooting distributed applications: Using traces and logs together for root-cause analysis - Take a look at how you can use Cloud Trace, Google Cloud’s distributed tracing tool, and Cloud Logging together to help you perform root-cause analysis.

How Deutsche Bank achieved high availability and scalability with Spanner - Learn about the benefits that Deutsche Bank achieved from migrating to Spanner, and some best practices it developed to reliably and efficiently scale the platform.

Configure forward proxying on Apigee X - This article provides a step-by-step guide on how to configure Apigee X to leverage a custom Forward Proxy for outbound traffic either to the Internet or to any VM or service within your private network.

Setting up a SOCKS Proxy Server with Dante on Google Cloud - A SOCKS proxy server can serve various purposes, and in this guide, we’ll explore how to set up one using Dante on Google Cloud.

Running Serverless Cron Jobs With Firebase - In this post, we’ll explore the seamless integration of Google Cloud Scheduler with Firebase Cloud Functions to streamline the process of writing, deploying, and executing code for scheduled tasks.

Automatic Transcription of Phone Calls w/ Google Speech-to-text API - An example of using Cloud Speech to convert hundreds of phone calls to text.

Deploy React App to Google App Engine with Github Actions CI/CD - A Complete Guide - This guide provides you a step-by-step process to deploy your React app efficiently to Google App Engine.

Big Data, Analytics, ML&AI

How the new Google Cloud to Neo4j Dataflow template streamlines data movement - In this blog post, we discuss how the Google Cloud to Neo4j template can help data engineers and data scientists who need to streamline the movement of data from Google Cloud to Neo4j database, to enable enhanced data exploration and analysis with the Neo4j database.

Mixpanel and BigQuery break down silos and democratize data insights - With Mixpanel's major release Warehouse Connectors, the integration between Google Cloud and Mixpanel is now easier to set up and maintain. This integration allows for enriching product and marketing data in Mixpanel with source-of-truth data from BigQuery, providing a comprehensive view that links product and marketing efforts to tangible business outcomes.

Optimizing BigQuery computational analysis costs - This blog post focuses on optimizing BigQuery computational costs through the utilization of the newly introduced BigQuery editions instead of the on-demand ($/TB) pricing model.

Ultimate Guide to Saving Time and Money with BigQuery - Whether you’re a seasoned data professional or just beginning to explore the capabilities of BigQuery, this guide is crafted to provide you with practical, actionable insights to enhance your experience and efficiency.

Three Key BigQuery Optimizations We Should All Be Using - Sharing three optimization tips that are extremely simple to start using but will save enormous time and money if used across all your models.

BigFrames — Scikit and Pandas for BigQuery SQL - This article aims to introduce BigFrames, a library that offers both a Pandas and SKLearn like API acting directly on BigQuery tables.

A complete analytics engineering project: visualize a bike-sharing service data - This article explains how to built a data pipeline, from ingesting data from a public API to visualizing the main insights.

Getting started with AI notebooks and Jupyterhub on GKE - A solution template that will help you get started with Jupyterhub on GKE.

Large Language Models(LLMs) in Google Cloud with VertexAI - From concept to code: Everything you need to know to start building an application with GenAI’s LLMs.

Gemini, the stylist… - A personalized haircut recommendation Application.

Continuous Integration and Deployment (CI/CD) for Custom Containerized ML Models on Vertex AI - In this tutorial, we will establish aCI/CD pipeline for our custom container model on Vertex AI. Here’s what we will cover:.

Fasten and Structure your Vertex AI Jobs with Hydra.cc - A comprehensive guide to fasten and structure your Vertex AI Jobs development using Hydra.cc.

Empowering Your Organisation with AI: Integrating Gemini Pro and PaLM2 into Google Sheets - A Comprehensive Guide to Using Gemini Pro and PaLM2 in Google Sheet on thousands of cells.

Practical Applications of Generative AI - After reading this article and getting oriented with the companion repository, you should have a better idea of what opportunities exist of using Generative AI.

Various

Laying the foundation for a career in platform engineering - Read on for an overview of the platform engineering field, including an introduction to what platform engineers do and the skills required.

Looking back at my 10 years in Google Cloud - Some of the most important Google Cloud services launched in the last 10 years.

Cracking the GCP Certified Professional Machine Learning Engineer Exam in a Week - Preparing for the Professional Machine Learning Engineer certification exam in a week.

GCP Professional Machine Learning Engineering Certification- Latest Tips and Tricks - Sharing tips and tricks to pass the Professional Machine Learning Engineering Certification exam.

Slides, Videos, Audio

Kubernetes Podcast - #216 NAIS, with Johnny Horvi and Frode Sundby.

Security Podcast - #154 Mike Schiffman: from Blueboxing to LLMs via Network Security at Google.

Releases

AppEngine Flexible - .NET - .NET 8 is now generally available.

Artifact Registry - Artifact Registry Tags are Generally Available (GA).

Cloud Composer - Environment snapshots now correctly work with the bring your own bucket (BYOB) feature. (Available without upgrading) Cloud Composer now creates an audit log on a successful completion of create and update environment operations. If an environment update operation fails, Airflow configuration overrides that were applied by this operation are rolled back. Cloud Composer 2.5.4 images are available: composer-2.5.4-airflow-2.5.3 composer-2.5.4-airflow-2.6.3 (default). The apache-airflow-providers-google package is upgraded to version 10.12.0 in images with Airflow 2.6.3 and 2.5.3. Cloud Composer versions 2.1.2 and 1.20.2 have reached their end of full support period.

Compute Engine - Google has patched several vulnerabilities that were discovered in the TianoCore EDK II UEFI firmware used in Google Compute Engine VMs. Generally available: NVIDIA L4 GPUs are now available in the following additional zone: Changhua County, Taiwan (asia-east1-b) For more information about using GPUs on Compute Engine, see GPU platforms.

Container Registry - Starting January 8, 2024, organizations that haven't previously used Container Registry will have any gcr.io repositories hosted on Artifact Registry by default.

Database Migration Service - Database Migration Service for Oracle to Cloud SQL for PostgreSQL and Oracle to AlloyDB for PostgreSQL migrations now support row sizes of up to 30 MB.

Datastore - The ability to create multiple databases per project is now supported at the General Availability level.

Dialogflow - Dialogflow CX Standard NLU agents will be migrated to Advanced NLU starting March 1, 2024.

Cloud Data Loss Prevention - The FINANCIAL_ACCOUNT_NUMBER infoType detector is available in all regions.

Document AI - The Custom Extractor with generative AI has General Availability and is ready for production workloads. To better support production workloads, we reduced prices for the Custom Extractor, Custom Classifier, Custom Splitter, and Form Parser. Developers can now specify pages Document AI should process within a document.

Cloud Firestore - The ability to create multiple databases per project is now supported at the General Availability level.

Cloud Functions - Cloud Functions now supports the .NET 8 runtime at the General Availability release level for 2nd gen functions. Cloud Functions (2nd gen) now supports custom service accounts for Cloud Build, at the Preview release level.

Google Kubernetes Engine - (2024-R01) Version updates GKE cluster versions have been updated. 1.29 is now available in the Rapid channel Kubernetes 1.29 is now available in the Rapid channel. New Features The beta ValidatingAdmissionPolicy feature can be opted into by enabling the beta APIs. New APIs flowcontrol.apiserver.k8s.io/v1 FlowSchema, PriorityLevelConfiguration Notable changes in flowcontrol.apiserver.k8s.io/v1: The PriorityLevelConfiguration spec.limited.nominalConcurrencyShares field only defaults to 30 when unspecified, and an explicit value of 0 is not changed to 30. Deprecated APIs The following Beta versions of graduated APIs are deprecated in 1.29 in favor of newer versions: flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema, PriorityLevelConfiguration Deprecated since 1.29 Instead, use flowcontrol.apiserver.k8s.io/v1, available since 1.29 The status.nodeInfo.kubeProxyVersion field in the Node API is deprecated and will not be populated starting in version 1.33. Removed APIs The following Beta versions of graduated APIs are removed in 1.29 in favor of newer versions: flowcontrol.apiserver.k8s.io/v1beta2 FlowSchema, PriorityLevelConfiguration Deprecated since 1.26 Instead, use flowcontrol.apiserver.k8s.io/v1beta3 available since 1.26, or flowcontrol.apiserver.k8s.io/v1, available since 1.29. Removed SHA-1 certificate support Starting from version 1.29, GKE no longer supports webhook backends that use TLS certificates signed with the insecure SHA-1 algorithm. Deprecated in-tree volume support The Ceph CephFS (kubernetes.io/cephfs) and RBD (kubernetes.io/rbd) volume plugins are deprecated in 1.28 and will be removed in a future release. GKE clusters in Autopilot mode no longer require workload-level logging to be enabled.

GKE - (2024-R01) Version updates The following control plane versions are now available: 1.24.17-gke.2472000 1.25.16-gke.1268000 1.26.12-gke.1111000 1.27.9-gke.1092000 1.28.5-gke.1217000 The following node versions are now available: 1.24.17-gke.2472000 1.25.16-gke.1268000 1.26.12-gke.1111000 1.27.9-gke.1092000 1.28.5-gke.1217000 The following control plane versions are no longer available: 1.24.16-gke.500 1.24.17-gke.2230000 1.24.17-gke.2347000 1.25.12-gke.500 1.25.16-gke.1020000 1.26.7-gke.500 1.26.10-gke.1073000 1.26.10-gke.1235000 1.27.7-gke.1056000 1.27.7-gke.1293000 1.28.4-gke.1083000 Control planes and nodes with auto-upgrade enabled will be upgraded from version 1.23 to version 1.24.17-gke.2266000 with this release.

Google Kubernetes Engine Rapid - (2024-R01) Version updates Version 1.28.3-gke.1286000 is now the default version in the Rapid channel. 1.29 is now available in the Rapid channel Kubernetes 1.29 is now available in the Rapid channel. New Features The beta ValidatingAdmissionPolicy feature can be opted into by enabling the beta APIs. New APIs flowcontrol.apiserver.k8s.io/v1 FlowSchema, PriorityLevelConfiguration Notable changes in flowcontrol.apiserver.k8s.io/v1: The PriorityLevelConfiguration spec.limited.nominalConcurrencyShares field only defaults to 30 when unspecified, and an explicit value of 0 is not changed to 30. Deprecated APIs The following Beta versions of graduated APIs are deprecated in 1.29 in favor of newer versions: flowcontrol.apiserver.k8s.io/v1beta3 FlowSchema, PriorityLevelConfiguration Deprecated since 1.29 Instead, use flowcontrol.apiserver.k8s.io/v1, available since 1.29 The status.nodeInfo.kubeProxyVersion field in the Node API is deprecated and will not be populated starting in version 1.33. Removed APIs The following Beta versions of graduated APIs are removed in 1.29 in favor of newer versions: flowcontrol.apiserver.k8s.io/v1beta2 FlowSchema, PriorityLevelConfiguration Deprecated since 1.26 Instead, use flowcontrol.apiserver.k8s.io/v1beta3 available since 1.26, or flowcontrol.apiserver.k8s.io/v1, available since 1.29. Removed SHA-1 certificate support Starting from version 1.29, GKE no longer supports webhook backends that use TLS certificates signed with the insecure SHA-1 algorithm. Deprecated in-tree volume support The Ceph CephFS (kubernetes.io/cephfs) and RBD (kubernetes.io/rbd) volume plugins are deprecated in 1.28 and will be removed in a future release.

Google Kubernetes Engine Regular - (2024-R01) Version updates The following versions are now available in the Regular channel: 1.24.17-gke.2364000 1.25.16-gke.1041000 1.26.11-gke.1055000 1.27.8-gke.1067000 1.28.3-gke.1286000 The following versions are no longer available in the Regular channel: 1.24.17-gke.2230000 1.25.13-gke.200 1.26.10-gke.1073000 1.27.7-gke.1056000 Control planes and nodes with auto-upgrade enabled in the Regular channel will be upgraded from version 1.23 to version 1.24.17-gke.2266000 with this release.

Google Kubernetes Engine Stable - (2024-R01) Version updates The following versions are now available in the Stable channel: 1.24.17-gke.2266000 1.25.15-gke.1115000 1.26.10-gke.1101000 1.27.7-gke.1121000 1.28.3-gke.1203001 The following versions are no longer available in the Stable channel: 1.24.16-gke.500 1.25.12-gke.500 1.26.7-gke.500 Control planes and nodes with auto-upgrade enabled in the Stable channel will be upgraded from version 1.23 to version 1.24.17-gke.200 with this release.

Cloud VPN - Cloud VPN lets you connect an HA VPN gateway to Compute Engine virtual machine (VM) instances.

Policy Intelligence - The requirement that customers have organization-level activations of Security Command Center to use certain Policy Intelligence features has been delayed until April 29, 2024.

Cloud Run - Tags configured on Cloud Run services are now available when exporting Cloud Billing data to BigQuery.

Security Command Center - Issue that caused finding severities to change unexpectedly is resolved Between December 11, 2023 and January 10, 2024, an issue might have changed the severities of some findings unexpectedly.

Service Mesh - Managed Anthos Service Mesh. Managed Anthos Service Mesh 1.17 is rolling out in the regular channel.

SAP Solutions - Google Cloud's Agent for SAP version 3.0 Version 3.0 of Google Cloud's Agent for SAP is generally available (GA). The Cloud Storage Backint agent for SAP HANA is deprecated, and is replaced by the Backint feature of version 3.0 of Google Cloud's Agent for SAP. Cloud Storage Backint agent for SAP HANA version 1.0.33 Version 1.0.33 of the Cloud Storage Backint agent for SAP HANA is available.

Cloud Speech-to-Text - Model adaptation is now available for latest_long models in 13 languages. Speech-to-Text has launched a new model, named chirp_telephony to bring the accuracy gains of our chirp model to telephony-specific use cases.

Vertex AI - Model tuning for the textembedding-gecko and textembedding-gecko-multilingual models is available in GA. AutoSxS evaluates LLMs side by side The automatic side-by-side (AutoSxS) evaluation tool is available in Preview to A/B test the performance of your LLMs or pre-generated predictions.

Virtual Private Cloud - Regional service endpoints that are used with Private Service Connect backends are now referred to as locational service endpoints. Private Service Connect endpoints support accessing regional service endpoints.

Workflows - Support for invoking a VPC Service Controls-compliant private endpoint is generally available (GA). Support for defining environment variables at deployment time is generally available (GA).