Welcome to issue #384 February 5th, 2024

News

Heita South Africa! The new Google Cloud region is now open in Johannesburg

---

Imagine having a direct line to over 150 senior cloud architects for any cloud-related question or issue you encounter. With thousands of cloud questions and issues resolved, DoiT is your gateway to world-class cloud expertise.

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

GKE plus Filestore - Improve training times for AI/ML workloads by up to 37% - This blog post focuses on the important role that Filestore can play in training AI/ML workloads.

Consuming Cloud Storage objects on GKE using the Kubernetes API - Part I - The Cloud Storage FUSE CSI driver is a fully-managed experience powered by the open-source Google Cloud Storage FUSE CSI driver. You get portability, reliability, performance, and out-of-the-box GKE integration.

IDC: Migrating to Google Cloud IaaS has a 318 percent ROI - IDC research study to provides better understanding of the tangible ROI of migrating to cloud infrastructure.

Cloud CISO Perspectives: Executive lessons from our response to Reptar

Cloud HPC made easy: A Blueprint Catalog for Google's Cloud HPC Toolkit - An overview of Cloud HPC Toolkit, that simplifies the deployment and management of HPC workloads and environments on Google Cloud.

Facts you should know about GKE (Google Kubernetes Engine) - you may save $$$ in your bill and avoid painful fixes!!

Operationalizing Google Cloud Media CDN - The purpose of this particular blog is to enable users of Media CDN to understand how to gain more visibility into Media CDN performance , monitor the various aspects of service , and (if needed) do a better and faster troubleshooting of Media CDN.

Google Cloud HA VPN to Compute Engine VM (NVA) config lab with VyOS - Google Cloud announced the GA of the Cloud HA VPN support of compute Instances with public IPs.

App Development, Serverless, Databases, DevOps

Google Cloud databases round-up: January 2024 - The latest feature releases and innovations within GCP database offerings.

ABAP SDK: Powering Alphabet’s AI/ML solutions on SAP S/4HANA

Moving to S/4 HANA on Google Cloud brings significant benefits for Deutsche Börse Group

How Visual Research reduced costs by 35 percent with the help of Google Cloud - By adopting Google Compute Engine, Cloud SQL for SQL Server, and Memorystore for Redis, Visual Research reduced its license costs by over 35%, eliminated manual tasks, improved performance, and gained the ability to scale to handle peak seasons.

Getting started with Svelte on Google Cloud

Book Review : Database Design and Modeling with Google Cloud by Abirami Sukumaran - This is a book review of Database Design and Modeling with Google Cloud by Abirami Sukumaran.

Centralized Log Monitoring for multiple GCP projects - A walk-through setting up a centralized log monitoring solution for multiple GCP projects, empowering you to aggregate logs from various sources into a single, easy-to-manage location.

Firebase Email Enumeration Enforcement: A Workaround For ‘fetchSignInMethodsForEmail’ - Handling correctly Firebase fetchSignInMethodsForEmail.

Host and monetize your Streamlit app cost-effectively - Hosting and monetizing a Streamlit app for many users can be quite tricky, but that's not as impossible as it looks like.

High throughput file transfers with Google Cloud Storage - This guide will teach you how to parallelize your Google Cloud Storage (GCS) file transfers for dramatically increased throughput vs. single stream transfers.

Big Data, Analytics, ML&AI

LookML or ELT? Three reasons why you need LookML

Synthesized creates accurate synthetic data to power innovation with BigQuery - Synthesized helps organizations gain faster access to data and navigate compliance restrictions by using generative AI to create shareable, compliant snapshots of large datasets.

Automated fraud detection with Fivetran and BigQuery - The combination of Fivetran and BigQuery provides a simple design to a complex problem — an effective fraud detection solution capable of real-time, actionable alerts.

Game on: Aiven for Apache Kafka and BigQuery - your ultimate gaming cheat code - By partnering with Google Cloud and Aiven on Google Cloud, the games industry can prepare for a worry-free launch while having the data to understand players.

A Technical Guide for Building BigQueryML Models in Looker - In this blog post, we will explore the capabilities and workflow for generating, evaluating, and utilizing machine learning (ML) models in BigQuery from Looker.

A Guide to Search Indexes in BigQuery - If you’re dealing with huge amounts of data or developing search-intensive applications, understanding and implementing this technique can….

Guide to Implementing Custom Docker Containers in Google Cloud Dataflow - In this extensive guide, we’ll walk through the detailed process of creating, building, and deploying custom Docker containers for Dataflow, ensuring enhanced performance and scalability of your data pipelines.

Data refinement in action: Apache Flink® for data transformation into Google BigQuery - Data refinement in action: Apache Flink® for data transformation into Google BigQuery.

Pub/Sub and Batching: Optimizing Cloud Messaging Performance - This post explores how batching messages in Pub/Sub can enhance performance, ensuring a more reliable, efficient, and cost-effective message handling process.

Paperless: Dataproc Serverless Meets Jupyter - Paperless is a Python package designed to streamline the execution of Jupyter Notebooks with a Spark kernel.

Technical and Business data alerts using Dataform - How to create and run data alerts using Dataform and send notifications to Slack.

JustSQL GPT: Making BigQuery Easier for Everyone - JustSQL makes complex data analysis accessible to everyone by translating natural language queries into SQL for BigQuery, eliminating the need for deep technical expertise in SQL without compromising on privacy and efficiency.

Build enterprise gen AI apps with Google Cloud databases - An overview of databases on GCP that can be used to store and query vector embeddings.

Automate public website indexing for efficient semantic search with Vertex AI - In this blog post you will learn the power of vector search and additionally, you will explore techniques for rapid ingestion of unstructured data, such as web pages, to enhance your search and chat systems efficiently.

Various

29 no-cost ways to leap ahead in your cloud career this February - Take a leap into learning this February, and make the most of the extra day! We’ve created a list of 29 no-cost ways for you to leap ahead in your cloud career, so take your pick and get learning something new!

Retrospective 2023: Between Conferences, Cloud Certifications, and Recognitions

Slides, Videos, Audio

Security Podcast - #157 EP157 Decoding CDR & CIRA: What Happens When SecOps Meets Cloud.

GCP Life Podcast - #57 “I was pwnd!“ – Know your customer, US CLOUD Act, Database Enhancements, Google Q4 Results, SYS:ALL Loophole, GKE Cluster Takeover, Medibank Hacker Sanctions, Optus Triple Zero Calls, Stolen Data on Genesis, Google Settled, Google Lumiere, Google and Hugging Face, Google and Samsung S24, Gemini API.

Releases

AlloyDB - AlloyDB Public IP is now available in Preview. Fixed the issue causing failed connections to certain AlloyDB instances when using Auth Proxy version 1.5.0.

Anthos clusters on bare metal - 1.15. Release 1.15.9 GKE on Bare Metal 1.15.9 is now available for download. Fixes: The following container image security vulnerabilities have been fixed in 1.15.9: Critical container vulnerabilities: CVE-2023-25775 High-severity container vulnerabilities: CVE-2021-41617 CVE-2023-5717 CVE-2023-5869 CVE-2023-6931 CVE-2023-6932 CVE-2023-27533 CVE-2023-29491 CVE-2023-35827 CVE-2023-39417 CVE-2023-51780 CVE-2023-51781 CVE-2023-51782 Medium-severity container vulnerabilities: CVE-2021-44879 CVE-2023-3446 CVE-2023-3817 CVE-2023-5870 CVE-2023-6121 CVE-2023-25165 CVE-2023-27535 CVE-2023-27536 CVE-2023-27538 CVE-2023-28321 CVE-2023-34324 CVE-2023-36054 CVE-2023-45863 CVE-2023-46218 CVE-2023-46343 CVE-2023-51385 CVE-2023-5868 CVE-2024-0584 Low-severity container vulnerabilities: CVE-2023-2975 CVE-2023-4527 CVE-2023-4911 CVE-2023-5178 CVE-2023-5197 CVE-2023-6531 CVE-2023-6817 CVE-2023-27534 CVE-2023-28322 CVE-2023-38545 CVE-2023-38546 CVE-2023-46813 CVE-2023-46862 CVE-2024-0193 CVE-2024-0641. Known issues: For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section. 1.28. Release 1.28.100-gke.146 GKE on Bare Metal 1.28.100-gke.146 is now available for download. Fixes: Fixed a rootless permission issue on file /var/lib/audit.log in 1.28.100, which might block control plane node upgrades. Known issues: For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section. 1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14 & 1.15 & 1.16 & 1.28. Security bulletin (all minor versions) A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods might be able to gain full access to the node filesystem. 1.16. Release 1.16.5 GKE on Bare Metal 1.16.5 is now available for download. Fixes: The following container image security vulnerabilities have been fixed in 1.16.5: Critical container vulnerabilities: CVE-2023-25775 High-severity container vulnerabilities: CVE-2021-3121 CVE-2021-41617 CVE-2022-4450 CVE-2022-29458 CVE-2023-0215 CVE-2023-0286 CVE-2023-0361 CVE-2023-0464 CVE-2023-5717 CVE-2023-5869 CVE-2023-6931 CVE-2023-6932 CVE-2023-27533 CVE-2023-29491 CVE-2023-35827 CVE-2023-39417 CVE-2023-51780 CVE-2023-51781 CVE-2023-51782 Medium-severity container vulnerabilities: CVE-2020-8565 CVE-2021-44879 CVE-2022-2097 CVE-2022-3821 CVE-2022-4304 CVE-2022-4415 CVE-2023-0465 CVE-2023-0466 CVE-2023-2650 CVE-2023-5868 CVE-2023-5870 CVE-2023-6121 CVE-2023-25165 CVE-2023-27535 CVE-2023-27536 CVE-2023-27538 CVE-2023-28321 CVE-2023-34324 CVE-2023-36054 CVE-2023-45863 CVE-2023-46343 CVE-2023-51385 CVE-2024-0584 Low-severity container vulnerabilities: CVE-2023-5178 CVE-2023-5197 CVE-2023-6531 CVE-2023-6817 CVE-2023-27534 CVE-2023-28322 CVE-2023-38545 CVE-2023-38546 CVE-2023-46813 CVE-2023-46862 CVE-2024-0193 CVE-2024-0641. Known issues: For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

Anthos clusters on VMware - GKE on VMware 1.15.8-gke.41 is now available. Upgraded etcd to v3.4.27-0-gke.1. The following issues are fixed in 1.15.8-gke.41: Fixed Seesaw crashing on duplicated service IP. A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods on Container-Optimized OS and Ubuntu nodes might be able to gain full access to the node filesystem.

Apigee X - On February 2, 2024, we released an updated version of Apigee. We modified or added these limits: Changed the maximum API proxy endpoints per API proxy from 5 to 10 Specified the maximum API base paths per organization as 21,250 See the Limits page for details. On February 1, 2024, we released an updated version of Apigee.

AppEngine Standard Go - Go 1.12, 1.13, 1.14, 1.15, 1.16, and 1.18 have reached end of support on January 30, 2024.

AppEngine Standard Java - Java 8 has reached end of support on January 31, 2024.

AppEngine Standard NodeJS - Node.js 10, 12, 14, and 16 have reached end of support on January 30, 2024.

AppEngine Standard PHP - PHP 5 has reached end of support on January 30, 2024.

AppEngine Standard PHP7 - PHP 7.2, 7.3, and 7.4 have reached end of support on January 30, 2024.

AppEngine Standard Python - Python 2.7 has reached end of support on January 31, 2024.

AppEngine Standard Python3 - Python 3.7 has reached end of support on January 30, 2024.

AppEngine Standard Ruby - Ruby 2.5, 2.6, and 2.7 have reached end of support on January 30, 2024.

Artifact Registry - Artifact Registry is available in the africa-south1 region (Johannesburg, South Africa).

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs.

Batch - You can configure custom status events, which describe important events for a job's runnables. You can write unstructured and structured task logs: An unstructured task log lets you define a log's message. You can run Batch jobs as a non-root user to meet workload or security requirements.

BigQuery - The following information schema views display the history of configuration changes to the options of your organization and projects: ORGANIZATION_OPTIONS_CHANGES view displays the configuration changes to an organization, including all organization and project-level changes. BigQuery now supports vector search and vector indexes. You can now use tags on BigQuery tables to conditionally grant or deny access with Identity and Access Management (IAM) policies. Cloud console updates: You can now sort query results by column.

Bigtable - The Bigtable Studio query builder is generally available (GA). Bigtable is available in the africa-south1 (Johannesburg) region.

Chronicle - The following log types were added to the Chronicle feed management API to create AWS data feeds. The bi-weekly release of Chronicle parsers will change to a more frequent release schedule to allow for more testing before parser changes automatically take effect in Parser Management.

Cloud Composer - In the first half of February, 2024, Cloud Composer 2 environments with Airflow 2.6.3 will start using Python 3.11: New and upgraded environments with Airflow 2.6.3 will switch to Python 3.11.

Compute Engine - Generally available: You can plan ahead for VM maintenance on C3, C3D, and Z3 Preview machine types by viewing their maintenance schedule notifications. Preview: You can create GPU VMs in a MIG by using resize requests. Generally available: Johannesburg, South Africa africa-south1-a,b,c has launched with E2, N2, N2D, and T2D general-purpose VMs in all three zones. Generally available: Persistent Disk Asynchronous Replication is available between the following region pairs: europe-west3 (Frankfurt, Germany) and europe-west8 (Milan, Italy) europe-west3 (Frankfurt, Germany) and europe-west10 (Berlin, Germany) us-east1 (Moncks Corner, South Carolina) and northamerica-northeast1 (Montréal, Québec) For the full list of available regions, see Supported region pairs. Preview: Z3 VMs, which offer the latest compute, networking, and storage innovations in one platform with a particular focus on high density, high performing Local SSD are now in Preview. Generally available: Snapshot settings are centralized configuration parameters for all snapshots in a project. Generally available: NVIDIA L4 GPUs are now available in the following additional region and zone: Zurich, Switzerland (europe-west6-b) For more information about using GPUs on Compute Engine, see GPU platforms.

Dataflow - Dataflow is available in Johannesburg, South Africa (africa-south1).

Dataproc Serverless - Dataproc on Compute Engine: Bucket ttl validation now also runs for buckets created by Dataproc. Dataproc on Compute Engine: Added a warning during cluster creation if the cluster Cloud Storage staging bucket is using the legacy fine-grained/ACL IAM configuration instead of the recommended Uniform bucket-level access controls. Dataproc Serverless for Spark: When dynamic allocation is enabled, the initial executor number is determined by max of spark.dynamicAllocation.initialExecutors and spark.executor.instances. New Dataproc on Compute Engine subminor image versions: 2.0.91-debian10, 2.0.91-rocky8, 2.0.91-ubuntu18 2.1.39-debian11, 2.1.39-rocky8, 2.1.39-ubuntu20, 2.1.39-ubuntu20-arm 2.2.5-debian12, 2.2.5-rocky9, 2.2.5-ubuntu22. New Dataproc Serverless for Spark runtime versions: 1.1.48 2.0.56 2.1.35 2.2.0-RC8. Dataproc on Compute Engine: Backported patches for HIVE-21214, HIVE-23154, HIVE-23354 and HIVE-23614. Dataproc is now available in the africa-south1 region (Johannesburg, South Africa). The GitHub Ops Agent initialization action installs the Ops Agent on a Dataproc cluster, and provides metrics similar to the metrics that were enabled with the --metric-sources=monitoring-agent-defaults setting available for use with Dataproc images versions prior to version 2.2.

Dataproc - Dataproc on Compute Engine: Bucket ttl validation now also runs for buckets created by Dataproc. Dataproc on Compute Engine: Added a warning during cluster creation if the cluster Cloud Storage staging bucket is using the legacy fine-grained/ACL IAM configuration instead of the recommended Uniform bucket-level access controls. Dataproc Serverless for Spark: When dynamic allocation is enabled, the initial executor number is determined by max of spark.dynamicAllocation.initialExecutors and spark.executor.instances. New Dataproc on Compute Engine subminor image versions: 2.0.91-debian10, 2.0.91-rocky8, 2.0.91-ubuntu18 2.1.39-debian11, 2.1.39-rocky8, 2.1.39-ubuntu20, 2.1.39-ubuntu20-arm 2.2.5-debian12, 2.2.5-rocky9, 2.2.5-ubuntu22. New Dataproc Serverless for Spark runtime versions: 1.1.48 2.0.56 2.1.35 2.2.0-RC8. Dataproc on Compute Engine: Backported patches for HIVE-21214, HIVE-23154, HIVE-23354 and HIVE-23614. Dataproc is now available in the africa-south1 region (Johannesburg, South Africa). The GitHub Ops Agent initialization action installs the Ops Agent on a Dataproc cluster, and provides metrics similar to the metrics that were enabled with the --metric-sources=monitoring-agent-defaults setting available for use with Dataproc images versions prior to version 2.2.

Datastore - Eventarc events and Firestore events for Cloud Functions (2nd gen) are now supported at the General Availability (GA) level.

Cloud Data Loss Prevention - You can now configure your discovery scans to reprofile data when the inspection template changes. Sensitive Data Protection is now available in Johannesburg, South Africa (africa-south1 region).

Eventarc - Eventarc is available in the africa-south1 (Johannesburg, South Africa) region. Eventarc support for creating triggers for direct events from Cloud Firestore is generally available (GA).

Cloud Firestore - Eventarc events and Firestore events for Cloud Functions (2nd gen) are now supported at the General Availability (GA) level.

Identity-Aware Proxy - Effective January 12, 2024, a BeyondCorp Enterprise license is no longer required to deploy internal applications with an internal load balancer when securing those applications with Identity-Aware Proxy.

Networking Interconnect - Dedicated Cloud Interconnect support is available in the following colocation facilities: Teraco Johannesburg Campus, South Africa Africa Data Centres, Johannesburg JHB2 For more information, see the Locations table.

KMS - Cloud KMS is available in the following region: africa-south1 For more information, see Cloud KMS locations.

Google Kubernetes Engine - FQDN network policies are now generally available with the following GKE versions: 1.26.4-gke.500 and later. You can now encrypt Pod-to-Pod traffic between nodes in the same cluster or in a multi-cluster environment natively with GKE. A security vulnerability, CVE-2024-21626, has been discovered in runc where a user with permission to create Pods on Container-Optimized OS and Ubuntu nodes might be able to gain full access to the node file system. The africa-south1 region in Johannesburg, South Africa is now available.

Cloud Logging - Fixed a bug that caused the audit log associated with an API that performs both Data Access and Admin Activity operations to be classified as a Data Access log.

Managed Microsoft AD - Managed Microsoft AD is available in the africa-south1 (Johannesburg) region.

Cloud Memorystore - Added new Memorystore for Redis region: Johannesburg (africa-south1).

Cloud Monitoring - New event types for VM instances and for GKE Pods, Clusters, and Nodes, are now available to display on your dashboards.

Cloud Interconnect - Dedicated Cloud Interconnect support is available in the following colocation facilities: Teraco Johannesburg Campus, South Africa Africa Data Centres, Johannesburg JHB2 For more information, see the Locations table.

Cloud VPN - Cloud VPN is now available in region africa-south1 (Johannesburg, South Africa). Cloud VPN support for IPv6-only HA VPN gateways is in Preview.

Cloud PubSub - Pub/Sub is available in Johannesburg, South Africa (africa-south1).

Cloud Run - The following new region is now available: africa-south1.

Secret Manager - Secret Manager is now available in the following region: africa-south1 For more information, see Secret Manager locations.

Security Command Center - Virtual Machine Threat Detection, a built-in service of Security Command Center, launched the Malware: Malicious file on disk (YARA) detector to Preview.

Service Mesh - Managed Anthos Service Mesh. In February 2024, Managed Anthos Service Mesh will begin creating new Google Cloud backend resources that relate to upcoming control plane enhancements. Managed Anthos Service Mesh 1.17 is rolling out in the stable channel.

SAP Solutions - New SAP HANA certification: Hyperdisk Balanced usage with M3 machine types For running SAP HANA on Google Cloud, SAP has certified using Hyperdisk Balanced with M3 machine types.

Cloud Spanner - You can create Spanner regional instances in Johannesburg, South Africa (africa-south1). Cloud Spanner directed reads is now available in Preview.

Cloud SQL MySQL - Private Service Connect now includes support for cross-region read replicas. Support for africa-south1 (Johannesburg) region. You can now use the MySQL Shell dumpInstance and loadDump utilities to export and import data for multiple files in parallel.

Cloud SQL Postgres - Private Service Connect now includes support for cross-region read replicas. Support for africa-south1 (Johannesburg) region. You can now use the pg_dump and pg_restore utilities to export and import data for multiple files in parallel.

Cloud SQL SQL Server - You can now use Private Service Connect to connect to a Cloud SQL for SQL Server instance. Support for africa-south1 (Johannesburg) region.

Cloud Storage - Cloud Storage is now available in Johannesburg, South Africa (africa-south1 region).

Vertex AI - Vertex Prediction You can now customize more deployment parameters when uploading your models, such as shared memory allocation and custom startup and readiness probes.

Virtual Private Cloud - Private Service Connect interfaces are available in General Availability. For auto mode VPC networks, added a new subnet 10.218.0.0/20 for the Johannesburg africa-south1 region.

Workflows - Workflows is available in the following additional region: africa-south1 (Johannesburg, South Africa).