News

GKE Enterprise, the next evolution of container platforms, is now generally available - With GKE Enterprise, companies can increase development and deployment velocity across multiple teams, easily and securely run their most important business-critical workloads, and reduce total cost of ownership with a fully integrated and managed solution from Google Cloud.

Cloud Storage Autoclass now available for existing Cloud Storage buckets

Advanced text analyzers and preprocessing functions in BigQuery - A public preview of a set of text analysis and preprocessing functions and capabilities in BigQuery. These new features will be an essential part of text processing on top of primitive string functions to further enhance the Search and ML experience in BigQuery.

Introducing Cloud SQL in-place upgrade: move from Enterprise to Enterprise Plus with ease - An in-place upgrade from Enterprise edition to Enterprise Plus edition, which provides a seamless in-place upgrade path with minimal disruption (anticipated downtime of < 60 seconds).

Announcing Cloud TPU v5e GA for cost-efficient AI model training and inference - Cloud TPU v5e is now generally available, as weel as Singlehost inference and Multislice Training technologies.

Introducing Accurate Quantized Training (AQT) for accelerated ML training on TPU v5e - Introduction of the open-source Accurate Quantization Training (AQT) library that provides the software support needed for easy tensor operation quantization in JAX.

Calculate SUM and AVG aggregations using Firestore - New feature of Cloud Firestore supports SUM and AVG aggregation functions.

Golang’s GORM support for Cloud Spanner is now Generally Available

Synthetic Monitoring in Cloud Monitoring is now Generally Available - Synthetic monitoring uses automated Node.js scripts to simulate user interactions with an application or service. This allows you to test the availability, consistency, and performance of your web applications, APIs, and key business workflows from the perspective of a real user, on a periodic basis.

Running AI and ML workloads with the Cloud HPC Toolkit - Google Cloud HPC Toolkit is a set of open-source tools and resources that help you create repeatable, turnkey HPC environments for your HPC, AI and machine learning workloads; you can quickly and easily use an existing blueprint, or create your own in a simple YAML file, to get a cluster up and running in minutes.

Google Cloud at SC23: Learn how to accelerate your HPC and AI workloads - Supercomputing conference, SC23 is set to take place this year from November 12th to 17th, 2023, in Denver, Colorado.

Google supports launch of DAOS Foundation, for next-generation HPC and AI storage - t Google Cloud is a founding member of the DAOS Foundation, launched today by the Linux Foundation.

Google Cloud offers new ways to build cloud skills and identify talent - Today, we’re excited to share that we've added three new skill badges, bringing our learning portfolio to 80+ skill badges covering in-demand topics ranging from security, data, and AI.

Safeguarding the bioeconomy, with Bio-ISAC - Google Cloud joined the Bioeconomy Information Sharing and Analysis Center (Bio-ISAC) as an Industry Member.

Google Cloud sponsors CyberGreen Institute to advance research in Cyber Public Health - Google Cloud is becoming an official sponsor of the CyberGreen Institute, a leading organization dedicated to advancing the field of cyber public health.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Google Cloud Cybersecurity Forecast 2024: A look at the cyber landscape in the year ahead - Published security report can help prepare security professionals and business leaders for the year ahead by providing forward-looking insights from cyber experts across Google Cloud, including leaders and experts on the frontlines of the latest and largest attacks.

Tips on building a network security policy in Google Cloud

Centralised management of VPCSC with Terraform and Cloud Asset Inventory - Centrally managing VPC Service Controls using Terraform and Cloud Asset Inventory.

Explaining Kubernetes Clusters VS Workloads Views in GCP Console - This article aims to explain the difference between the view of Cluster and Workload as encountered in the GKE console.

App Development, Serverless, Databases, DevOps

Build a Java Spring Boot app in IntelliJ with Duet AI assistance - This article explains how to create a simple Java Spring Boot web application from scratch with the help of Duet AI inside Jetbrains IntelliJ IDEA.

3 new ways Duet AI can help you get things done fast in the Google Cloud console - Complexity can hold organizations back from realizing the full potential of cloud. Duet AI, a new AI-powered collaborator for Google Cloud, can help you reduce complexity and overcome obstacles by providing assistance with a variety of common tasks.

Using FFmpeg with Google Cloud Speech-to-Text - Using FFmpeg library to decode audio file to correct format for Speech-to-Text processing.

The pitfalls of deploying a Node.js backend with GCP Cloud Build - Handling various issues in the Cloud Build deployment pipeline for the Cloud Run application.

Core RAG Architecture with AlloyDB AI - Retrieval Augmented Generation Architecture and Implementation Example.

Rate limiting and Cloud Run dynamic IP address pool - Fixing rate-limiting of Shopify app that runs on Cloud Run using a dedicated IP.

Dynamically determine Cloud Run Job numbers and execute it via Workflow - Using Cloud Workflows to dynamically launch Cloud Run jobs.

Cloud Functions Combed - An architectural approach for growing Cloud Function projects.

Big Data, Analytics, ML&AI

Migrating IBM DataStage to Google Cloud - IBM DataStage is a data integration tool that helps you design, build, and run jobs to move and transform data. It supports both ETL (extract, transform, load) and ELT (extract, load, transform) patterns. IBM DataStage supports a variety of data sources to a variety of targets, including the Netezza database.

No more double vision: How Miinto improved its customer experience using Vertex AI Vision - Solving inventory duplication using Vertex AI Vision.

Dataflow Streaming Engine autotuning: Improve autoscaling for your streaming jobs - An overview of new capabilities for more intelligent autotuning of autoscaler behavior for Dataplex streaming jobs.

Kafka migration from on-prem to Confluent - The purpose of this technical blog is to guide readers through the migration process from on-prem to Confluent Kafka and to shed light on the data ingesting/transformation processes carried through data pipelines to finally store data to BigQuery.

Using Correlated Subqueries in BigQuery - There are several interesting concepts in BigQuery once you get past the basics. One such concept is a correlated subquery. In this short walkthrough, we’re going to look at what they are and where are they used, as well as things to pay attention to.

10 pitfalls to know when using your GA4 Data in BigQuery - Covering common mistakes and issues when working with Google Analytics in BigQuery.

Google launched Bag of Word for BigQuery & BigQuery ML - How you can now do Text Analysis easily.

Google Cloud’s approach to trust and transparency in AI

Google Cloud demonstrates the world’s largest distributed training job for large language models across 50000+ TPU v5e chips

AssemblyAI leverages Google Cloud TPU v5e for leading price-performance on large-scale AI inference

Bringing capacity assurance and faster startup times to Vertex AI Training - In this blog, we explain how you can use persistent resources to ensure compute resource availability and achieve faster startup times to run your critical model training applications.

Deploying Kedro Pipelines on Vertex AI: The MLOps journey of a Life Company - Easy way to scale Kedro projects in production.

Integrating langchain4j and PaLM 2 Chat Bison Model - Converting Python to Java code that invokes Chat Bison PaLM2 model hosted on Vertex AI.

Duet AI Review: My Perception and Use Cases — Part 1 - This article is Part 1 of a two-part review on Duet AI, presenting my perceptions about the use of the tool.

Slides, Videos, Audio

Security Podcast - #147 Special: 2024 Security Forecast Report.

Releases

AlloyDB - The extension pgvector is updated to version 0.5.0. The extension oracle_fdw is added to the extensions supported by AlloyDB.

Anthos clusters on bare metal - 1.14. Release 1.14.10 Anthos clusters on bare metal 1.14.10 is now available for download. Functionality changes: Added NODEPOOL-NAME, NODEPOOL-NAMESPACE, and STATUS columns for the InventoryMachine resource to improve troubleshooting. Fixes: Fixed an issue where CoreDNS Pods can get stuck in an unready state. Fixes: The following container image security vulnerabilities have been fixed in version 1.14.10: Critical container vulnerabilities: CVE-2023-38408 High-severity container vulnerabilities: CVE-2021-20206 CVE-2021-25741 CVE-2022-3064 CVE-2023-28840 Medium-severity container vulnerabilities: CVE-2019-11250 CVE-2020-8554 CVE-2020-8561 CVE-2020-21047 CVE-2021-4235 CVE-2021-25736 CVE-2022-46146 CVE-2023-28841 CVE-2023-28842 CVE-2023-39347 CVE-2023-41333 Low-severity container vulnerabilities: CVE-2021-25740 CVE-2021-32292 CVE-2023-34242 CVE-2023-41332 GHSA-qq97-vm5h-rrhg. Known issues: For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

Anthos clusters on Azure - A vulnerability (CVE-2023-4004) has been discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes.

Anthos clusters on VMware - A vulnerability (CVE-2023-4004) has been discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes.

Apigee X - As of November 10, 2023, Configurable API Proxies (preview) is no longer available. On November 10, 2023 we released an updated version of Apigee. Apigee is now available in a new region: Middle East - Dammam (me-central2).

AppEngine Flexible Python - Python 3.12 is now generally available.

AppEngine Standard Python3 - Python 3.12 is now generally available.

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, and Feed APIs: ContactCenter AI Insights contactcenterinsights.googleapis.com/IssueModel contactcenterinsights.googleapis.com/PhraseMatcher contactcenterinsights.googleapis.com/View.

Batch - You might experience latency when listing jobs in projects that contain more than 10,000 jobs.

BigQuery ML - The following BigQuery ML features for time series forecasting are now generally available (GA): Ensure forecasted values fall within specified limits and Custom holiday modeling.

BigQuery - The batch SQL translator has added enhancements when viewing SQL translation reports. The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery.

BigTable - Cloud Bigtable app profiles now let you configure request priorities to prioritize certain workload data requests over others.

Chronicle - Detection Engine has added support for rule statuses for Chronicle YARA-L rules running on live data.

Cloud Composer - The apache-airflow-providers-google package is upgraded to version 10.10.1 in images with Airflow 2.6.3 and 2.5.3. The apache-airflow-providers-cncf-kubernetes package was upgraded to version 7.6.0. Cloud Composer 2.5.1 images are available: composer-2.5.1-airflow-2.5.3 composer-2.5.1-airflow-2.6.3 (default). Cloud Composer versions 2.0.31, 2.0.30, 1.19.14, and 1.19.13 have reached their end of full support period.

Compute Engine - Preview: In a managed instance group (MIG), you can turn off repairs to inspect failed and unhealthy VMs, to implement your own repair logic, or to monitor the application health without triggering repairs by MIG. Generally available: A replica recovery checkpoint of a regional Persistent Disk volume represents the most recent crash-consistent point in time of the fully replicated disk.

Dataproc Metastore - v1. The Data Catalog Sync feature is generally available (GA).

Dataproc Serverless - Announcing the General Availability (GA) release of Dataproc Jupyter Plugin and its availability in Vertex AI Workbench instance notebooks. Announcing the release of Workflow Template CMEK (Customer Managed Encryption Key) encryption. You can now use Dataproc Serverless autoscaling V2 to help you manage Dataproc Serverless workloads, improve workload performance, and save costs. Set spark.shuffle.mapOutput.minSizeForBroadcast=128m to fix SPARK-38101 when Dataproc Serverless Spark dynamic allocation is enabled.

Dataproc - Announcing the General Availability (GA) release of Dataproc Jupyter Plugin and its availability in Vertex AI Workbench instance notebooks. Announcing the release of Workflow Template CMEK (Customer Managed Encryption Key) encryption. You can now use Dataproc Serverless autoscaling V2 to help you manage Dataproc Serverless workloads, improve workload performance, and save costs. Set spark.shuffle.mapOutput.minSizeForBroadcast=128m to fix SPARK-38101 when Dataproc Serverless Spark dynamic allocation is enabled.

Datastore - Support for Firestore in Datastore mode point-in-time recovery (PITR) feature that provides protection against accidental deletion or writes is now generally available (GA).

Datastream - You can now stream the following large object data types for Oracle sources: BLOB CLOB NCLOB Note: For Datastream to stream large object data types to the destination, you need to pass the streamLargeObjects flag in your request to create or update a stream. You can now recover a permanently failed stream. You can now start a stream from a specific binary log position for MySQL sources using the Datastream API.

Cloud Deploy - Configuring Google Cloud operations suite alerts is now supported in the Cloud Deploy console.

Dialogflow - Dialogflow CX now has a new language code for Hebrew: he-il.

Cloud Data Loss Prevention - The following changes were made to the COUNTRY_DEMOGRAPHIC infoType detector: The sensitivity score was changed from HIGH to MODERATE.

Cloud Firestore - Support for Firestore point-in-time recovery (PITR) feature that provides protection against accidental deletion or writes is now generally available (GA).

Cloud Functions - Cloud Functions now supports on-deployment security updates (1st gen and 2nd gen) and fully automatic security updates (1st gen only). Cloud Functions now supports the Python 3.12 runtime at the General Availability release level.

Cloud Healthcare API - Configuring Blob storage settings is now available in Preview.

Google Kubernetes Engine - A vulnerability (CVE-2023-4004) has been discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes. The Observability tab for a GKE deployment now shows application performance metrics if the metrics are available. (2023-R23) Version updates GKE cluster versions have been updated. GKE Infrastructure Dashboards and Metrics Packages are now generally available for both GKE Autopilot and Standard clusters with control plane version 1.27.2-gke.1200 and later. New inference-focused Cloud Tensor Processing Unit (TPU) v5e machine types are available in GKE. Cloud Tensor Processing Unit (TPU) v5e is generally available in clusters running GKE version 1.27.2-gke.2100 and later. A set of vulnerabilities (CVE-2023-4015, CVE-2023-4623, CVE-2023-4623, CVE-2023-4921) have been discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes. GKE begins automatically upgrading clusters still running version 1.24 to version 1.25 after 1.24 reaches end of life on January 8, 2024.

GKE - (2023-R23) Version updates The following control plane and node versions are now available: 1.24.17-gke.2198000 1.24.17-gke.2211000 1.25.15-gke.1033000 1.25.15-gke.1049000 1.26.10-gke.1024000 1.26.10-gke.1038000 1.27.7-gke.1038000 1.27.7-gke.1056000 The following control plane versions are no longer available: 1.24.14-gke.2700 1.27.6-gke.1506000 Control planes and nodes with auto-upgrade enabled will be upgraded from version 1.23 to version 1.24.15-gke.1700 with this release.

Google Kubernetes Engine Rapid - (2023-R23) Version updates Version 1.27.5-gke.200 is now the default version in the Rapid channel.

Google Kubernetes Engine Regular - (2023-R23) Version updates The following versions are now available in the Regular channel: 1.24.17-gke.200 1.25.13-gke.200 1.26.8-gke.200 1.27.5-gke.200 The following versions are no longer available in the Regular channel: 1.24.15-gke.1700 1.25.11-gke.1700 Control planes and nodes with auto-upgrade enabled in the Regular channel will be upgraded from version 1.23 to 1.24.16-gke.500 with this release.

Google Kubernetes Engine Stable - (2023-R23) Version updates Version 1.24.14-gke.2700 is no longer available in the Stable channel.

Cloud Logging - Log buckets in the following regions can now be upgraded to use Log Analytics: me-central2 For more information, see Supported regions.

Cloud Monitoring - You can now display events, such as the crash of a GKE pod, on your dashboards. Observability for Google Kubernetes Engine: The curated set of kube state metrics is now Generally Available. Observability for Google Kubernetes Engine: The Observability tab for a GKE deployment now shows application performance metrics if the metrics are available.

Network Connectivity Center - Connecting VPC networks by using Network Connectivity Center is now generally available.

Network Intelligence Center - Connectivity Tests now supports connecting VPC networks by using Network Connectivity Center.

Policy Intelligence - You can use the Google Cloud console to analyze organization policies.

Resource Manager - You can use the Google Cloud console to analyze organization policies.

Cloud Run - Setting custom audiences on your Cloud Run services is now at general availability (GA).

Security Command Center - Support for VPC Service Controls released to Preview You can now protect Security Command Center using VPC Service Controls perimeters.

Cloud Spanner - Cloud Spanner now supports batch-oriented scans. Cloud Spanner now supports the Go programming language ORM, GORM, with GoogleSQL-dialect databases.

Cloud Speech-to-Text - Speech-to-Text has launched two models, named telephony and telephony_short.

Cloud SQL MySQL - You can now upgrade Enterprise edition instances to Enterprise Plus edition in place with minimal disruption. Cloud SQL Enterprise Plus edition now supports asia-southeast2 (Jakarta).

Cloud SQL Postgres - You can now upgrade Enterprise edition instances to Enterprise Plus edition in place with minimal disruption. Data cache is now available for Cloud SQL for PostgreSQL Enterprise Plus edition instances. Cloud SQL Enterprise Plus edition now supports asia-southeast2 (Jakarta).

Cloud Text-to-Speech - As of November 13 2023, speaker en-US-Studio-M will no longer be available.

Traffic Director - Traffic Director advanced load balancing, which is in Preview, is updated to include failover health threshold configuration.

Vertex AI - Generative AI on Vertex AI Security controls are available for additional Generative AI on Vertex AI features. Training on TPU VMs is generally available (GA).