Welcome to issue #368 October 16th, 2023

News

Simplify Dataflow development using Cloud Code plugin for IntelliJ IDE - Simplify your dataflow pipeline with integrations for your development environment and time-saving plugins.

Getting to know Systems insights, a simplified database system monitoring tool - The story of how we simplified database system monitoring for generalists while making it flexible enough for specialists.

Safeguard your VM workloads with new GCVM Protected - The new GCVE Protected offers bundled pricing for both Google Cloud VMware Engine and Google Cloud’s Backup & DR Service.

Sanitas achieves database modernization with a true DevOps operational model - By migrating from a commercial database to Cloud SQL for PostgreSQL, Sanitas benefits from the open-source developer ecosystem.

AlloyDB Omni, the downloadable edition of AlloyDB, is now generally available - AlloyDB Omni, the Downloadable Edition of AlloyDB, is now generally available and includes AlloyDB AI, new pricing, and the Kubernetes operator.

Cloud Spanner is now half the cost of Amazon DynamoDB, and with strong consistency and single-digit ms latency - Alongside lower costs, Cloud Spanner provides single-digit ms latencies and strong consistency across multiple availability zones in the same region.

Active Assist change risk recommenders: Introducing a new way to prevent misconfigurations - Active Assist change risk recommendations help prevent and detect common misconfigurations to help reduce risk, and improve operational resilience.

Easier log management for multi-tenancy through new routing features - Cloud Logging’s Log Router can now send log sinks to a Google Cloud Project, to provide greater flexibility for routing logs.

Google mitigated the largest DDoS attack to date, peaking above 398 million rps - Google Cloud stopped the largest known DDoS attack to date, which exploited HTTP/2 stream multiplexing using the new “Rapid Reset” technique.

New Vertex AI Feature Store built with BigQuery, ready for predictive and generative AI - The new Vertex AI Feature Store is in Public Preview, fully powered by BigQuery and ready for predictive and generative AI workloads at any scale.

Get a head start on 2024 with AI and more at Google Cloud Security Talks - At our Security Talks on Oct. 25, Google Cloud will bring experts together to share insights, best practices, and ways to help increase resilience against modern risks and threats.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How Deutsche Bank manages certificates in Google Cloud at scale - Deutsche Bank partnered with Google Cloud Professional Services to manage the encryption of data in transit for hundreds of the company’s applications. Here’s how.

Additional signals for enforcing Context Aware Access for Android - BeyondCorp Enterprise, Workspace CAA, and Cloud Identity can now receive critical Android device security signals for advanced and basic managed devices.

Connecting hybrid and multicloud workloads - Networking Architecture - Explore two options for setting up hybrid networking within your Google Cloud environment.

How it works: The novel HTTP/2 ‘Rapid Reset’ DDoS attack - Learn how the new DDoS attack technique Rapid Reset works, and how to mitigate it.

Google Cloud Migrations-Generate quick TCO reports for fast cloud adoption. - This article walks you through using Migration Center, a native Google Cloud Platform tool that can help you run an assessment and produce a Total Cost of Ownership report.

App Development, Serverless, Databases, DevOps

The power of AlloyDB AI in AlloyDB Omni - GenAI capabilities help you expand descriptions, generate code, answer questions or summarize text, inside of your database.

Spotify keeps engineers and code in tune with fleet management - At Spotify, the rise of fleet management lets developers deliver secure, scalable, easy-to-manage apps and services faster than they ever have before.

Zero-downtime migrations to Memorystore for Redis Cluster - Learn how to migrate your existing Redis cluster to Memorystore for Redis Cluster with RIOT with this step-by-step guide.

How to use PostgreSQL drivers with Cloud Spanner at scale with Cloud Run - Cloud Spanner’s PostgreSQL interface provides developers with access to Spanner’s consistency and availability.

How Snap reduced latency by 96 percent with KeyDB database on Google Cloud - KeyDB hosted in Google Cloud caches frequently requested data to avoid repetitive cross-cloud calls and minimize latency to services in other clouds.

How to use the MongoDB connector with Application Integration - Using MongoDB Atlas as your core operational database with Application Integration can help your organization automate business processes.

Simple observability for Cloud Run applications with GCP and OTLP - Cloud Run application that uses OpenTelemetry to collect telemetry data.

Spring Boot integration tests with Firebase Local Emulator Suite - Using Firebase emulator suite for unit tests.

Deep Dive into Google Cloud SQL Connector for Node.js - Features of Cloud SQL Connector; dive into internal workings; usage with database libraries; list of supported drivers and libraries.

Big Data, Analytics, ML&AI

Formula E chooses generative AI to inform drivers and engage fans - Formula E combined race and car telemetry data with gen AI to provide a conversational interface for drivers and fans to ask wide-ranging questions.

Generate additional revenue streams with an enterprise data platform using Google Cloud - Pythian’s Google Cloud-based enterprise data platform provided the customer with demand modeling, better product recommendations, and anonymized data.

Build AI/ML and generative AI applications in Python with BigQuery DataFrames - Learn how to perform analytics on BigQuery data using BigQuery DataFrames and its bigframes.pandas and bigframes.ml APIs.

Query fresh Google Ads data in BigQuery, via Apache Beam and Dataflow - Now, you can write Google Ads data to BigQuery using Dataflow, enabling you to make data-driven decisions on campaign strategies in real-time.

Mastering Apache Beam: Essential Transformations in Java for Google Cloud Dataflow - This article explains most common transformations in Apache Beam using Java samples.

Building an Event-Driven Serverless Data Pipeline with Google Cloud Workflows and Functions - This blog post explains how to create an event-driven system that loads data from Google Cloud Storage to BigQuery and curates it based on where it came from.

Choosing the right tool while building your Data Platform: DBT vs. Spark (By example) - Table of contents.

Finally, Data Cube Aggregation Can Work Directly in Google BigQuery - Syntax Support for Grouping by Cubes Now Available in Google BigQuery since October 2023.

Guarding BigQuery: Enhancing Data Security with VPC Service Control - This article delves into the world of VPC Service Control and how it serves as a robust shield for your data in BigQuery, striking a balance between accessibility and security.

Using Datastream to replicate PostgreSQL tables to BigQuery partitioned tables in GCP - Replicating PostgreSQL tables to BigQuery partitioned tables using Datastream.

Mage, BigQuery, and Bundled-Up Bike Trips - A brief end-to-end data project on Montreal year-round cycling.

Using Dynamic SQL in BigQuery - A brief overview of dynamic queries in BigQuery.

Five use cases for manufacturers to get started with generative AI

How to enrich product data with generative AI using Vertex AI - LLMs can support PIM by generating product descriptions, translating product descriptions, and extracting product attributes.

From localhost to shareable Vertex AI Notebook demo - This post explains how to share the Vertex AI Notebooks.

Slides, Videos, Audio

Security Podcast - #142 Cloud Security Podcast Ask Me Anything #AMA 2023.

Releases

Access Approval - Access Approval supports Access Context Manager in the GA stage.

AlloyDB - AlloyDB Omni is now generally available (GA). In AlloyDB Omni version 15.2.1 and earlier, after a failover, when you promote a standby instance, incremental backups from the newly promoted instance might conflict with the existing backup files, and the backups might fail.

Anthos clusters on VMware - Anthos clusters on VMware 1.15.5-gke.41 is now available. The following issues are fixed in 1.15.5-gke.41: Fixed the issue that server-side preflight checks fail to validate container registry access on clusters with a private network and no private registry.

Apigee X - On October 13, 2023, we released an updated version of Apigee (1-11-0-apigee-6). Bug ID Description 304681330 Security fix for apigee-ingress.This addresses the following vulnerability:CVE-2023-44487 305127632 Security bulletin published.GCP-2023-032. Description A Denial-of-Service (DoS) vulnerability was recently discovered in multiple implementations of the HTTP/2 protocol (CVE-2023-44487), including the Apigee Ingress (Anthos Service Mesh) server used by Apigee X.

Cloud Asset Inventory - New searchable fields are now available. The following resource types are now publicly available through the analyze policy APIs (AnalyzeIamPolicy and AnalyzeIamPolicyLongrunning).

Batch - Job limits have increased to 100,000 tasks per task group and 5,000 parallel tasks per job.

BeyondCorp Enterprise - The BeyondCorp Enterprise Policy Remediator is in Preview.

BigQuery - The following geography functions are now generally available (GA): ST_LINESUBSTRING: Gets a segment of a single linestring at a specific starting and ending fraction. Queries now support additional ways to work with grouping sets, which include: GROUP BY GROUPING SETS clause (new): Produce aggregated data for one or more grouping sets. Adding descriptions to the columns of a view is now generally available (GA). BigQuery is now available in the Dammam (me-central2) region. BigQuery Data Transfer Service is now available in the Dammam (me-central2) region.

Cloud Build - Users can now set an IP range size and starting IP address for private connections in Bitbucket Data Center using the peeredNetworkIpRange.

Chronicle - While creating a custom parser, you can use the preview option to view the UDM output.

Access Transparency - Access Transparency supports Access Context Manager in the GA stage.

Compute Engine - Generally available: C3 VMs support Compute Engine flexible committed use discounts (CUDs). If you want to modify a future reservation request using the Compute Engine API, the paths query parameter is deprecated. Preview: You can now use workforce identity federation with OS Login. Preview: The following metrics are now available to help you monitor your Persistent Disk and Hyperdisk volume performance: Average I/O latency (compute.googleapis.com/instance/disk/average_io_latency) Average I/O queue depth (compute.googleapis.com/instance/disk/average_io_queue_depth) To learn more about these metrics and how to view them, see Review disk metrics. Generally available: You can configure stateful IP addresses in a managed instance group. When you install the Ops Agent on a Compute Engine VM by using the Observability tab on a Compute Engine VM details page, the agent is now installed with an Ops Agent OS policy. Generally available: H3 VMs, designed for compute-intensive high performance computing (HPC) workloads, are now generally available.

Container Registry - Starting October 10, 2023, mirror.gcr.io is transitioning to being hosted on Artifact Registry.

Dataproc Serverless - Announcing the General Availability (GA) release of Dataproc Serverless for Spark Interactive sessions.

Dataproc - New Dataproc Serverless for Spark runtime versions: 1.1.35 2.0.43 2.1.22.

Deep Learning Containers - M112 release Miscellaneous bug fixes and improvements.

Deep Learning VM - M112 release CUDA 12.1 VM images are available with the following image names: common-cu121-debian-11-py310 common-cu121-ubuntu-2004-py310 Miscellaneous bug fixes and improvements.

Dialogflow - Dialogflow CX generative feedback now supports more languages. Dialogflow CX launched generative playbooks with restricted access. Dialogflow CX spelling correction now supports all regions, but is limited to five languages.

Cloud Functions - Cloud Functions (2nd gen) now supports Shared VPC ingress at the General Availability release level.

Networking Interconnect - MACsec for Cloud Interconnect is now generally available.

Google Kubernetes Engine - (2023-R20) Version updates GKE cluster versions have been updated. Starting in GKE 1.28.1-gke.1066000, two new TPU usage metrics are available: TensorCore utilization and Memory Bandwidth utilization. Containers running in nodes in GKE version 1.28.1-gke.201 or later don't need to have privileged mode enabled to access TPUs. A Denial-of-Service (DoS) vulnerability was recently discovered in multiple implementations of the HTTP/2 protocol (CVE-2023-44487), including the golang HTTP server used by Kubernetes. If you are using a third generation machine series (for example, C3), GKE configures Local SSD volumes as the local ephemeral storage by default.

GKE - (2023-R20) Version updates The following control plane and node versions are now available: 1.24.17-gke.2113000 1.25.14-gke.1421000 1.26.9-gke.1437000 1.27.6-gke.1445000 The following control plane versions are no longer available: 1.24.17-gke.1963000, 1.25.14-gke.1256000, 1.26.9-gke.1256000.

Google Kubernetes Engine Rapid - (2023-R20) Version updates The following versions are now available in the Rapid channel: 1.24.17-gke.2113000 1.25.14-gke.1421000 1.26.9-gke.1437000 1.27.6-gke.1445000 The following versions are no longer available in the Rapid channel: 1.24.17-gke.1963000, 1.25.14-gke.1256000, 1.26.9-gke.1256000.

Cloud Logging - You can now configure the format of the timestamp in your query results in the Logs Explorer. When you install the Ops Agent on a Compute Engine VM by using the Cloud Monitoring VM Instances dashboard or the Observability tab on a Compute Engine VM details page, the agent is now installed with an Ops Agent OS policy.

Migrate for Compute Engine - 5.0. Preview: Migrate to Virtual Machines now supports migrating VMs to the C3, H3, and M3 machine types. 5.0. Generally Available: Migrate to Virtual Machines from an Azure source lets you migrate VM instances running on Azure to Google Cloud Compute Engine.

Cloud Monitoring - When you install the Ops Agent on a Compute Engine VM by using the Cloud Monitoring VM Instances dashboard or the Observability tab on a Compute Engine VM details page, the agent is now installed with an Ops Agent OS policy.

Cloud Interconnect - MACsec for Cloud Interconnect is now generally available.

Cloud Run - Shared VPC ingress is now at general availability (GA).

Security Command Center - Cloud IDS threat detections available in Security Command Center Threats that are detected by Cloud IDS, a Google Cloud intrusion detection service, are now included in the findings that are issued by the Event Threat Detection service of Security Command Center.

Service Mesh - 1.18.x. 1.18.4-asm.0 is now available for in-cluster Anthos Service Mesh. 1.17.x. 1.17.7-asm.0 is now available for in-cluster Anthos Service Mesh. 1.16.x. 1.16.7-asm.10 is now available for in-cluster Anthos Service Mesh.

SAP Solutions - Cloud Storage Backint agent for SAP HANA version 1.0.30 Version 1.0.30 of the Cloud Storage Backint agent for SAP HANA is available.

Cloud Spanner - Cloud Spanner has made improvements that provide higher throughput for instances located in select Spanner regional and multi-region instance configurations. Cloud Spanner batch write is now available in Preview. Cloud Spanner Vertex AI integration now supports Vertex AI Generative AI text embeddings and the text-bison model.

Cloud SQL SQL Server - The cross db ownership chaining flag is deprecated for all SQL Server versions.

Cloud Storage Transfer - You can now transfer data from Amazon S3 via your CloudFront domain.

Cloud Storage - The Node.js and Python client libraries now have parallelized upload and download options, improving their performance.

Vertex AI - Colab Enterprise is now generally available (GA). M112 release The M112 release of Vertex AI Workbench user-managed notebooks includes the following: Miscellaneous bug fixes and improvements.