Welcome to issue #424 November 11th, 2024

News

Mandatory MFA is coming to Google Cloud. Here’s what you need to know - Google Cloud is implementing mandatory multi-factor authentication (MFA) to enhance security for all users. The phased rollout will begin in November 2024 and will be completed by the end of 2025. Users are encouraged to enable MFA as soon as possible to protect their accounts from unauthorized access.

Now run your custom code at the edge with the Application Load Balancers - Google Cloud's Service Extensions plugins for Application Load Balancers allow users to run custom code directly in the request/response path in a fully managed Google environment. These plugins support various use cases such as header addition, manipulation, security policies, custom logging, exception handling, and HTML rewriting.

Google is a Leader in The Forrester Wave™: Translytical Data Platforms, Q4 2024 - Google Cloud's AlloyDB has been recognized as a Leader in The Forrester Wave™: Translytical Data Platforms, Q4 2024 report. AlloyDB stands out with its differentiated architecture that combines the performance of a traditional relational database with the scalability and flexibility of cloud-first technology.

How to simplify building RAG pipelines in BigQuery with Document AI Layout Parser - BigQuery now offers simplified document preprocessing for retrieval-augmented generation (RAG) pipelines through its integration with Document AI. The ML.PROCESS_DOCUMENT function, now generally available, can access new processors, including Document AI's Layout Parser processor, allowing you to parse and chunk PDF documents with SQL syntax. The blog provides a step-by-step example of building a RAG pipeline in BigQuery using Document AI's Layout Parser to analyze a complex financial document.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Flare-On 11 Challenge Solutions - The eleventh Flare-On challenge, a global cybersecurity competition, concluded with over 5,300 participants and only 275 completing all 10 stages. All challenge binaries and solutions are now available on the Flare-On website.

Elevate network security and migrate your legacy VPC firewall rules to Cloud NGFW - Google Cloud has enhanced its network security offerings with the launch of Cloud Next-Generation Firewall (NGFW). Customers are encouraged to transition from legacy VPC firewall rules to Cloud NGFW's powerful and flexible firewall policies.

My FinOps Collection - A collection of articles about FinOps.

Backend Access Control with Envoy Proxy and Google Cloud Service Accounts - Envoy Proxy can be configured to add an authorization header to all incoming requests, enhancing security and streamlining application architecture. The GCP Authentication Filter simplifies authentication for GCP applications by fetching service account credentials and attaching them to requests.

From Autopilot to Standard GKE: The Key to 15x Cheaper Istio - Istio proxy costs for 10-nodes GKE cluster reduced from $3065 to $185 per month.

Proactive Cloud Security: Learning from Google Cloud Armor Vulnerability Discovery - Google Cloud Armor, a security service within Google Cloud Platform, had a critical vulnerability that allowed bypassing of its Web Application Firewall (WAF) rules. The flaw was specific to HTTP/2 requests and involved handling whitespace in query parameters. The vulnerability was reported to Google and a fix was released.

(In)tuned to Takeovers: Abusing Intune Permissions for Lateral Movement and Privilege Escalation in Entra ID Native Environments - Mandiant Red Team discovered a novel way adversaries can move laterally and elevate privileges within Microsoft Entra ID when organizations use Intune-managed Privileged Access Workstations (PAWs) by abusing Intune permissions (DeviceManagementConfiguration.ReadWrite.All) granted to Entra ID service principals.

Counting the cost of carbon in the cloud - Lack of transparent data makes cloud service providers’ green claims hard to measure; we should demand more, writes MPB’s Neil Haffenden.

App Development, Serverless, Databases, DevOps

Flipping out: Modernizing a classic pinball machine with cloud connectivity - Backlogged Pinball is a custom pinball game that connects to the cloud for a variety of services, including keeping track of data about current and completed games, updating leaderboards, and more.

GitHub Action — Build, Push Docker Image to Artifact Registry - This blog post introduces a public GitHub action that builds and pushes a Docker Image to Google Cloud’s Artifact Registry.

All the ways to scrape Prometheus metrics in Google Cloud - Look into methods of scraping Prometheus metrics emitted by your application on Google Cloud. Reduce costs and get rid of toil.

Etsy’s Service Platform on Cloud Run cuts deployment time from days to under an hour - Etsy, an online marketplace, migrated its infrastructure to Google Cloud and created a customized service platform called ESP running on Google Cloud Run. By leveraging Cloud Run, Etsy was able to focus on core platform functionality while Cloud Run handled the complexities of running containerized services. ESP also integrates with Etsy's existing tools and services, providing a consistent developer and operational experience.

Deploying OAuth2-Proxy as a Cloud Run sidecar container - In this article, we will show you how to secure access to a Cloud Run web application with OAuth2 without modifying the application code. We will deploy the OAuth2-Proxy container as a sidecar in front of the Cloud Run microservice, decoupling the OAuth authorization flow from the web application. This approach allows for a modular and reusable architecture. We will guide you through the steps of registering your app with the Google OAuth2 provider, preparing the OAuth2-Proxy configuration, deploying the sidecar OAuth2 container, and testing the OAuth flow.

Big Data, Analytics, ML&AI

Deutsche Telekom designs the telco of tomorrow with BigQuery - Deutsche Telekom is modernizing its data platform with Google Cloud to improve customer experience, security, and efficiency. They built a data lakehouse architecture using BigQuery, Cloud Storage, and BigLake, and implemented fine-grained data governance controls using Dataplex. With the help of Gemini Pro 1.5 models, they were able to migrate their codebase and build new AI capabilities.

Getting started with NL2SQL (natural language to SQL) with Gemini and BigQuery - Natural Language to SQL (NL2SQL) translates human language questions into structured SQL queries, enabling non-technical users to explore data and gain insights without specialized SQL knowledge. Google Cloud offers solutions to address these challenges, including BigQuery vector search for embedding and retrieval, contribution analysis for multi-step reasoning, and Gemini for ambiguity checks and user feedback.

Integrating Open Source Unity Catalog with GCP workloads - Open source Unity Catalog, a data and AI governance solution, can be integrated with GCP workloads like Spark on Dataproc. This blog provides a step-by-step guide to set up Unity Catalog in GCP, including hosting the Unity Catalog server and webserver UI, and interacting with Unity Catalog using Spark workloads.

A practical guide to synthetic data generation with Gretel and BigQuery DataFrames - This guide demonstrates how to generate synthetic data using Gretel AI and BigQuery DataFrames, ensuring data privacy and compliance. It includes steps for de-identifying data with Gretel Transform v2, fine-tuning a model with Navigator Fine Tuning, evaluating data quality and privacy, and writing synthetic data back to a BigQuery table.

How to deploy and serve multi-host gen AI large open models over GKE - Google Cloud Platform now supports deploying and serving open models, such as the Llama 3.1 405B FP16 LLM, over Google Kubernetes Engine (GKE).

Can AI eliminate manual processing for insurance claims? Loadsure built a solution to find out - Loadsure, an InsurTech firm, leveraged Google Cloud's generative AI and Document AI to automate claim management, reducing processing time from 30-60 minutes to near real-time. This automation improved accuracy, reduced the risk of human error, and enhanced customer satisfaction. Loadsure is now exploring expanding these technologies to other insurance processes like underwriting and pricing.

Elia Group helps businesses and consumers reduce carbon emissions with Vertex AI - Elia Group, a leading European transmission system operator, is using Google Cloud's Vertex AI to measure and forecast the carbon intensity of electricity across its grid. This information helps businesses make informed decisions about their energy consumption and reduce their carbon emissions. Elia Group built a comprehensive MLOps platform using Vertex AI, enabling them to control and manage multiple forecasting models simultaneously.

Generative AI with enterprise controls for business users in 24 Hours - Aible, a leader in generating business impact from AI, partners with Google Cloud to enable customers to build, train, and deploy generative AI models on their own data securely. Aible leverages Vertex AI, Google Cloud's AI platform, to ensure customers have control over their data and access when developing, training, or fine-tuning AI models. Aible's ChatAible system simplifies the process for business users, allowing them to "just start chatting" with their enterprise data without extensive training.

ResuMate Pro: Revolutionizing Resume Review with Gemini-Powered Multimodal Analysis - ResuMate Pro revolutionizes resume review with Gemini-powered multimodal analysis, providing personalized feedback in seconds. Leveraging Google Vertex AI and BigQuery, it offers high-speed processing, scalability, and data-driven improvements.

Serving Models with Ray Serve - Ray Serve is a scalable model serving library built on Ray. It allows you to deploy and serve machine learning models at scale, handling incoming requests with low latency while dynamically scaling model replicas.

Chrome Built-in AI Experiments: Code Explanation Chrome Extension - This blog post explores the use of Chrome Built-in AI to build a Chrome extension that explains code selected on a web page. The extension utilizes the experimental Prompt API to generate explanations in bullet points.

Talk to Your Cronos Data: AI Agent based User Experiences for Blockchain Insights - Talk to Your Cronos Data: AI Agent based User Experiences for Blockchain Insights explores the use of AI agents to simplify complex blockchain data and enhance its accessibility. By leveraging Google Cloud's Vertex AI and Agent Builder, users can interact with on-chain data using natural language queries, making blockchain insights readily available to a broader audience.

RAG and Long-Context Windows: Why You need Both - Combining RAG and Long-Context Windows achieves performance at a lower cost.

Automating Infrastructure as Code with Vertex AI - The article showcases how they leveraged Google's Vertex AI with Gemini 1.5 to implement multimodal input for automating the creation of infrastructure as code.

Slides, Videos, Audio

Security Podcast - #197 SIEM (Decoupled or Not), and Security Data Lakes: A Google SecOps Perspective.

Releases

AlloyDB - AlloyDB Omni version 15.7.0 is generally available (GA). In AlloyDB Omni versions 15.5.5 and earlier, parameterized view features were available in the alloydb_ai_nl extension. Since the google_ml_integration.enable_model_support flag is enabled by default, if you are using the google_ml_integration extension version 1.3, your ability to query Vertex AI models using the embedding() function might be impacted.

Google Distributed Cloud Bare Metal - 1.28. Release 1.28.1200-gke.83 Google Distributed Cloud for bare metal 1.28.1200-gke.83 is now available for download. Fixes: Fixed an issue where the registry mirror reachability check fails for a single unreachable registry mirror. The following container image security vulnerabilities have been fixed in 1.28.1200-gke.83: High-severity container vulnerabilities: CVE-2021-33194 CVE-2022-27664 CVE-2022-41723 Medium-severity container vulnerabilities: CVE-2021-31525 CVE-2022-41717 CVE-2023-23931 Low-severity container vulnerabilities: CVE-2023-49083. Known issues: For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

GKE attached clusters - You can now launch clusters with the following Kubernetes versions: 1.30.0-gke.3, 1.29.0-gke.6, 1.28.0-gke.9.

GKE on AWS - You can now launch clusters with the following Kubernetes versions: 1.30.5-gke.200, 1.29.8-gke.1800, 1.28.14-gke.200.

Anthos clusters on Azure - You can now launch clusters with the following Kubernetes versions.

Anthos clusters on VMware - Google Distributed Cloud (software only) for VMware 1.28.1200-gke.83 is now available for download. The following issue is fixed in 1.28.1200-gke.83: Fixed the issue that additional manual steps are needed after disabling always-on secrets encryption with gkectl update cluster.

GDCV for VMware - Google Distributed Cloud (software only) for VMware 1.28.1200-gke.83 is now available for download. The following issue is fixed in 1.28.1200-gke.83: Fixed the issue that additional manual steps are needed after disabling always-on secrets encryption with gkectl update cluster.

Anthos GKE on AWS - You can now launch clusters with the following Kubernetes versions.

BigQuery - BigQuery now offers the following Gemini-enhanced SQL translation features: In interactive translation mode, you can use Gemini-enhanced SQL translations to customize translated GoogleSQL queries. Dataplex automatic discovery lets you scan data in Cloud Storage buckets to extract and catalog metadata. The BigQuery Data Transfer Service data source change log provides details about upcoming changes to data source schemas and schema mappings.

Chronicle Security Operations - Check the release note page to see which new parser documentation is now available.

Cloud Composer - A new Cloud Composer release has started on November 06, 2024. (Cloud Composer 3) Fixed an issue that affected the speed of PyPI package installation. (Airflow 2.9.3 and 2.7.3) The docutils package was removed from preinstalled packages. New Airflow builds are available in Cloud Composer 3: composer-3-airflow-2.9.3-build.6 (default) composer-3-airflow-2.7.3-build.22. New images are available in Cloud Composer 2: composer-2.9.10-airflow-2.9.3 (default) composer-2.9.10-airflow-2.7.3. Cloud Composer version 2.5.1 has reached its end of support period.

Compute Engine - Generally available: An updated version of the gVNIC driver for Windows offers improved network performance and support for Jumbo frames. Preview: You can create GPU VMs all at once in a regional managed instance group (MIG) by using resize requests.

Contact Center AI Platform - Version 3.29 is released All release notes published on this date are part of version 3.29. Call Waiting Call waiting lets agents handle multiple inbound calls. Skip CRM account and record creation (Zendesk and ServiceNow) You can now skip CRM account and record creation for Zendesk and ServiceNow. Agent status localization You can translate the default, system, and custom agent statuses for the languages supported by Google Cloud Contact Center as a Service (CCaaS). Generative session summarization using Agent Assist Agent Assist now supports generative session summarization for chat and voice sessions. Generative knowledge assist using Agent Assist Agents can now view knowledge articles while on a call or chat. Queue transfer restrictions You can control which queues or teams that agents can transfer sessions to. Support for direct SIP REFER in virtual agent call transfers Virtual assistant call transfers now support the direct SIP REFER method. Alvaria WFM for chat Customers with Alvaria integrations can now receive chat session data. Clear the voicemails in a queue You can now clear the voicemails in any queue from the Call settings page. Queue status API We added two new API endpoints that let you check whether a queue is in After Hours (AH) or Overcapacity (OC) status. SDK parameters in the custom CRM lookup flow You can now use SDK data parameters in the CRM lookup flow. Use the admin user for CRM API calls with Salesforce Using Salesforce, you can now use the admin user for all CRM API calls for record creation and updating, while still allowing agents to retain ownership of CRM-specific actions. New Agent_Assist_Started event A new Agent_Assist_Added event is now available. Chat transcripts download You can now download a chat transcript using the web SDK. Display email session ID in the email adapter and email subject You can now display the session ID in the email adapter and in the subject line of an email thread. Clickable authentication icon The authentication icon in the agent adapter can now be clicked by the agent to mark the customer as either authenticated or unauthenticated. Fixed an issue where the session summary wouldn't automatically scale with the height of the chat adapter. Fixed an issue where wrap-up and disposition settings were not following destination queue settings when calls and chats were transferred. Fixed an issue where transferring calls to a parent queue sometimes caused calls to drop. Fixed an issue that sometimes prevented agents from going into Available status after wrapping up a call while still in a chat session. Fixed an issue where the wait time for transferred sessions sometimes displayed incorrectly on the "Queued Calls" and "Queued Chats" dashboards. Improved the user interface for the email transcript capability.

Database Migration Service - Database Migration Service now supports MySQL minor version 8.0.39 for homogeneous MySQL migrations.

Dataplex - Dataplex automatic discovery is available in public preview. Project-based semantic search offered by Dataplex Search is available in Preview.

Datastore - You can now use the managed bulk delete service to delete entities in bulk.

Cloud Data Loss Prevention - You can configure discovery to save sample findings to a BigQuery table.

Eventarc - Eventarc Standard is available in the northamerica-south1 (Mexico, North America) region.

Cloud Firestore - You can now use the Firestore managed bulk delete service to delete documents in bulk.

Gemini - You can now purchase Gemini Code Assist Enterprise edition on the Admin for Gemini page within the Google Cloud console.

Google Kubernetes Engine - The machine family of N1 custom machine types (like custom-1-1024) is now accurately labeled as "N1" for all node versions later than 1.31.2-gke.1115000. GKE clusters running version 1.28 or later now support automatic application monitoring in public preview. The GKE Volume Populator is generally available on GKE clusters running version 1.31.1-gke.1729000 or later. (2024-R43) Version updates GKE cluster versions have been updated. Generally available: In GKE version 1.26 and later, Hyperdisk Balanced volumes can be created in Confidential mode for custom boot disks and persistent volumes and attached to Confidential GKE Nodes. Cloud TPU v6e machine types are now in public preview for GKE clusters running version 1.30.4-gke.1167000 or later.

GKE new features - GKE clusters running version 1.28 or later now support automatic application monitoring in public preview. The GKE Volume Populator is generally available on GKE clusters running version 1.31.1-gke.1729000 or later. Generally available: In GKE version 1.26 and later, Hyperdisk Balanced volumes can be created in Confidential mode for custom boot disks and persistent volumes and attached to Confidential GKE Nodes. Cloud TPU v6e machine types are now in public preview for GKE clusters running version 1.30.4-gke.1167000 or later.

Live Stream API - The Live Stream API is now available in asia-south1 and europe-north1.

Load Balancing - Percentage-based request mirroring is now supported for the global and regional external Application Load Balancers (classic is not supported).

Cloud Logging - Audit Logging now populates the status.details field in the audit log with the google.rpc.ErrorInfo and google.rpc.Help proto payload types in cases where an API returns an error status and that status includes one of those types in the details field.

Looker - Looker (Google Cloud core) and Looker (original) changes. Looker 24.20 includes the following changes, features, and fixes: Expected Looker (original) deployment start: Monday, November 11, 2024 Expected Looker (original) final deployment and download available: Thursday, November 21, 2024 Expected Looker (Google Cloud core) deployment start: Thursday, November 7, 2024 Expected Looker (Google Cloud core) final deployment: Thursday, November 14, 2024. In the Looker application API, for methods that include a query_id field, or, in the case of Query APIs, an id field, the query_id and id fields no longer accept a numeric value and now require a query slug value. Users no longer need the download_without_limit permission to select the All Results option when they schedule Looks and dashboards. The Chart Config Editor now supports creating a Dependency Wheel visualization. The Chart Config Editor now supports creating an Item visualization. The New Project page in Looker has been replaced with the Create a Model page. An issue has been fixed where renaming a project using a bare repository could prevent deploying to production for that project. An issue has been fixed where editing a model set could take a long time to load. An issue has been fixed where the Actions page could fail to reflect recently saved settings. An issue has been fixed where Sankey charts could ignore series values if they matched other series values. An issue has been fixed where conditional formatting could fail to apply to total rows if the value was zero. An issue has been fixed where Looker could generate datagroup names with dashes even though dashes aren't allowed in datagroup names. An issue has been fixed where certain System Activity queries could time out. The PDF and PNG rendering software has been upgraded to the latest stable version. An issue has been fixed where visualizations that were created with the Chart Config Editor could fail to be displayed in an embedded context. An issue has been fixed where the LookML Validator would not display an error message if the convert_tz parameter was used in an invalid context. An issue has been fixed where selecting the word cloud visualization could cause Looker to display a blank page. Tooltips have been added for truncated progress values in single value visualizations. An issue has been fixed where progress values in single value visualizations were unnecessarily truncated. An issue has been fixed where modifying dashboard filters after deleting a tile could cause Looker to display an error. An issue has been fixed where progress bars in single value visualizations could disappear when the visualization was resized. An issue has been fixed where relative date filters could misinterpret numbers with more than three digits (such as "in the last 1000 minutes") as dates. An issue has been fixed where killing queries on BigQuery Standard SQL could be unnecessarily expensive. An issue has been fixed where special characters (such as < and >) in pivoted dimension values could cause Looker to incorrectly truncate legend labels. An issue has been fixed where downloading a dashboard tile with an invalid hex color code as an Excel spreadsheet could cause the download to fail. An issue has been fixed where location type fields could not be used in custom filter expressions. An issue has been fixed where invalid "set" or "when" LookML fields could cause the LookML Validator to fail with a 500 error. An issue has been fixed where a locale value of fr would fall back to fr-CA instead of fr-FR, which was causing text to be translated incorrectly. An issue has been fixed where the LookML IDE did not persist line wrap settings. Looker (original) only changes. Upon upgrade to Looker 24.20, support access will be disabled on Looker (original) instances. Looker (original) deployments can now use the Redshift 2.1.0.30 driver. A new Labs feature is available, New Database Connection Setup. Google Cloud Technical Support access has updated duration settings of 0 to 48 hours. A new Labs feature is available, Tiered Support Access, which defaults to enabled. A new legacy feature is available, Use Legacy Project Creation Page. A new Labs feature is available, Complex Filters UI Configuration for Explores. Looker (Google Cloud core) only changes. Google Cloud Technical Support access is now available for Looker (Google Cloud core) instances. An issue has been fixed where logging in to an instance using IP Allowlist could take a long time.

Memorystore for Redis Cluster - Added support for multiple VPC networks (Preview).

Cloud PubSub - General availability: You can now create Cloud Storage import topics in Pub/Sub that lets you ingest data from Cloud Storage into Pub/Sub. General availability: You can now enable Google Cloud platform logs to help you troubleshoot issues when you are using Cloud Storage import topics to ingest data.

Cloud Run - You can now specify mount options when you configure Cloud Storage volume mounts for both Cloud Run services and jobs.

Security Command Center - The v2 Security Command Center API is generally available (GA).

Sensitive Data Protection - You can configure discovery to save sample findings to a BigQuery table.

Service Mesh - Managed Cloud Service Mesh. The following images are now rolling out for managed Cloud Service Mesh: 1.19.10-asm.21 is rolling out to the rapid release channel. 1.23.x. 1.23.3-asm.1 is now available for in-cluster Cloud Service Mesh. 1.22.x. 1.22.6-asm.1 is now available for in-cluster Cloud Service Mesh. 1.21.x. 1.21.5-asm.10 is now available for in-cluster Cloud Service Mesh. 1.20.x. 1.20.8-asm.9 is now available for in-cluster Cloud Service Mesh. 1.20.x & 1.21.x & 1.22.x. This release fixes a bug in the following versions where the default user for distroless proxy was changed to root; As a result of this fix, the default user is now back to non-root 1.20.8-asm.6 1.20.8-asm.7 1.21.5-asm.5 1.21.5-asm.7 1.22.3-asm.1 1.22.4-asm.0 1.22.5-asm.1 This change may affect some gateway deployments which rely on the root user to expose a privileged port for ingress or egress. Managed Cloud Service Mesh. Patches fixing a bug where the default user for distroless proxy was changed to root will be rolling out to all release channels.

Cloud Spanner - Spanner now supports client-side metrics for Java and Go applications.

Cloud SQL - You can now view the size of a backup for a Cloud SQL instance.

Cloud Storage - You can now restore soft-deleted buckets.

Cloud Translation - The translation LLM now supports Polish, Turkish, Indonesian, Dutch, Vietnamese, Thai and Czech.