Welcome to issue #411 August 12th, 2024

News

Real-time in no time: Introducing BigQuery continuous queries for up-to-the-minute insights - BigQuery continuous queries, now available in preview, enables real-time data analysis and event-driven processing using SQL. It simplifies real-time pipelines, unlocks AI use cases, streamlines reverse ETL, and provides scalability and performance. With BigQuery continuous queries, businesses can gain real-time insights, make informed decisions, and deliver exceptional customer experiences.

Introducing Approximate Nearest Neighbor (ANN) search to Spanner - Spanner now supports Approximate Nearest Neighbor (ANN) search, enabling fast and scalable vector search for large datasets. This capability is particularly useful for unstructured data like images, text, or audio, where traditional search methods may not be as effective. Spanner's ANN search leverages Google Research's ScaNN algorithm and offers optimizations such as tree-like structure clustering, quantized embeddings, and optimized distance calculation.

Announcing LangChain on Vertex AI for AlloyDB and Cloud SQL for PostgreSQL - AlloyDB and Cloud SQL for PostgreSQL now support LangChain on Vertex AI, enabling developers to build, deploy, query, and manage AI agents and reasoning frameworks in a secure, scalable, and reliable way. This integration unlocks new database use cases such as querying databases, knowledge retrieval, chat bots, and tool use. It also provides fast knowledge retrieval, secure authentication and authorization, chat history context, and fast prototyping.

Announcing Apache Airflow operators for Google generative AI - Apache Airflow now has operators to interact with Vertex AI's generative models. These operators enable the integration of Vertex AI's generative models into data pipelines orchestrated by Apache Airflow and Cloud Composer.

Create a powerful Kubernetes security duo with Custom Org Policy and Policy Controller - Google Cloud offers custom Org Policy and Policy Controller to help customers implement defense in depth strategies for Google Kubernetes Engine (GKE).

Join Google Cloud Security Summit 2024 for a masterclass in modernizing security - Join Google Cloud Security Summit 2024 for a masterclass in modernizing security. Discover the latest innovations and strategies to protect your business, customers, and cloud transformation from emerging threats. Learn about secure cloud platforms, integrating dynamic intelligence, and security operations.

---

Technology and expertise from a Google Cloud Premier Partner

As a trusted Google Cloud partner of over a decade, DoiT delivers cloud optimization technology and unrivaled cloud expertise to help customers take full advantage of the public cloud. From migration and infrastructure to usage insights and analysis, DoiT has the experience and tools to help companies optimize Google Cloud use to accelerate return. Learn More

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Level up your Kubernetes security with the CIS GKE Benchmarks - Google Cloud has partnered with the Center for Internet Security (CIS) to release updated CIS Benchmarks for GKE and GKE Autopilot. The benchmarks include over 80 recommended controls, addressing the latest security challenges and best practices, and are aligned with the latest CIS Kubernetes Benchmark version.

Check the last time a Service Account was used on GCP - Use a GCP tool to analyze account activities.

Cloud Adoption: Enabling Your Consumers and Tenants, and the Project Factory - How to enable your tenants, with folder hierarchy, project factory, sandboxes, and support documentation.

Exposing gRPC application on GKE through Global External Gateway and Google Cloud Load Balancing - This guide demonstrates how to expose a gRPC application on Google Kubernetes Engine (GKE) using the Gateway API and Google Cloud Load Balancing. It covers creating a certificate map, reserving a static external IP address, setting up a GKE cluster, deploying the gRPC application, creating Kubernetes services, gateways, HTTP routes, and health check policies.

App Development, Serverless, Databases, DevOps

Apigee and the Interoperability Model (ModI) for the Italian Public Administration - Apigee API Management enables the Italian Public Administration and its customers, suppliers, and system integrators to achieve full compliance with the new Interoperability Model (ModI) written and designed by Agency for Digital Italy (AgID).

Is Your Google Cloud Storage a Sitting Duck? Lock it Down with VPC Service Controls - This article explains how to protect your Google Cloud Storage buckets from unauthorized access with VPC Service Controls, Google Cloud's API-level firewall.

The Bigtable Advantage: How Flipkart gets the most out of every byte - Flipkart's transition to Bigtable has brought unprecedented efficiency and flexibility, ensuring a seamless shopping experience for their millions of customers. Scaling up or down is simplified with a single click in the user interface, enhancing the resource management experience — the platform scaled up 4X during its Big Billion Day event with no capacity concerns or performance impact.

Continuous Delivery on Google Cloud with Gitlab CI/CD and Cloud Deploy - This article explains how to create a Continuous Delivery pipeline to automate software delivery from code commit to production release on Cloud Run using Gitlab CI/CD and Cloud Deploy, leveraging the recently released Gitlab Google Cloud integration.

IoT Data Logging and Device Control with GCP: Setting Up Google Cloud Functions and BigQuery for Teracom TCG140–4 - Tutorial with step-by-step setup of Teracom TCG 140–4 with Google Cloud Function and BigQuery database for remote data logging.

Cloud Notifications: Automating Email Alerts for Deployments with Google Cloud Build and msmtp in GCP - Cloud messaging involves sending notifications, alerts, or emails from cloud-based applications or services. For our cloud build notifications, we use msmtp, a lightweight SMTP client that allows sending emails using an external SMTP server. To securely store your SMTP password, use Google Secret Manager. Once configured, the pipeline will send an email notification upon successful build completion.

Controlling Network Egress for Cloud Workstations with Secure Web Proxy - Learn how to control egress traffic for Cloud Workstations with a cloud-first Secure Web Proxy for a more delightful developer on-boarding.

How-To: Create a Google Cloud Organization Using Cloud Identity for Free (Step-by-Step Guide) - Learn how to create a Google Cloud Organization using Cloud Identity for free.

Big Data, Analytics, ML&AI

Validate Your BigQuery SQL Table Metadata Using Just 1 Word - Leverage BigQuery SQL table metadata to deduplicate, partition and delete data — all using only one word.

Implementing SCD Type 2 in BigQuery - Implementing SCD Type 2 in BigQuery using MERGE statement.

Building an Advanced Date Calendar in BigQuery: A Step-by-Step Guide - This article provides a step-by-step guide to creating an advanced date calendar in BigQuery. It explains the benefits of having a date calendar table and the various columns included in it, such as day of the week, month name, quarter, and more. The guide includes instructions for manually creating the date calendar table using SQL scripts and also introduces the JustFunctions open-source library, which offers convenient functions to simplify the process.

Google launched JSON Keys and Paths for BigQuery - News Functions when working with JSON in BigQuery SQL.

Version control in BigQuery: a quick and dirty way - Without migrations, Liquibase or DBT.

Understanding Google Cloud Dataflow: Common Mistakes with Flex-Templates - What I got Wrong About Dataflow Flex Templates.

Unlocking the Power of Cloud Composer: 5 Hidden Gems to Supercharge Your Workflow Orchestration - Discover five tips and tricks for Cloud Composer to simplify DAG development, share data between tasks, manage secrets securely, optimize DAG performance with caching, and monitor DAGs with built-in alerts.

Discovering and Classifying Your Data with GCP’s Sensitive Data Protection (DLP and Terraform!) - In this article, we explore how Google Cloud Platform (GCP) can help businesses understand and control their data, enabling them to deliver personalized customer experiences while ensuring data privacy and regulatory compliance. We focus on Cloud Data Loss Prevention (DLP), a key component of GCP's Sensitive Data Protection suite, and demonstrate how to automate DLP scans using Terraform, making data protection an integral part of infrastructure management. By automating sensitive data protection, businesses can save time and resources, improve accuracy, reduce risk, and enhance their overall security posture.

Chat on Confluence/Jira/Sharepoint … Data with access control. - This article explores how to integrate Agent Builder Search with Confluence to create a personalized and contextually relevant search experience for users.

Deep dive into function calling in Gemini - Function calling in Gemini allows you to augment language models with real-time data from external APIs. You can define custom functions and the model can delegate tasks to these functions. This enables the model to complete its response to the user's prompt with up-to-date information.

Assigning a Static IP to Vertex AI Pipelines or Workbench Managed Notebooks - This guide provides a workaround for assigning a static IP to Vertex AI Pipelines or Workbench Managed Notebooks. It involves creating internal and external networks, setting up a Cloud Router with Cloud NAT enabled, configuring a NAT VM for NAT functions, and creating routes to forward traffic to the NAT instance. By following these steps, you can ensure that your Pipelines traffic originates from specific IP addresses.

Part 1: Let’s Build an Operational MLOps Framework from Scratch - A step-by-step tutorial to transform a simple use case into an operational and reusable MLOps framework.

Slides, Videos, Audio

Kubernetes Podcast - #232 OpenTofu, with Ohad Maislish.

Security Podcast - #184 One Week SIEM Migration: Fact or Fiction?

GCP Life Podcast - #70 In this episode we discuss; Architect Certification, Optus Award, Crowdstrike, GDG Sydney, Q2 Results,Wiz Deal,Open Tofu, VMWare Costs, XERO Outage, Office 365 Outage, Google Project Hijacks, Malicious Python Packages, Apple AI, Open AI Search, VS Code AI Assitant, Google HR using AI.

Releases

AlloyDB - Enhanced Query Insights and active queries are now available in Preview for read pool instances. AlloyDB Omni version 15.5.5 is now generally available (GA).

Google Distributed Cloud Bare Metal - 1.29. Release 1.29.400-gke.86 Google Distributed Cloud for bare metal 1.29.400-gke.86 is now available for download. GA: Added support in version 1.29.400-gke.86 and higher for Red Hat Enterprise Linux (RHEL) version 9.2. Fixes: The following container image security vulnerabilities have been fixed in 1.29.400-gke.86: Fixed the following vulnerabilities: High-severity container vulnerabilities: CVE-2019-16884 CVE-2021-30465 CVE-2021-33194 CVE-2021-38561 CVE-2021-43565 CVE-2021-43816 CVE-2022-23648 CVE-2022-27191 CVE-2022-27664 CVE-2022-32149 CVE-2022-41723 CVE-2022-43945 CVE-2024-36971 CVE-2024-38583 CVE-2024-39480 CVE-2024-39495 CVE-2024-40902 Medium-severity container vulnerabilities: CVE-2019-19921 CVE-2021-31525 CVE-2021-32760 CVE-2021-41103 CVE-2021-43784 CVE-2022-23471 CVE-2022-29162 CVE-2022-29526 CVE-2022-31030 CVE-2022-40735 CVE-2022-41717 CVE-2023-25153 CVE-2023-25173 CVE-2023-28642 CVE-2023-48795 CVE-2024-27019 CVE-2024-36288 CVE-2024-38662 CVE-2024-38780 CVE-2024-39292 CVE-2024-39475 CVE-2024-39476 CVE-2024-39482 CVE-2024-39484 CVE-2024-39489 CVE-2024-39493 CVE-2024-42070 CVE-2024-42076 CVE-2024-42077 CVE-2024-42082 GHSA-7ww5-4wqc-m92c Low-severity container vulnerabilities: CVE-2023-25809 GHSA-5j5w-g665-5m35 GHSA-77vh-xpmg-72qh GHSA-c9cp-9c75-9v8c. Known issues: For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

Anthos clusters on VMware - Google Distributed Cloud for VMware 1.28.800-gke.109 is now available for download. Existing Seesaw load balancers now require TLS 1.2. Fixed The following vulnerabilities are fixed in 1.28.800-gke.109: High-severity container vulnerabilities: CVE-2020-22218 CVE-2022-48622 CVE-2024-0567 Ubuntu vulnerabilities: CVE-2023-52752 CVE-2024-25742 CVE-2024-26886 CVE-2024-26952 CVE-2024-27017 CVE-2024-36016.

GDCV for VMware - Google Distributed Cloud for VMware 1.28.800-gke.109 is now available for download. Existing Seesaw load balancers now require TLS 1.2. Fixed The following vulnerabilities are fixed in 1.28.800-gke.109: High-severity container vulnerabilities: CVE-2020-22218 CVE-2022-48622 CVE-2024-0567 Ubuntu vulnerabilities: CVE-2023-52752 CVE-2024-25742 CVE-2024-26886 CVE-2024-26952 CVE-2024-27017 CVE-2024-36016.

Apigee Advanced API Security - On August 5, 2024 we released an updated version of Advanced API Security. Shadow API Discovery, which is in preview, now supports the use of tags to label and organize observation results.

Apigee X - On August 8, 2024, we announced an increase in the recommended number of API basepaths per Apigee environment or environment group. Bug ID Description 329304975, 301845257 Limit on number of basepaths per environment Fixed issue with the number of total basepaths per environment causing potential failures when deploying API proxy revisions. On August 7, 2024, we published new documentation explaining how to integrate Apigee with a Security Information and Event Management (SIEM) solution.

Backup and DR Service - Management console is now available in the London (europe-west2), Mumbai (asia-south1), and Los Angeles (us-west2) regions. Backup and DR Service 11.0.12.322 is now available to update your backup/recovery appliance. Backup and DR service now supports restoring Oracle database to any target. Backup and DR Service now supports migrating manual protection to dynamic protection using tags through management console.

BigQuery - The JSON_KEYS function, which extracts unique JSON keys from a JSON expression, is in Preview. Some JSON functions that take a JSONPath let you specify a mode that allows flexibility in how the JSONPath matches the JSON data structure. An updated version of JDBC driver for BigQuery is now available. You can now create a materialized view over Apache Iceberg table that is partition aligned with the base table.

Chronicle SOAR - Release 6.3.15 is currently in Preview. Unable to upload ZIP files to the Case wall.

Colab - Fixed an issue in which users weren't able to access the Colab Enterprise UI when Colab Service Status was OFF for everyone in Google Workspace.

Cloud Composer - The kubelet read-only port in GKE clusters (TCP port 10255) is not used by Cloud Composer.

Compute Engine - Preview: An HTTPS metadata server endpoint is now available that provides added security for transmission of information between the metadata server and the VM.

Confidential VM - AMD has notified Google about 3 new (2 medium risk, 1 high risk) firmware vulnerabilities affecting SEV-SNP in AMD EPYC 3rd generation (Milan) and 4th generation (Genoa) CPUs (CVE-2023-31355, CVE-2024-21978, and CVE-2024-21980).

Config Connector - Config Connector version 1.121.0 is now available. The state-into-spec is default to Absent in any new ConfigController clusters. DataformRepository (Alpha) now uses direct reconciliation. BigTable When autoscaling is enabled (spec.cluster[].autoscalingConfig.), does not use numNodes (spec.cluster[].numNodes=2) as that applies only to manual scaling. BigQueryConnection Added status.observedState field to store the output-only fields which are previously mistakenly defined in spec.

Contact Center AI Platform - Web SDK 2.22 is released Web SDK 2.22 includes the following updates: Resize text. Version 3.23 is released All release notes published on this date are part of version 3.23. Cancel partial responses for virtual agents You can now configure virtual agents (VAs) to enable cancellation of the playback of partial-responses fulfillment when the final fulfillment is returned. New force logout endpoint in the Apps API You can now do a force logout on agents, using agent ID or IP address, by making a POST call to a new Apps API endpoint: POST /apps/api/v1/agent_statuses/force_logout. Updated agent statuses endpoint The agent statuses endpoint, apps/api/v1/agent_statuses, now returns an agent's IP address. Contact lists Agents can now use contact lists, which improve the organization and accessibility of external contacts. Auto answer settings are relocated The following auto answer settings for calls and chats have been relocated. End-of-session transfers for virtual agents You can now set up a virtual agent (VA) to handle end-of-session interactions. Queue-level whisper settings for calls Whisper settings for calls are now available at the queue level. Workforce management Workforce management (WFM) provides simple, flexible, real-time forecasting, scheduling, and adherence monitoring. Queue-level configuration of the cascade group timer You can now configure the cascade group timer at the queue level, instead of only at the global level. Queue-level configuration of the percent allocation group timer You can now configure the percent allocation group timer at the queue level, instead of only at the global level. Settings version control Settings version control is a new feature that lets administrators more easily migrate the queue menu structure and settings from one tenant to another. New quality management (QM) session event field CCAIP now passes a new field in the session event payload for calls. Fixed an issue where calls would sometimes not ring for an agent while connecting to a caller. Fixed an issue where agents with chat concurrency set to 1 were incorrectly receiving new chat offers while already handling a chat. Fixed a co-browse display issue. Fixed an issue where chats continued to be translated after being transferred to a queue that had live translation turned off. Fixed an issue where the agent adapter was not triggering events in an iframe when starting or ending a co-browse session. Fixed an issue where the Telnyx API was throwing an error when verifying a number, preventing BYOC numbers from being added.

Dataform - Access Approval supports Dataform in the GA stage.

Google Distributed Cloud Edge - Distributed Cloud connected 1.7.1. This is a patch release of Google Distributed Cloud connected (version 1.7.1). Security mitigations for the following vulnerabilities have been implemented in this release of Google Distributed Cloud connected: CVE-2024-36971, CVE-2024-36901, CVE-2024-36969, CVE-2024-36902 CVE-2024-36893, CVE-2024-36897, CVE-2024-35984, CVE-2024-35997, CVE-2024-6387 (GCP-2024-040), CVE-2024-38433, CVE-2024-0172. The following Google Distributed Cloud connected components have been updated: GKE on Bare Metal has been updated from version 1.28.500 to version 1.28.700. The following issues have been resolved in this release of Google Distributed Cloud connected: Nodes no longer get stuck in Ready,SchedulingDisabled state after applying configuration changes. This release of Google Distributed Cloud connected contains the following known issues: Virtual machine management can fail after a node has been powered down for an extended time.

Cloud Data Loss Prevention - The ORGANIZATION_NAME infoType detection model that was previously only accessible by setting InfoType.version to latest has been promoted to be the default detection model for this infoType. The STREET_ADDRESS infoType detection model that was previously only accessible by setting InfoType.version to latest has been promoted to be the default detection model for this infoType.

Cloud Networking Products - You can now select internal proxy Network Load Balancers as a health checked target for DNS routing policies in Preview.

Google Kubernetes Engine - (2024-R28) Version updates GKE cluster versions have been updated. You can now keep a GKE Standard cluster on a minor version for longer with the Extended release channel.

GKE new features - You can now keep a GKE Standard cluster on a minor version for longer with the Extended release channel.

GKE - Version 1.29.6-gke.1326000 is now the default version.

Google Kubernetes Engine Rapid - The following versions are now available in the Rapid channel: 1.27.16-gke.1082000, 1.28.12-gke.1090000, 1.29.7-gke.1174000, 1.30.3-gke.1451000.

Google Kubernetes Engine Regular - Version 1.29.6-gke.1326000 is now the default version in the Regular channel.

Google Kubernetes Engine Stable - The following versions are now available in the Stable channel: 1.27.15-gke.1154000, 1.28.11-gke.1172000, 1.29.6-gke.1254000.

Load Balancing - Regional external Application Load Balancer, regional internal Application Load Balancer, and cross-region internal Application Load Balancer support mutual TLS (mTLS).

NetApp - For the Flex service level, Google Cloud NetApp Volumes offers its services in the following regions: europe-west8 (Milan) europe-west9 (Paris) europe-west12 (Turin) To learn more, see NetApp Volumes key features.

reCAPTCHA Enterprise - reCAPTCHA Mobile SDK v18.6.0-beta01 is now available for iOS.

Sensitive Data Protection - The ORGANIZATION_NAME infoType detection model that was previously only accessible by setting InfoType.version to latest has been promoted to be the default detection model for this infoType. The STREET_ADDRESS infoType detection model that was previously only accessible by setting InfoType.version to latest has been promoted to be the default detection model for this infoType.

Service Mesh - Managed Cloud Service Mesh. Configuring Cloud Service Mesh for either proxyless gRPC or Envoy proxy deployments with the Kubernetes Gateway API is now available as a preview feature.

Cloud Storage - You can now use parallel downloads with Cloud Storage FUSE to accelerate read performance of large files over 1 GB in size.

Traffic Director - Managed Cloud Service Mesh. Configuring Cloud Service Mesh for either proxyless gRPC or Envoy proxy deployments with the Kubernetes Gateway API is now available as a preview feature.

Vertex AI Workbench - The M124 release of Vertex AI Workbench managed notebooks includes the following: Fixed a bug that prevented kernels from appearing when the Cloud Resource Manager API is turned off and Dataproc is enabled. Fixed a bug that prevented kernels from appearing when the Cloud Resource Manager API is turned off and Dataproc is enabled.

VMware Engine - VMware Engine ve2-mega-64 node type is generally available in the australia-southeast1 region.