Welcome to issue #417 September 23rd, 2024

News

Accelerate Ray in production with new Ray Operator on GKE - The Ray Operator on Google Kubernetes Engine (GKE) simplifies the deployment and management of Ray clusters for distributed AI/ML workloads. It offers declarative APIs, integrated logging and monitoring, TPU support, and features to reduce startup latency. With Ray on GKE, organizations can scale their AI applications efficiently and reliably, taking advantage of GKE's managed container orchestration service.

Find key insights with contribution analysis in BigQuery ML - Contribution analysis in BigQuery ML helps organizations find insights and patterns in their data by identifying combinations of contributors that cause unanticipated changes. It supports both summable metrics (e.g., revenue) and summable ratio metrics (e.g., earnings per share). The model uses pruning optimizations to reduce the search space and quickly find relevant segments.

Chat with your business data - Conversational Analytics comes to Gemini in Looker - Conversational Analytics, a new feature in Looker, allows users to ask questions of their data in natural language and receive insights powered by AI. It connects to BigQuery and Google Sheets, enabling users to directly ask questions of their data stored in these locations.

Which app should you modernize first? New Application Rationalization Dashboard can help - The Application Rationalization Dashboard is a data-driven tool that provides insights into your application portfolio and helps you identify which applications are best suited for modernization and migration to the cloud. It combines data from CAST Highlight and Google Cloud Migration Center to provide a variety of visualizations that help you see the big picture of your applications’ footprint.

Announcing expanded CIEM support to reduce multicloud risk in Security Command Center - Security Command Center now supports AWS IAM identities for AWS, and Entra ID (Azure AD) and Okta identities on Google Cloud. With multicloud, multi-identity CIEM support, customers can more easily discover which identities have access to which cloud resources across more of their cloud footprint.

Build and scale faster with new Blockchain RPC Service built on Google infrastructure - Google Cloud has launched a new Blockchain RPC service that provides developers with a streamlined way to interact with blockchain data. The service offers enterprise-grade reliability, cost-effectiveness, compatibility, and scalability. It is now globally available in preview and supports Ethereum mainnet and testnets, with plans to expand to several additional chains over the next year.

Chrome Enterprise Improves Management and Productivity Capabilities for Google Workspace users - Chrome Enterprise enhances management and productivity capabilities for Google Workspace users. New profile management features provide admins with more control and flexibility, while new productivity tools for users include a daily Google Calendar overview on new tabs, site search shortcuts, and automatic tab group saving and syncing across devices.

Google is a Leader in the 2024 Gartner® Magic Quadrant™ for Container Management - Google Cloud has been recognized as a Leader in the 2024 Gartner® Magic Quadrant™ for Container Management for the second consecutive year.

Activating your defender's advantage at mWISE ‘24 - Security experts from around the world are convening at mWISE Conference 2024 to collaborate on the latest solutions, compare experiences, and bolster their defenses. Google is announcing new ways to help customers stay safe, including updated best practices, expanded managed services, and additional avenues for threat intelligence sharing.

Join us for Gemini at Work - Join us for Gemini at Work, our first digital event dedicated to showcasing the transformative power of AI in the workplace. Learn how Gemini can help your business at our digital event on September 24 at 9am PT.

---

Technology and expertise from a Google Cloud Premier Partner

As a trusted Google Cloud partner of over a decade, DoiT delivers cloud optimization technology and unrivaled cloud expertise to help customers take full advantage of the public cloud. From migration and infrastructure to usage insights and analysis, DoiT has the experience and tools to help companies optimize Google Cloud use to accelerate return. Learn More

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Routing in Google Cloud: Where can I send my IP packet from a VM? - This blog explores various routing options from a virtual machine perspective, enabling seamless access to applications, managed services, SaaS solutions, on-premises services, and public or private services. Policy-based routes facilitate traffic inspection within a VPC.

UNC1860 and the Temple of Oats: Iran’s Hidden Hand in Middle Eastern Networks

How to prevent account takeovers with new certificate-based access - Certificate-based access (CBA) uses mutual TLS (mTLS) to ensure that user credentials are bound to a device certificate before authorizing access to cloud resources. CBA provides strong protection requiring X.509 certificates as device identifiers and verifies devices with user context for every access request to cloud resources.

An Offer You Can Refuse: UNC2970 Backdoor Deployment Using Trojanized PDF Reader - UNC2970, a cyber espionage group suspected to have a North Korea nexus, targeted victims in the U.S. critical infrastructure sectors using a trojanized PDF reader. The group sent malicious ZIP archives containing a password-protected PDF and a modified version of SumatraPDF, a legitimate PDF viewer. This technique did not exploit a vulnerability in the original SumatraPDF source code.

Cloud CISO Perspectives: The high value of cross-industry communication - Taking a look at how GCP initiatives to drive cybersecurity collaboration across industries, regulators and governments, IT consortia, and researchers and universities can help make everyone safer online.

Optimizing Cloud Costs on Google Cloud: A Practical Guide to the FinOps Open Cost and Usage Specification - Learn How to Save Money and Gain Insights with FOCUS on Google Cloud.

Next-Gen Cloud Network Security: Design Notes - Next-Gen Cloud Network Security (NGFW Enterprise) brings inspection capabilities to the workload level, offering greater flexibility and scalability.

Google Kubernetes Engine Cost Optimization Best Practices - In order to optimize costs for Google Kubernetes Engine (GKE), there are couple of approaches as best practices.

Cost management for AI/ML platforms with Google Kubernetes Engine - This blog discusses some of the capabilities and cost-saving initiatives engineered specifically into Google Kubernetes Engine (GKE) for running AI/ML workloads.

Terraform for GCP Security: security-posture Resource - The article provides a step-by-step guide on creating a basic security posture using Terraform, including enforcing uniform bucket-level access and checking for unencrypted BigQuery tables.

Accessing Private Resources & DNS On Your Google Cloud VPC w/ Tailscale - Learn how to securely access private resources and DNS in your Google Cloud VPC using Tailscale. A step-by-step guide.

App Development, Serverless, Databases, DevOps

Integrate Five9 VoiceStream with Google Cloud CCAI Agent Assist - Five9 VoiceStream can be integrated with Google Cloud CCAI Agent Assist using a gRPC streaming supported solution. This integration allows you to listen to real-time voice conversations and bring AI-powered suggestions to human agents.

From keywords to relationships: Reveal deeper insights with full-text search and Spanner Graph - Spanner Graph combines the power of graph databases and full-text search in a unified system. It enables you to model complex relationships and efficiently retrieve relevant information from unstructured data.

AlloyDB Omni on Kubernetes: Anywhere - AlloyDB Omni, a PostgreSQL-compatible database from Google Cloud, now offers a downloadable edition for deployment anywhere.

How does GCP Database Migration Service handles Oracle object conversions to Postgres - GCP Database Migration Service handles Oracle to Postgres migrations. It can convert Oracle objects without direct Postgres equivalents, such as tables without primary keys, partitioned tables, indexes, interval partitioning, composite partitioned tables, synonyms, packages, global variables, and global temporary tables.

Create a CI/CD Pipeline using GitHub Actions and Google Cloud - CI/CD with GitHub Actions, building and testing Python, and deploying to Google Cloud Functions and Google Cloud Run.

Big Data, Analytics, ML&AI

Tell a more complete data story with customized Looker charts and visualizations - Looker's Chart Config Editor enables you to customize Looker visualizations and create impactful data experiences. You can configure each line's appearance and labels in a line chart, enable inline scrolling for visualizations, and fully customize the data labels for a pie chart and other chart visualizations.

GenOps: the evolution of MLOps for gen AI - Learn how to build and scale Generative AI solutions with GenOps, an evolution of MLOps for Gen AI. GenOps combines DevOps principles with ML workflows to deploy, monitor, and maintain Gen AI models in production, ensuring scalability, reliability, and continuous improvement.

From Terabytes to Insights: NEON's Journey to Earth Engine with Google Cloud - The National Ecological Observatory Network (NEON) is one of the largest ecological observation facilities in the world. NEON is working with Google Public Sector to share its data globally on Google Cloud, including through Earth Engine, with researchers.

Apache Airflow ETL in Google Cloud - Apache Airflow is a popular choice for running complex tasks like ETL or data analytics pipelines. There are three different ways to run Apache Airflow on Google Cloud: Compute Engine, GKE Autopilot, and Cloud Composer. Each approach has its own advantages and disadvantages in terms of cost, performance, and availability.

Critical GCP Composer Flaw Exposed: How CloudImposer Almost Led to Remote Code Execution - Google Cloud Platform's Composer, a workflow orchestration service, had a critical vulnerability named CloudImposer that could have led to remote code execution.

Working with Gigantic Google BigQuery Partitioned Tables in DBT - This article describes approach to reduce scanned data for DBT tables that are incremental and partitioned.

ELT with DBT on Cloud Run job - This article presents a real-world use case of ELT (Extract, Load, Transform) with DBT (Data Build Tool) on Cloud Run Job. It explains how to build a modern ELT pipeline using DBT and deploy it on Cloud Run Job, combining the flexibility and ease of use of DBT with the scalability and cost-effectiveness of Cloud Run.

Unleashing the Power of AI: Automating SQL Query Generation and Real-Time Data Streaming with Confluent and Google Cloud - This blog post explores how to integrate large language models (LLMs) with Confluent and Google Cloud Vertex AI to create a powerful, end-to-end solution for real-time data processing and insights.

Building a Dynamic Podcast Generator Inspired by Google’s NotebookLM and Illuminate - Building a dynamic podcast generator that transforms written articles into engaging audio conversations. Using AI language models and speech synthesis technologies, this project demonstrates the power of combining these technologies to create innovative content consumption methods.

Using TPUs for fine-tuning and deploying LLMs with dstack - Dstack, an open-source container orchestrator, now supports using TPUs with Google Cloud. You can use dstack to fine-tune and deploy large language models (LLMs) on TPUs, leveraging open-source tools like Hugging Face's Optimum TPU and vLLM.

Architecting GenAI applications with Google Cloud - Architecting GenAI applications with Google Cloud involves selecting the appropriate application hosting infrastructure, such as Cloud Run or GKE, and choosing a model hosting option like Vertex AI or GKE. It also includes selecting the right model from Vertex AI's Model Garden, considering factors like task, dataset size, and budget. Additionally, grounding solutions like retrieval-augmented generation (RAG) can be implemented using vector databases to ensure informed and accurate model responses.

AI on your Laptop with AlloyDB Omni and Ollama. - AlloyDB Omni, a fully-managed PostgreSQL-compatible database service, can be integrated with Ollama, an open-source tool for running large language models locally, to generate embeddings from user inputs stored in databases. The process involves setting up Ollama on the local laptop, integrating AlloyDB Omni with Vertex AI, setting up input and output transformation, and loading the local model into AlloyDB Omni.

Various

Google's AI First Accelerator: Boosting Europe's innovator scene - Google for Startups Accelerator: AI First is a 10-week program that provides startups with expertise, mentorship, and resources from Google Cloud and Google DeepMind to develop and deploy innovative AI solutions.

Meet the second Google for Startups Accelerator: Climate Change Cohort in Europe - Google for Startups Accelerator in Europe: Climate Change Cohort announced, empowering startups leveraging technology to combat climate change.

Slides, Videos, Audio

Kubernetes Podcast - #236 Dagger, with Solomon Hykes.

Security Podcast - #190 Unraveling the Security Data Fabric: Need, Benefits, and Futures.

Releases

AlloyDB - The AlloyDB Omni operator is now available in Preview on Google Distributed Cloud (GDC) connected. You can now add the predefined CMEK organization policy for your AlloyDB clusters and backups. The postgres_ann extension has been renamed to alloydb_scann. Added a tutorial that shows you how to set up a connection from an application running in a Google Kubernetes Engine autopilot cluster to an AlloyDB instance. AlloyDB Omni Kubernetes operator version 1.1.1 is now available. Upgrading to version 1.1.1 of the AlloyDB Omni Kubernetes operator might result in a brief interruption to all database clusters.

Anthos clusters on VMware - Google Distributed Cloud (software only) for VMware 1.29.500-gke.160 is now available for download. Fixed the following issues in 1.29.500-gke.160: Fixed the known issue where updating DataplaneV2 ForwardMode didn't automatically trigger anetd DaemonSet restart.

GDCV for VMware - Google Distributed Cloud (software only) for VMware 1.29.500-gke.160 is now available for download. Fixed the following issues in 1.29.500-gke.160: Fixed the known issue where updating DataplaneV2 ForwardMode didn't automatically trigger anetd DaemonSet restart.

Apigee UI - On September 18, 2024, we released an updated version of the Apigee UI. Bug ID Description 349284447 All API products associated with a key now displayed in the UI All API products associated with a key can now be viewed in the App detail page of the UI using pagination.

Apigee X - On September 20, 2024, we released an updated version of Apigee (1-13-0-apigee-5). Bug ID Description 366039324 Fixed PEM parsing error in JWT/JWS policies Resolved a PEM parsing error in JWT/JWS policy execution caused by a problematic PEM format. On September 18, 2024 we released an updated version of Apigee Note: Rollouts of this release to production instances will begin within two business days and may take four or more business days to complete across all Google Cloud zones. Release of Cloud IAM-based authorization and authentication and the VerifyIAM policy.

Cloud Architecture Center - (New guide) Migrate from Amazon RDS and Amazon Aurora for PostgreSQL to Cloud SQL and AlloyDB for PostgreSQL: Describes how to design, implement, and validate a plan to migrate from Amazon Relational Database Service (RDS) or Amazon Aurora for PostgreSQL to Cloud SQL. (New guide) Scalable BigQuery backup automation: Build a solution to automate recurrent BigQuery backup operations at scale, with two backup methods: BigQuery snapshots and exports to Cloud Storage. Design an optimal storage strategy for your cloud workload: Updated guidance about storage recommendations and storage options decision tree with information about Hyperdisk ML and Hyperdisk Balanced.

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs.

BigQuery - You can perform model monitoring in BigQuery ML. You can now batch migrate classic saved queries to saved queries. You can now use a CREATE MODEL statement to create a contribution analysis model in BigQuery ML. You can store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency.

Chronicle Security Operations - Google SecOps has updated the list of supported default parsers.

Chronicle SOAR - Release 6.3.19 is currently in Preview. Case Report can now be exported in PDF format. The comment count on the case wall is not updating correctly. The HTML widget refresh is not affecting the JS code.

Cloud Composer - A new Cloud Composer release has started on September 18, 2024. Airflow 2.9.3 is available in Cloud Composer images. (Cloud Composer 2) Fixed the issue where environment create and update operations could fail in rare cases because of the scheduler probe timeouts. (Cloud Composer 3) Fixed the issue that caused KubernetesPodOperator tasks to fail if they ran for longer than 15 minutes. New Airflow builds are available in Cloud Composer 3: composer-3-airflow-2.9.3-build.0 composer-3-airflow-2.9.1-build.7 (default) composer-3-airflow-2.7.3-build.16. Cloud Composer 2.9.4 images are available: composer-2.9.4-airflow-2.9.3 composer-2.9.4-airflow-2.9.1 (default) composer-2.9.4-airflow-2.7.3. Support dates for previous Cloud Composer 3 builds are available. Cloud Composer versions 2.4.2 and 2.4.3 have reached their end of support period.

Compute Engine - You can determine the number of running VMs and reservations that match the properties of a future reservation request. You can create a future reservation request by reusing the properties of an existing VM.

Data Fusion - The SAP SLT No RFC Replication plugin version 0.11.3 is available in Cloud Data Fusion version 6.8.0 and later.

Database Migration Service - Database Migration Service for homogeneous PostgreSQL migrations to Cloud SQL now automatically enables point-in-time recovery (PITR) for the destination instance when you promote the migration job.

Dataproc - New Dataproc on Compute Engine subminor image versions: 2.0.118-debian10, 2.0.118-rocky8, 2.0.118-ubuntu18 2.1.66-debian11, 2.1.66-rocky8, 2.1.66-ubuntu20, 2.1.66-ubuntu20-arm 2.2.32-debian12, 2.2.32-rocky9, 2.2.32-ubuntu22.

Dialogflow - Dialogflow CX and Vertex AI: The text-bison-002 model will be deprecated on September 30, 2024 and automatically upgraded to the gemini-1.5-flash-001 model. Dialogflow CX & ES: Text-to-speech Journey Voices now supports MULAW output audio_encoding (CX, ES) in addition to LINEAR16. Dialogflow CX: Cloud Text-to-Speech europe-west1 and europe-west3 regions for Neural2 voices will temporarily use the eu mulit-region instead. Dialogflow CX and Vertex AI Agents: Generative features will migrate to the gemini-1.5-flash-001 model on September 30, 2024.

Cloud Data Loss Prevention - The POLITICAL_TERM infoType detector is available in all regions. The NEW_ZEALAND_NHI_NUMBER infoType detector is available in all regions.

Terraform on Google Cloud - Multiple Terraform samples added to BigQuery documentation.

Anti Money Laundering AI - A new major engine version is now available for Retail and Commercial lines of business, within the v4 tuning version.

IAM - Privileged Access Manager (PAM) is now released to General Availability.

Identity-Aware Proxy - Preview: You can now use authorization policies to delegate authorization to Identity-Aware Proxy (IAP) and Identity and Access Management (IAM).

Integration Connectors - The Microsoft Teams connector is now generally available (GA). The BigQuery connector now supports OAuth 2.0 - Authorization code. Connection names cannot exceed 49 characters. The OneDrive connector is now generally available (GA).

Load Balancing - Envoy-based Application Load Balancers now support authorization policies that let you establish access control checks for incoming traffic.

Cloud Logging - You can now query your log data from the Log Analytics page by using reserved BigQuery slots. You can now create and manage your log scopes by using the Logging API in addition to using the Cloud Console. There is a new Cloud Observability Overview page in the Google Cloud Console.

Media CDN - HTTP method filtering for specific route rules is now Generally Available.

Memorystore for Redis Cluster - Added support for 1, 2, and 4 shard instance shapes (Preview).

Migration Center - The discovery client 6.3.3 is available with new features and bug fixes. Added a notification that appears when a new version of the discovery client is available. Added technical fit assessment for BitLocker encryption as part of "Shift to GCE" and "Shift to GDC" assessment. MSI installer privileges changes: Users are no longer required to run the MSI installer as administrators. Fixed an issue that caused Linux guest collections to report the wrong number of CPU cores in some cases. Reduced the resource consumption of the Windows collection script on the target VMs. Fixed an issue that caused VMs using Windows NLB of the same cluster to be represented as a single VM in Migration Center. Fixed an issue that caused the discovery client to wrongly classify Java processes as JBoss applications.

Cloud Monitoring - There is a new Cloud Observability Overview page in the Google Cloud Console.

NetApp - Large capacity volumes in Preview is now generally available for allow-listed users.

Cloud Run - The Direct VPC egress feature of Cloud Run now supports Secure Web Proxy. You can now apply custom constraints for projects that get enforced by organization policies on your Cloud Run services and jobs (in Preview).

Sensitive Data Protection - The POLITICAL_TERM infoType detector is available in all regions. The NEW_ZEALAND_NHI_NUMBER infoType detector is available in all regions.

Service Extensions - Authorization extensions help you configure Cloud Load Balancing authorization policies to use custom authorization engines. You can now also host an extension on a backend service that uses serverless NEGs pointing to Cloud Run services.

Service Mesh - 1.23.x. 1.23.2-asm.2 is now available for in-cluster Cloud Service Mesh. 1.22.x. 1.22.5-asm.1 is now available for in-cluster Cloud Service Mesh. 1.21.x. 1.21.5-asm.7 is now available for in-cluster Cloud Service Mesh. 1.20.x. 1.20.8-asm.7 is now available for in-cluster Cloud Service Mesh. 1.20.x & 1.21.x & 1.22.x. Cloud Service Mesh with a Traffic Director control plane implementation is still incompatible with Envoy version v1.31.0.

Cloud SQL MySQL - Cloud SQL is discontinuing support for legacy high availability (HA) instance configuration on January 6, 2025.

Cloud SQL Postgres - You can now use gcloud or the Cloud SQL Admin API to switch the storage location of the transaction logs used for point-in-time recovery on your instance without downtime to Cloud Storage.

Traffic Director - 1.23.x. 1.23.2-asm.2 is now available for in-cluster Cloud Service Mesh. 1.22.x. 1.22.5-asm.1 is now available for in-cluster Cloud Service Mesh. 1.21.x. 1.21.5-asm.7 is now available for in-cluster Cloud Service Mesh. 1.20.x. 1.20.8-asm.7 is now available for in-cluster Cloud Service Mesh. 1.20.x & 1.21.x & 1.22.x. Cloud Service Mesh with a Traffic Director control plane implementation is still incompatible with Envoy version v1.31.0.

AutoML Translation - AutoML Translation API is deprecated and will no longer be available on Google Cloud after September 30, 2025.

Vertex AI - To ensure that VM resources are available when your custom training and prediction jobs need them, you can now use Compute Engine reservations. To reduce the cost of running your training and prediction jobs, you can now use Spot VMs. Schedule Vertex AI custom training jobs based on resource availability.

Workstation - Cloud Workstations preconfigured base images use Ubuntu 24.04. Cloud Workstations preconfigured base images default to Python 3.12.3.