Welcome to issue #382 January 22nd, 2024

News

Announcing Workflows execution steps history - You can now view step level debugging information for each execution from the Google Cloud console or the REST API.

AlloyDB AI powers gen AI applications with seamless Vertex AI integration - You can enable AlloyDB access to Vertex AI with AlloyDB AI using the google_ml_integration extension, which allows you to run predictions with your data in AlloyDB using custom models in Vertex AI or models from Model Garden.

Ten new generative AI trainings to upskill in 2024 with Duet AI - A recommended list of 10 new trainings on Duet AI for Google Workspace and Duet AI for Google Cloud to learn how to use generative AI to improve your productivity and efficiency.

Introducing granular cost insights for GKE, using Cloud Monitoring and Billing data in BigQuery - With Cloud Monitoring metrics in BigQuery (now in Preview), you can combine billing data with resource utilization metrics, empowering you to perform detailed analyses in BigQuery.

Personalized Service Health is now generally available: Get started today - Personalized Service Health begins processing and publishing relevant incidents to your Service Health dashboard in the Google Cloud console.

Get timely networking health updates with Personalized Service Health emerging incidents - Emerging incidents are machine-driven alerts that are communicated simultaneously to you and internal Google SRE teams, significantly reducing the time-to-first-meaningful post about an incident.

Introducing Bulikula and Halaihai, subsea cables to connect the central Pacific - Bulikula will connect Guam with Fiji, Halaihai, will link Guam and French Polynesia.

Our Solar API now covers more rooftops worldwide - The Solar API is being used to optimize solar panel arrays, make solar assessments and proposals more accurate and efficient, and to educate the public about transitioning to solar energy by showing homeowners the feasibility for their individual properties.

Unleashing the Power of AI for Government and Education: Join Public Sector GenAI Live & Labs in NYC! - A half day event designed for public agencies, research institutions, and academia in the region to discover how Google Cloud's AI technologies can fundamentally accelerate their work and impact.

Standardize your cloud billing data with the new FOCUS BigQuery view - a new BigQuery view that leverages the recent FOCUS (FinOps Open Cost and Usage Specification) (in a preview) to help simplify cloud cost management across clouds.

---

Imagine having a direct line to over 150 senior cloud architects for any cloud-related question or issue you encounter. With thousands of cloud questions and issues resolved, DoiT is your gateway to world-class cloud expertise.

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Figuring out microservices running on your GKE cluster with help from Duet AI

Optimizing Your Google Kubernetes Engine Costs — Part 2: The Bin Packing Problem - This article explores the bin packing problem in Kubernetes with special reference to GKE.

Configuring GCP Workload Identity Federation for Bitbucket Pipelines - This blog will cover how to securely and seamlessly connect GCP to Bitbucket using Workload Identity Federation.

Automating CI/CD Workflows: Executing Kubernetes Commands on GCP from GitHub Actions - Optimizing DevOps Pipelines: Tips and Tricks for Kubernetes in Google Cloud Platform.

Cross-Project Cloud SQL Connection with Private Service Connect and Terraform - This article explores how to establish a cross-project Cloud SQL connection using Private Service Connect.

App Development, Serverless, Databases, DevOps

ChromeOS: A New Year’s Resolution for Your Business - ChromeOS is the simple-to-manage operating system that keeps your business data secure, employees productive, and all while reducing your total cost of ownership.

Get your services back online quickly with Duet AI - Duet AI, an assistive AI tool,can help you make sense of the error messages and also speed up your investigation.

Google Cloud mobile app: A troubleshooting and management companion for your cloud applications - With the Google Cloud mobile app, you can easily monitor the status and access services.

Secure CloudSQL Connectivity with Google Cloud SQL Auth Proxy as Systemd - This guide will guide you through installing and configuring Cloud SQL Auth Proxy as Systemd on Ubuntu in two ways.

Spanner: A differentiated database for non-relational workloads

How to create a multilingual chatbot that queries AlloyDB with Langchain, Streamlit, LLMs, and Google Translate

Google Cloud HPC for computer aided engineering: Accelerate your design and simulation workflows - Google has developed a Computer Aided Engineering solution that brings together the right technologies to run major CAE applications efficiently. The solution harnesses Google Cloud’s HPC capabilities and is tailored to the simulation and analysis steps of CAE workflows.

Linear optimizes data and scalability with vector search support on Google Cloud SQL - Leveraging the power of Cloud SQL for PostgreSQL, Linear was able to keep pace with its expanding customer base–improving the efficiency, scalability, and reliability of data management, scaling up into the tens of terabytes without increasing engineering effort.

Navigating the Cloud API Maze: Apigee, GCP API Gateway, or GCP Cloud Endpoints? - An overview of API solutions on GCP.

Big Data, Analytics, ML&AI

NEAR accelerates Web3 adoption with simplified app development for blockchain - NEAR chose to work with Google Cloud and partner Databricks to improve data querying experiences and dramatically accelerate retrieving analytics from apps in its NEAR blockchain operational system.

Dealing with PII Data in Dataflow with Cloud DLP API - In this guide, we’ll walk through the process of creating a Dataflow pipeline to read data from Google Cloud Storage (GCS), apply transformations, data masking using Cloud DLP API, and then write the transformed data to a BigQuery table.

Infrastructure failures during big data processing - This blog post explains how to handle hardware failures when running Spark jobs.

Optimizing BigQuery for astronomy datasets using HealPix Index - This article covers how to enhance query performance on an astronomy dataset employing clustering of the records by HEALPix index. Although this article specifically refers to astronomy data, the techniques could be useful for any user of the BigQuery GIS platform.

Integrating BigQuery data into your LangChain application - See how to integrate your BigQuery data into LLM solutions.

Serverless ELT with GCS, BigQuery and Cloud Workflows - This article presents a complete use case with a Serverless ELT pipeline with Cloud Storage, BigQuery and Cloud Workflows.

How to Low-Pass Filter in Google BigQuery - This article shows how to implement a low-pass filter in SQL / BigQuery that can come in handy when improving ML features.

Why Partitioning Tables is Essential in BigQuery (2024) - Part 1 of in-depth series on maximizing efficiency in Google BigQuery.

Enterprise MLOps with Google Cloud Vertex AI (part 1) - This article is the first in a series where we will demonstrate how to establish a comprehensive MLOps framework on Google Cloud.

Analyzing images with Gemini Pro Vision and BigQuery - Learn how to access the Gemini models from BigQuery to analyze your big data, including images.

How to build an Anomaly Detector using BigQuery - Creating the anomaly detector with BigQueryML.

Deploy, Don’t Delay: Streamlining model deployment with Vertex AI Deployer - This article provides an overview on how to use Cloud Deploy with Vertex AI for model deployment.

How Wayfair is improving its feature engineering system with Vertex AI

How Vertex AI Endpoints helped Wayfair achieve real-time model serving

Slides, Videos, Audio

Security Podcast - #155 Cyber, Geopolitics, AI, Cloud - All in One Book?

Releases

AlloyDB - AlloyDB database server compatibility with PostgreSQL version 15 is now generally available (GA).

Anthos clusters on Azure - The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS nodes: CVE-2023-6111.

BigQuery - You can now use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions.

Cloud Build - Cloud Build is introducing a new organization policy constraint.

Chronicle - The supported default parsers have changed, check the release page for more details.

Data Fusion - Cloud Data Fusion version 6.10.0 is available in Preview. Source control management using Github is generally available (GA) in Cloud Data Fusion version 6.10.0. In versions before 6.10.0, you can only establish connectivity between private Cloud Data Fusion instances and customer resources with VPC Peering. You can control access to Google Cloud resources at the Cloud Data Fusion namespace level using an IAM service account. Changes in Cloud Data Fusion 6.10.0: Role-based access control in Cloud Data Fusion supports using the Cloud Data Fusion API Service Agent service account while emitting lineage events in RBAC-enabled instances. To support the Private Service Connect integration, the following permissions are added to the Cloud Data Fusion API Service Agent role: compute.networkAttachments.get compute.networkAttachments.update compute.networkAttachments.list. Fixed in Cloud Data Fusion 6.10.0: Fixed an issue in the Postgres DB plugin causing macros to be unsupported for database configuration (PLUGIN-1681). Dataproc 1.5 isn't supported in Cloud Data Fusion version 6.10.0. The Spark compute engine running on Scala 2.11 is not supported in Cloud Data Fusion version 6.10.0.

Database Migration Service - Database Migration Service support for Oracle to AlloyDB for PostgreSQL migrations is now available in Preview.

Dataproc Serverless - New Dataproc Serverless for Spark runtime versions: 1.1.46 2.0.54 2.1.33 2.2.0-RC6. Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.19 version in 1.1, 2.0, and 2.1 runtimes. Beginning March 31, 2024, when you submit a Dataproc Serverless Batch with a CMEK key: In addition to encrypting disk and Cloud Storage data, Dataproc Serverless will use your CMEK to also encrypt batch job arguments. New Dataproc Serverless for Spark runtime versions: 1.1.45 2.0.53 2.1.32 2.2.0-RC5. Dataproc Serverless for Spark: Upgraded Spark RAPIDS to version 23.12.1 Upgraded the following components to the following versions in the latest 2.2 runtime: Spark BigQuery connector version 0.35.0 Cloud Storage connector version 3.0.0 Conda version 23.11.

Dataproc - New Dataproc on Compute Engine subminor image versions: 2.0.88-debian10, 2.0.88-rocky8, 2.0.88-ubuntu18 2.1.36-debian11, 2.1.36-rocky8, 2.1.36-ubuntu20, 2.1.36-ubuntu20-arm 2.2.2-debian12, 2.2.2-rocky9, 2.2.2-ubuntu22. Dataproc on Compute Engine: Changed the Hive Server2 and MetaStore maximum default JVM heap size to 32GiB. Dataproc on Compute Engine: Backported the patch for YARN-10975 in the latest 2.0 images. Upgraded the following components to the following versions in the latest 2.2 runtime: Spark BigQuery connector version 0.35.0 Cloud Storage connector version 3.0.0 Conda version 23.11. Dataproc Spark Enhancements are now available in the Google Cloud console Dataproc cluster and job creation pages.

Datastream - The maximum event size that Datastream supports when streaming data to BigQuery is now increased to 20 MB.

Deep Learning VM - M115 release TensorFlow 2.15 with CUDA 12.1 and Python 3.10 images are now available.

Dialogflow - To temporarily opt out from the speech model migration, you can now use the opt_out_conformer_model_migration field in the runtime Dialogflow API requests. Dialogflow CX now allows cancellation of partial response playback. Vertex AI Conversation has launched generative agents in the Vertex AI Conversation console.

Error Reporting - Announcing a common navigation pane in the Cloud console for Logging, Monitoring, Trace, and Error Reporting.

Cloud Filestore - Zonal tier backups are now generally available.

Cloud Functions - Cloud Functions (1st gen) now uses Artifact Registry as the default container registry.

IAM - IAM deny policies let you deny groups of permissions for certain services.

Identity-Aware Proxy - A BeyondCorp Enterprise license is no longer required when configuring Identity-Aware Proxy with an internal load balancer.

KMS - Bare Metal HSM is generally available for customers with specific business and technical requirements in limited regions.

Google Kubernetes Engine - The following vulnerability was discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS nodes: CVE-2023-6111 For instructions and more details, see the GKE security bulletin.

Cloud Logging - You can now edit Log Analytics charts that are saved to a dashboard directly in the Dashboards page. Announcing a common navigation pane in the Cloud console for Logging, Monitoring, Trace, and Error Reporting.

StratoZone - Updated the Assessment Summary and Proposal, and the Detailed Pricing report generation to redirect to Migration Center when an account is integrated with Migration Center. Updated Google Cloud pricing. Updated some CloudFit scores for operating systems that have reached End of Life (EOL). Fixed an issue where drive type manual override was not being applied in certain situations. Fixed an issue with the Microsoft License Assessment report that occasionally would cause a blank Detail report when requesting both Summary and Detail reports at the same time.

Cloud Monitoring - Ops Agent version 2.46.0 introduces a new health check, LogPingOpsAgent, an informational payload message written every 10 minutes to the ops-agent-health log. Observability for Google Kubernetes Engine: You can now enable the package of kube state metrics from the Observability tab of a GKE deployment. Announcing a common navigation pane in the Cloud console for Logging, Monitoring, Trace, and Error Reporting.

Cloud NAT - Cloud NAT support for Inter-VPC communications is available in General Availability.

Cloud Run - You can now mount a Cloud Storage bucket as a storage volume for services, and also for Cloud Run jobs. Direct VPC egress (Preview) is now available in the following additional regions: asia-east1 asia-east2 asia-northeast2 asia-south2 australia-southeast1 europe-north1 europe-southwest1 europe-west12 europe-west4 me-west1 northamerica-northeast1 northamerica-northeast2 southamerica-east1 us-east4 us-south1 us-west1 us-west4.

Service Mesh - 1.19.x. 1.19.6-asm.2 is now available for in-cluster Anthos Service Mesh. 1.18.x. 1.18.7-asm.0 is now available for in-cluster Anthos Service Mesh. 1.17.x. 1.17.8-asm.12 is now available for in-cluster Anthos Service Mesh.

Cloud Storage - Newly created organizations now have the uniformBucketLevelAccess constraint enabled by default.

Cloud Trace - Announcing a common navigation pane in the Cloud console for Logging, Monitoring, Trace, and Error Reporting.

Vertex AI - The M115 release of Vertex AI Workbench user-managed notebooks includes the following: Added support for TensorFlow 2.15 with Python 3.10 on Debian 11, fixed the BigQuery connector within PySpark containers, added support for venv kernels.