Welcome to issue #391 March 25th, 2024

News

Anthropic’s Claude 3 Sonnet and Claude 3 Haiku are now generally available on Vertex AI - Claude 3 Sonnet and Claude 3 Haiku are generally available to all customers on Vertex AI.

Combine data across BigQuery and Salesforce Data Cloud securely with zero ETL - Bidirectional data sharing between BigQuery and Salesforce Data Cloud is now generally available.

Unify analytics with Spark procedures in BigQuery, now generally available

Introducing new BigQuery features to simplify time-series data analysis - New features simplify writing queries that perform two of the most common time series operations: windowing and gap filling.

At least once Streaming: Save up to 70% for Streaming ETL workloads - Introducing at-least-once streaming mode and comparison with exactly-once processing for streaming jobs.

Take control of GKE scaling with new quota monitoring

Introducing Cloud Run volume mounts: connect your app to Cloud Storage or NFS - With volume mounts, mounting a volume in a Cloud Run service or job is a single command. You can mount a Cloud Storage bucket or an NFS share, like a Cloud Filestore instance.

Rocky Linux 8 and CentOS 7 versions of HPC VM image now generally available - With these HPC VM images, it's easy to build an HPC-ready VM instances.

Unlock cloud savings with new Looker Studio Dashboard and Terraform scripts - An OSS Recommendations Dashboard to view cost savings recommendations for optimizing Google Cloud resources.

Google Cloud VMware Engine supercharged with Google Cloud NetApp Volumes - By combining the simplicity, performance, and advanced data management capabilities of NetApp Volumes with the reliability and operational efficiency of Google Cloud VMware Engine, organizations can achieve agility and cost savings.

Leveraging Couchbase connector and Application Integration in the Google ecosystem - Application Integration now supports Couchbase connector, to empower users to efficiently manage Couchbase NoSQL databases within their integration flows.

Introducing stronger default Org Policies for our customers - With the release of secure-by-default organization resources, potentially insecure postures and outcomes are addressed with a bundle of organization policies that are enforced as soon as a new organization resource is created.

Automatic driver installation simplifies using NVIDIA GPUs in GKE - GKE can now automatically install NVIDIA GPU drivers, making it easier for customers to take advantage of GPUs.

Google named a Leader in The Forrester Wave: AI Infrastructure Solutions, Q1 2024

---

Imagine having a direct line to over 150 senior cloud architects for any cloud-related question or issue you encounter. With thousands of cloud questions and issues resolved, DoiT is your gateway to world-class cloud expertise.

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How to secure Ray on Google Kubernetes Engine - Sharing security insights and hardening techniques for training AI/ML workloads on Ray framework.

How Deutsche Bank uses Google Distributed Cloud for its Autobahn FX real-time trading platform

How Commerzbank safeguards its data with VPC Service Controls

How to improve resilience to DDoS attacks with Cloud Armor Advanced rate limiting capabilities - Google Cloud Armor's always-on Layer 3 and Layer 4 DDoS defense, Web Application Firewall (WAF), Adaptive Protection, bot management, threat intelligence, and rate-limiting capabilities can help enterprises build a comprehensive DDoS mitigation strategy.

How to set compliance controls for your Google Cloud Organization

Enter the FinOps Universe: The Team-Up of Finance and Cloud Tech - Some thoughts on how to organize FinOps for your organization.

Controll K8s logging Cost on GCP - How to cut down 80% on k8s logging on cloud?

Accessing Google APIs via Private Service Connect and Private Google Access - Using PSC and PGA to for accessing Google APIs privately.

How to install and use Istio Ambient Mesh on GKE - A complete guide on how to install Istio in Ambient Mesh mode on Google Kubernetes.

App Development, Serverless, Databases, DevOps

NetRise elevates their user experience with Google Cloud

Choosing a suitable database for your startup: An overview of AlloyDB and Spanner

Statsig supports up to 7.5 million QPS with Memorystore for Redis Cluster

How to connect your Google load balancing with Google Cloud Run - This article explains how to connect Cloud Run to Load Balancer.

Push code with GitHub Actions to Google Cloud’s Artifact Registry - Push code from GitHub to Google Cloud Artifact Registry (without using service account keys) with Workload Identity Federation.

Deploy A Production-Ready Streamlit App with Cloud Run and Cloud Build - How to ship containerized applications on a serverless architecture and over a CICD pipeline.

Modernizing cloudbuild.yaml for Container Builds - Minimizing square brackets in Cloud Build yaml file.

How to verify your Google App

Big Data, Analytics, ML&AI

Built with BigQuery: How Pendo Data Sync maximizes ROI on your data

How Palo Alto Networks uses BigQuery ML to automate resource classification

How RealTruck drives data reliability and business growth with Masthead and BigQuery

Why GKE for your Ray AI workloads? Portability, scalability, manageability, cost - This blog post explains the numerous benefits that running Ray on GKE brings to the table — scalability, cost-efficiency, fault tolerance, isolation, and portability, to name a few — and resources on how to get started.

Advanced scheduling for AI/ML with Ray and Kueue - In this blog dives into how KubeRay and Kueue work together to orchestrate advanced scheduling for Ray applications on GKE.

Accelerate your generative AI journey with NVIDIA NeMo framework on GKE - This blog post shows how generative AI models can be adapted to your use cases by demonstrating how to train models on Google Kubernetes Engine (GKE) using NVIDIA accelerated computing and NVIDIA NeMo framework.

In-Place LLM Insights: BigQuery & Gemini for Structured & Unstructured Data Analytics - Introduction.

Building Knowledge Graphs from Scratch Using Neo4j and Vertex AI

BigQuery customers save up to 54% in TCO compared to alternative cloud data platforms - A report from TechTarget’s Enterprise Strategy Group.

DELETE + INSERT vs MERGE in BigQuery - How do you merge changes from staging tables into target tables in BigQuery?

Table Snapshots: BigQuery vs dbt - How do you know which one to use?

RANGE data type in BigQuery - Explanation of new BigQuery RANGE datatype.

Various

The future of infrastructure modernization: how Google Cloud Innovators are embracing the cloud

Slides, Videos, Audio

Kubernetes Podcast - #221 Creating Envoy, with Matt Klein.

Security Podcast - #164 Quantum Computing: Understanding the (very serious) Threat and Post-Quantum Cryptography.

Releases

Chronicle - Chronicle has added a new rule set to Cloud Threat Detections , called Serverless Threats, that detects activity associated with potential compromise or abuse of server-less resources in Google Cloud, such as Cloud Run and Cloud Functions. Chronicle now supports direct ingestion and parsing of reCAPTCHA Enterprise logs from Google Cloud. There is no longer a limit on the number of feeds you can create for the same log type in Feed Management.

Cloud Composer - The Logs in Cloud Logging only feature is gradually rolled out to all regions: New Cloud Composer environments now save Airflow task logs only in Cloud Logging by default. Cloud Composer 2.6.5 release started on March 19, 2024. Airflow 2.7.3 is available in Cloud Composer images. Fixed an issue where past Airflow task instances could be marked as failed in some cases. Fixed an issue where Airflow task logs for the first try of a task might not be visible in Airflow UI. BigQueryInsertJobOperator now correctly handles ephemeral tables created with tableDefinitions. In BigQueryInsertJobOperator, fixed the handling of parsing errors during Lineage emission when the query is too long or deeply nested. The apache-airflow-providers-google package is upgraded to version 10.16.0 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version. The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.0.1 in images with Airflow 2.6.3, and images with Airflow 2.7.3 have this version. Cloud Composer 2.6.5 images are available: composer-2.6.5-airflow-2.7.3 composer-2.6.5-airflow-2.6.3 (default) composer-2.6.5-airflow-2.5.3. Cloud Composer versions 2.1.10, 2.1.9, 1.20.10, and 1.20.9 have reached their end of full support period.

Compute Engine - Generally available: Disaster recovery with Persistent Disk Async Replication has been expanded to allow you to replicate data on a disk in one region to any other region within the same continent. Generally available: In a managed instance group (MIG), you can set metadata and labels for all VMs in the group without the need to create a new instance template. Generally available: In a managed instance group (MIG), you can turn off repairs to inspect failed and unhealthy VMs, to implement your own repair logic, or to monitor the application health without triggering repairs by MIG. Generally available: The organization-wide patch status dashboard and organization-wide OS policy compliance reports in VM Manager are now generally available.

Container Registry - Container Registry is scheduled to be shut down and superseded by Artifact Registry on March 18, 2025.

Dataproc Serverless - Announcing the Preview release of Dataproc Serverless for Spark 1.2 runtime: Spark 3.5.0 BigQuery Spark Connector 0.35.1 Cloud Storage Connector 3.0.0 Conda 23.11 Java 17 Python 3.12 R 4.3 Scala 2.12. New Dataproc Serverless for Spark runtime versions: 1.1.55 1.2.0-RC1 2.0.63 2.1.42 2.2.0-RC15. Dataproc Serverless for Spark: Upgraded Spark RAPIDS plugin to version 24.2.0 in the latest runtimes.

Dataproc - New Dataproc on Compute Engine subminor image versions: 2.0.96-debian10, 2.0.96-rocky8, 2.0.96-ubuntu18 2.1.44-debian11, 2.1.44-rocky8, 2.1.44-ubuntu20, 2.1.44-ubuntu20-arm 2.2.10-debian12, 2.2.10-rocky9, 2.2.10-ubuntu22.

Deep Learning Containers - M118 release Pytorch 2.1.0 with CUDA 12.1 and Python 3.10 container images are now available.

Deep Learning VM - M118 release Restored legacy gpu image families for TensorFlow 2.12 through 2.14, and for PyTorch 2.0.

Cloud Data Loss Prevention - The discovery and inspection services, which support BigQuery, now support tables that contain columns with INTERVAL, RANGE, RANGE, and RANGEdata types.

Eventarc - Eventarc support for creating triggers for direct events from Network Services is generally available (GA).

Google Kubernetes Engine - (2024-R08) Version updates GKE cluster versions have been updated. Cilium cluster-wide network policies are now generally available with the following GKE versions: 1.28.6-gke.1095000 or later 1.29.1-gke.1016000 or later You can now control your GKE workloads' ingress and egress traffic cluster-wide, without being bound to a namespace for your network policies. Workloads running on GKE clusters with COS-based nodes may experience DNS resolution issues.

Cloud Monitoring - You can now use Duet AI for Developers to help you create a synthetic monitor.

reCAPTCHA Enterprise - reCAPTCHA Enterprise platform logs are now available in Chronicle. reCAPTCHA Enterprise Mobile SDK v18.5.0-beta02 is now available for Android and iOS.

Cloud Run - Direct VPC egress (Preview) is now available in the following additional regions: africa-south1 asia-south1 asia-southeast2 australia-southeast2 europe-central2 europe-west2 europe-west6 europe-west8 europe-west9 europe-west10 me-central1 me-central2 southamerica-west1 us-east5 us-west2 us-west3 us-west8. Cloud Run services can now connect to a Firestore database using integrations (Preview). Cloud Run services can now connect to Vertex AI to access generative AI models using integrations (Preview). You can now mount an NFS file share as a volume for Cloud Run services and jobs.

Cloud Spanner - Leader-aware routing now dynamically routes read-write transactions to the leader region in Spanner multi-region instances, reducing latency and improving performance. Statistics for active partitioned data manipulation language (DML) queries are now generally available.

Cloud SQL - Cloud SQL Enterprise Plus edition now supports the me-central2 (Dammam) region.

Cloud Storage - You can now use the GCS FUSE file cache feature, a client-based read cache that lets repeat file reads to be served from a faster cache storage of your choice.

Vertex AI - Vector Search heuristics-based compaction Vector Search uses heuristics-based metrics assess whether to trigger compaction. The M118 release of Vertex AI Workbench user-managed notebooks includes the following: Pytorch 2.1.0 with CUDA 12.1 and Python 3.10 user-managed notebooks instances are now available. The M118 release of Vertex AI Workbench managed notebooks includes the following: Updated Nvidia drivers to R535, which fixed a bug where the latest PyTorch 2.0 kernel didn't work due to outdated drivers. The M118 release of Vertex AI Workbench instances includes the following: Updated Nvidia drivers to R535.

VPC Service Controls - Beta stage support for the following integration: App Hub.

AlloyDB - Updated the default major version of PostgreSQL compatibility for new AlloyDB clusters to PostgreSQL 15. The Quotas documentation is updated to include additional guidance on setting the maximum number of concurrent connections for your database instance size.

Anthos Config Management - Config Controller now uses the following versions of its included products: Config Connector v1.113.0, release notes Anthos Config Management v1.17.2, release notes.

Anthos Config Management - 1.17.3. The constraint template library includes a new template: K8sPSSRunAsNonRoot. Policy Controller bundles have been updated to the following versions: cis-gke-v1.4.0: 202402.0-preview, nist-sp-800-190: 202402.0, nist-sp-800-53-r5: 202402.0, pci-dss-v3.2.1: 202402.0, pss-baseline-v2022: 202402.0, pss-restricted-v2022: 202402.0. Fixed a regression introduced in 1.16.0 that limits the length of the Secret name referenced in the spec.git.secretRef.name field of the RootSync object. Fixed a regression introduced in 1.17.0 that caused Config Sync to sometimes fail to pull the latest commit from a Git branch by upgrading git-sync (Config Sync dependency for pulling from git) from v4.1.0 to v4.2.1.

Anthos clusters on bare metal - 1.28. Release 1.28.300-gke.131 GKE on Bare Metal 1.28.300-gke.131 is now available for download. Functionality changes: Updated preflight checks to add a check for networking kernel modules. Fixes: Fixed an issue with configuring a proxy for your cluster that required you to manually set HTTPS_PROXY and NO_PROXY environment variables on the admin workstation. The following container image security vulnerabilities have been fixed in 1.28.300-gke.131: High-severity container vulnerabilities: CVE-2022-28948 CVE-2023-29499 Medium-severity container vulnerabilities: CVE-2023-3446 CVE-2023-3817 CVE-2023-32611 CVE-2023-32665 CVE-2023-49290 CVE-2024-21664 GHSA-2c7c-3mj9-8fqh Low-severity container vulnerabilities: CVE-2021-25743 CVE-2023-2975. Known issues: For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section. 1.15. Release 1.15.11 GKE on Bare Metal 1.15.11 is now available for download. The following container image security vulnerabilities have been fixed in 1.15.11: Medium-severity container vulnerabilities: CVE-2023-46218 CVE-2023-49290 CVE-2024-21664 GHSA-2c7c-3mj9-8fqh Low-severity container vulnerabilities: CVE-2021-25743. Known issues: For information about the latest known issues, see GKE on Bare Metal known issues in the Troubleshooting section.

Anthos clusters on VMware - GKE on VMware 1.28.300-gke.123 is now available. Increased the default memory limit for node-exporter. The following issues are fixed in 1.28.300-gke.123: Fixed the issue where the admin cluster backup did a retry on non-idempotent operations.

Artifact Registry - Effective March 22, 2024, Artifact Registry npm repositories enforce not including uppercase letters in package names in order to match npmjs naming rules. Fixed the issue causing images copied to Artifact Registry from Container Registry with the automatic migration tool to fail to propagate their creation time to Artifact Registry.

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs.

Bare Metal Solution - You can now create and manage VRFs for the networks in your Bare Metal Solution environment.

BigQuery - The March 20, 2024 release notes announced the preview for user-defined aggregate functions, but user-defined aggregate functions are not yet supported. You can now add Salesforce Data Cloud data to BigQuery. Incremental materialized views now support LEFT OUTER JOIN and UNION ALL. You can use the CREATE AGGREGATE FUNCTION statement to create user-defined aggregate functions. The maximum notebook size has been increased from 10 MB to 20 MB. You can now view lists of all saved queries and all notebooks in your project. You can now create and run Spark stored procedures that are written in Python, Java, and Scala. The minimum duration between scheduled queries has been reduced from 15 minutes to 5 minutes. You can now undelete a dataset that is within your time travel window to recover it to the state that it was in when it was deleted. These BigQuery features are now generally available (GA): Text analysis configuration options for the following: CREATE SEARCH INDEX DDL Existing LOG_ANALYZER and new PATTERN_ANALYZER analyzers, which are used in various functions, including SEARCH The TEXT_ANALYZE function.

BigQuery ML - The following advanced processing functions: ML.BAG_OF_WORDS ML.TF_IDF BAG_OF_WORDS TF_IDF COSINE_DISTANCE EUCLIDEAN_DISTANCE EDIT_DISTANCE. You can now perform hierarchical forecasts in BigQuery ML time series models, which let you aggregate and roll up values for all time series in the model.

BigTable - You can now view Bigtable cost data with instance granularity in the Google Cloud Billing detailed export to BigQuery. You can now create daily backups of your Bigtable table by enabling automated backup.