Welcome to issue #393 April 8th, 2024

News

Privacy-preserving data sharing now generally available with BigQuery data clean rooms - BigQuery data clean rooms are now generally available, empowering businesses to securely share and analyze sensitive data in low-trust environments. With BigQuery's robust foundation, data owners can protect their assets through analysis rules such as join restrictions and differential privacy.

GKE Autopilot mode gets burstable workloads and smaller Pod sizes - Now featuring burstable workloads and CPU increments as low as 50m, you can optimize resource utilization and run even high-density applications cost-effectively. Plus, with flexible Pod sizing and automated Vertical Pod Autoscaling, you can effortlessly scale your workloads and reduce infrastructure management overhead.

Google Cloud Backup and DR upgrade: VM protection made easier - You can now leverage the power of Google Cloud tags, including inheritance, to easily configure backup policies for Compute Engine VMs, ensuring consistent protection of your dynamic cloud environments.

Attention DevOps engineers: Top managed container sessions to add to your Next ‘24 agenda

Cloud architects, don’t miss these Google Cloud Next sessions

Developers, check out these app dev sessions at Next ‘24

Get excited about what's coming for data professionals at Next ‘24

What startups can look forward to at Google Cloud Next '24

15 must-attend security sessions at Next '24

Showcase your skills: Discover new ways to skill up with Google Cloud Credentials

Get inspired: Database success stories at Google Cloud Next

Your Public Sector Guide to Next '24: Sessions, Demos, Lightning Talks

---

Imagine having a direct line to over 150 senior cloud architects for any cloud-related question or issue you encounter. With thousands of cloud questions and issues resolved, DoiT is your gateway to world-class cloud expertise.

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Cutting Edge, Part 4: Ivanti Connect Secure VPN Post-Exploitation Lateral Movement Case Studies - Mandiant has responded to incidents involving exploited Ivanti Connect Secure VPN appliances. This blog post outlines post-exploitation activity observed, including lateral movement and malware deployment. Mandiant recommends patching and following Ivanti's guidance to mitigate the vulnerabilities.

DNS providers for Google Kubernetes Engine explained - Discover the different DNS options available for Google Kubernetes Engine (GKE) clusters: kube-dns, Cloud DNS, and NodeLocal DNSCache. Learn how to choose the best solution for your workload's performance and reliability requirements, and improve the overall efficiency of your service discovery and resolution.

GCP Security — Finding Zero Trust Policy issues using IAM policy Recommander — Big Data Processing - Identifying security issues within GCP environment using Google Recommender and BigQuery.

Design your Landing Zone — Design Considerations Part 4— IaC, GitOps and CI/CD (Google Cloud Adoption Series) - LZ design considerations and decisions you need to make, relating to IaC, GitOps and CI/CD.

Deploying MySQL Databases in Google Kubernetes Engine : A step-by-step guide - A step-by-step process of setting up a MySQL database on GKE, covering everything from provisioning persistent storage to configuring MySQL settings and launching the database instance.

App Development, Serverless, Databases, DevOps

Boosting data cyber-resilience for your Cloud Storage data with object retention lock - WORM - "write once, read many" storage is easily achievable with the new object retention lock for Cloud Storage.

Securing GCS Buckets: disable directory listing! - Comparing Cloud Storage IAM roles and their access.

Working with GCP Cloud Functions and Box webhooks - Using Cloud Functions for Box webhooks.

How Google Admins can Save Money by Understanding the Relationship between Google Cloud Identity, Google Workspace, and Google Cloud - Learn how to avoid unnecessary licensing costs by understanding the difference between Workspace and Cloud Identity licenses. Discover how to disable automatic licensing and manually assign licenses to optimize your spending. Also, find out how to get Cloud Identity Premium for free with a Google Workspace Enterprise Edition subscription.

Brewing Security into Your Deployments: SLSA, Cloud Build, and a Shot of Efficiency - Integrate SLSA, the software supply chain security framework, with Cloud Build and Cloud Deploy to enhance your security posture.

Automating Docker Image Builds and Pushes to GCP Artifact Registry with GitHub Actions

GCP’s Signed URLs & A Browser Oddity - A view into my experience developing a signed URL with Go in GCP and Cloud Storage.

Introducing Konfig: GitLab and Google Cloud preconfigured for startups and enterprises - Konfig provides a turnkey platform for migrating and modernizing cloud applications, leveraging GitLab and GCP.

Using AlloyDB Go connector for automatic IAM authentication (service account) - This blog post walks you through the process of configuring your application and AlloyDB Instance to use AlloyDB Go connector so that your application can use a service account to connect to AlloyDB database.

Use GCP Workflows to Secure an FTP-to-Google Cloud Storage Data Flow - Example of using Workflows to orchestrate file transfer.

Quick Notes: Everything About CloudShell in 5 minutes - An overview of Cloud Shell capabilities.

Big Data, Analytics, ML&AI

Google enabled History based Optimization for Queries - How to save Time and Costs trough automated History based Optimization.

Measure your BigQuery costs by table - Using INFORMATION_SCHEMA to get information about query costs per table.

Unlock the Power of Machine Learning Without Coding: A Beginner’s Guide to BigQuery ML - Demystifying machine learning for data analysts — build predictive models directly within your data warehouse.

Deterministic Generative AI with Gemini Function Calling in Java - Gemini's Function Calling blends the flexibility of generative AI with the precision of traditional programming. This allows for controlled and predictable outputs, as demonstrated in the Java Cloud Function that standardizes addresses using Gemini and an external API.

Making sense of Vector Search and Embeddings across GCP products - An overview of vector search and embedding through the GCP products.

From GenAI to Insights from Your Customers (Part 1) - Analyzing customer complaints with Gen AI models.

Shh, It’s Free: But Let’s Not Tell Google! Exploring Gemini’s Multimodal Capabilities on Vertex AI - Consider this your backdoor pass into a free club, where the only membership requirement is your curiosity.

Calling Gemma with Ollama, TestContainers, and LangChain4j - To run the Gemma language model locally without installing Ollama, use TestContainers to create a container that pulls in the Gemma model. Then, use LangChain4j to interact with the container and generate responses from Gemma. This approach provides easy integration with Java applications while eliminating the need for local Ollama installation.

Create multimodal conversational experiences with Google Cloud Dialogflow CX and Gemini Vision - Boosting your digital assistant by analyzing images with Gemini Vision.

Navigating Data Transformations: Insights from a Lead Data Engineer on Dataform, dbt in BigQuery - Comparing experience of using dbt and Dataform with BigQuery.

How to Augment Text Data with Gemini through BigQuery DataFrames - Data augmentation is a technique used in machine learning to increase the size of a dataset by creating new data out of existing data.

Slides, Videos, Audio

Security Podcast - #166 Workload Identity, Zero Trust and SPIFFE (Also Turtles!).

Releases

Advisory Notifications - Advisory Notifications for users using Google Cloud without an organization is now in General Availability.

AlloyDB - The following extensions are added to the extensions supported by AlloyDB: autoinc, insert_username, moddatetime, pg_background, pg_squeeze, tcn. The extension pgvector is updated to version 0.6.0. AlloyDB Studio is now generally available (GA).

Google Distributed Cloud Bare Metal - 1.15 & 1.16 & 1.28. A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes.

GDCV for VMware - A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes.

Apigee X - On April 3, 2024, we released an updated version of Apigee. With this release, Apigee expanded its support for data residency to additional regions in Asia-Pacific and the Middle East. On April 2, 2024, we announced an increase in the rate limits for the Spike Arrest policy. On April 1, 2024, we released an updated version of Apigee. With this release, Apigee expanded its support for data residency to additional regions in Canada.

Cloud Architecture Center - (New guide) Infrastructure for a RAG-capable generative AI application using GKE: Design the infrastructure to run a generative AI application with retrieval-augmented generation (RAG) using GKE, Cloud SQL, and open source tools like Ray, Hugging Face, and LangChain.

Google Cloud Armor - The Cloud Armor premium service tier "Cloud Armor Managed Protection Plus" has been renamed to "Cloud Armor Enterprise." This change is being made to reflect the evolution of Cloud Armor's enterprise features. Cloud Armor Enterprise Paygo (formerly Managed Protection Plus Paygo) is now Generally Available.

Artifact Registry - Artifact Analysis automatic scanning for Ruby, Rust, .NET and PHP vulnerabilities in container images is now generally available.

Assured Workloads for Goverment - The new control packages are now available in Preview. You can now create Assured Workloads folders from Resource Manager's Manage resources page in the Google Cloud console.

Backup and DR Service - Backup and DR Service added support to automatically protect your compute engine instances using Google Cloud tags.

Batch - You can set maximum time limits for tasks and runnables. When a job fails due to exceeding a timeout, the job's logs don't indicate whether the failure was caused by the relevant task's timeout or the relevant runnable's timeout.

BigQuery ML - The following BigQuery ML features are now in preview: Performing supervised tuning on a remote model based on a Vertex AI text-bison large language model (LLM).

BigQuery - Differential privacy is now generally available (GA). You can now use BigLake to access Delta Lake tables. The allow_non_incremental_definition option and max_staleness option for materialized views are now generally available (GA). You can now perform model monitoring in BigQuery ML. BigQuery data clean rooms with analysis rules and enhanced usage metrics are now generally available (GA). Join restrictions, list overlap, differential privacy with privacy budgeting, and aggregation thresholding are now enforceable in BigQuery data clean rooms using analysis rules. Collation now supports the following generally available (GA) features: The underscore in the LIKE operator. You can now configure materialized views with tables enabled for change data capture (CDC) streaming update and delete operations. You can now enable, disable, and analyze history-based optimizations for queries. BigQuery Studio is now available in the South Carolina (us-east1) region to manage versions of code assets such as notebooks and saved queries. The BigQuery Data Transfer Service for Search Ads 360 now supports the new Search Ads 360 Reporting API.

Carbon Footprint - Scope 2 market-based emissions data is now launched in Preview.

Chronicle - Curated Detections has been enhanced with new detection content for Cloud Threats category. On or after May 1, 2024, in an effort to improve enrichment quality, the enrichment process using telemetry events and entities will prioritize values set by parsers over values from aliases in unenriched events.

Compute Engine - Generally available: Simplify block storage management for Compute Engine instances with Hyperdisk Storage Pools. Compute Engine is not affected by CVE-2024-3094.

Config Connector - Config Connector version 1.115.0 is now available. Improved support for AlloyDB, by adding new fields to AlloyDBCluster and AlloyDBInstance. AlloyDBCluster Added spec.clusterType field. AlloyDBInstance Added spec.instanceTypeRef field.

Data Fusion - Cloud Data Fusion is available in the africa-south1 region. The Google Sheets plugin version 1.4.2 (bundled with the Google Drive plugins) is available in all Cloud Data Fusion versions.

Database Migration Service - Database Migration Service now supports physical backup files created by using the Percona XtraBackup utility for homogeneous MySQL to Cloud SQL for MySQL migrations.

Dataflow - The following Dataflow templates are generally available (GA): Google Cloud to Neo4j Pub/Sub to Datadog.

Dataform - You can now use Customer-Managed Encryption Keys (CMEK) to protect repositories in Dataform.

Dataproc Metastore - Dataproc Metastore now supports managed migrations. Dataproc Metastore now supports autoscaling.

Dataproc Serverless - New Dataproc Serverless for Spark runtime versions: 1.1.57 1.2.1 2.0.65 2.1.44 2.2.1. Added bigframes Python package by default in the Dataproc Serverless for Spark runtime versions 1.2 and 2.2.

Dataproc - The following previously released sub-minor versions of Dataproc on Compute Engine images have been rolled back and can only be used when updating existing clusters that already use them: 2.0.97-debian10, 2.0.97-rocky8, 2.0.97-ubuntu18 2.1.45-debian11, 2.1.45-rocky8, 2.1.45-ubuntu20, 2.1.45-ubuntu20-arm 2.2.11-debian12, 2.2.11-rocky9, 2.2.11-ubuntu22.

Datastore - Support for Customer-managed encryption keys (CMEK).

Dialogflow - Vertex AI Conversation: You can now create a data store in one language that is connected to an agent that uses different languages. Dialogflow CX: Language auto detect is now available for chat conversations. Dialogflow CX: Call companion is now generally available with new user interface settings. Dialogflow CX: Dialogflow CX phone gateway is now generally available. Vertex AI Conversation: Data stores now support parse and chunk configuration.

Cloud Data Loss Prevention - If you opted to publish your data profiles to Security Command Center, you can configure Security Command Center to prioritize resources automatically according to the sensitivity of the data that the resources contain.

Document AI - Fine tuning generative AI models within the Custom Extractor is now supported in GA.

Eventarc - Eventarc support for creating triggers for direct events from Cloud Deploy is generally available (GA).

Cloud Firestore - Support for Customer-managed encryption keys (CMEK). You can now use Firestore to perform K-nearest neighbor (KNN) vector searches.

Gemini - The Gemini Code Assist code generation feature now allows users to generate code repeatedly at the same cursor location. Fixed an issue where handling very small .csv, .tsv, and .jsonl files crashed the Gemini Cloud Code extension. Fixed an issue where empty files weren't being included as chat context. Fixed telemetry data issue in the VSCode Gemini Code Assist plug-in.

Integration Connectors - The following connectors are now generally available (GA): CockroachDB Email MailChimp Office365 SingleStore To view the list of all the GA connectors, see Connectors in GA. Integration Connectors is now available in the following additional regions: Delhi (asia-south2) Warsaw (europe-central2) Toronto (northamerica-northeast2) Columbus (us-east5) Las Vegas (us-west4) Dallas (us-south1) Santiago (southamerica-west1) For the list of all the supported regions, see Locations.

Google Kubernetes Engine - GPU NVIDIA Multi-Process Service (MPS) is available in version 1.27.7-gke.1088000 and later, which allows multiple workloads to share a single NVIDIA GPU hardware accelerator with NVIDIA MPS. A Denial-of-Service (DoS) vulnerability (CVE-2023-45288) was recently discovered in multiple implementations of the HTTP/2 protocol, including the golang HTTP server used by Kubernetes. The GKE compliance dashboard now offers compliance evaluation for CIS Kubernetes Benchmark 1.5, Pod Security Standards (PSS) Baseline, and PSS Restricted standards in Preview. GKE threat detection is now available in Preview. Observability for Google Kubernetes Engine: Added a dashboard for Tensor Processing Unit (TPU) metrics on the Observability tab of both the cluster listing and cluster details pages for GKE clusters.

Load Balancing - The cross-region internal Application Load Balancer supports backends in multiple regions, provides seamless cross-region failover using Cloud DNS routing policies, and is globally accessible by clients from any Google Cloud region, on premise, or other clouds. The cross-region internal proxy Network Load Balancer supports backends in multiple regions, provides seamless cross-region failover, and is globally accessible by clients from any Google Cloud region, on premise, or other clouds. You can now configure advanced traffic management using flexible pattern matching.

Cloud Logging - The Logging query language now supports the cast and regexp_extract functions.

Memorystore for Redis Cluster - Added support for new node types, including smaller and larger nodes. Added support for AOF and RDB persistence (Preview). Added support for instance configurations (Preview).

Policy Intelligence - Policy Troubleshooter for IAM currently doesn't fetch tags for regional resources, such as Google Kubernetes Engine (GKE) clusters.

Secret Manager - Preview: Delayed destruction of secret versions is now available in Preview.

Security Command Center - Enterprise tier released to General Availability The Enterprise tier, which transforms Security Command Center into a cloud-native application protection platform (CNAPP) that combines cloud security and enterprise security operations with multicloud support, is released to General Availability. With the Enterprise tier, severity levels of certain findings are now variable In the Enterprise tier of Security Command Center, the default severity level of an active vulnerability or misconfiguration finding can change if the finding's attack exposure score changes. The following Security Health Analytics misconfiguration detectors have changed to check for overly restrictive flag values that might prevent error messages from being written to the logs: SQL_LOG_ERROR_VERBOSITY SQL_LOG_MIN_ERROR_STATEMENT_SEVERITY For the flag values that the detectors check for, see: SQL log error verbosity SQL log min error statement severity.

Sensitive Data Protection - If you opted to publish your data profiles to Security Command Center, you can configure Security Command Center to prioritize resources automatically according to the sensitivity of the data that the resources contain.

Service Extensions - Service Extensions is Generally Available for callout extensions for most Google Cloud Application Load Balancers.

SAP Solutions - SAP BTP edition of the ABAP SDK for Google Cloud Version 1.0 of the SAP BTP edition of ABAP SDK for Google Cloud is generally available (GA). BigQuery Connector for SAP version v2.6 Version 2.6 of the BigQuery Connector for SAP is generally available (GA). ABAP SDK for Google Cloud version v1.6 Version 1.6 of the ABAP SDK for Google Cloud is generally available (GA). Google Cloud's Agent for SAP version 3.2 Version 3.2 of Google Cloud's Agent for SAP is generally available (GA).

Cloud SQL MySQL - For Cloud SQL Enterprise Plus edition, you can now use advanced disaster recovery (DR) to simplify recovery and fallback processes after you perform a cross-regional failover. You can now migrate your external MySQL 5.7 and 8.0 databases into Cloud SQL for MySQL by using Percona XtraBackup physical files. You can now scale up the compute size (vCPU, memory) of a Cloud SQL Enterprise Plus edition primary instance with near-zero downtime. If your Cloud SQL Enterprise edition instance stores the transaction logs used for point-in-time recovery (PITR) on disk, then when you do an in-place upgrade to Cloud SQL Enterprise Plus edition, the storage location for the transaction logs is switched to Cloud Storage.

Cloud SQL Postgres - You can now migrate data in the tables of your database in parallel. Version 0.6.0 of the pgvector extension that's listed in the March 27 release note isn't available yet. You can now scale up the compute size (vCPU, memory) of a Cloud SQL Enterprise Plus edition primary instance with near-zero downtime.

Cloud SQL SQL Server - You can now use SSL mode instead of the legacy require_ssl setting to specify the encryption requirements for connections to your Cloud SQL for SQL Server instances.

Cloud Storage - Custom constraints for Cloud Storage are now available. Managed folders are now available in the Google Cloud Console. Google Cloud Storage now offers Dual-region Google Egress Bandwidth quotas per dual-region location.

Virtual Private Cloud - You can use Packet Mirroring to collect IPv6 traffic.