Welcome to issue #421 October 21st, 2024

News

Introducing Google Cloud’s new Vulnerability Reward Program - Google Cloud launched a new Vulnerability Rewards Program (VRP) specifically for its products and services. The program aims to incentivize security researchers to find and report vulnerabilities in Google Cloud, with a top reward of $101,010.

Accelerating partner growth with Earnings Hub and new AI resources - Google Cloud partners can now use the Earnings Hub platform to track and optimize their incentives, including rebates, discounts, funds, and credits. The platform provides a unified dashboard, incentive analysis, filterable data, and CSV download capabilities.

Google Cloud Marketplace private offer enhancements unlock enterprise and AI use cases - Google Cloud Marketplace private offers now support enterprise AI purchasing models, including provisioned throughput for generative AI models on Vertex AI. Customers can now tailor offers and payment schedules with multiple orders for the same product, meeting the needs of different business units within an enterprise.

From AI to Zero Trust: Google Cloud Security delivers comprehensive solutions for the public sector - Google Cloud Security offers comprehensive solutions to help public sector organizations strengthen their defenses against cyber threats. New announcements include FedRAMP High authorization for Google Security Operations, expanded compliance control packages for Assured Workloads, Google Cloud Audit Manager to simplify compliance audits, Gemini for Google Workspace to enhance productivity with AI, Mandiant ThreatSpace for immersive cyber range experience, and tailored cybersecurity assistance for rural hospitals.

A new era of partnership: Unveiling Google Public Sector's enhanced Partner Program - Google Public Sector launches an enhanced Partner Program to empower partners in the public sector market. The program focuses on accelerating growth, fostering collaboration, and maximizing profitability for partners. Key pillars include improved incentives, stronger training and badging, improved co-marketing, accelerated go-to-market, clear delivery framework, and commitment to quality outcomes.

Revolutionizing SQL with pipe syntax, now available in BigQuery and Cloud Logging - Pipe syntax is a new feature in BigQuery and Cloud Logging that makes SQL simpler, more concise, and more flexible. It supports the same underlying operators as standard SQL, with the same semantics and mostly the same syntax, but allows applying operators in any order, any number of times.

Get up to 100x query performance improvement with BigQuery history-based optimizations - BigQuery history-based optimizations, now generally available, can improve query performance by up to 100x. It uses statistical data from previous executions of similar queries to identify and apply additional improvements to the query execution. These optimizations are self-tuning and self-correcting, requiring no user intervention.

Beyond the basics: Build real-world gen AI skills with the latest learning paths from Google Cloud - Google Cloud Skills Boost offers four new learning paths to equip developers with real-world generative AI skills. These paths cover building and modernizing applications, integrating generative AI into data workflows, deploying and managing generative AI models, and generating smarter generative AI outputs.

Welcome to Google Public Sector Summit 2024 - Google Public Sector Summit 2024 focused on AI, cybersecurity, and data analysis for government customers. Google announced Gemini in Google Distributed Cloud for Secret and Top Secret workloads, achieving IL4/5 ATO for Air Force Cloud One, and launching a $15 million AI upskilling grant program. The summit also featured the launch of the Google Cloud NIH STRIDES Marketplace and several partnership announcements.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How to benchmark application performance from the user's perspective - In this blog post, we discuss why it's important to incorporate end-user perceived performance benchmarking into modern application development. We also show you how to use the open-source Locust tool to simulate complex user behavior as part of your end-to-end benchmarking practice. Benchmarking helps developers identify and address performance bottlenecks early, optimize performance continuously, and bridge the gap between development and production.

Sustainable silicon to intelligent clouds: collaborating for the future of computing - Google VP and Technical Fellow Parthasarathy Ranganathan and Principal Engineer Amber Huffman discuss the past and future of hyperscale computing, emphasizing the importance of cross-disciplinary co-design and collaboration. They highlight progress and opportunities in sustainability, trusted silicon, AI accelerators, and systems infrastructure, showcasing Google's contributions to the Open Compute Project (OCP). The article also explores the potential of robotics and automation in data centers to meet the next level of scale required by AI infrastructure.

Cloud CISO Perspectives: AI vendors should share vulnerability research. Here’s why - Google Cloud CISO Phil Venables discusses the importance of AI vendors sharing vulnerability research to enhance security standards and foster collaboration in the AI industry. He emphasizes the need for transparency, open discussions, and collective efforts to address AI vulnerabilities and ensure the security of AI systems and applications. Venables highlights Google's commitment to AI security research and encourages developers to normalize sharing AI security research to collectively work towards a future where AI is secure by default.

How Low Can You Go? An Analysis of 2023 Time-to-Exploit Trends - In 2023, Mandiant analyzed 138 vulnerabilities that were exploited in the wild, with a striking 97 of them being zero-days. This indicates a significant increase in zero-day exploitation compared to previous years. The average time-to-exploit (TTE) dropped dramatically to just five days, highlighting the urgency for organizations to prioritize patching and implementing robust security measures. While exploit releases and media attention do not directly predict exploitation timelines, they should still be considered alongside other factors when assessing vulnerability risk.

How We Reduced Costs on GCP by Optimizing a Single SKU: Network Inter Zone Data Transfer Out - Significantly reduced GCP costs by optimizing a single SKU: Network Inter Zone Data Transfer Out. By analyzing data transfer patterns, consolidating workloads, and leveraging zonal resources, a 45% reduction was achieved in inter-zone data transfer, resulting in monthly savings of approximately 20% for the GCP bill.

Best Practices for deploying Google Cloud VMware Engine Protected Part 2 - The article discusses best practices for deploying Google Cloud VMware Engine Protected Part 2. It covers setting up the GCBDR management console, creating storage pools, and configuring backup templates. The author also introduces Backup Vaults, a new Google Cloud service that supports immutable and retention enforced backups for GCBDR and GCVE.

App Development, Serverless, Databases, DevOps

From Cassandra to Bigtable: Database migration tips from Palo Alto Networks - Palo Alto Networks migrated their Advanced WildFire product from Apache Cassandra to Bigtable, achieving 5x lower latency and cutting their total cost of ownership by half.

Reltio's Data Plane Transformation with Spanner on Google Cloud - Reltio, a leading provider of AI-powered data unification and management solutions, migrated its data infrastructure from self-managed Cassandra to Spanner on Google Cloud. The successful migration showcases the benefits of a joint design, well-executed database migration, and deep operational collaboration between application and data layers.

How The Home Depot is innovating with Google Cloud using the ABAP SDK - The Home Depot, a leading home improvement retailer, has been using Google Cloud since 2017 to enhance customer experience, optimize SAP applications, and build an efficient supply chain. With the ABAP SDK, The Home Depot has implemented various use cases, such as boosting financial accounting accuracy with near real-time inventory data, safeguarding sensitive financial data in integrations, cutting costs and improving performance in data sharing.

Can I use Cloud Armor with Cloud Run? - Google Cloud Armor is a tool that can be used to secure Cloud Run, but it has some limitations. For example, it can be bypassed if you don't have a load balancer or backend in front of your Cloud Run service. Additionally, Cloud Armor can be expensive, with the enterprise tier costing €200 per month plus additional expenses. Cloudflare is a more affordable alternative that offers similar features and benefits.

Implementing Private Service Connect for Cloud Run Applications - Private Service Connect (PSC) allows you to securely expose Cloud Run services to other VPC networks without relying on public endpoints. This guide provides step-by-step instructions, including Terraform snippets, to configure PSC for Cloud Run applications.

AlloyDB: Effortless Embedding Generation with SQL and a quick App with Streamlit - This article provides a step-by-step guide to setting up an AlloyDB cluster and instance, enabling APIs, and creating a table for storing thought embeddings. It also demonstrates how to build a simple Streamlit app that captures user thoughts, generates embeddings using Google's embedding model, and retrieves similar thoughts from the database.

Big Data, Analytics, ML&AI

How Shopify improved consumer search intent with real-time ML - Shopify improved consumer search intent with real-time machine learning (ML) using embeddings, which translate textual and visual content into numerical vectors. These embeddings enable more accurate and context-aware search results. Shopify processes roughly 2,500 embeddings per second across image and text pipelines in near real time using Google Cloud's streaming analytics service Dataflow. This helps merchants boost sales and offer positive interactive experiences for consumers.

Demystifying BigQuery reservations — part 2 - This blog post discusses how unexpected on-demand usage can occur in a capacity-based BigQuery organization, it explores the challenges and provides best practices, including separating compute and storage projects, and monitoring roles with the bigquery.jobs.create permission, and avoiding unexpected locations.

Running a Modern Data Platform at Nando’s — Part 2/2 — Data Mesh and Data Contracts - Nando's journey to a modern data platform involved consolidating data into a single source of truth, initially managed by a central data team. However, they realized the limitations of this approach and adopted a data mesh architecture, empowering domain teams to own and transform their data using tools like Dataform and BigQuery. To ensure data consistency and track data consumption, they created a Data Contract Service that automatically detects and manages external and cross-pond contracts.

Enhancing BigQuery UX with a new Chrome Extension a data analytics - The "Enhance BigQuery UX" Chrome extension simplifies daily interactions with BigQuery's interface. It offers keyboard shortcuts for tab navigation, on-demand compute pricing estimation, a SQL formatter for readability, and easy tab renaming. By integrating this extension, users can save time, manage costs effectively, and enjoy a more efficient and organized BigQuery experience.

Mastering BigQuery’s Pipe Syntax: Streamlining Your SQL Queries - BigQuery’s Pipe Syntax modernizes SQL, making it more intuitive and easier to work with. It simplifies query writing, improves readability, and maintains the strengths of traditional SQL.

dbt-bigquery-monitoring: Monitoring BigQuery compute and storage with dbt - An introduction to the dbt package designed to help you with BigQuery monitoring.

How to standardize GA4 data with dbt - This article provides a guide to transforming raw GA4 data into a polished data mart using dbt. It covers setting up the GA4 to BigQuery connection, applying data transformations, and building refined tables for marketing insights.

BigQuery time-series forecasting using ARIMA_PLUS and ARIMA_X - BigQuery ML's ARIMA_PLUS and ARIMA_PLUS_XREG models offer powerful tools for time series forecasting within the familiar SQL environment. These models automate complex processes and incorporate valuable features, making sophisticated forecasting accessible to a wider audience. Whether predicting future sales, analyzing website traffic, or forecasting resource needs, these models empower data-driven decisions with the scalability and reliability of BigQuery.

MLOps made easy with Dataform & BigQuery ML— Part 1 - Dataform and BigQuery ML can be used together to create end-to-end machine learning pipelines that are easy to use for data analysts and analytics engineers. Dataform orchestrates the pipeline, while BigQuery ML handles the heavy lifting of training, evaluating, and predicting models. This combination allows for repeatable and versioned ML pipelines that can be scheduled to run at regular intervals.

Various

A founder’s guide to Google for Startups programs - Google for Startups offers a variety of programs tailored to different stages and needs of startups, including accelerators, founder funds, cloud program, growth academy, AI academy, founders on campus, and flex and enterprise agreement programs. These programs provide expert mentorship, training, cloud credits, support, and access to Google's resources to help startups accelerate their growth and tackle specific challenges.

Slides, Videos, Audio

Kubernetes Podcast - #239 Container Security, with Michele Chubrika.

Security Podcast - #194 Deep Dive into ADR - Application Detection and Response.

Releases

Google Distributed Cloud Bare Metal - 1.28. Release 1.28.1100-gke.94 Google Distributed Cloud for bare metal 1.28.1100-gke.94 is now available for download. Fixed an issue where the control plane VIP might become unavailable because Keepalived didn't check correctly that the VIP is on a node with a responsive HAProxy. The following container image security vulnerabilities have been fixed in 1.28.1100-gke.94: High-severity container vulnerabilities: CVE-2024-39487 CVE-2024-41040 CVE-2024-41046 CVE-2024-41049 CVE-2024-41059 CVE-2024-41070 CVE-2024-42104 CVE-2024-42148 Medium-severity container vulnerabilities: CVE-2016-3709 CVE-2024-7264 CVE-2024-36901 CVE-2024-36938 CVE-2024-41009 CVE-2024-41012 CVE-2024-41055 CVE-2024-41063 CVE-2024-41064 CVE-2024-42101 CVE-2024-42102 CVE-2024-42131 CVE-2024-42137 CVE-2024-42152 CVE-2024-42153 CVE-2024-42154 CVE-2024-42157 CVE-2024-42161 CVE-2024-42223 CVE-2024-42224 CVE-2024-42229 CVE-2024-42232 CVE-2024-42236 CVE-2024-42244 CVE-2024-42247 Low-severity container vulnerabilities: CVE-2022-2309 CVE-2024-41007 GHSA-xr7q-jx4m-x55m. Known issues: For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

GDCV for VMware - Google Distributed Cloud (software only) for VMware 1.28.1100-gke.91 is now available for download. The following issue is fixed in 1.28.1100-gke.91: Fixed the known issue that caused gkectl to display false warnings on admin cluster version skew.

App Hub - App Hub supports global infrastructure resources with global applications in Preview.

Artifact Registry - Artifact Registry remote repositories support setting standard Artifact Registry repositories as upstreams for supported formats. Organization policy constraints for Artifact Registry is available in General Availability.

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs.

Backup and DR Service - Backup and DR Service 11.0.13.278 is now available to update your backup/recovery appliance. Backup and DR Service added support to view connector version logs in Cloud Logging. Backup and DR Service added support to view connector version reports in BigQuery.

BigQuery - You can now use fine-grained DML to optimize the execution of UPDATE, DELETE, and MERGE statements on tables.

Chronicle - Google SecOps has updated the list of supported default parsers.

Chronicle Security Operations - Google SecOps has updated the list of supported default parsers.

Cloud Composer - A new Cloud Composer release has started on October 16, 2024. Fixed a bug where upgrading a private IP environment could fail because of an invalid CIDR range. The default version of Airflow is changed to 2.9.3. Airflow 2.9.1 is no longer included in Cloud Composer images and builds. New Airflow builds are available in Cloud Composer 3: composer-3-airflow-2.9.3-build.3 (default) composer-3-airflow-2.7.3-build.19. Cloud Composer 2.9.7 images are available: composer-2.9.7-airflow-2.9.3 (default) composer-2.9.7-airflow-2.7.3.

Compute Engine - End of life: On October 31, 2024, SLES 12 SP5 and SLES 12 SP5 for SAP are reaching end of life and the images will be deprecated on Google Cloud. Generally available: In addition to the A3 High machine type that has 8 NVIDIA H100 GPUs attached, we now have smaller machine types available that have 1, 2, or 4 NVIDIA H100 GPUs attached.

Config Connector - Config Connector version 1.124.0 is now available. The direct resource development guide is now available for contributors To improve the Config Connector resource development process, we have a new development guide to contributing resources to Config Connector with the direct reconciliation process. RedisCluster is promoted from alpha to beta (Direct Reconciler). CertificateManagerDNSAuthorization Add the spec.Location field. ComputeForwardingRule Added spec.target.googleApisBundle field (allowed values are all-apis or vpc-sc). CertificateManagerDNSAuthorization is migrated from the Terraform-based to the new Direct controller to enhance reliability and performance. New Alpha Resources (Direct Reconciler) PrivilegedAccessManagerEntitlement BigQueryAnalyticsHubDataExchange.

Contact Center AI Platform - Version 3.27 is released All release notes published on this date are part of version 3.27. Spelling and grammar check The agent adapter now provides spelling and grammar checking. Voice detection for auto-answer You can now configure auto-answer to listen for an agent's voice after a call is connected. New destinations for incoming SIP header data and CCAI Platform metadata You can now pass incoming SIP header data and CCAI Platform metadata to session metadata files and CRM records. Emergency calling Agents in the US and Canada can now make calls to emergency services. Conversational Agents (Dialogflow CX) is supported in additional regions Conversational Agents (Dialogflow CX) is now supported in additional regions. Fixed an issue where queue-level caller announcements were not working properly. Fixed an issue where multiple contacts could be created for the same contact.

Dataplex - Some of the BigQuery metadata that is stored in Dataplex Catalog is changing. Dataplex is available in Dammam (me-central2).

Dataproc - Dataproc Clusters created with image versions 2.0.57+, 2.1.5+, or 2.2+: Secondary workers' control plane operations are made by the Dataproc Service Agent service account (service-@dataproc-accounts.iam.gserviceaccount.com).

Datastream - Datastream is now available in the us-south1 (Dallas) region.

Cloud Deploy - You can now automatically retry failed rollouts, and automatically roll back to the most recent successful rollout, in preview.

Anti Money Laundering AI - The API is now available in the australia-southeast1 region.

Gemini - General bug fixes and improvements to code transformation for the VS Code Gemini Code Assist extension. Improved error handling for the IntelliJ Gemini Code Assist plugin.

Integration Connectors - The Configure private connectivity documentation has been restructured to include information about the following networking patterns: Public network connectivity Private network connectivity Network connectivity in Google Cloud VPC Private connectivity for on-premise or other cloud providers Network connectivity in Google Cloud managed services Connectivity to services hosted in serverless environment.

KMS - You can now use custom constraints with Organization Policy to provide more granular control over specific fields for some Cloud KMS resources.

Google Kubernetes Engine - (2024-R40) Version updates GKE cluster versions have been updated. You can now use NVIDIA H100 80GB GPUs on GKE in the following smaller machine types: a3-highgpu-1g (1 GPU) a3-highgpu-2g (2 GPUs) a3-highgpu-4g (4 GPUs) These machine types are available through Dynamic Workload Scheduler Flex Start mode, Spot VMs in GKE Standard mode clusters, or Spot Pods in GKE Autopilot mode clusters. The new release of the GKE Gateway controller (2024-R2) is now generally available. In GKE clusters with the control plane running version 1.29.1-gke.1425000 or later, TPU slice nodes support SIGTERM signals that alert the node of an imminent shutdown. In GKE version 1.31.1-gke.1621000 and later, the kube_pod_resource_request metric and the kube_pod_resource_limit metric are exported as part of the the scheduler metrics package. On GKE Autopilot clusters running version 1.30 and later, partner workloads that set AppArmor profiles might unexpectedly be rejected at admission. You can now create workloads with multiple network interfaces in GKE Autopilot clusters running version 1.29.5-gke.1091000 and later or version 1.30.1-gke.1280000 and later. For newly-created VPC Peering-based clusters running version 1.27 or later, traffic from the kube-apiserver to nodes routes through the Konnectivity service.

GKE new features - You can now use NVIDIA H100 80GB GPUs on GKE in the following smaller machine types: a3-highgpu-1g (1 GPU) a3-highgpu-2g (2 GPUs) a3-highgpu-4g (4 GPUs) These machine types are available through Dynamic Workload Scheduler Flex Start mode, Spot VMs in GKE Standard mode clusters, or Spot Pods in GKE Autopilot mode clusters. The new release of the GKE Gateway controller (2024-R2) is now generally available. In GKE clusters with the control plane running version 1.29.1-gke.1425000 or later, TPU slice nodes support SIGTERM signals that alert the node of an imminent shutdown. You can now create workloads with multiple network interfaces in GKE Autopilot clusters running version 1.29.5-gke.1091000 and later or version 1.30.1-gke.1280000 and later.

Load Balancing - You can now use the Google Cloud Console to create the following load balancers in Premium Tier: Regional external Application Load Balancer Regional external proxy Network Load Balancer Previously, only Standard Tier support was available in the Console. Previously, the classic external Application Load Balancer had lenient HTTP/2 request parsing that did not reject requests containing certain invalid characters in the request path.

Cloud Monitoring - You can now use the Monitoring API to configure a metric-based alerting policy to send notifications when incidents are closed.

Secret Manager - Creating regional secrets using Secret Manager is now in Generally Availability (GA).

Security Command Center - The VMTD disabled finding category from Virtual Machine Threat Detection is no longer available. Toxic combination findings are generally available.

SAP Solutions - New SAP certification for operating system: SLES 15 SP6 for SAP For use with SAP HANA and SAP NetWeaver on Google Cloud, SAP has certified the operating system SUSE Linux Enterprise Server (SLES) 15 SP6 for SAP.

Cloud Spanner - Spanner Graph now supports the following functions: DESTINATION_NODE_ID(): gets a unique identifier for a graph edge's destination node. Spanner now supports customer-managed encryption keys (CMEK) to protect databases in custom, dual-region, and multi-region instance configurations. Spanner now offers usage statistics for database splits along with the associated System insights dashboard to help you identify hotspots on affected rows in your database. Directed reads are Generally Available. Query Optimizer version 7 is generally available and is the default optimizer version.

Cloud SQL MySQL - Cloud SQL for MySQL now supports minor version 8.0.39.

Cloud Storage - Hierarchical namespace for Cloud Storage buckets is generally available (GA).

VMware Engine - Stretched private clouds using `ve2' node types are now available in the following region: Frankfurt, Germany, Europe (europe-west3). Added missing release notes for stretched private cloud availability using ve2 node types in Sydney, Australia, APAC (australia-southeast1-b).

VPC Service Controls - Updated the correct support status for the following integration in the Supported products and limitations page: Dialogflow is in Preview stage. Preview stage support for the following integration: Audit Manager.