Welcome to issue #416 September 16th, 2024

News

Introducing backup vaults for cyber resilience and simplified Compute Engine backups - Google Cloud Backup and DR service introduces backup vaults for immutable and indelible backups, ensuring data security against tampering and unauthorized deletion.

Cut through the noise with new log scopes for Cloud Observability - Log scopes in Cloud Observability allow you to manage and analyze your organization's logs more efficiently. They are named collections of logs of interest within the same or different projects, made up of groups of log views that control and grant permissions to a subset of logs in a log bucket. Log scopes can be used to correlate metrics with logs from the same application or isolated environments, providing a more focused and relevant view of your telemetry data.

Announcing the 11th Annual Flare-On Challenge - The Flare-On Challenge is an annual reverse engineering contest held by the FLARE team. This year marks its 11th year running and will feature 10 challenges covering various architectures including Windows, Linux, JavaScript, .NET, YARA, UEFI, Verilog, and Web3. The contest will run for six weeks from September 27th to November 8th, 2024. Successful participants will receive a prize and have their names etched into the Hall of Fame on the Flare-On website.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How EA Sports protects their game servers with Cloud Armor - Electronic Arts (EA) chose Google Cloud Armor to protect its game servers and enhance its DDoS resiliency. EA Sports uses advanced network DDoS protection in conjunction with custom network edge security policies to create a set of security rules to allow or deny traffic at the edge of the network according to user-specified filters.

Want to save on GKE block storage costs? Hyperdisk Storage Pools can help - Hyperdisk Storage Pools, a pre-purchased collection of capacity, throughput, and IOPS, can be provisioned to applications as needed, optimizing operations and cost. Hyperdisk Storage Pools help lower storage-related Total Cost of Ownership (TCO) by as much as 30-50% and are available for use on Google Kubernetes Engine (GKE).

Safer by default: Automate access control with Sensitive Data Protection and conditional IAM - Google Cloud’s Sensitive Data Protection can automatically discover sensitive data assets and attach tags to your data assets based on sensitivity. Using IAM conditions, you can grant or deny access to data based on the presence or absence of a sensitivity level tag key or tag value.

Protecting Multi-Cloud Resources in the Era of Modern Cloud-Based Cyberattacks - In the era of multi-cloud adoption, organizations face new security challenges due to expanded attack surfaces and complex permission structures. Mandiant's white paper explores critical risks and provides a framework for establishing a robust security posture in multi-cloud environments. The paper examines real-world attack scenarios and introduces a cloud-agnostic tiered security model to protect privileged access to critical assets.

Insights on Cyber Threats Targeting Users and Enterprises in Mexico - Mexico faces a complex cyber threat landscape with global and local threats targeting critical sectors and exploiting digital infrastructure. Cyber espionage operations from multiple nations, including China, North Korea, and Russia, target users and organizations in Mexico.

Networking in Google Cloud — Network Connectivity Center - Exploring Google Cloud’s Network Connectivity Center to integrate multiple VPC based on their use case.

Next-Gen Cloud Network Security: TLS inspection with NGFW Enterprise - NGFW Enterprise enables TLS inspection by acting as a "man-in-the-middle," splitting client-server connections and inspecting TLS traffic. TLS inspection policies link NGFW Enterprise to Certificate Authority Service (CAS) and Certificate Manager Trust-Configs for certificate management. To inspect TLS traffic, create TLS inspection policies and enable TLS inspection for specific traffic patterns within Firewall Policies. Trusting CAS certificates requires manual distribution to VMs, as they are not automatically trusted within GCP.

Using VPC Service Controls to isolate data analytics use cases in Google Cloud - Pushing the limits of VPC Service Controls to implement complex data access requirements.

GKE Enterprise: A platform engineered for success with Kubernetes - GKE Enterprise is a multi-cluster management platform that simplifies the management of multiple Kubernetes clusters. It provides visibility into the cluster landscape, reduces operational overhead, enforces desired state for compliance and governance, and offers cost optimization recommendations. Key features include cluster onboarding, fleet management, policy enforcement, configuration synchronization, and cost visibility.

Self-managed GPU Monitoring Stack on Google Cloud with DCGM, Prometheus, and Grafana - Overview.

App Development, Serverless, Databases, DevOps

Using Google Workflows and Cloud Functions to build an LLM-based app - This article describes the architecture of Biglang.app, a language-learning tool that leverages LLMs to enable users to create exercises from any content they like.

Cloud Run job with a Python Module - The article explains how to create a Cloud Run job with a Python module, including the project structure, code logic, CI/CD setup, and how to execute the job directly or via a Cloud Scheduler.

Managing Python Dependencies in Cloud Functions - Python Packaging to Google Artifact Registry.

Deploy NodeJS Typescript to Google App Engine - A walk-through deploying a NodeJS Typescript application to Google App Engine.

End-to-End Database ChangeManagement on CloudSQL with Liquibase and Github Actions - This article demonstrates how Liquibase can make changes to our database schema on Cloud SQL with Github Actions.

Is Your CCAIP/DFCX Monitoring Strategy Solid? - Are you monitoring the right metrics for your CCAIP/DFCX solution? This guide covers the key metrics you need to track.

Cloud Spanner — Performance comparison between SQL and READ - Cloud Spanner provides two APIs to query data: SQL API and READ API. This article compares the performance of both APIs in terms of CPU usage, latency, and QPS for different types of queries: searching for a single key, searching ten rows using range scan, and searching ten rows using a list of keys. The results show that there is no significant difference in performance for single key search, but READ API has a significant advantage in CPU usage for range scan and multi-splits queries.

Harnessing Google AI’s Potential Within SAP: Introducing the Vertex AI SDK for ABAP - The Vertex AI SDK for ABAP enables developers to integrate Google Cloud's Vertex AI platform with SAP systems. Developers can build data ingestion pipelines, create Retrieval Augmented Generation (RAG) based workflows, augment model context with SAP function module calls, manage feature stores, and prototype SAP use cases.

Big Data, Analytics, ML&AI

Brewing innovation: Google Cloud and Oracle powering AI-driven experiences - Coffee Nirvana, an AI-powered solution built on Oracle Database on Google Cloud, simplifies the search for the perfect coffee by understanding nuanced taste preferences and providing real-time inventory information. It leverages Retrieval Augmented Generation (RAG) to deliver personalized recommendations and integrates with Google Maps for visualizing stockists.

Regnology Automates Ticket-to-Code with agentic GenAI on Vertex AI - Regnology, a global technology pioneer in regulatory reporting solutions, has developed an innovative AI-powered tool called the Ticket-to-Code Writer. This tool automates the conversion of bug tickets into actionable code, significantly streamlining the software development process.

Experimenting with BigQuery data compression - This blog post explores factors that influence compression ratios in BigQuery, including record reordering, dictionary usage, string sorting, and comparisons with other file formats.

Data Conversations with BigQuery Connectors and Looker Studio - This blog post explores how to create a system that allows users to ask questions about their Jira or Confluence data in natural language, without the need for SQL expertise. By utilizing BigQuery, Looker Studio, and BigQuery Data Connectors, this approach democratizes data access and makes data insights accessible to a wider audience within the organization.

My Learning Journey on LookML - Sharing a learning journey on LookML, a modeling language used in Looker for building data models and generating SQL queries.

The AI detective: The Needle in a Haystack test and how Gemini 1.5 Pro solves it - The Needle in the Haystack test challenges AI models to retrieve specific information from large amounts of data. Google's Gemini 1.5 Pro excels in this test, demonstrating near-perfect recall of information within vast contexts of text, video, and audio. With its advanced architecture, multimodal capabilities, and innovative training techniques, Gemini 1.5 Pro pushes the boundaries of AI's ability to understand and engage in meaningful conversations.

Next-gen search and RAG with Vertex AI - Vertex AI offers a comprehensive suite of tools and services to build next-gen search applications. It provides out-of-the-box solutions for building semantic and hybrid search applications, as well as DIY APIs for developers who want to construct their own end-to-end RAG solutions. Vertex AI Search can be used to tackle analytical queries, and it integrates with other Google Cloud capabilities such as Vertex AI Agent Builder and BigQuery data canvas.

Test it out: an online shopping demo experience with Gemini and RAG - An online shopping demo showcases how Gemini, a large language model, can enhance the shopping experience by providing personalized recommendations. Retrieval-Augmented Generation (RAG) improves the accuracy of Gemini's responses by incorporating relevant data from an external database, ensuring that recommendations are based on actual products in the store's inventory.

15 BigQuery Cheat Sheets I wish I knew Earlier - BigQuery Tips For Data Engineers.

Building a Powerful GCP RAG System with LangChain, Vertex AI, and Ensemble Retrieval - The blog covers integrating Vertex AI Retriever for document retrieval, combining multiple data sources with Ensemble Retriever, and using Vertex AI LLMs and LangChain for response generation. It emphasizes the importance of prompt engineering and provides a code snippet for a complete RAG pipeline.

A Patent Search Agent with LangChain and Reasoning Engine - This blog post presents a method for building a patent search agent using LangChain, AlloyDB, and Vertex AI Reasoning Engine. The agent leverages the power of large language models (LLMs) to retrieve and analyze relevant patents based on user search criteria.

Google Cloud RAG API - 30 lines of code is all you need for RAG. The easiest way to get started with RAG.

Intro to Ray on GKE - An overview of Ray and Ray Operator for GKE.

Lessons Learned from Migrating Huge BigQuery Datasets Across Regions

Various

How I Passed the Google Cloud Professional Data Engineer Certification Exam — August 2024 - A simple and Comprehensive guide to becoming a GCP Data Engineer.

Acing the Google Cloud ML Engineer Exam: Field-Tested Strategies for Success

Slides, Videos, Audio

Security Podcast - #189 How Google Does Security Programs at Scale: CISO Insights.

GCP Bytes Podcast (former GCP Life Podcast) - #1 We are back! - Luna Lake CPU, Chromebook Showcase, GDG Sydney Study Jam, Melbourne Devfest, Cloud Run Functions, Backup Data Vault, Pluralsight, Redis & ValKey, NBN Speeds, DC Investment, Android15, EU Antitrust, Imagen3, GROK2, Flux.1, Amazon AI Hires, Tech Mahindra & Google.

Releases

Access Approval - Access Approval supports Database Center in the Preview stage. Access Approval supports Cloud Data Fusion in the GA stage.

Google Distributed Cloud Bare Metal - 1.29. Release 1.29.500-gke.163 Google Distributed Cloud for bare metal 1.29.500-gke.163 is now available for download. The following container image security vulnerabilities have been fixed in 1.29.500-gke.163: High-severity container vulnerabilities: CVE-2024-7348 GHSA-87m9-rv8p-rgmg CVE-2023-47038 Medium-severity container vulnerabilities: CVE-2024-6104 GHSA-mh55-gqvf-xfwm CVE-2023-5981 Low-severity container vulnerabilities: CVE-2022-48303. Known issues: For information about the latest known issues, see Google Distributed Cloud for bare metal known issues in the Troubleshooting section.

GDCV for VMware - Google Distributed Cloud (software only) for VMware 1.28.900-gke.113 is now available for download. The following issues are fixed in 1.28.900-gke.113: Fixed the known issue where updating DataplaneV2 ForwardMode doesn't automatically trigger anetd DaemonSet restart.

Apigee Advanced API Security - Delay in score generation for Risk Assessment v2 with VPC-SC-enabled organizations only This issue impacts Risk Assessment v2 only, which is in preview. On September 10, 2024 we released an updated version of Advanced API Security. Proxy-specific security actions You can now create security actions that apply only to one or more specified proxies.

Apigee X - On September 12, 2024, we released an updated version of Apigee. With this release, Apigee supports Workforce Identity Federation. Bug ID Description 338285095 Fixed a problem where apps associated with an AppGroup did not appear in the Apps list in the Apigee UI in Cloud Console. PEM parsing error in JWT/JWS policies due to non-standard format For Apigee and Apigee hybrid versions 1.13 and higher, any deviations in the required PEM format of keys used in Apigee JWS or JWT policies may result in a parsing error.

Apigee Hybrid - v1.12.2. hybrid v1.12.2 On September 13, 2024 we released an updated version of the Apigee hybrid software, 1.12.2. Bug ID Description 362305438 You can now add additional env variables to the runtime component. Bug ID Description N/A Security fixes for apigee-cassandra-backup-utility and apigee-hybrid-cassandra.

Application Integration - The XSLT Transform data transformer function is now available.

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs. The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs.

Assured Workloads Access Approval - Access Approval supports Database Center in the Preview stage. v1. Access Approval supports Cloud Data Fusion in the GA stage.

Assured Workloads Access Transparency - Access Transparency supports Database Center in the Preview stage.

BigQuery - You can now use the partial ordering mode in BigQuery DataFrames to generate more efficient queries. You can now use Terraform to manage IAM tags on datasets and tables. The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery, including Creative, Partner, and Advertiser tables.

Chronicle - The following new YARA-L 2.0 functions are available in Rules and Search: arrays.concat arrays.join_string arrays.max arrays.min arrays.size arrays.index_to_int cast.as_bool cast.as_float math.ceil math.floor math.geo_distance math.is_increasing math.pow math.random strings.contains strings.count_substrings strings.extract_domain strings.extract_hostname strings.from_hex strings.ltrim strings.reverse strings.rtrim strings.trim strings.url_decode timestamp.as_unix_seconds timestamp.now The following new YARA-L 2.0 functions are available in Rules: hash.sha256 window.avg window.first window.last window.median window.mode window.stddev window.variance Details on function signatures and behavior can be found in YARA-L2.0 Function Syntax Reference Documentation.

Chronicle SOAR - Release 6.3.17 is now in General Availability. Release 6.3.18 is currently in Preview. Due to technical issues, the SOAR version has been rolled back to Release 6.3.16.

Access Transparency - Access Transparency supports Database Center in the Preview stage.

Config Connector - Config Connector version 1.122.0 is now available. The state-into-spec field now defaults to Absent in all Config Controller clusters. RedisCluster (Alpha) now uses direct reconciliation. SQLInstance now uses direct reconciliation. Added RedisCluster (Alpha) resource for service Redis. ContainerCluster The spec.nodeConfig.taint can be updated in place in lieu of destroying and recreating the object. ContainerNodePool The spec.nodeConfig.taint can be updated in place in lieu of destroying and recreating the object. SQLInstance Add the spec.cloneSource field to clone a SQLInstance. RunJob Add the spec.template.template.volumes[].cloudSqlInstance field to configure Cloud SQL instance.

Dataform - You can now set a default Dataform customer-managed encryption keys (CMEK) key for your project to encrypt multiple Dataform repositories with the same CMEK key.

Dataproc Serverless - New Dataproc Serverless for Spark runtime versions: 1.1.78 1.2.22 2.2.22. Dataproc Serverless for Spark: Fixed a bug that caused some batches and sessions to fail to start when using the premium compute tier.

Cloud Data Loss Prevention - The discovery service of Sensitive Data Protection now supports Amazon S3. The DOD_ID_NUMBER infoType detector is available in all regions.

Document AI - The custom extractor models pretrained-foundation-model-v1.2-2024-05-10 and pretrained-foundation-model-v1.3-2024-08-31 now support ML Processing in US/EU regions.

Integration Connectors - Connectors for Google services When you use OAuth 2.0 - Authorization code authentication in Google services connectors, you can select the applicable scopes from the drop-down. The following connectors are now generally available (GA): Confluence Dataverse DocuSign Excel Online GitHub IBM Db2 Jira Service Management SAP Business One UKG WooCommerce To view the list of all the GA connectors, see Connectors in GA.

Google Kubernetes Engine - (2024-R35) Version updates GKE cluster versions have been updated. For GPU node pools created in GKE Standard clusters running version 1.30.1-gke.115600 or later, GKE automatically installs the default NVIDIA GPU driver version corresponding to the GKE version if you don't specify the gpu-driver-version flag. We previously identified a potential issue that could cause downtime for traffic directed to your GKE-managed internal passthrough Network Load Balancers after certain cluster operations, like node upgrades.

Looker - Looker (Google Cloud core) and Looker (original) changes. Looker 24.16 includes the following changes, features, and fixes: Expected Looker (original) deployment start: Monday, September 16, 2024 Expected Looker (original) final deployment and download available: Thursday, September 26, 2024 Expected Looker (Google Cloud core) deployment start: Monday, September 16, 2024 Expected Looker (Google Cloud core) final deployment: Monday, September 30, 2024. Beginning in Looker 24.18, the October 2024 Looker release, Google Maps will be the only visualization engine for all map visualizations. The LookML Validator now checks for incompatible types in Liquid comparison expressions and, if it finds them, returns an error. You can change the width of the panels in the Looker IDE, both the feature panel (which contains File Browser, Object Browser, and Git Actions) and the side panel (which contains Project Health, Quick Help, and Metadata). The Chart Config Editor now supports sunburst visualizations. The Redshift driver is now configured with AWS's recommended TCP keep-alive settings. The content_summary API endpoint is now generally available. Comprehensive API support for Looker Connected Sheets is now accessible through both AppsScript and the Google Sheets APIs. Looker instances with the Redshift license feature enabled will now use the driver version 2.1.0.30. An issue has been fixed where measures would remove COALESCE SQL expressions from dimensions during query generation. CJK characters are now displayed properly in mobile browsers when they are included within inline table email attachments. An issue has been fixed that was causing the Collapse All Folders button in the Looker IDE to not work correctly. An issue has been fixed where some schedules would fail to send if a PDT was rebuilding. An issue where downloaded queries would not show error messages has been fixed. An issue has been fixed where the progress bar on single value visualizations could overlap with the visualization note. The LookML validator no longer forces the full_suggestions parameter to be enabled in certain situations involving Liquid variables and derived tables. The Chart Config Editor now displays a more informative error message if you try to use an unsupported visualization type. An issue has been fixed where the LookML Validator would return incorrect errors on cancel_grouping_fields in Explores with joins. An issue has been fixed where the Looker SQL Interface could not connect to Tableau using OAuth. Internal database calls during LookML validation have been reduced. An issue where the LookML Validator could crash if a LookML file incorrectly referenced a dimension_group in a filters parameter has been fixed. An issue has been fixed where Looker was incorrectly sanitizing some of the allowed CSS properties. The child_count property can now be omitted from dashboard and Look API responses when a feature flag is enabled. An issue has been fixed with the TRUNC function on some Denodo 8 dialects. An issue has been fixed where query metrics were not appearing in the Explore list. An issue has been fixed where the LookML validator would not return an error when value_format and named_value_format were both defined for a field. Looker (Google Cloud core) only changes. The render event has been added to the audit log list. An issue with SAML authentication has been fixed. The audit log buffer is now persisted to minimize log data loss. Looker (original) only changes. A new Labs feature, Delegate Model Set Management, lets admins grant a new permission, manage_modelsets_restricted.

Memorystore for Redis Cluster - Added support for vector store and vector search capabilities (Preview).

Cloud Memorystore - Added support for CMEK organization policies.

Cloud Monitoring - Table and TopList widgets can now display the results of multiple queries.

reCAPTCHA Enterprise - reCAPTCHA Mobile SDK v18.6.1 is now available for Android.

Secret Manager - Secret Manager is now enabled for use with Cloud KMS Autokey.

Secure Source Manager - Secure Source Manager branch protection is Generally Available. Secure Source Manager integration with Cloud Build lets you define your Cloud Build configuration and build triggers in your Secure Source Manager repository.

Security Command Center - Validate updates to integrations in the Security Command Center Enterprise use case Updates to the threat response playbook blocks and use case flows are available in the SCC Enterprise - Cloud Orchestration & Remediation use case for Security Command Center Enterprise. New configuration options for Vulnerability Assessment for AWS When configuring Vulnerability Assessment for AWS, you can customize the scan settings by defining the scan interval, specific regions, specific tags, and specific instance IDs.

Sensitive Data Protection - The discovery service of Sensitive Data Protection now supports Amazon S3. The DOD_ID_NUMBER infoType detector is available in all regions.

SAP Solutions - New SAP certification for operating system For use with SAP HANA and SAP NetWeaver on Google Cloud, SAP has now certified the operating system Red Hat Enterprise Linux (RHEL) for SAP 8.10. ABAP SDK for Google Cloud version v1.8 (On-premises or any cloud edition) Version 1.8 of the on-premises or any cloud edition of ABAP SDK for Google Cloud is generally available (GA).

Cloud SQL MySQL - Cloud SQL now supports near-zero downtime planned maintenance on standalone Cloud SQL Enterprise Plus edition primary instances. You can now upgrade your instances to Cloud SQL Enterprise Plus edition with near-zero downtime. You can now upgrade the minor version of a Cloud SQL for MySQL Enterprise Plus edition instance with near-zero downtime. You can now provide access to Cloud SQL Studio by granting a new IAM role, Cloud SQL Studio User (roles/cloudsql.studioUser), instead of using the Cloud SQL Admin IAM role.

Cloud SQL Postgres - Cloud SQL now supports near-zero downtime planned maintenance on standalone Cloud SQL Enterprise Plus edition primary instances. You can now upgrade your instances to Cloud SQL Enterprise Plus edition with near-zero downtime. You can now provide access to Cloud SQL Studio by granting a new IAM role, Cloud SQL Studio User (roles/cloudsql.studioUser), instead of using the Cloud SQL Admin IAM role.

Cloud SQL SQL Server - For Cloud SQL Enterprise Plus edition, you can set the number of days of retained transaction logs from 1 to 35. You can now provide access to Cloud SQL Studio by granting a new IAM role, Cloud SQL Studio User (roles/cloudsql.studioUser), instead of using the Cloud SQL Admin IAM role.

Cloud Storage - You can now specify United States regions when using regional endpoints. You can now use the Google Cloud console to do the following: Create buckets with hierarchical namespace enabled.

Cloud Text-to-Speech - Journey Voices is now in Preview and supports text streaming.

Vertex AI - Ray cluster's autoscaling feature is now supported.

Vertex AI Workbench - The ability to back up and restore data on a Vertex AI Workbench instance is now available in Preview.

VPC Service Controls - Preview stage support for the following integration: Privileged Access Manager.

Virtual Private Cloud - You can use Private Service Connect endpoints to access the regional service endpoints of supported Google APIs.

Workflows - The maximum number of concurrent workflow executions has increased from 7,500 to 10,000.