Welcome to issue #414 September 2nd, 2024

News

Google Cloud launches Memorystore for Valkey, a 100% open-source key-value service - Google Cloud launches Memorystore for Valkey, a fully-managed, open-source key-value service. It offers high performance, scalability, and adaptability, and is compatible with Redis 7.2 and popular Redis clients.

Instant snapshots: protect Compute Engine workloads from errors and corruption - Google Cloud introduces instant snapshots for Compute Engine, providing near-instantaneous, high-frequency, point-in-time checkpoints of a disk that can be rapidly restored as needed.

Announcing Terraform Google Provider 6.0.0: More Flexibility, Better Control - The Terraform Google Provider v6.0.0 is now generally available. Key changes include an opt-out default label "goog-terraform-provisioned" to identify resources created by Terraform, deletion protection fields added to multiple resources to prevent accidental deletion, and the ability to reduce the suffix length in "name_prefix" for more flexible resource names.

GenOps: learning from the world of microservices and traditional DevOps - Generative AI applications have unique operational requirements compared to traditional applications. They require a new operational platform, called GenOps, that can manage their non-deterministic nature, model compliance, prompt version management, model evaluation, model security, and centralized tool management. Google Cloud provides a range of managed services to support GenOps, including Model Garden, Model Armor, prompt management capabilities, model evaluation services, and Vertex AI Search.

Magic partners with Google Cloud to train frontier-scale LLMs - Magic, a generative AI startup, has partnered with Google Cloud to build two new cloud-based supercomputers to support its mission of developing code assistants with a context window reaching 100 million tokens.

Your ultimate guide to the latest in generative AI on Vertex AI - Announcements list of Generative AI topics and Vertex AI.

Get started with the new generally available features of Gemini in BigQuery - Gemini in BigQuery, now generally available, brings AI-powered experiences to enhance data discovery, preparation, analysis, and insight generation. Features include SQL and Python code generation with natural language prompts, explanations for complex queries, a data canvas for intuitive exploration, data insights with pre-validated queries, and recommendations for partitioning and clustering to optimize performance.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Cloud CISO Perspectives: What you’re missing when you miss out on mWISE Conference - Google Cloud’s VP/GM SecOps, Peter Bailey, discusses the upcoming Mandiant Worldwide Information Security Exchange (mWISE) Conference in Denver, Colorado, on September 18-19. mWISE is an annual cybersecurity and knowledge-sharing conference that brings together security leaders and practitioners to share expertise and best practices.

A Measure of Motive: How Attackers Weaponize Digital Analytics Tools - To defend against these attacks, defenders should implement automated analysis around link shorteners, IP geolocation utilities, and CAPTCHA tools. They should also be cautious when clicking on ads or links in ads, double-checking the website address (URL) of the destination to make sure it matches the company or product in the ad and doesn't contain typos.

Google IT adopting Google Cloud services - Google's enterprise IT teams migrated their infrastructure to Google Cloud Platform (GCP) to take advantage of its scale and operational standards. The migration involved a lift-and-shift phase followed by cloud modernization efforts. The team encountered challenges such as multi-environment enterprise, new gaps, and being early adopters of new features.

Need a higher cache hit rate? Media CDN origin offload does the trick - Google Cloud's Media CDN offers efficient origin offload, reducing the load on origin servers and delivering content faster with lower latency. It leverages a series of distributed caches, high-performance network infrastructure, and intelligent features to minimize the need to reach origin servers.

I Spy With My Little Eye: Uncovering an Iranian Counterintelligence Operation - Mandiant has uncovered an Iranian counterintelligence operation that targets Iranian dissidents, activists, and Farsi speakers. The operation uses fake social media accounts to spread a network of fake recruiting websites that collect personal information from users. The campaign has been active since at least 2017 and has targeted individuals in Iran, Syria, and Lebanon.

GKE Made Easy: The Essential Guide for Businesses (part 2) - This article explores the autoscaling capabilities of Google Kubernetes Engine and modes of operation.

GKE Introduction: How to Set It Up with Terraform and Visualize Metrics on Grafana - Learn how to deploy a private GKE cluster with Terraform, enable Prometheus monitoring, and visualize metrics on Grafana.

Hashicorp Vault — gcs + gcpckms - Secret Management and Why It’s Important.

App Development, Serverless, Databases, DevOps

A Practical Guide to using PgCat with AlloyDB - PgCat is a PostgreSQL connection pooler and proxy that scales PostgreSQL databases beyond a single instance. This blog post demonstrates how to configure and use PgCat with AlloyDB.

Convert2PDF: An API to Transform Microsoft Office Files into PDF - Let’s use LibreOffice to convert MS Office files to PDF format and deploy the API on GCP Cloud Run.

Google Compute metadata — Efficient Usage in Shell Script - If you are familiar with Google Compute metadata and you are wondering how to integrate it with startup script, this article is for you!

Level up your codebase with Gemini’s long context window in Vertex AI - Gemini, a Generative AI model with a 2 million token context window, can help developers understand and work with unfamiliar codebases. By packaging all source code into a single prompt, developers can ask questions about the code, such as summarizing its content or implementing a feature request.

Mirror GitHub Repositories into Google Cloud Source Repository - Automated Repository Backup: Ensuring Your Codebase is Secure and Recoverable.

How to Propagate Span Context Using OpenTelemetry from Publisher to Subscriber with Google Cloud Pub/Sub and Go - Learn how to propagate span context using OpenTelemetry in a Go app with Google Cloud Pub/Sub.

Big Data, Analytics, ML&AI

Uncover the upcoming BigQuery Price change in autumn `24 - Google‘s upcoming storage transfer cost and network changes for BigQuery and how to get ready for them!

Datastream and BigQuery — Append mode experiments - In this article we illustrate some recipes for processing Datastream append mode change tables.

Effective Tactics for End-to-End Pipeline Testing - Using self-made BigQuery emulator in tests.

Parallel & Serverless CSV Ingestion to CloudSQL Using Cloud Dataflow - This blog post explores how to solve this problem efficiently using a Dataflow pipeline powered by Apache Beam.

Another Video Search Engine with BigQuery and Gemini - AI-Powered Video Search: A Step-by-Step Guide.

A developer’s guide to getting started with Imagen 3 on Vertex AI - Imagen 3 on Vertex AI is a powerful text-to-image model that generates stunning visuals with exceptional composition, sharpness, color accuracy, and resolution. Imagen 3 is now generally available with an allowlist, and developers can sign up for access through a form.

Grounding Analytical AI Agents with Looker’s Trusted Metrics - Bytecode, a Google Cloud partner, explains how to integrate Looker's trusted metrics with analytical AI agents to provide fast and reliable insights for AI-powered Business Intelligence. By combining large language models with Looker's semantic layer, businesses can quickly fine-tune and ground their analytical agents in their business truths, enabling end users to access trusted insights easily.

The modern marketer’s strategic advantage: AI-powered data clean rooms - AI-powered data clean rooms offer a secure environment for sharing, collaborating, and analyzing sensitive data without compromising privacy. They enable modern marketers to unlock valuable insights from collaborative data analysis across various industries, driving strategic decision-making and business growth.

A multimodal search solution using NLP, BigQuery and embeddings - Multimodal search using NLP, BigQuery, and embeddings combines natural language processing and multimodal embeddings to enable users to search for images or videos in the same way they would with text-based content. The solution leverages Google Cloud Storage for storing media files, BigQuery object tables for referencing these files, and a pre-trained multimodal embedding model to generate semantic embeddings for efficient similarity searches.

Vertex AI Function Calling - LLMs are turning into reasoning engines using capabilities like web search and calling external APIs.

Gemini’s Power for Tourism Apps: A Function Calling Tutorial - In this tutorial, we will create a Tourism web app where users can query the weather for future days in a specific location.

Decoding the Price Tag: Estimating Google Gemini AI Costs - Navigate the complexities of Google Gemini AI pricing and make informed decisions for your business.

Various

Emerging startups working with Google Cloud to fuel the next wave of innovation - Google Cloud is committed to supporting emerging startups by providing mentorship, technical expertise, community building, exposure, and funding opportunities. If you're an emerging startup interested in collaborating with Google Cloud, reach out to us to explore how we can help you build faster and easier.

Slides, Videos, Audio

Security Podcast - #187 Conquering SOC Challenges: Leadership, Burnout, and the SIEM Evolution.

GCP Life Podcast - #72 In this episode we discuss GDG Sydney, C4 Machines, LLM on Cloud Run, Google invests in Texas,Cloud Spend, Cloud Migrations, Google Privacy Class Action, Google New Security Features, Australian Cyber Security, AI Stunning Cost.

Releases

AlloyDB - Query federation between BigQuery and AlloyDB is now generally available (GA). Database server compatibility with PostgreSQL version 16 is now available in Preview.

Anthos Config Management - 1.19.0. The spec.git and spec.enableLegacyFields fields of the ConfigManagement object have been removed. Optimized Config Sync resource usage by implementing watch filtering with ApplySet ("applyset.kubernetes.io/" labels and annotations). Config Sync now enables loading files from directories beyond the Kustomize root during rendering. Improved support for private registries. Upgraded bundled Helm version from v3.14.4 to v3.15.3 to pick up vulnerability fixes. Upgraded the Open Telemetry image from 0.102.0 to 0.103.0 to pick up vulnerability fixes. Fixed some inaccuracies in status updates and metrics reports.

GKE on AWS - The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes: CVE-2024-36978 For more details, see the GCP-2024-049 security bulletin.

GDCV for VMware - Google Distributed Cloud (software only) for VMware 1.30.0-gke.1930 is now available for download. GA: StatefulSet CSI Migration Tool GA: Migrate clusters to use recommended features Preview: The gcloud beta container fleet memberships get-credentials command uses a preview feature of the Connect gateway that lets you run the kubectl attach, cp, and exec commands. For admin and user clusters created at 1.30 and later versions, loadBalancer.Kind needs to be set to either MetalLB or ManualLB. Version changes in 1.30.0-gke.1930: Existing Seesaw load balancers now require TLS 1.2. ​​The following issues were fixed in 1.30.0-gke.1930: Fixed the known issue where cluster creation failed due to the control plane VIP in a different subnet. The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes: CVE-2024-39503 For more details, see the GCP-2024-047 security bulletin.

Apigee X - On August 30, 2024, we released an updated version of Apigee (1-13-0-apigee-4). Bug ID Description N/A Updates to security infrastructure and libraries. Clarification: On July 26 we announced monetization support with data residency. On, August 26, 2024, Apigee announced the GA launch of its non-VPC provisioning option. With the non-VPC peering provisioning approach, you are not required to provide networks and IP ranges during the Apigee provisioning process.

Cloud Architecture Center - (New guide) Select a managed container runtime environment: Learn about managed runtime environments and assess your requirements to choose between Cloud Run and GKE Autopilot.

Artifact Registry - Updates to the Artifact Registry API are as follows: Add or update file annotations with UpdateFile(). Artifact Registry records metrics and logs for your projects.

Cloud Asset Inventory - The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs. The following resource types are now publicly available through the ExportAssets, ListAssets, BatchGetAssetsHistory, QueryAssets, Feed, and Search (SearchAllResources, SearchAllIamPolicies) APIs.

BigQuery - The BigQuery Data Transfer Service now supports incremental transfers when you migrate your data from your Teradata data warehouses to BigQuery. Delta Lake BigLake tables are now generally available (GA). You can now use the GROUP BY clause and the SELECT DISTINCT clause with the ARRAY and STRUCT data types. You can now query data in AlloyDB using a federated query. The following Gemini in BigQuery features are now generally available (GA): Data insights Data canvas SQL and Python code assistance features: Use the SQL generation tool Prompt to generate SQL queries Explain a SQL query Generate Python code Partitioning and clustering recommendations To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery. Phrase support for the SEARCH function is now generally available (GA). You can now create remote models in BigQuery ML based on the Anthropic Claude model in Vertex AI. You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable (reverse ETL).

Bigtable - The Preview of Data Boost for Bigtable has been expanded to let you use Data Boost compute resources when you read Bigtable data using a Spark application.

Chronicle - The prioritization logic of Applied Threat Intelligence (ATI) rule set has been improved to remove alerts from events that have a specified security result action of BLOCKED or QUARANTINED.

Chronicle SOAR - Release Notes 6.3.17 is currently in Preview. Last Close comment and Last Close Root Cause not showing up in BigQuery. Alert names that are too long cover the time remaining on the alert SLA. Unable to edit, delete or export custom integration (ID #52403533). Multi Select option not working in Custom Actions. Playbook shows failed step even though it's not being used by the playbook. Playbook export contains archived blocks.

Cloud Composer - (Cloud Composer 2) Starting on September 26, 2024, all Cloud Composer environments within a VPC SC perimeter will have no access to public PyPI repositories by default.

Compute Engine - Generally available: When applying a spread placement policy to VMs, you can specify the availability domain in which to place the VMs. The fleetwide and per-instance Observability tabs on the Compute Engine VM instances page now include charts for GPU metrics the from NVIDIA Management Library (NVML). The fleetwide and per-instance Observability tabs on the Compute Engine VM instances page now include charts for GPU metrics the from NVIDIA Management Library (NVML).

Contact Center AI Platform - VPC Service Controls and private egress You can now use VPC Service Controls with CCAI Platform to help mitigate the risk of data exfiltration from your contact center.

Database Migration Service - Database Migration Service for homogeneous migrations to Cloud SQL for SQL Server now supports differential backup files.

Dataplex - Data insights is generally available (GA).

Dataproc - New Dataproc on Compute Engine subminor image versions: 2.0.115-debian10, 2.0.115-rocky8, 2.0.115-ubuntu18 2.1.63-debian11, 2.1.63-rocky8, 2.1.63-ubuntu20, 2.1.63-ubuntu20-arm 2.2.29-debian12, 2.2.29-rocky9, 2.2.29-ubuntu22.

Dialogflow - Dialogflow CX & ES: Text-to-Speech Journey Voices will get an update in the week of Sept 3.

Cloud Data Loss Prevention - Regional endpoints are available for Sensitive Data Protection.

Cloud Functions - The Cloud Run functions (cloud functions v2 API) build process now supports cross-project Artifact Registry repositories for greater configurability.

Integration Connectors - The following connectors are now generally available (GA): Active Directory QuickBooks Online. The following connectors are now generally available (GA): Azure AD Azure Synapse Dropbox Instagram SAP ASE (Sybase) To view the list of all the GA connectors, see Connectors in GA. The TIBCO EMS connector is now generally available (GA).

Google Kubernetes Engine - (2024-R32) Version updates GKE cluster versions have been updated. Starting from version 1.30.3-gke.1451000, new and upgraded GKE clusters support the GKE Metrics Server updates where the addon-resizer runs in the cluster's control plane instead of worker nodes. The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes: CVE-2024-36978 For more details, see the GCP-2024-049 security bulletin. The following vulnerabilities were discovered in the Linux kernel that can lead to a privilege escalation on Container-Optimized OS and Ubuntu nodes: CVE-2024-41009 For more details, see the GCP-2024-048 security bulletin.

GKE new features - Starting from version 1.30.3-gke.1451000, new and upgraded GKE clusters support the GKE Metrics Server updates where the addon-resizer runs in the cluster's control plane instead of worker nodes.

Load Balancing - The Global external Application Load Balancer and the Classic Application Load Balancer will no longer support TLS sessionID resumption.

Looker - Looker (Google Cloud core) only changes. To create a Looker (Google Cloud core) instance with Private Service Connect, it is no longer necessary to be added to an allowlist.

Network Connectivity Center - Preset topologies and include export filters are generally available.

Network Intelligence Center - Network Analyzer now includes an insight indicating that the next hop IP address is not assigned to any forwarding rule in the relevant networks.

reCAPTCHA Enterprise - reCAPTCHA Mobile SDK v18.6.0 is now available for iOS.

Cloud Run - The following Cloud Run volume types are now generally available (GA): Mounting an NFS file share as a volume for Cloud Run services and jobs. You can now configure traffic routing between Cloud Run and Cloud Run, Google Kubernetes Engine, and Google Compute Engine services using Cloud Service Mesh service routing APIs.

Security Command Center - Dynamic mute rules are generally available Security Command Center now supports dynamic mute rules, which allow you to mute future and existing findings temporarily until a specified date or indefinitely until a finding no longer matches the configuration. Documentation is available for the Security Posture REST API.

Sensitive Data Protection - Regional endpoints are available for Sensitive Data Protection.