Welcome to issue #415 September 9th, 2024

News

Cut costs and boost efficiency with Dataflow's new custom source reads - Dataflow's new custom source reads feature helps cut costs and boost efficiency in streaming environments by better distributing workloads and proactively relieving overwhelmed workers with load balancing.

Google named a leader in the Forrester Wave: AI/ML Platforms, Q3 2024 - Google Cloud has been named a Leader in The Forrester Wave™: AI/ML Platforms, Q3 2024 report, tying for the highest score in the Strategy category. Google Cloud's Vertex AI platform provides a unified platform to manage the lifecycle of AI investments, from data preparation to model deployment, and addresses the full spectrum of AI needs: predictive, generative, and agentic. Google Cloud's open and flexible approach, pioneering innovation, comprehensive tools for AI teams, and enterprise-readiness make it an attractive choice for enterprises looking to accelerate their AI strategy.

BigQuery and Anthropic’s Claude: A powerful combination for data-driven insights - Google Cloud has integrated Anthropic's Claude models with BigQuery, allowing organizations to leverage advanced AI capabilities directly within their data platform. This integration enables tasks like text generation, summarization, translation, and more, to be performed directly on data in BigQuery.

Ensuring internet resilience: Catchpoint IPM now available on Google Cloud Marketplace - Catchpoint IPM, a suite of Internet Performance Monitoring (IPM) solutions, is now available on the Google Cloud Marketplace. It provides proactive monitoring of your entire Internet Stack, including all your Google Cloud services.

A radically simpler way to plan and manage block storage performance - Google Cloud introduces Hyperdisk Storage Pools with Advanced Performance, a new way to provision IOPS and throughput in aggregate for block storage. This simplifies performance planning and management, allowing customers to achieve 40-55% lower TCO without sacrificing application success or resource efficiency. A.

---

Technology and expertise from a Google Cloud Premier Partner

As a trusted Google Cloud partner of over a decade, DoiT delivers cloud optimization technology and unrivaled cloud expertise to help customers take full advantage of the public cloud. From migration and infrastructure to usage insights and analysis, DoiT has the experience and tools to help companies optimize Google Cloud use to accelerate return. Learn More

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

DeFied Expectations — Examining Web3 Heists - In the realm of Web3, where money flows, crime follows. This article delves into the escalating heists targeting decentralized finance (DeFi), surpassing traditional finance in both scale and frequency.

Next-Gen Cloud Network Security: Mastering NGFW Enterprise in GCP - NGFW Enterprise series - Dive into NGFW Enterprise: what it is, how it works and some recommendations for its deployment.

10 GKE networking problems and how to resolve them - This article discusses common networking issues in Google Kubernetes Engine (GKE) and provides troubleshooting steps to resolve them. It covers problems such as control plane connectivity issues, external communication issues, cross-VPC and on-premises communication issues, and communication beyond cluster VPCs.

GKE and RBAC: A Deep Dive into Securing Your Clusters with Google Groups - In this article, we delve into securing Kubernetes clusters in GKE using RBAC and Google Groups. We guide you through setting up RBAC, enabling Google Groups for RBAC, and managing access based on roles like DevOps, developers, and viewers.

Disabling Public GKE Cluster Creation in GCP Using Custom Organization Policy Constraints - In this article, we'll show you how to disable the creation of public Google Kubernetes Engine (GKE) clusters within your Google Cloud Platform (GCP) organization using custom constraints in organization policies. Custom constraints allow you to enforce granular control over resource configurations, ensuring adherence to your organization's security and compliance policies. We'll provide step-by-step instructions on how to set up the necessary constraints and test their enforcement.

Monitoring Google Cloud Armor Traffic: A Comprehensive Dashboard - Creating a Cloud Monitoring dashboard to better understand and monitor incoming traffic and attacks evaluated by Cloud Armor.

App Development, Serverless, Databases, DevOps

Galxe migrates to Google Cloud AlloyDB for PostgreSQL, cutting costs by 40% - Galxe, a Web3 company, migrated from Amazon Aurora to Google Cloud's AlloyDB for PostgreSQL, cutting costs by 40%.

Getting Started with Firestore:CRUD Operations on Google Firebases NOSQL Platform - In My Opinion.FireBase Firestore is a good free NOSQL Database that you can use for your side projects.

Demystifying the Google Cloud Marketplace - A Deep Dive into Architecture and Technical Considerations.

Hey Google (Cloud API), trust my AWS Application - Learn how to use Workload Identity Federation to authenticate AWS apps for accessing Google Cloud Services with a focus on EC2 and ECS.

Big Data, Analytics, ML&AI

BigTesty: Integration testing library with BigQuery - BigTesty is an integration testing library for BigQuery that allows users to execute tests in the real infrastructure, ensuring that SQL queries work as expected and return the intended data. It can be installed as a Python package or used as a Docker image.

Google launched better Delta Lake BigQuery Integration - Delta Lake BigLake Tables are now generally available.

Getting Ice Cream Recommendations at Scale with Gemini, Embeddings, and Vector Search - This blog post demonstrates how to build a scalable recommendation system for ice cream products using Gemini, a multimodal foundation model from Google. It leverages text embeddings and vector search within BigQuery to find similar products based on their descriptions and review summaries. The resulting recommendations are presented in an interactive Looker Studio dashboard, allowing users to explore and discover new ice cream flavors and combinations.

Multimodal Document Processing - How to process 10251 documents for just 1$. Built within 15 minutes.

Tuning Information Retrieval in Agent Builder Search applications with Google Search Adaptor. - This article discusses how to improve the precision and relevance of information retrieval systems using Search Adaptor, a tool that allows users to modify embeddings on the fly without the need for time-consuming reindexing. The integration of Search Adaptor into Vertex AI Search applications is also explored, showcasing its practical application in building private, customized search experiences tailored to enterprise requirements.

Is Your Business Ready and Enabled For Generative AI? - Generative AI (Gen AI) is a game-changing technology that can drive business value, improve decision-making, and enhance competitiveness. However, successful Gen AI adoption requires careful planning and assessment of an organization's readiness.

String Matching with BigQuery Machine Learning: A Deep Dive into Solving the No-Matching-Key Puzzle - String matching is a fundamental operation in data analysis, but it can be challenging when keys are misaligned or missing. BigQuery Machine Learning (BQML) offers powerful tools to address this issue, even without matching keys. This article provides a step-by-step guide to using BQML for string matching, including data preparation, creating embeddings, calculating cosine distance, and generating a unified table with top matching records.

Brewing Up a Storm: Real-time Order Processing for a Next-Gen Coffee Shop - The article delves into the technical architecture, highlighting how Google Cloud services like Pub/Sub, Dataflow, and BigQuery enable efficient order matching, barista location tracking, and accurate delivery time estimation. It also explores potential enhancements and the broader implications of cloud technologies in revolutionizing traditional industries and enhancing customer experiences.

BigQuery-Powered Natural Language Image Search: Multimodal Embeddings Within SQL Environment - BigQuery now functions as a vector database, enabling media embedding within its SQL environment. This advancement allows for efficient multimedia search capabilities, particularly in processing and retrieving visual content through natural language queries. By leveraging multimodal embeddings and advanced natural language processing techniques, users can search for images or videos, and even specific information within them, as intuitively as they would with text.

Various

GCP Partner Service Funds (PSF) and how to apply for it - Jumpstart your GCP projects with the support from Google Cloud and it’s partners.

Slides, Videos, Audio

Kubernetes Podcast - #235 Ray & KubeRay, with Richard Liaw and Kai-Hsun Chen.

Security Podcast - #188 Beyond the Buzzwords: Identity's True Role in Cloud and SaaS Security.

Releases

AlloyDB - AlloyDB for PostgreSQL is now available in africa-south1 (Johannesburg) and in me-central2 (Damman). The extension pgvector is updated to version 0.7.2.

Apigee UI - On September 5, 2024, we released an updated version of the Apigee UI. An informational message was added to the action creation flow for Apigee Security actions, informing users that actions can't be edited or deleted. Bug ID Description 349284447 Correct sorting for revisions in Duplicate Proxy The Duplicate Proxy experience now correctly sorts the list of existing revisions.

Apigee Hybrid - 1.13.0-hotfix.1. hybrid 1.13.0-hotfix.1 On September 5, 2024 we released an updated version of the Apigee hybrid software, 1.13.0-hotfix.1. Bug ID Description 362690729 Fix for aggressive scaling of runtime pods & cpu spike.

Cloud Architecture Center - (New guide) Enterprise application with Oracle Database on Compute Engine: Provides a reference architecture to host an application that uses an Oracle database, deployed on Compute Engine VMs.

Backup and DR Service - Backup and DR Service added support to automatically protect your Google Cloud VMware Engine VMs using vSphere tags. Backup and DR Service added support to view mounted image logs in Cloud Logging. Backup and DR Service added support to view mounted image reports in BigQuery.

BigQuery - You can now use vector search and vector index features in BigQuery.

Capacity Planner - Preview: You can view the on-demand reservations and future reservation requests available for consumption in your project, folder, or organization. Preview: You can use the Capacity Planner API to export usage and forecast data of the VMs, Persistent Disk volumes, or GPUs in your project, folder, or organization.

Chronicle SOAR - Release 6.3.17 is now in General Availability. Release 6.3.18 is currently in Preview. Playbooks are getting stuck in the queue. Release Notes 6.3.16 is now in General Availability. Remote Agents 2.1.0 is now in General Availability.

Cloud Composer - A new Cloud Composer release has started on September 2, 2024. Added a new metric: composer.googleapis.com/workflow/task_instance/queued_duration. Fixed an issue where an upgrade of a PSC-based private IP environment failed leaving the environment in an inconsistent state. (Cloud Composer 3) Fixed the cause of false-positive failures of Airflow Celery workers reported by the liveness health check. The apache-airflow-providers-google package was upgraded to version 10.22.0 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. The apache-airflow-providers-cncf-kubernetes package was upgraded to version 8.4.0 in Cloud Composer 2 images and Cloud Composer 3 builds with Airflow 2.9.1. New Airflow builds are available in Cloud Composer 3: composer-3-airflow-2.9.1-build.6 (default) composer-3-airflow-2.7.3-build.15. Cloud Composer 2.9.3 images are available: composer-2.9.3-airflow-2.9.1 (default) composer-2.9.3-airflow-2.7.3.

Compute Engine - Generally available: Multi-writer support for Hyperdisk Balanced disks. Generally available: You can use the performance monitoring unit (PMU) to monitor low-level CPU events and metrics in VMs that use a C4 machine type.

Contact Center AI Platform - Version 3.24 is released All release notes published on this date are part of version 3.24. Raw data export With raw data export, you can export detailed CCAI Platform session data to an external storage bucket. Virtual agents can create and assign tickets for custom CRM apps Virtual agents can now create and assign incoming tickets for custom CRM apps prior to a human agent taking the ticket. Barge is available for chat Barge, which lets supervisors join or take over calls with end-users, is now available for chat. Reserved data attributes With reserved data attributes, you can tag sessions with one of the following labels: Verified Customer, Bad Actor, or Repeat Customer. Fixed an issue where listening to a voicemail would occasionally lead to the creation of a duplicate entry at the top of the list. Fixed an issue that prevented emails from being sent to external storage. Fixed an issue where integrating the Salesforce CRM with CCAI Platform was failing.

Dataproc Metastore - Dataproc Metastore supports custom region configurations.

Dataproc - New Dataproc on Compute Engine subminor image versions: 2.0.117-debian10, 2.0.117-rocky8, 2.0.117-ubuntu18 2.1.65-debian11, 2.1.65-rocky8, 2.1.65-ubuntu20, 2.1.65-ubuntu20-arm 2.2.31-debian12, 2.2.31-rocky9, 2.2.31-ubuntu22. Dataproc on Compute Engine: The latest 2.2 image versions now support Hudi 0.15.0. Dataproc on Compute Engine: The latest 2.2 image versions support Hudi Trino integration natively. Dataproc on Compute Engine: Dataproc image version 2.2 will become the default Dataproc on Compute Engine image version on September 6, 2024. New Dataproc on Compute Engine subminor image versions: 2.0.116-debian10, 2.0.116-rocky8, 2.0.116-ubuntu18 2.1.64-debian11, 2.1.64-rocky8, 2.1.64-ubuntu20, 2.1.64-ubuntu20-arm 2.2.30-debian12, 2.2.30-rocky9, 2.2.30-ubuntu22,. Dataproc on Compute Engine: Apache Spark upgraded to version 3.5.1 in image version 2.2 starting with image version 2.2.30.

Cloud Data Loss Prevention - The SEXUAL_ORIENTATION infoType detector is available in all regions.

Cloud Firestore - You can now use Firestore to perform K-nearest neighbor (KNN) vector searches.

Integration Connectors - The Neo4j connector is now generally available (GA).

Memorystore for Redis Cluster - Added support for Maintenance Windows (Preview).

Cloud Monitoring - The Metrics management page in Cloud Monitoring now shows you the sources of metric reads and lets you exclude unneeded metrics entirely, eliminating the cost of ingesting them. The fleetwide and per-instance Observability tabs on the Compute Engine VM instances page now include charts for GPU metrics the from NVIDIA Management Library (NVML). You can now import Grafana dashboards into Cloud Monitoring by using the console.

Cloud Run - Deterministic URLs, which let you predict a Cloud Run service URL before the service is created, is now in general availability (GA).

Security Command Center - Install new version of the Security Command Center Enterprise use case The installation and configuration of a new version of the SCC Enterprise - Cloud Orchestration & Remediation use case in the Security Operations console is required for the toxic combination functionality of Security Command Center Enterprise.

Sensitive Data Protection - The SEXUAL_ORIENTATION infoType detector is available in all regions.

Cloud SQL Postgres - You can now use point-in-time recovery to restore your zonal instance to a preferred primary zone and your regional instance to both a preferred primary zone and a preferred secondary zone.

Cloud SQL SQL Server - When you clone your zonal instance, you can now specify a preferred zone for the instance.

VMware Engine - VMware Engine now offers GA support for VPC Service Controls.

Workflows - Support for execution backlogging is available in Preview.