Welcome to issue #353 July 3rd, 2023

News

Introducing the Hive-BigQuery open-source Connector - With the open-source Hive-BigQuery Connector, you now can let Apache Hive workloads read and write to BigQuery and BigLake tables.

Making insights actionable with new Looker Studio scheduling capabilities - When scheduling a report, Looker Studio now lets you apply distinct filter conditions, and create and manage multiple schedules.

Improving the Slurm on Google Cloud Experience - The latest Slurm for Google Cloud adds new resource types and operating systems, simpler deployment, improved error reporting, and more.

Protect data from disasters using new Asynchronous Replication - The new Persistent Disk Asynchronous Replication replicates Compute Engine workloads between Google Cloud regions, enabling low RPO and RTO.

Cloud Next 2023 session catalog is live, covering all of your key cloud topics - Now that the session catalog for Cloud Next 2023 is live, you can register for sessions on AI, security, app dev, infrastructure operations and more.

AlloyDB for PostgreSQL with Database Migration Service is now Generally Available - The Database Migration Service for AlloyDB for PostgreSQL is now GA, and makes it easy to move from on- prem, self-managed and cloud databases.

Announcing vector support in PostgreSQL services to power AI-enabled applications - Use AlloyDB and Cloud SQL to store and index vector embeddings generated by large language models (LLMs), via the pgvector PostgreSQL extension.

GKE Security Posture dashboard now generally available with enhanced features - Strengthen your Google Kubernetes Engine (GKE) cluster security with advanced features, expanded capacity, and Autopilot integration.

Launching Address Descriptors to make it easier to find addresses using landmarks in Indian cities

Google named a Leader in The Forrester Wave™: Cloud Data Warehouses, Q2 2023 - Forrester says Google’s Data and AI Cloud offers a unified analytics offering, an open data ecosystem, and built-in machine learning intelligence.

Google is named a Leader in Forrester Functions-As-A-Service Wave - Google achieved the highest scores possible in Forrester’s Q2 2023 Functions As A Service Wave for Vision, Adoption and Observability criteria.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Cloud CISO Perspectives: Late June 2023 - What does the threat landscape look like this summer? Guest author Sandra Joyce discusses for boards and business leaders what their organizations are facing right now.

Unlocking cloud cost optimization: A guide to Google FinOps Resources - Simple resources for beginning your cloud FinOps journey to maximize value and ROI from your IT.

Sharing the inaugural State of Kubernetes Cost Optimization report - The first ever State of Kubernetes Cost Optimization report presents large-scale quantitative research on anonymized data from Kubernetes clusters.

Networking 101 Google Cloud reference sheet 2023 v2: Networking basics - This is a quick networking 101 reference guide. Includes general networking terms and Google Cloud networking services.

Reducing GKE production costs - A few tips to reduce GKE costs.

Implementing GCP VPC Service controls using Terraform (Terragrunt) - Implementing VPC service controls in a GCP environment with a shared VPC network.

Testing GCP VPC Service controls in Shared VPC network environment - Testing some scenarios related to VPC Service control in a Shared VPC network .environment.

Event-Driven Autoscaling in Kubernetes: Harnessing the Power of KEDA - How to use KEDA event driven approach to autoscale production Kubernetes clusters.

Terraform with Workspaces on Google Cloud - The goal of this article is showing a complete example with the use of Terraform Workspaces created on Google Cloud.

Application Secrets Encryption in Kubernetes and Anthos Products - Here is Yet Another Summary of the tips and tricks for keeping application secret values encrypted at rest in Kubernetes applications. This time covering Google Anthos product flavours as well.

App Development, Serverless, Databases, DevOps

How to migrate sensitive data with confidence using Google Cloud’s CDMC-certified architecture - New and existing Google Cloud customers can migrate their sensitive data to the cloud with greater confidence thanks to our newly CDMC-certified architecture.

How PLAID put the 'real' in real-time user analytics with Bigtable - PLAID recently re-architected its real-time user analytics engine using Cloud Bigtable, achieving latencies within 10 milliseconds.

Querying flexibly in Firestore with OR operator - Your application may often need to compare one or more fields with two or more values and select the documents that match at least one of the values.

The Modernization Imperative: Is your platform engineering project doomed to fail? - Just because you’ve built a platform, doesn’t mean your users will come. Luckily, there are ways to give an internal technology platform a fighting chance.

Three unique ways to add maps to custom, native, no code mobile applications - Thunkable gives users the ability to create custom, native, no code mobile applications for Android, iOS, and mobile web—without any technical expertise required. In this post John McMullan, Head of Marketing, shares how their customers are incorporating maps into their mobile application experiences.

How to run Python scripts with GCP Batch - Using GCP new batch service to run long-running scripts.

Docker made simple: A comprehensive guide to Artifact Registry - Get ready to learn about Docker images, Artifact Registry, and how they’ll boost your development process.

Getting your Docker containers to talk to each other in your Google Cloud Build CI/CD pipeline - At Kudos we use Google Cloud Build to create build pipelines for our apps and microservices. GCB pipelines are made up of Docker….

Practical Guide: How To Create A Docker Image In GCP Artifact Registry - Step-by-step guide to create a custom docker image in Artifact Registry using Cloud Build yaml file in your Git Repo.

How to inject custom log files into GCP Cloud Logging? - Using a small example to demonstrate how to inject custom logs into Cloud Logging.

Big Data, Analytics, ML&AI

Building AI-powered apps on Google Cloud databases using pgvector, LLMs and LangChain - Learn how to add generative AI features to your applications with just a few lines of code using pgvector, LangChain and LLMs on Google Cloud.

The rise of GenEng: How AI changes the developer role - This post highlights how the bulk of the innovation in applying Generative AI to solve business problems will be driven by application developers.

Unveiling the first generation data architecture of a newspaper - This article describes how NZZ, Switzerland’s German-speaking newspaper of record, developed and improved its first data cloud architecture powering various data products. Use-case driven, iteratively, and modular.

Built with BigQuery: How to supercharge your product data with Google Cloud and Harmonya - Harmonya relies on BigQuery to build and maintains data pipelines and train and serve machine learning models for its product enrichment service.

Google launches Java and Scala Procedures for BigQuery - Using stored procedures for Apache Spark with Java or Scala.

Hidden Gems of BigQuery — P6 — Time-traveling and clones - This post is about such BigQuery features as Time-travel window, Fail-safe period, Snapshots, Clones.

Modern data pipeline building with BigQuery Dataform - As ELT is becoming more and more popular, BigQuery Dataform enables modern SQL based transformation within BigQuery’s environment.

Achieving Differential Privacy using BigQuery - In this blog post, you will see an introduction to Differential Privacy framework and how to apply on your data in BigQuery.

Getting started with Chirp, the Google’s Universal Speech Model (USM) on Vertex AI - This article provides three most exciting facts about Chirp and a step-by-step guide on how to get started with Chirp on Vertex AI.

Event-driven Data Pipeline with Cloud Workflows and Serverless Spark - In some situations, we need to process individual files as soon as they arrive to our platform in order to minimise the time between….

Onboard large data science teams to GCP from on-prem cloud - Learn how to onboard large data science teams to Google Cloud (GCP) from an on-prem cloud.

Various

Google Workspace Training Now Available on LinkedIn Learning - Google Cloud announces a new partnership with LinkedIn Learning. Users can access Google Workspace training to increase their productivity & collaboration.

Cybrary: Closing the cybersecurity skills gap with affordable tools and training

Slides, Videos, Audio

Kubernetes Podcast - #204 Platform Engineering with Nicholas Eberts.

Security Podcast - #127 Is IAM Really Fun and How to Stay Ahead of the Curve in Cloud IAM?

Releases

AlloyDB - Vectorized join is available in Preview. Fault injection lets you test the resilience of a cluster's primary instance by simulating a sudden outage of its active node. IAM authentication for AlloyDB is available in Preview. The columnar engine now supports columns with the following data types: boolean bytea enum uuid.

Anthos Config Management - Config Controller now uses the following versions of its included products: Config Connector v1.105.0, release notes Anthos Config Management v1.15.1, release notes.

Anthos Config Management - 1.15.2. The constraint template library includes a new template: K8sRequireBinAuthZ. The constraint template library includes a new template: K8sRestrictAutomountServiceAccountTokens. The constraint template library includes a new template: K8sRestrictRoleRules. Fixed a formatting issue in nomos status --name.

Anthos clusters on bare metal - 1.13. Release 1.13.9 Anthos clusters on bare metal 1.13.9 is now available for download. Fixes: The following container image security vulnerabilities have been fixed: CVE-2019-17594 CVE-2019-17595 CVE-2021-20206 CVE-2021-36222 CVE-2021-37750 CVE-2022-2097 CVE-2022-3821 CVE-2022-4304 CVE-2022-4415 CVE-2022-4450 CVE-2022-29178 CVE-2022-29179 CVE-2022-29458 CVE-2023-0215 CVE-2023-0286 CVE-2023-0361. ISSUE Known issues: For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section. 1.6 & 1.7 & 1.8 & 1.9 & 1.10 & 1.11 & 1.12 & 1.13 & 1.14 & 1.15. Security bulletin (all minor versions) A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM).

Anthos clusters on Azure - With CVE-2023-31436, an out-of-bounds memory access flaw was found in the Linux kernel's traffic control (QoS) subsystem in how a user triggers the qfq_change_class function with an incorrect MTU value of the network device used as lmax. A new vulnerability (CVE-2023-2235) has been discovered in the Linux kernel that can lead to a privilege escalation on the node.

Anthos clusters on VMware - Security bulletin A number of vulnerabilities have been discovered in Envoy, which is used in Anthos Service Mesh (ASM). Security bulletin With CVE-2023-31436, an out-of-bounds memory access flaw was found in the Linux kernel's traffic control (QoS) subsystem in how a user triggers the qfq_change_class function with an incorrect MTU value of the network device used as lmax. Security bulletin A new vulnerability (CVE-2023-2235) has been discovered in the Linux kernel that can lead to a privilege escalation on the node.

Apigee X - On June 27, 2023 we released an updated version of Apigee X. Public preview of AppGroups Introduces the concept of AppGroups, which represent a relationship between one or more apps that are managed by the same set of people.

AppEngine Flexible NodeJS - Node.js 20 is now generally available.

AppEngine Flexible PHP - PHP 7.4, 8.1, and 8.2 are now available in preview.

AppEngine Standard NodeJS - The Node.js 20 runtime for App Engine standard environment is now generally available.

Artifact Registry - Go repositories are now generally available.

Assured Workloads for Goverment - v1. The EU Regions and Support with Sovereignty Controls compliance program now supports the following products. The IL5 compliance program is now generally available. v1. The ITAR compliance program now supports BigQuery.

BigQuery - Metadata caching is now available for BigLake tables that reference Amazon S3 data. Support for the following compliance programs is now generally available (GA): EU Regions and Support with Sovereignty Controls Sovereign Controls by Partners International Traffic in Arms Regulation (ITAR). You can now create stored procedures for Apache Spark using Java or Scala.

BigTable - Reverse scans in Cloud Bigtable are now available in Preview. The maximum retention period for a Cloud Bigtable backup has been increased from 30 days to 90 days, giving you more robust data protection and data quality control. The Cloud Bigtable metric Five-second maximum requests per minute is now generally available (GA). You can now enable batch write flow control when you use Dataflow to send batch writes to Cloud Bigtable.

Chronicle - The following supported default parsers have changed.

Compute Engine - You can suspend and resume E2 VMs. Preview: You can use instant snapshots to take in-place disk backups that can be restored to new disks under a minute. Preview: c3-standard and c3-highmem machine types are now available for general-purpose C3 VMs. Generally Available: Persistent Disk Asynchronous Replication (PD Async Replication) is now generally available. Generally available: NVIDIA A100 80GB GPUs are now available in the following additional regions and zones: Ohio, North America: us-east5-b For more information about using GPUs on Compute Engine, see GPU platforms. Generally available: For managed instance groups (MIGs), Google Cloud Console provides you with an improved way to configure autoscaling based on Cloud Monitoring metrics.

Container Registry - Container Registry API requests that reference a non existent project respond with 403 (permission denied) instead of 400 (bad request) status.

Dataproc Serverless - New Dataproc Serverless for Spark runtime versions: 1.1.21 2.0.29 2.1.8. Added support for Premium compute and storage pricing tiers for Dataproc Serverless Spark workloads. New Dataproc on Compute Engine subminor image versions: 2.0.68-debian10, 2.0.68-rocky8, 2.0.68-ubuntu18 2.1.16-debian11, 2.1.16-rocky8, 2.1.16-ubuntu20. Backported ZEPPELIN-5755 to Zeppelin 0.10 in 2.1 images for Spark 3.3 support. Added Dataproc Serverless Templates for batch creation: Cloud Storage to BigQuery Cloud Storage to Cloud Spanner Hive to Cloud Storage JDBC to BigQuery JDBC to Cloud Storage.

Datastore - Eventarc events and Firestore in Datastore mode events for Cloud Functions (2nd gen) now available in Preview.

Datastream - Support for ENUM and CITEXT data types is now added for PostgreSQL sources.

Deep Learning VM - M109 release Pytorch 2.0 on Debian 11 with Python 3.10 and CUDA 11.8 images are now available.

Cloud Deploy - Cloud Deploy support for parallel deployment is now generally available. Cloud Deploy support for the canary deployment strategy is now generally available.

Dialogflow - Dialogflow CX now supports flow scoped parameters. Dialogflow CX has added the following system functions: IS_CREDIT_CARD_NUMBER IS_DATE IS_FUTURE_DATE IS_PAST_DATE IS_PHONE_NUMBER NESTED_FIELD ROUND TO_OBJECT TO_PHONE_NUMBER UPPER.

Document AI - v1.4. Identity Document AI (IDAI) photo copy detection in ID proofing (Preview) Updated the pretrained-id-proofing-v1.1-2023-05-18 ID proofing processor for all Document AI users. v1.4. The following document OCR features are Generally Available (GA). Support for DOCX is in Preview. Added fixes to our doc.proto-to-vision.proto conversion tool, which facilitates migration from Vision API Text Detection to document OCR. The document OCR native text from digital PDF feature contains the following known issues: For a small number of documents, word order in lines of text that are reported by native text extraction might be inaccurate.

Cloud Filestore - Support for revert snapshot operations is now available for high scale SSD instances (Preview).

Cloud Functions - Cloud Functions now supports performance recommendations that analyze cold starts and suggest setting up minimum instances to improve function performance at the General Availability release level. The Node.js 20 runtime is now available for Google Cloud Functions at the GA release level.

Google Kubernetes Engine - FQDN Network Policy, currently in Public Preview, can now be enabled on GKE Autopilot clusters, by updating your clusters. With CVE-2023-31436, an out-of-bounds memory access flaw was found in the Linux kernel's traffic control (QoS) subsystem in how a user triggers the qfq_change_class function with an incorrect MTU value of the network device used as lmax. A new vulnerability (CVE-2023-2235) has been discovered in the Linux kernel that can lead to a privilege escalation on the node. (2023-R14) Version updates GKE cluster versions have been updated. Managed Service for Prometheus is enabled by default in new GKE Standard clusters running version 1.27 and later. Starting June 26, 2023, Cloud DNS becomes the default DNS provider for new GKE Autopilot clusters created with version 1.25.9-gke.400 or later or version 1.26.4-gke.500 or later, effectively replacing kube-dns.

GKE - (2023-R14) Version updates Version 1.26.5-gke.1200 is now the default version.

Google Kubernetes Engine Rapid - (2023-R14) Version updates The following versions are now available in the Rapid channel: 1.23.17-gke.7700 1.24.14-gke.2100 1.25.10-gke.2100 1.26.5-gke.2100 1.27.2-gke.2100 The following versions are no longer available in the Rapid channel: 1.23.17-gke.6800 1.24.14-gke.1400 1.25.10-gke.1200 1.26.5-gke.1200 Control planes and nodes with auto-upgrade enabled in the Rapid channel will be upgraded from version 1.22 to version 1.23.17-gke.7000 with this release.

Google Kubernetes Engine Regular - (2023-R14) Version updates Version 1.26.5-gke.1200 is now the default version in the Regular channel.

Google Kubernetes Engine Stable - (2023-R14) Version updates The following versions are now available in the Stable channel: 1.22.17-gke.12700 1.23.17-gke.5600 1.24.14-gke.1200 1.25.9-gke.2300 The following versions are no longer available in the Stable channel: 1.22.17-gke.8000 1.23.17-gke.2000 1.24.12-gke.500 Control planes and nodes with auto-upgrade enabled in the Stable channel will be upgraded from version 1.21 to version 1.22.17-gke.12700 with this release.

Load Balancing - Global external Application Load Balancers now support outlier detection for serverless NEG backends.

Migrate for Compute Engine - 5.0. Generally Available: Migrate to Virtual Machines lets you migrate your VM instances running on Google Cloud VMware Engine to VM instances running on Compute Engine.

StratoZone - StratoProbe 5.0.5.8 - Added the ability to scan MongoDB clusters. Added Cloud fit and pricing details for MongoDB. Added database Cloud fit report. Updated Microsoft License Report with improved layout and additional cost saving opportunities for SQL Server. Updated default package selection to exclude A2 and M3 preemptible families. Updated Content Security Policy to enable Trusted Types. Updated the default visibility of AWS and Azure pricing catalogs. Updated SQL Server Cloud fit rules to accommodate for new CloudSQL for SQL Server features. Fixed an issue where the sole-tenant overcommit surcharge was not being added when setting the overcommit options in pricing preferences. Fixed an issue where bin packing results were not refreshed after changes to some user rightsizing preferences. Fixed an issue where invalid custom shapes were selected in some cases for sole-tenant nodes.

Retail Recommendations AI - Retail API: Data export for analytics and other use cases is in GA Exporting retail data into BigQuery is now generally available (GA), allowing you to extract insights from your data. Entities are available as a way to subdivide your retail organization into more than one segment. The Data quality page assesses the quality of your product catalog and user event data and shows you which search performance tiers you have unlocked for Retail Search.

Security Command Center - As of June 20, 2023, Security Command Center Asset API endpoints and dependent functionality are deprecated and will be removed from the product for all users on or after June 20, 2024.

SAP Solutions - Cloud Storage Backint agent for SAP HANA version 1.0.26 Version 1.0.26 of the Cloud Storage Backint agent for SAP HANA is available.

Cloud Spanner - A monthly digest of client library updates from across the Cloud SDK.

Cloud SQL MySQL - For our preferred partners and allowlisted customers, Private Service Connect is now available.

Cloud SQL Postgres - For our preferred partners and allowlisted customers, Private Service Connect is now available.

Cloud SQL SQL Server - Cloud SQL now supports SQL Server 2022.

Cloud Storage Transfer - Transfers from Amazon S3 no longer require s3:GetBucketLocation permission on the source bucket.

Cloud Storage - You can now have a maximum of 10 HMAC keys per service account. Cloud Storage FUSE is now generally available.

Cloud Text-to-Speech - Studio voices now support SSML, except for the following tags: mark, emphasis, prosody, and lang.

Media Translation - Media Translation is deprecated and will no longer be available on Google Cloud after July 1, 2024.

Vertex AI - Vertex Explainable AI Support for example-based explanations is now generally available (GA). Vertex AI data labeling is deprecated and will no longer be available on Google Cloud after July 1, 2024. Vertex AI Codey APIs The Vertex AI Codey APIs are now generally available (GA). M109 release The M109 release of Vertex AI Workbench user-managed notebooks includes the following: Pytorch 2.0 with Python 3.10 and CUDA 11.8 user-managed notebooks instances are now available. The M109 release of Vertex AI Workbench managed notebooks includes the following: Fixed a bug that caused high cpu utilization due to excessive internal diagnostic tool processes.

Video Stitcher API - The Video Stitcher API now requires that a live stream source manifest references at least one valid segment file.

VMware Engine - Google Cloud VMware Engine now supports ESXi syslog forwarding, including distributed firewall logs, which provides more visibility into security events on VMware Engine instances. Google Cloud VMware Engine now supports Terraform for private cloud, cluster, and network management. Pay-as-you-go (PAYG) licenses for Windows Server are now available in Preview.

Virtual Private Cloud - You can use custom constraints to provide more granular and customizable control over specific fields for some VPC resources.

Workflows - Support for invoking a VPC Service Controls-compliant private endpoint is available in Preview. Three functions are available: map.merge takes two maps, creates a copy of the first map, and adds items from the second map to the copy; map.merge_nested recursively adds items from a map to a copy of another map; uuid.generate returns a random universally unique identifier. v1. Support for Customer-Managed Encryption Keys (CMEK) is generally available (GA).