Welcome to issue #361 August 28th, 2023

News

Introducing BigQuery text embeddings for NLP tasks - You can now generate text embeddings in BigQuery and apply them to downstream application tasks using familiar SQL commands.

Keep a closer eye on Google Cloud costs with new Budgets for project users - Now, Google Cloud project users can easily create budgets and stay on top of their cloud spend, and Project Owners can receive timely alerts.

Scaling machine learning with BigQuery ML inference engine - BigQuery ML inference engine lets you run inference over custom models, remote models, and pretrained models within your machine learning workflow.

Introducing Google Cloud NetApp Volumes: Fully managed file storage for enterprise workloads - Fully managed file storage for enterprise workloads.

Cloud Storage FUSE is now optimized for GKE and AI workloads - Cloud Storage FUSE enables storage buckets to be mounted as a local file system for applications that need file system semantics.

Unleash your AI workloads with Google Cloud’s latest storage solutions - Google Cloud’s new storage solutions include the Parallelstore parallel file system, Cloud Storage FUSE, and Google Cloud NetApp Volumes.

Improving sustainability with our Earth Engine and BigQuery connector - The new BigQuery connector to Google Earth Engine improves ease-of-use and enables new analyses that combine raster and tabular data.

---

Expert support, on demand. Imagine having a direct line to over 150 senior cloud architects for any cloud-related question or issue you encounter. With thousands of cloud questions and issues resolved, DoiT is your gateway to world-class cloud expertise. All from a Google Cloud Partner. Learn more

---

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Enter the Hardware-verse: Google hardware comes to life at Google Cloud Next - Visitors to the Hardware-verse at Google Cloud Next can see Google Distributed Cloud (GDC) and Cloud Tensor Processing Units (TPUs) up close.

Hands-On Guide: Google Cloud VPC Network Peering to Connect Two VMs and Check Nginx Server Access. - A demonstration of VPC network peering within Google Cloud Platform.

Deploy using Helm Charts on GKE: Continuous Delivery Pipeline using Google Cloud Build & Google Cloud Deploy - This blog discusses the steps required to create a continuous integration and continuous delivery (CI/CD) pipeline for multiple Google Kubernetes Engine (GKE) clusters for multiple environments using Helm, Cloud Build, and Cloud Deploy.

Everything Your Cloud Provider Won’t Tell You About Service Accounts - Learn service account logic, use cases and the unavoidable business problem they solve.

Exposing Applications with GKE Gateway Controller - An overview and example of the usage of GKE Gateway controller.

App Development, Serverless, Databases, DevOps

Calling all DevOps, IT Ops, Platform Engineers and SREs: 5 can’t-miss breakout sessions at Next ‘23 - There’s no lack of breakout sessions for DevOps, IT Ops, Platform Engineers, and Site Reliability Engineers (SREs) at Google Cloud Next 2023.

How Codeway uses Cloud Storage FUSE for their generative AI avatar portrait maker - Codeway uses Cloud Storage FUSE and GKE for their Generative AI avatar portrait maker.

The Modernization Imperative (TMI): The beauty in boring - Durability needs to be a cornerstone of the application modernization conversation, so your applications can be around for the long haul.

Setting Up Strapi on Google Cloud Run with Cloud SQL and GCP Bucket - Configuring Strapi V3/V4 on Google Cloud Run, intricately weaving in the capabilities of Cloud SQL and Cloud Bucket integration.

Maximizing Throughput and Minimizing Costs with Cloud Run’s Direct VPC Egress - This article explains how to use the new Direct VPC Egress with Cloud Run.

Flutter for data engineering and data science! - Deploying Flutter app (written in Python?!) to Cloud Run.

Build Flutter application in Python to chat in ANY language with Google Cloud LLMs available on… - Creating & deploying chat application on GCP using PaLM 2 model.

Big Data, Analytics, ML&AI

10 must-attend sessions for data professionals at Google Cloud Next ‘23 - Check out these breakout sessions for data professionals at Google Cloud Next ‘23.

Supporting generative AI development with our data cloud partners - Google Cloud’s open, flexible partner ecosystem helps organizations innovate with generative AI.

Talk to the map: How CARTO is making GIS more conversational using BigQuery - Carto’s conversational GIS relies on BigQuery to provide information to generative AI, while BigQueryML connects to Vertex AI and PaLM2.

Built with BigQuery: Sustainable supply chains start with a data sharing ecosystem - TraceMark’s sustainable sourcing monitoring platform provides transparency into global supply chains about the sourcing of raw materials.

Built with BigQuery: How Atlas AI helps build a more resilient future - Atlas AI ‘s geospatial artificial intelligence platform that helps organizations anticipate changing societal conditions to help them make investment decisions.

Migrating BigQuery across regions with dataset replication - Leverage on the new cross-region dataset replication feature to migrate your BQ datasets across regions.

dbt & Machine Learning? It is possible! - Using BigQuery ML with DBT.

Demystifying BigQuery Information Schema: A Comprehensive Guide with Real-World Examples - Examples of queries to query Information Schema data in BigQuery.

Migrating SQL Workflows from Teradata to BigQuery using Python - This article explores how to create a migration workflow for Batch SQL Translation from Teradata to BigQuery using Python.

Special Characters handling in BigQuery - This blog post focuses on understanding encoding support for BigQuery, how to detect special characters in BigQuery and how to handle special characters.

What is Multimodal Search: "LLMs with vision" change businesses - Adding vision capabilities to LLMs.

The Science of Architectural Design: How Foundational Models can help to group building styles - How image, language, and multimodal embedding models can help with image clustering.

Generative AI — Deploy and inference of Llama 2 in Vertex AI Prediction - This post shows how to deploy a Llama 2 chat model (7B parameters) in Vertex AI Prediction with a T4 GPU.

Seeing is explaining. How Example-Based Explanations help improving a car damage classifier - How Example-Based Explanations help improving a car damage classifier.

Various

Five sessions IT Pros can’t miss at Next ‘23 (and a bevy of hidden pop culture references) - Google Cloud Next ‘23 will be a tremendous event for architects and IT pros — here are the ‘must see’ breakout sessions during the show.

Google Cloud Next ’23: What startups can expect at the event - Google Cloud Next ’23: learn about the Google for Startups Cloud program, network with cloud AI experts and learn to scale your business faster.

Google Cloud Next ’23 must-see experiences - Attending Next ’23 in-person? Be sure to visit these must-see experiences.

Slides, Videos, Audio

Kubernetes Podcast - #206 LeakSignal with Wesley Hales and Max Bruce.

Security Podcast - #135 AI and Security: The Good, the Bad, and the Magical.

Releases

Access Approval - Access Approval supports Memorystore for Redis in the GA stage.

AlloyDB - You can now improve connection security by enforcing applications to connect to AlloyDB instances through AlloyDB connectors, such as the Auth Proxy. You can now configure SSL enforcement mode to ensure that all database connections to an instance use SSL encryption.

Anthos Config Management - 1.16.0. The constraint template library's GkeSpotVMTerminationGrace template adds the includePodOnSpotNodes parameter, which requires enabling referential constraints. The Config Sync feature to sync directly from Helm repositories (including OCI-based ones) is generally available (GA). Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 616aa8f). ExpansionTemplates APIs promoted to v1beta1. Policy Controller add new bundles: nist-sp-800-190, nist-sp-800-53-r5, and nsa-cisa-k8s-v1.2. Added a new field spec.helm.valuesFileRefs in RootSync and RepoSync to allow specifying Helm values files in ConfigMaps. Added a new configsync.gke.io/deletion-propagation-policy annotation for use on RootSync and RepoSync, for configuring foreground cascading deletion as a preview feature. The Helm chart version field spec.helm.version in RootSync and RepoSync can now be specified as a range of values from which Config Sync pulls the latest values. Added the spec.override.logLevels field to RootSync and RepoSync, for configuring the log level of the containers of reconciler Pods. The constraint template library's K8sRequireDaemonsets template now supports restricting the use of NodeSelector in required Daemonset using the new restrictNodeSelector parameter. Policy Controller bundles have been updated to the following versions: cis-k8s-v1.5.1: 202307.1, pci-dss-v3.2.1: 202307.0, policy-essentials-v2022: 202307.1, pss-baseline-v2022: 202307.0, pss-restricted-v2022: 202307.0. Upgraded bundled Kustomize version from v5.1.0 to v5.1.1 to pick up vulnerability fixes. Reduced the resource requirements for the reconciler Pod when no Kustomize rendering is needed. Increased the default timeout for Kubernetes API requests from 5 seconds to 15 seconds. Fixed an issue causing Config Sync to prematurely stop tracking managed resource objects when pruning them before they are deleted from the cluster. Fixed an issue preventing Config Sync from pruning managed resources correctly when the apiGroup of those resources is renamed in the source of truth. Known Issue for Policy Controller: If you're using ExpansionTemplates and mutators, Policy Controller will not apply the mutators for the generated resources. There is a memory issue in the reconciler due to the discovery client loading the OpenAPI for schema validations.

Anthos clusters on bare metal - 1.16. Release 1.16.0 Anthos clusters on bare metal 1.16.0 is now available for download. Version 1.13 end of life: In accordance with the Anthos Version Support Policy, version 1.13 (all patch releases) of Anthos clusters on bare metal has reached its end of life and is no longer supported. Cluster lifecycle: Upgraded to Kubernetes version 1.27.4. Functionality changes: Updated constraint on NodePool spec.upgradeStrategy.concurrentNodes to be the smaller of 15 nodes or 50% of the size of the node pool. Fixes: Fixed an issue where the apiserver could become unresponsive during a cluster upgrade for clusters with a single control plane node. The following container image security vulnerabilities have been fixed: Critical container vulnerabilities: CVE-2022-29155 CVE-2022-29458 CVE-2023-0464 CVE-2023-0465 CVE-2023-0466 CVE-2023-2283 CVE-2023-2650 High-severity container vulnerabilities: CVE-2019-19906 CVE-2020-8032 CVE-2022-4450 CVE-2022-4904 CVE-2022-24407 CVE-2022-29154 CVE-2022-32190 CVE-2023-0045 CVE-2023-0215 CVE-2023-0286 CVE-2023-0361 CVE-2023-0386 CVE-2023-0461 CVE-2023-1077 CVE-2023-1078 CVE-2023-1118 CVE-2023-1281 CVE-2023-1670 CVE-2023-1829 CVE-2023-1989 CVE-2023-2454 CVE-2023-3567 CVE-2023-23559 CVE-2023-28466 CVE-2023-31436 CVE-2023-32233 Medium-severity container vulnerabilities: CVE-2018-1099 CVE-2019-9511 CVE-2019-9513 CVE-2020-11080 CVE-2020-13844 CVE-2021-3468 CVE-2022-2097 CVE-2022-3707 CVE-2022-3821 CVE-2022-4129 CVE-2022-4304 CVE-2022-4382 CVE-2022-4415 CVE-2022-23524 CVE-2022-23525 CVE-2022-23526 CVE-2022-36055 CVE-2023-0458 CVE-2023-0459 CVE-2023-1073 CVE-2023-1074 CVE-2023-1076 CVE-2023-1079 CVE-2023-1667 CVE-2023-1855 CVE-2023-1859 CVE-2023-1990 CVE-2023-1998 CVE-2023-2162 CVE-2023-2194 CVE-2023-2455 CVE-2023-2985 CVE-2023-3161 CVE-2023-3220 CVE-2023-3358 CVE-2023-23916 CVE-2023-26545 CVE-2023-28328 CVE-2023-28484 CVE-2023-29469 CVE-2023-30456 CVE-2023-32269 CVE-2023-33203 CVE-2023-33288 Low-severity container vulnerabilities: CVE-2009-5155 CVE-2015-8985 CVE-2019-17594 CVE-2019-17595 CVE-2021-3468 CVE-2022-2196 CVE-2022-3424 CVE-2022-4379 CVE-2023-1513 CVE-2023-1611 CVE-2023-1872 CVE-2023-2163 CVE-2023-21102 CVE-2023-22998 CVE-2023-23004 CVE-2023-25012 CVE-2023-30772. Known issues: For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section. 1.14. Release 1.14.8 Anthos clusters on bare metal 1.14.8 is now available for download. Fixes: The following container image security vulnerabilities have been fixed: High-severity container vulnerabilities: CVE-2017-11468 Medium-severity container vulnerabilities: CVE-2019-9511 CVE-2019-9513 CVE-2020-11080 CVE-2020-13844 Low-severity container vulnerabilities: CVE-2009-5155 CVE-2015-8985. Known issues: For information about the latest known issues, see Anthos clusters on bare metal known issues in the Troubleshooting section.

Anthos clusters on VMware - Anthos clusters on VMware 1.16.0-gke.669 is now available. Preview: You can migrate from the Seesaw load balancer to MetalLB. Version changes: Upgraded VMware vSphere Container Storage Plug-in from 3.0 to 3.0.2. The following issues are fixed in 1.16.0-gke.669: Fixed the known issue that caused intermittent ssh errors on non-HA admin master after update or upgrade. The following vulnerabilities are fixed in 1.16.0-gke.669: Critical container vulnerabilities: CVE-2022-29155 High-severity container vulnerabilities: CVE-2023-0286 CVE-2023-2828 CVE-2023-27561 CVE-2022-29458 CVE-2023-3138 CVE-2020-7712 CVE-2015-3276 CVE-2020-8032 CVE-2023-0215 CVE-2023-0361 CVE-2022-4450 CVE-2023-2454 CVE-2022-29154 CVE-2023-1999 Container-optimized OS vulnerabilities: CVE-2023-2609 CVE-2023-0386 CVE-2023-1872 CVE-2023-27561 CVE-2023-3090 CVE-2023-24329 Windows vulnerabilities: CVE-2022-41723 CVE-2022-41725.

Google Cloud Armor - Adaptive Protection suggested rules can now be deployed automatically in General Availability.

Artifact Registry - Artifact Registry is now available in the europe-west10 region (Berlin, Germany).

Batch - Batch is available in the europe-west10 (Berlin) region. Batch is available in the following regions: asia-south2 (Delhi) asia-southeast2 (Jakarta) europe-southwest1 (Madrid) me-central1 (Doha) For more information, see Locations.

BigQuery ML - The following text embedding features are now available in preview: Creating a BigQuery ML remote model that references the Vertex AI PaLM APIs for embeddings (textembedding-gecko). The following BigQuery ML inference features are now generally available (GA): Importing ONNX, XGBoost, and TensorFlow Lite models so that you can run them within the BigQuery ML inference engine. BigQuery now allows you to create your own masking routines for your data. You can now scan tables to create data profiles and monitor data quality. Analytics Hub now lets you manage subscriptions.

BigQuery - BigQuery now allows you to create your own masking routines for your data. You can now scan tables to create data profiles and monitor data quality. Analytics Hub now lets you manage subscriptions.

Billing - Budgets for project users now available (in preview) Now project users in Google Cloud can create budgets and stay on top of their cloud spend, without needing additional permissions to access Cloud Billing accounts. We added a new BigQuery example that helps you join your Price and Detailed Cost Data exports, so that you can compare your costs with detailed information from your Pricing BigQuery export.

Chronicle - Chronicle has updated the rules engine's YARA-L 2.0 language compiler to report warnings. The following supported default parsers have changed.

Key Access Justifications - Access Approval supports Memorystore for Redis in the GA stage.

Cloud Composer - Composer 1 (all versions): Creating Cloud Composer 1 environments in projects with VPC Service Controls enabled might fail due to missing access in the security perimeter. Cloud Composer 2 is now available in Dallas (us-south1), Santiago (southamerica-west1), Melbourne (australia-southeast2), and Madrid (europe-southwest1).

Compute Engine - Preview: You can reduce network latency between VMs by using compact placement policies to specify the maximum distance between VMs. Generally available: Berlin, Germany, Europe europe-west10-a,b,c has launched with E2, N2, N2D, and T2D VMs available in all three zones. Generally available: When a managed instance group (MIG) repairs a failed or an unhealthy VM, you can apply the latest instance template and per-instance configuration to recreate the VM instead of applying the configuration originally used to create the VM. Generally available: Hyperdisk Throughput is now available in the following additional regions and zones: Council Bluffs, Iowa: us-central1 Moncks Corner, South Carolina: us-east1 Ashburn, Virginia: us-east4-b, c Eemshaven, Netherlands: europe-west4-a, c Jurong West, Singapore: asia-southeast1 Mumbai, India: asia-south1-a.

Config Connector - Config Connector version 1.108.0 is now available. Added support for customization on cnrm-unmanaged-detector pods resource requests/limits. Added support for customization on cnrm-controller-manager pods resource requests/limits in namespaced mode. Added support for BigQueryReservationCapacityCommitment resource(v1alpha1). Resource BigQueryDataset(v1beta1): Added spec.storageBillingModel field. Resource BigQueryTable(v1beta1): Added spec.externalDataConfiguration.jsonOptions field. Resource BigtableTable(v1beta1): Added spec.changeStreamRetention field. Resource CertificateManagerCertificate(v1alpha1): Added spec.managed.issuanceConfig field. Resource CloudFunctions2Function(v1alpha1): Added status.url field. Resource ComputeDisk(v1beta1): Added spec.enableConfidentialCompute field. Resource ComputeExternalVPNGateway(v1beta1): Added status.labelFingerprint field. Resource ComputeForwardingRule(v1beta1): Added spec.noAutomateDnsZone field. Resource ComputeInstance(v1beta1): Added spec.params field. Resource ComputeInstanceTemplate(v1beta1): Added spec.networkInterface.items.networkAttachment field. Resource ComputeTargetHTTPProxy(v1beta1): Added spec.httpKeepAliveTimeoutSec field. Resource ComputeTargetHTTPSProxy(v1beta1): Added spec.httpKeepAliveTimeoutSec field. Resource ComputeURLMap(v1beta1): Added spec.pathMatcher.items.routeRules.items.matchRules.items.pathTemplateMatch field. Resource ContainerCluster(v1beta1): Added spec.nodeConfig.guestAccelerator.items.gpuDriverInstallationConfig field. Resource ContainerNodePool(v1beta1): Added spec.nodeConfig.guestAccelerator.items.gpuDriverInstallationConfig field. Resource DataformRepository(v1alpha1): Added spec.workspaceCompilationOverrides field. Resource GKEBackupBackupPlan(v1alpha1): Added status.state field. Resource HealthcareFHIRStore(v1alpha1): Added spec.complexDataTypeReferenceParsing field. Resource RedisInstance(v1beta1): Added status.maintenanceSchedule field. Resource RunJob(v1beta1): Added spec.annotations field. Resource SQLInstance(v1beta1): Added spec.settings.dataCacheConfig field. Resource VertexAIFeaturestoreEntityTypeFeature(v1alpha1): Added status.region field. Resource VertexAIIndex(v1alpha1): Added spec.metadata.config.shardSize field.

Dataflow - Dataflow is available in Berlin (europe-west10).

Dataplex - Dataplex automatic data quality and data profiling are generally available.

Dataproc Serverless - Fixed a Dataproc Serverless issue where Spark batches failed with unhelpful error messages.

Dataproc - Dataproc is now available in the europe-west10 region (Berlin).

Datastore - You can now view and list multiple databases using the Google Cloud console. Scheduled backups now available in Preview.

Cloud Deploy - Cloud Deploy support for deploy parameters is now generally available.

Cloud Firestore - Scheduled backups now available in Preview. You can now view and list multiple databases using the Google Cloud console.

Networking Interconnect - Dedicated Cloud Interconnect support is available in the following colocation facilities: Level(3) Berlin, Berlin NTT Berlin 1 Data Center (BER1), Berlin For more information, see the Locations table.

KMS - Cloud KMS is available in the following region: europe-west10 For more information, see Cloud KMS locations.

Google Kubernetes Engine - GKE now delivers insights and recommendations to ensure your workloads are ready for disruption using features such as Pod Disruption Budgets. The europe-west10 region in Berlin, Germany is now available.

Load Balancing - Internal passthrough Network Load Balancers can now be configured to handle private IPv6 traffic within your VPC.

Cloud Logging - You can now view your query results as a chart in the Log Analytics page. Identity and Access Management (IAM) deny policies can now include Cloud Logging permissions.

StratoZone - Added Migration Center informational banner. Added support for Google Cloud Hyperdisk. Updated Google Cloud pricing. Updated demo account assessment capabilities to prevent misuse.

Cloud Monitoring - You can now add pie charts to your custom dashboards.

Cloud NAT - Cloud NAT support for Inter-VPC communications is in Preview.

Cloud Interconnect - Dedicated Cloud Interconnect support is available in the following colocation facilities: Level(3) Berlin, Berlin NTT Berlin 1 Data Center (BER1), Berlin For more information, see the Locations table.

Cloud VPN - Cloud VPN is now available in region europe-west10 (Berlin, Germany).

Cloud PubSub - Pub/Sub is now available in europe-west10 (Berlin, Germany).

Cloud Run - The following new region is now available: europe-west10.

Secret Manager - Secret Manager is now available in the following region: europe-west10 For more information, see Secret Manager locations.

Security Command Center - inIpRange() function released to General Availability You can now specify a range of IP addresses by using the inIpRange() function in query statements to filter findings that contain IPv4 or IPv6 addresses within the specified range.

Cloud Spanner - Cloud Spanner has added 13 new PostgreSQL functions and operators, including: REGEXP_MATCH(string, pattern) function REGEXP_SPLIT_TO_ARRAY(string, pattern) function SUBSTRING(string, pattern) function TO_CHAR(value, format) function For more information, see Supported PostgreSQL functions. Cloud Spanner now supports integer sequences and bit reversal. Cloud Spanner now supports generating a UUID (v4) as part of a table's primary key DEFAULT expression using the GENERATE_UUID function in GoogleSQL or generate_uuid() in PostgreSQL-dialect databases. You can create Cloud Spanner regional instances in Berlin, Germany (europe-west10). Spanner Studio enhances the Spanner query editor in the Google Cloud console, with full support for SQL, DML, and DDL operations.

Cloud SQL - Support for europe-west10 (Berlin) region. Private Service Connect is now GA for Cloud SQL for PostgreSQL.

Cloud Storage - The Storage Object User role (roles/storage.objectUser) is now available. Cloud Storage is now available in Berlin, Germany (europe-west10 region).

Vertex AI - Vertex AI custom training has launched persistent resources in Preview.

Virtual Private Cloud - For auto mode VPC networks, added a new subnet 10.214.0.0/20 for the Berlin europe-west10 region.