Welcome to issue #363 September 11th, 2023

News

Firestore adds point-in-time recovery and scheduled backups - Firestore adds support for Point-in-time recovery and Scheduled Backups and Restores for both Datastore and Firestore Native Customers.

Actuate your data in real time with new Bigtable change streams - With the new Cloud Bigtable change streams feature, you can track changes to your Bigtable data and integrate it with other systems.

How Vertex AI empowered our Supply Chain Science team at Wayfair - Once it decided to migrate to Vertex AI, Wayfair data scientists learned to use the platform by implementing a delivery-time prediction use case.

Delivering greater insights for insurance underwriters with BigQuery geospatial analytics - CNA worked with Google Cloud and several third-party data vendors to develop a solution to address challenges with underwriting flood risk assessment.

Freestar harnesses Memorystore for ultra-low latency in global service delivery - Whenever an event occurs in the primary Memorystore cluster, the Envoy proxy propagates writes to every regional cluster.

Google Cloud Next 2023 FinOps product announcements recap - At Google Cloud Next 2023, we made several Google Cloud FinOps product announcements around Google Cloud Billing.

Improving Trust in AI and Online Communities with PaLM-based Moderation - How Google Cloud’s Text Moderation service, powered by the Cloud Natural Language API, can protect brands.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Effective alerting in Google Cloud - Understanding the relevance of what we’re monitoring can help us support triage in advance.

Increasing robustness of serving public DNS names using multi-provider setups - New Terraform scripts make it easy to use Cloud DNS as a second authoritative DNS server for public DNS hosting.

Configure CAI Feed for Real Time Notifications about GCP Resource Changes - Configuring Cloud Asset Inventory Feed to get notifications about changes to Google Cloud resources.

DevOps Engineering: How to Create Google Kubernetes Engine (GKE) Cluster - Introduction.

App Development, Serverless, Databases, DevOps

Develop and Test Spring Boot Applications Consistently - Learn how to consistently build and test with environment parity from development to production with Emulators and open-source Testcontainers.

Cloud SQL regional migration - Planning and migrating Cloud SQL database from one region to another, with minimalistic downtime and management overhead.

Is your data protected? Backup and DR service integrates with logging and monitoring tools - Google Cloud’s Backup and DR now integrates with Cloud Logging and Cloud Monitoring tools so you can monitor your backups with familiar tools.

Using Cloud SQL Auth Proxy to Connect to a Private SQL Instance Outside Its VPC - This post explains how to connect to a private Cloud SQL instance from outside its VPC using Cloud SQL Auth Proxy.

Reimagining the developer experience with Google Cloud - The experience your developers have at your organization can give you a competitive advantage. Explore how Google Cloud can help.

Using Apigee API Proxy to streamline a GenAI Hackathon - Using Apigee API Proxy to create custom APIs.

How to Optimize PHP Performance on Google Cloud Run - Discover ways to enhance PHP performance on Google Cloud Run.

Utilizing the MinIO client for Google Cloud Storage integration. - Establishing a connection to Cloud Storage through Minio client.

Step-by-step tutorial: SMB file storage using Google Cloud NetApp Volumes - Step by step tutorial to create and use NetApp Volumes.

Big Data, Analytics, ML&AI

Built with Google Cloud AI: How AI can improve data quality and observability for startups and enterprises - How AI can improve data quality and observability for startups and enterprises.

Tune Spark properties to optimize Dataproc Serverless jobs - Dataproc Serverless uses Spark properties to determine the compute, memory, and disk resources to allocate to your batch workload.

Deploying Kubeflow Pipelines using Python SDK - This is the A to Z Guide, I’ll even help you with your utility functions too!

How Resistant AI uses Document AI for fraud-resilient automated document processing - Resistant AI’s Document Forensics is an easy-to-deploy API that can automate document fraud checks using Google Cloud’s Document AI.

How to create alert for BigQuery Slowness or Bottleneck — Yong-Jin’s Blog - A simple alerting for slow BigQuery queries on Cloud Monitoring.

Vertex AI Workbench Tips - Guide to configuring properly your Workbench on Google Cloud.

Meeting Security Requirements for Dataflow pipelines — Part 1/3 - This article focuses on the Internal assessment of tenants must be private of common Dataflow security requirements.

Meeting Security Requirements for Dataflow pipelines — Part 2/3 - This article focuses on the "every tenant must be isolated and dedicated to a specific system of services" of common Dataflow security requirements.

GA4 and BigQuery: how to match the Console traffic grouping metrics - Google Analytics 4 offers excellent functionality with its GA4 BigQuery dataset. However, you might find it challenging to align your GA4 UI figures with the unnested sources, mediums, and campaigns in BigQuery.

A day in the life of a Google Cloud data user - Data analysis is key for any organization to make better decisions using data. So, how will your team interact with this data platform?

Automate BigQuery reservations and assignments using Dataform - This will guide you how to use BigQuery’s reservation SQL APIs with Dataform to automate pricing plan adjustments, based on time periods.

Dataplex — An Overview of Google Cloud Dataplex - Firstly in this article we will see what is Dataplex , How does it works to better organize your Data Lakes and warehouses and what kind of benefits Dataplex offers.

Various

Google Cloud Next 2023 — Experience, Announcements, and Summaries of Favourite Sessions - My summary of key announcements, and write-ups of favourite sessions. Plus, a little of my experience in SF.

Slides, Videos, Audio

Kubernetes Podcast - #207 Kubernetes 1.28 with Grace Nguyen.

Security Podcast - #137 Next 2023 Special: Conference Recap - AI, Cloud, Security, Magical Hallway Conversations.

GCP Life Podcast - #48 In this episode we discuss; Data Centre Outages, Kasna Partner Of the Year, Next ‘23, NetApp On GCP, Ampre, Uni Of Sydney Breach, Australian Security Authorities, ChatGPT For Business, AI Tools For Enterprise.

Releases

AlloyDB - When creating an AlloyDB cluster, you can now specify an IP range for private services access. Updated the PostgreSQL-compatible server running on AlloyDB instances to version 14.7. Updated the following extensions related to the PostGIS extension to version 3.2.3: postgis postgis_raster postgis_sfcgal postgis_tiger_geocoder postgis_topology address_standardizer address_standardizer_data_us. The following extensions are updated: Updated pg_hint_plan to version 1.4.1.

Anthos clusters on Azure - You can now launch clusters with the following Kubernetes versions: 1.25.8-gke.500 1.26.7-gke.500 1.27.4-gke.1600. 1.27 GA: Enabled node auto repair. 1.25 Expanded the list of metrics collected from node pools to include gke-metrics-agent, cilium-agent, cilium-operator, coredns, fluentbit-gke, kubelet, and konnectivity-agent. This release fixes the following vulnerabilities: CVE-2021-43565 CVE-2022-3821 CVE-2022-4415 CVE-2022-21698 CVE-2023-24539 CVE-2023-24540 CVE-2023-29400.

AppEngine Flexible Go - Go 1.21 is now generally available.

Cloud Asset Inventory - The following resource types are now publicly available through the Search APIs (SearchAllResources, SearchAllIamPolicies).

Batch - Documentation has been added to explain how to automatically retry some or all of the failed tasks for a job.

BigTable - Dataflow templates are available to stream Cloud Bigtable data change records to BigQuery or Pub/Sub.

Cloud Build - Users can now use manual triggers, webhook triggers, and Pub/Sub triggers to build Bitbucket Server and Bitbucket Data Center repositories through Cloud Build repositories (1st gen).

Chronicle - The supported default parsers have changed, more info in release notes page.

Access Transparency - Access Transparency logs are enhanced with an eventID that signifies the incident that resulted in the access by Google personnel.

Cloud Composer - Cloud Composer 2.4.2 release started on September 7, 2023. Maintenance windows are now generally available (GA). (Cloud Composer 2) An update that enforces VPC Service Controls perimeter settings for accessing Airflow UI and Airflow REST API is rolled out to all regions.

Config Connector - Config Connector version 1.109.0 is now available. Added name validation for the customizable ControllerResource CRDs. Field spec.containers is no longer required in the customizable ControllerResource CRDs. Added support for CloudIOTDeviceRegistry (v1alpha1) resource. Added support for ComputeRegionSSLPolicy (v1alpha1) resource. Added support for VertexAIIndexEndpoint (v1alpha1) resource. Resource BigtableGCPolicy(v1beta1): Fixed a bug that generates unexpected diff when mode and gcRules are both specified. Resource AlloyDBCluster(v1alpha1): Added spec.continuousBackupConfig field. Resource ArtifactRegistryRepository(v1beta1): Added spec.cleanupPolicies field. Resource BigQueryTable(v1beta1): Added spec.maxStaleness field. Resource CloudBuildTrigger(v1beta1): Added spec.gitFileSource.bitbucketServerConfigRef field. Resource CloudFunctions2Function(v1alpha1): Added spec.kmsKeyName field. Resource ComputeAddress(v1beta1): Added spec.ipv6EndpointType field. Resource ComputeBackendService(v1beta1): Added spec.securityPolicy field. Resource ComputeInstance(v1beta1): Added spec.networkInterface.items.ipv6AccessConfig.items.name field. Resource ComputeInstanceTemplate(v1beta1): Added spec.disk.items.provisionedIops field. Resource ComputeSecurityPolicy(v1beta1): Added spec.advancedOptionsConfig.userIpRequestHeaders field. Resource ComputeTargetInstance(v1beta1): Added spec.securityPolicyRef field. Resource ComputeTargetPool(v1beta1): Added spec.securityPolicyRef field. Resource ContainerCluster(v1beta1): Added spec.allowNetAdmin field. Resource ContainerNodePool(v1beta1): Added spec.networkConfig.additionalNodeNetworkConfigs field. Resource DNSManagedZone(v1beta1): Removed spec.privateVisibilityConfig.required field. Resource EventarcTrigger(v1beta1): Added spec.eventDataContentType field. Resource FirebaseAndroidApp(v1alpha1): Added spec.apiKeyId field. Resource FirebaseWebApp(v1alpha1): Added spec.apiKeyId field. Resource HealthcareFHIRStore(v1alpha1): Added spec.defaultSearchHandlingStrict field. Resource IAMWorkforcePoolProvider(v1beta1): Added spec.oidc.clientSecret field. Resource MonitoringAlertPolicy(v1beta1): Added spec.conditions.items.conditionPrometheusQueryLanguage field. Resource PubSubSubscription(v1beta1): Added spec.cloudStorageConfig field. Resource RunJob(v1beta1): Added status.createTime field. Resource SecretManagerSecret(v1beta1): Added spec.annotations field. Resource SpannerDatabase(v1beta1): Added spec.enableDropProtection field. Resource SQLInstance(v1beta1): Added spec.settings.ipConfiguration.pscConfig field. Resource WorkstationsWorkstationCluster(v1alpha1): Added spec.privateClusterConfig.allowedProjects field. Fixed spec.webhookConfig.secretRef field in CloudBuildTrigger.

Data Catalog - Data Catalog is now available in Berlin (europe-west10).

Data Fusion - Cloud Data Fusion version 6.9.2 is generally available (GA). Features in Cloud Data Fusion 6.9.2: Editing deployed batch pipelines is generally available (GA). Changes in Cloud Data Fusion 6.9.2: Cloud Data Fusion supports setting custom scopes when creating a Dataproc cluster (CDAP-19428). Fixed in Cloud Data Fusion 6.9.2: Added a retry for Pub/Sub snapshot creation and deletion in a real-time pipeline with a Pub/Sub source when a retryable internal error is thrown (PLUGIN-1660). The SAP ODP plugin version 0.7.5 is available in Cloud Data Fusion versions 6.6.0 to 6.8.0. The SAP SuccessFactors Batch Source plugin version 1.2.1 is available in Cloud Data Fusion.

Dataflow - The following Dataflow templates are generally available (GA): BigQuery to Bigtable Pub/Sub to Splunk.

Dataproc Serverless - New Dataproc Serverless for Spark runtime versions: 1.1.30 2.0.38 2.1.17. Dataproc Serverless Interactive sessions detail and list pages are now available in the Google Cloud console.

Dataproc - Dataproc Auto zone placement for clusters is now available in the Google Cloud console by selecting the "Any" option for the cluster zone. New Dataproc on Compute Engine subminor image versions: 2.0.75-debian10, 2.0.75-rocky8, 2.0.75-ubuntu18 2.1.23-debian11, 2.1.23-rocky8, 2.1.23-ubuntu20, 2.1.23-ubuntu20-arm. The Apache Spark version has been upgraded from 3.3.0 to 3.3.2 in Dataproc on Compute Engine 2.1 images. Announcing the General Availability (GA) release of Data Lineage for Dataproc, which captures data transformations (lineage events) in Dataproc Spark jobs, and publishes them to Dataplex Lineage.

Datastream - The Datastream BigQuery Migration Toolkit is now launched.

Error Reporting - Troubleshooting errors with Duet AI assistance is now available in Preview.

Cloud Functions - Cloud Functions now supports the Go 1.21 runtime at the General Availability release level.

Google Kubernetes Engine - Three vulnerabilities (CVE-2023-3676, CVE-2023-3955, CVE-2023-3893) have been discovered in Kubernetes where a user that can create Pods on Windows nodes may be able to escalate to admin privileges on those nodes. With the recently released version of GKE, version 1.28, you cannot yet use the SidecarContainer feature with alpha clusters. For GKE clusters using Dataplane V2, upgrades to GKE version 1.26 might cause GKE to temporarily be unable to configure workloads. (2023-R18) Version updates GKE cluster versions have been updated. 1.28 is now available in the Rapid channel Kubernetes 1.28 is now available in the Rapid channel. New APIs The kubectl auth whoami command and the authentication.k8s.io/v1 SelfSubjectReview API enables checking the authenticated user information as seen by the server. Deprecated API versions These APIs are still served in version 1.28 but are in a deprecation period: The following Beta versions of graduated APIs will be removed in 1.29 in favor of newer versions: flowcontrol.apiserver.k8s.io/v1beta2 FlowSchema, PriorityLevelConfiguration deprecated since 1.26 use flowcontrol.apiserver.k8s.io/v1beta3 instead, available since 1.26. Deprecated in-tree volume support The Ceph CephFS (kubernetes.io/cephfs) and RBD (kubernetes.io/rbd) volume plugins are deprecated in 1.28 and will be removed in a future release. The recently released version of GKE, version 1.28, has a known regression in behavior. To deliver a better default price-performance for applications, all GKE Autopilot clusters with control plane version 1.27.5 and later will use an SSD-based persistent disk for ephemeral storage. It is not yet possible to enable Gateway API support in GKE 1.28 (Standard or Autopilot), this will be fixed in an upcoming patch release to GKE 1.28.

GKE - (2023-R18) Version updates The following control plane and node versions are now available: 1.24.17-gke.200 1.25.13-gke.200 1.26.8-gke.200 1.27.5-gke.200 The following control plane versions are no longer available: 1.23.17-gke.8400, 1.23.17-gke.10000, 1.23.17-gke.10700, 1.24.14-gke.1400, 1.24.14-gke.2100, 1.25.10-gke.2100, 1.26.5-gke.1400 Control planes and nodes with auto-upgrade enabled will be upgraded from version 1.23 to version 1.24.14-gke.2700 with this release.

Google Kubernetes Engine Rapid - With the recently released version of GKE, version 1.28, you cannot yet use the SidecarContainer feature with alpha clusters. (2023-R18) Version updates Version 1.27.4-gke.900 is now the default version in the Rapid channel. 1.28 is now available in the Rapid channel Kubernetes 1.28 is now available in the Rapid channel. New APIs The kubectl auth whoami command and the authentication.k8s.io/v1 SelfSubjectReview API enables checking the authenticated user information as seen by the server. Deprecated API versions These APIs are still served in version 1.28 but are in a deprecation period: The following Beta versions of graduated APIs will be removed in 1.29 in favor of newer versions: flowcontrol.apiserver.k8s.io/v1beta2 FlowSchema, PriorityLevelConfiguration deprecated since 1.26 use flowcontrol.apiserver.k8s.io/v1beta3 instead, available since 1.26. Deprecated in-tree volume support The Ceph CephFS (kubernetes.io/cephfs) and RBD (kubernetes.io/rbd) volume plugins are deprecated in 1.28 and will be removed in a future release.

Google Kubernetes Engine Regular - (2023-R18) Version updates The following versions are now available in the Regular channel: 1.24.16-gke.500 1.25.12-gke.500 1.26.7-gke.500 1.27.4-gke.900 The following versions are no longer available in the Regular channel: 1.23.17-gke.10000, 1.24.14-gke.2700, 1.25.10-gke.2700, 1.26.5-gke.2700 Control planes and nodes with auto-upgrade enabled in the Regular channel will be upgraded from version 1.23 to version 1.24.15-gke.1700 with this release.

Google Kubernetes Engine Stable - (2023-R18) Version updates The following versions are now available in the Stable channel: 1.24.16-gke.500 1.25.12-gke.500 1.26.7-gke.500 1.27.4-gke.900 The following versions are no longer available in the Stable channel: 1.23.17-gke.8400, 1.24.14-gke.1400, 1.25.10-gke.2100, 1.27.3-gke.1700 Control planes and nodes with auto-upgrade enabled in the Stable channel will be upgraded from version 1.23 to version 1.24.14-gke.2700 with this release.

Load Balancing - Cloud Load Balancing is introducing new advanced cost, latency, and resiliency optimizations for your global external Application Load Balancer.

Cloud Monitoring - Ops Agent version 2.39.0 introduces support for Compute Engine Arm VMs that are running Ubuntu 20.04 LTS (Focal Fossa). Starting with version 2.39.0, the Ops Agent supports ingesting Prometheus metrics with unknown types.

Anthos Service Mesh - 1.16.x. 1.16.7-asm.7 is now available for in-cluster Anthos Service Mesh. 1.17.x. 1.17.5-asm.9 is now available for in-cluster Anthos Service Mesh. 1.18.x. 1.18.2-asm.4 is now available for in-cluster Anthos Service Mesh. Managed Anthos Service Mesh. Anthos Service Mesh will begin creating Network Endpoint Groups (NEGs) for all services. Managed Anthos Service Mesh 1.16 is rolling out in the stable channel.

SAP Solutions - Cloud Storage Backint agent for SAP HANA version 1.0.28 Version 1.0.28 of the Cloud Storage Backint agent for SAP HANA is available.

Cloud Spanner - A Cloud Spanner multi-region instance configuration is now available in Asia - asia2 (Mumbai/Delhi/Singapore).

Vertex AI - Vertex AI Prediction You can now use A2 Ultra machines to serve predictions in us-central1, us-east4, europe-west4, and asia-southeast1. Vertex AI Prediction The following prebuilt containers for prediction have been updated: tf2-cpu.2-12 tf2-gpu.2-12 tf2-cpu.2-11 tf2-gpu.2-11 tf2-cpu.2-10 tf2-gpu.2-10 tf2-cpu.2-9 tf2-gpu.2-9 tf2-cpu.2-8 tf2-gpu.2-8 sklearn-cpu.1-2 xgboost-cpu.1-7 pytorch-cpu.2-0 pytorch-gpu.2-0 pytorch-cpu.1-13 pytorch-gpu.1-13 To update your containers, redeploy your models.

Virtual Private Cloud - Support for IPv6 static routes with the following next hops is generally available (GA): next-hop-gateway next-hop-instance.