Welcome to issue #260 September 20th, 2021

News

The new Google Cloud region in Toronto is now open - Google Cloud now has two regions in Canada: one in Montreal, and another in Toronto, providing customers with enhanced choice and data sovereignty.

Announcing Backup for GKE: the easiest way to protect GKE workloads - The new Backup for GKE is a first-party backup.

Announcing Filestore Enterprise, for your most demanding apps - Filestore Enterprise is a fully-managed file storage service built for business-critical applications like SAP, with regional availability and NFS.

To serve and protect: New storage features help ensure data is never lost - Google Cloud makes enterprise application storage easier to migrate and manage with product launches for Filestore, GKE and Cloud Storage.

Run code samples directly in the Google Cloud documentation - The Google Cloud documentation has a new feature that lets you run code samples in Cloud Shell, without leaving the page.

Run more workloads on Cloud Run with new CPU allocation controls - New always-on CPU allocation for Cloud Run opens the doors to running workloads that use background processes.

Google Cloud announces new Cloud Digital Leader training and certification - Learn to leverage knowledge of Google Cloud to drive business impact and improve collaboration between business and technical professionals.

Just released: The Google Cloud Next session catalog is live. Build your custom playlists. - Google Cloud Next session catalog is live.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Google Cloud disaster recovery using Actifio, VMware Engine and Zerto - There are subtle differences between the ways you protect applications on-premises and in the cloud.

Making VMware migrations to Google Cloud simpler than ever - The Google Cloud Catalyst Program provides Google Cloud VMware Engine users financial flexibility when migrating to Google Cloud.

Automate Application Migration with GKE Autopilot and Migrate for GKE - Migrating applications to Kubernetes can be challenging, but by combining the functionalities of Migrate for GKE and GKE Autopilot the process can be greatly simplified.

Network security threat detection - Comparison of analytics methods - Compares & contrasts 5 mechanisms for cultivating network signal telemetry for forensics, security analytics, advanced threat detection, and hunting.

Leveraging Network Telemetry for Forensics in Google Cloud - Network Forensics and Telemetry solution to enable Threat Hunting; brings together Packet Mirroring, Open source and your choice of SIEM (like Chronicle).

Multi Cluster Ingress — GKE - Creating Multi Cluster Ingress in GKE.

Google Cloud: Importing resources from the gcloud to IaaC in Terraform - Using gcloud command to export resources to Terraform format.

Quick Cloud Cost Hygiene in GCP - Reduce cloud cost and improve cloud hygiene by locating and identifying cost impactful resources in Google Cloud Platform using gcloud commands.

Enforcing best practice on self-serve infrastructure with Terraform, Atlantis and Policy As Code - Using open-source tools to build an end-to-end pipeline that centrally manages Terraform and enforces best practices using Policy as Code on GCP.

App Development, Serverless, Databases, DevOps

Understanding Cloud SQL Maintenance: how long does it take? - A technical breakdown of the Cloud SQL maintenance workflow helps illuminate instance updates and downtime periods.

VMs and their relevance to a cloud-native future: A conversation - Do VMs Even Matter? A conversation, based on a new Podcast "VM End-To-End”, about VMs and their relevance to a cloud-native future.

Push your code and see your builds happening in your terminal with "git deploy" - By extending git on your local machine with just one small Python script, you can see your Cloud Build triggers deployed as you push your code up to GitHub.

Deployment models for the Cloud Spanner emulator - This is the first of a three-part series of blog posts, which together will form a solution guide for developers using the Cloud Spanner emulator. In this series, after a quick introduction to the Cloud Spanner emulator, we will explore the usage of the emulator with a Cloud Spanner sample application called OmegaTrade.

Upgrade Postgres with pglogical and Database Migration Service - Google’s Database Migration Service simplifies upgrading to the next major version of Postgres.

SQL Server SSRS, SSIS packages with Google Cloud BigQuery - The following blog details patterns and examples on how Data teams can use SQL Server Integration Services (SSIS) and SQL Server Reporting Services (SSRS) with Google BigQuery.

The differences between synchronous web APIs and asynchronous stateful APIs - Revisits and distinguishes between the concepts of (stateful) APIs.—crafted as products—and enterprise integration patterns, in which APIs are just an additional technical endpoint, providing insights on the context in which each of these 2 concepts should be used.

Provisioning Cloud Spanner using Terraform - Provisioning and scaling Cloud Spanner and deploying an application on Cloud Run using Terraform templates.

Cloud SQL IAM database authentication — Manage user access to the databases in an efficient way - How to use IAM accounts to connect to Cloud SQL database.

Setup GCP Cloud functions Triggering by Cloud Schedulers with Terraform - This is a step by step tutorial on how to setup GCP cloud functions, trigger them automatically by Cloud Schedulers, all with Terraform.

Google Cloud Functions — A Brief Tutorial - Extracting data loaded from Cloud Storage to BigQuery with Cloud Functions.

Exploring Google Cloud API Gateway with Google Cloud Functions - Creating a secure API Gateway that can handle query params and form-data (file uploads) and forward them to appropriate Cloud Functions.

Reading Configuration Data from Google Firestore using REST API - Reading database configuration parameters stored in Google Firestore.

Big Data, Analytics, ML&AI

What type of data processing organization are you? - Data driven transformation requires technical and organizational changes. Success depends on what tech you choose, but also the type of data driven organization you are.

Airflow 2 Development Environment on GCP Cloud Shell - Setting up an automated and feature-rich Airflow 2 development environment on GCP Cloud Shell Code Editor.

Recommendations AI data ingestion - Recommendations AI uses your product catalog and user events to create machine learning models and deliver personalized product recommendations to your customers.

Dark launching real-time price alerts - Ingesting data from Pub/Sub.

Migrating from AWS Glue to BigQuery for ETL - Reasons to switch from AWS Glue pipeline to BigQuery.

How to Deal with NULL Values in Standard SQL - This article explains how to handle null values in BigQuery.

My BigQuery Query Cheat Sheet - Explains Query template, searches for Struct and Array field.

The fastest way to fetch BigQuery tables - A benchmark of the fastest methods used to fetch tables from BigQuery. Also introducing bqfetch: an easy-to-use tool for fast fetching.

PyTorch on Google Cloud: How to deploy PyTorch models on Vertex AI - In this blog post, we show how you can deploy PyTorch models on Google Cloud Vertex AI with the Vertex Prediction service to serve predictions from the trained model artifacts.

Chefkoch whips up handwritten recipes in the cloud with text detector - Chefkoch uses Google Cloud ‘s ML capabilities to parse and digitize handwritten recipes.

Distributed ML with Dask and Kubernetes on GCP - Example of running Dask on GKE.

Various

Optimizing Waze ad delivery using TensorFlow over Vertex AI - With the help of Google Cloud’s Vertex AI, Waze can run scalable ML to optimize ad delivery.

One on One with Google Cloud Product Director Irina Farooq - An interview with Irina Farooq about Data solutions on GCP.

Slides, Videos, Audio

GCP Podcast - #276 Cloud Migration with Txture and Accenture.

Kubernetes Podcast - #163 Prodfiler, with Thomas Dullien.

Security Podcast - #31 Cloud Certifications, and Cloud Security with TheCertsGuy.

Releases

Anthos clusters on AWS - A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files & directories outside of the volume, including on the host filesystem.

Cloud Run for Anthos - Preview: Installing Cloud Run for Anthos as an Anthos feature is now available as a Preview.

BigQuery ML - BigQuery ML documentation has been updated with the following improvements: The end-to-end user journey now includes an overview of the machine-learning workflow for each available model.

BigQuery - BigQuery now supports the following geospatial data functions: ST_EXTERIORRING: Returns a linestring geography that corresponds to the outermost ring of a polygon geography. ST_INTERIORRINGS: Returns an array of linestring geographies that corresponds to the interior rings of a polygon geography. ST_ANGLE: Returns the angle between two intersecting lines. ST_AZIMUTH: Returns the azimuth of a line segment formed by two points. ST_NUMGEOMETRIES: Returns the number of geometries in a geography. ST_GEOMETRYTYPE: Returns the Open Geospatial Consortium (OGC) geometry type that describes a geography as a string. These functions are generally available (GA).

Billing - Cost breakdown report now supports new filters and report sharing In the Cloud Billing Console Cost breakdown report, you can now select the costs you want to analyze using the Time range and other report filters, such as projects, services, and SKUs.

Cloud Composer - Airflow 2 in Cloud Composer is now generally available (GA). HA Scheduler in Cloud Composer is now generally available (GA). We plan to stop offering Airflow 1.10.14 and 1.10.12 in new versions of Cloud Composer. Java Runtime in Airflow workers and schedulers is updated from version 8 to version 11. The default value for the visibility-timeout Airflow configuration option, which determines the amount of time after which the task is re-picked by another worker, is increased from 6 hours to 7 days. If an Airflow configuration option is blocked in the image version specified for an upgrade operation, and this option has an override in your environment, the upgrade operation is rejected. In Airflow UI, menu items for Configuration and DAG dependencies pages are now correctly displayed for users with the Op role. New versions of Cloud Composer images: composer-1.17.0-airflow-2.1.2 composer-1.17.0-airflow-2.0.2 composer-1.17.0-airflow-1.10.15 (default) composer-1.17.0-airflow-1.10.14 composer-1.17.0-airflow-1.10.12 composer-2.0.0-preview.1-airflow-2.1.2 composer-2.0.0-preview.1-airflow-2.0.2. Airflow 2.1.1 is no longer included in Cloud Composer images.

Compute Engine - Generally Available: NVIDIA® T4 GPUs are now available in the following additional regions and zones: Las Vegas, Nevada,: us-west4-a,b Los Angeles, California: us-west2-b,c For more information about using GPUs on Compute Engine, see GPUs on Compute Engine.

Dataproc Metastore - v1beta1. For new projects, Dataproc Metastore service creations with CMEK enabled fail if a service without CMEK enabled has never been created before.

Dataproc - New sub-minor versions of Dataproc images: 1.4.71-debian10, 1.4.71-ubuntu18, 1.5.46-centos8, 1.5.46-debian10, 1.5.46-ubuntu18, 2.0.20-centos8, 2.0.20-debian10, 2.0.20-ubuntu18. Added support for enabling/disabling Ubuntu Snap daemon with cluster property dataproc:dataproc.snap.enabled. HIVE-21018: Grouping/distinct on more than 64 columns should be possible.

Eventarc - Eventarc can be configured for data location and is supported as a resource location.

Cloud Functions - Cloud Functions now supports PHP 7.4 at the General Availability release level.

IAM - You can now disable and enable service account keys.

Google Kubernetes Engine - (2021-R29) Version updates GKE cluster versions have been updated. In GKE versions 1.21.0-gke.1500 and later, VPC-native is the default network mode during cluster creation. For Autopilot clusters, starting with GKE version 1.21.3-gke.900: Users can also create mutating webhooks. With GKE versions 1.21.4-gke.30 and later, users can create ServiceAttachment resources to provision Private Service Connect (PSC) for internal LoadBalancer Services. Multi-cluster Ingress now supports SSL policies and HTTPS redirects using the FrontendConfig resource. GKE versions 1.19.14-gke.301 and later fix the issue with v1beta1 of the Backendconfig API, where a Cloud Armor security policy was inadvertently deleted from the backend Service of an Ingress resource.

GKE - (2021-R29) Version updates Version 1.20.9-gke.1001 is now the default version.

Google Kubernetes Engine Rapid - (2021-R29) Version updates Version 1.21.4-gke.301 is now the default version in the Rapid channel.

Google Kubernetes Engine Regular - (2021-R29) Version updates Version 1.20.9-gke.1001 is now the default version in the Regular channel.

Google Kubernetes Engine Stable - (2021-R29) Version updates Version 1.19.13-gke.701 is now the default version in the Stable channel.

Cloud Run for Anthos - Preview: Installing Cloud Run for Anthos as an Anthos feature is now available as a Preview.

KF - 2.5.3. Improved kf doctor reliability for Anthos on-prem clusters. Fixed an error that can occur during the initialization of the subresource API.

Cloud Run - You can now configure Cloud Run services to have CPU allocated for the entire lifetime of container instances.

Security Command Center - Event Threat Detection, a built-in service of Security Command Center Premium, has launched new detectors in public preview.

Anthos Service Mesh - 1.9.x & 1.10.x. 1.9.8-asm.3 and 1.10.4-asm.9 are now available.

SAP Solutions - Google Cloud monitoring agent for SAP NetWeaver, version 2.0 is generally available Version 2.0 of the monitoring agent for SAP NetWeaver is now generally available (GA).

Cloud SQL MySQL - Cloud SQL for MySQL now supports custom formatting controls for CSVs.

Cloud SQL Postgres - Cloud SQL for PostgreSQL has enhanced the support for multiline log entries in postgres.log. Cloud SQL for PostgreSQL now supports custom formatting controls for CSVs.

Vertex AI - Vertex Explainable AI is generally available (GA). You can use a pre-built container to serve predictions from TensorFlow 2.6 models.

Virtual Private Cloud - Enabling or disabling PROXY protocol after a Private Service Connect service attachment is created now correctly changes the configuration. Full control over which protocols are mirrored by Packet Mirroring is now available in General Availability.

Workflows - Call logging is available in Preview. Connectors are now generally available (GA).

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko