Welcome to issue #235 March 29th, 2021

A new GCP region in Warsaw / Poland is rolling out (unofficially), BigQuery is a leader as Cloud Data Warehouse solution, but we knew that already :)

News

Introducing Network Connectivity Center: A revolution in simplifying on-prem and cloud networking - With Network Connectivity Center, you can connect and manage VPNs, interconnects, third-party routers and SD-WAN across on-prem and cloud networks.

Batter up! Anthos on bare metal helps MLB gear up for upcoming season - Anthos on bare metal offers lower operational overhead than Anthos running on a virtualization layer, giving MLB much-needed flexibility.

High throughput VM logging and metrics agent now in Preview - Announcing the Preview of the new Ops Agent, which combines the metrics and logging agents into one, easy to install VM agent that supports a higher throughput.

Getting Compute Engine resources for batch processing just got easier - You can now obtain Compute Engine resources for batch processing workloads that account for availability of resources in zones of a region.

Google BigQuery is a Leader in The 2021 Forrester Wave: Cloud Data Warehouse - Forrester places Google as a Leader in the 2021 Forrester Wave™ Cloud Data Warehouse.

Run a transformed supply chain—see how at Google’s Digital Supply Chain summit - Check out how to build modern digital supply chain and logistics platforms and check out Google’s Supply Chain Summit.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Rethinking ‘rehost, replatform, rearchitect’: Cloud migration for the real world - An overview of different migration approaches.

A Google Cloud VMware Engine cheat sheet - A walk through VMware Engine, highlighting its benefits, features, and use cases.

The past, present and future of custom compute at Google - To meet users’ performance needs at low power, we’re doubling down on custom chips that use System on a Chip (SoC) designs.

How to automate the setup of a Kubernetes cluster on GCP - Using Ansible to install, setup, and configure a Google Kubernetes Cluster (GKE) on Google Cloud Platform.

Anthos on Bare Metal and Akri — Managing Leaf Devices on Edge Kubernetes Clusters from Cloud - Example of Anthos on Bare Metal usage.

Display GKE logs in a text format with kubectl - Script to format logs to display on CLI from GKE and Cloud Logging.

Secret Manager: protect your secrets from inside threats - Secret manager protect efficiently the secrets on Google Cloud. But when several teams need admin role, the security concerns occur.

Breaking up with our Secure-Drive, Google Cloud KMS to the rescue! - Using Cloud KMS to encrypt local files.

App Development, Serverless, Databases, DevOps

Check, please! Billing in Cloud Storage - So far, we've talked a lot about how to use Cloud Storage—from managing data to optimizing performance, uploading, downloading, and buckets. But there's one important topic that we haven't talked about: the price tag.

Limit file size when uploading to Cloud Storage - Set maximum size of a file that will be uploaded through signed URL to Cloud Storage bucket.

Handle multiple Google Cloud Shell using Windows Terminal - Using Windows terminal for multiple GCP projects.

How to mirror a GitLab project to Google Cloud Source Repositories - Setting up Cloud Source repo from a GitLab project.

Cloud Bigtable: What Is It And Why Might We Need It One Day? - Cloud Bigtable overview.

Securing a Google cloud API gateway with Identity provider like Keycloak - Using Keycloak (open source Identity and Access Management solution) for Auth in API Gateway.

Using Google’s Speech-to-Text API with Python - This post provides steps and python syntax for utilizing the Google Cloud Platform speech transcription service.

Big Data, Analytics, ML&AI

Cloud Composer/Apache Airflow, Dataform & BigQuery - Example of triggering Dataform transformation from Cloud Composer.

Node-RED solution deployment on GCP - Securing Node-RED deployments on GCP.

Trigger Cloud Data Fusion job using Cloud Functions - Triggering Data Fusion pipeline using Cloud Function when a file is loaded to Cloud Storage.

A Dataflow Journey: from PubSub to BigQuery - Exploiting Google Cloud Services to build a custom real time streaming data pipeline.

Understanding Customer Mobile App Journey Using Firebase Events and BigQuery. - Querying Firebase user events in BigQuery.

Using BigQuery Administrator for real-time monitoring - Resource Charts for BigQuery Administrator makes it easy to understand historical patterns across slot consumption, job concurrency, & job performance.

How to process large BigQuery tables/job result in a single memory machine with python - Python library to load large amount of data from BigQuery.

LTV — lifetime value - Analyzing user behavior in BigQuery.

Dekart - open-source data visualization and analytical tool - Dekart is an open-source analytical tool for large scale geo-spatial data. Dekart is based on Kepler.gl visualization and supports BigQuery as a data source.

How to train and deploy a Vaex model pipeline on Google Cloud Platform - No-pipeline deployments with Vaex.

Various

Road to recovery: How Google Cloud is helping states get the COVID-19 vaccine to more people - Google Cloud is proud to partner with a number of states across the U.S. to support vaccination efforts at scale.

Slides, Videos, Audio

Picture this: 10 whiteboard sketch videos that bring Google Cloud to life - The Google Cloud Tech YouTube channel has a series of whiteboard drawing board videos that explain cloud tools for technical practitioners in about 5 minutes each.

GCP Podcast - #252 GKE Cost Optimization with Kaslin Fields and Anthony Bushong.

Kubernetes Podcast - #143 Replicated, with Grant Miller.

Lunch & Learn BigQuery & Firebase from other Google Cloud customers

Terraforming with Freeman: Episode Five - Cloud Shift - Using Terraform on Cloud Platform.

Machine Learning with Spark on Google Cloud Dataproc Workshop - In this workshop, you will learn how to prepare the Spark interactive shell on a Google Cloud Dataproc cluster, create a training dataset for machine learning using Spark, develop a logistic regression machine learning model using Spark, and evaluate the predictive behaviour of a machine learning model using Spark on Google Cloud Datalab.

Releases

Cloud Dataflow - Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in europe-central2 (Warsaw). Dataflow SQL now supports user-defined functions (UDFs) written using SQL.

Dataproc - Dataproc is now available in the europe-central2 region (Warsaw). The default Dataproc image is now image version 2.0. Announcing the General Availability (GA) release of Dataproc cluster Stop/Start. New sub-minor versions of Dataproc images: 1.3.88-debian10, 1.3.88-ubuntu18, 1.4.59-debian10, 1.4.59-ubuntu18, 1.5.34-centos8, 1.5.34-debian10, 1.5.34-ubuntu18, 2.0.7-centos8, 2.0.7-debian10, and 2.0.7-ubuntu18. Image 2.0: Updated Iceberg to version 0.11.0. Image 2.0: HIVE-22373: File Merge tasks fail when containers are reused. Fixed a bug that caused Hive jobs to fail on Ranger-enabled clusters. Fixed a bug where Spark event logs directory and history server directory could not be set to Cloud Storage correctly. Fixed a bug where Presto property value with ';' could not be set correctly in the config file. CVE-2020-13957: SOLR-14663: ConfigSets CREATE does not set trusted flag. CVE-2020-1926: HIVE-22708: Test fix for http transport.

Cloud Networking Products - The API for updating ResourceRecordSets in Cloud DNS is now available in GA.

KMS - The europe-central2 region in Warsaw is now available.

Google Kubernetes Engine - The europe-central2 region in Warsaw is now available. Starting tomorrow, March 24, 2021, the mechanism we use to create GKE release notes will change. Workload Identity for Windows Server nodes is now available in GKE versions 1.18.16-gke.1200, 1.19.8-gke.1300, 1.20.4-gke.1500, and later. Windows Server, version 1909 is reaching end of support on May 11, 2021.

Google Kubernetes Engine Rapid - Starting tomorrow, March 24, 2021, the mechanism we use to create GKE release notes will change. Windows Server, version 1909 is reaching end of support on May 11, 2021.

Load Balancing - Subsetting for internal TCP/UDP load balancers lets you scale your internal TCP/UDP load balancer to support a larger number of backend VM instances per internal backend service.

Cloud PubSub - Pub/Sub is now available in the europe-central2 region (Warsaw).

Recommender - v1. Recommender pricing is now available and determines how you can process and view recommendations using the API and BigQuery export.

Resource Manager - The Resource Manager v3 API has been released into public preview.

Cloud Spanner - Cloud Spanner regional instances can now be created in Warsaw (europe-central2).

Cloud Speech-to-Text - Speech-to-Text now allows you to upload your longrunning transcription results directly into a Cloud Storage bucket.

Cloud SQL Postgres - Support for europe-central2 (Warsaw ).

Cloud Storage - Warsaw region (europe-central2) launched.

Traffic Director - Traffic Director support for xDS clients that connect and request configuration using the xDS x3 API is now Generally Available.

Cloud Vision API - EXIF rotation featured fixed EXIF rotation is now disabled.

Virtual Private Cloud - For auto mode VPC networks, added a new subnet 10.186.0.0/20 for the Warsaw europe-central2 region. The ability to connect VM interfaces other than nic0 to a Shared VPC is now available in General Availability for instance templates and managed instance groups.

Cloud VPN - Cloud VPN is now available in region europe-central2 (Warsaw, Poland).

Anthos GKE deployed on-prem - Anthos clusters on VMware 1.7.0-gke.16 is now available. Cluster lifecycle improvements The cluster upgrade process has changed. Platform enhancements GA: Node auto repair is now generally available and enabled by default for newly created clusters. Security enhancements: Preview: You can run Container-Optimized OS on your user cluster worker nodes. Simplify Day-2 operations: GA: Support for vSphere folders is now generally available. Functionality changes: gkectl diagnose cluster now includes validation load balancing, including F5, Seesaw, and manual mode. Breaking changes: The following Google Cloud API endpoints must be allowlisted in network proxies and firewalls. Fixes: Fixed a bug where Grafana dashboards based on the container_cpu_usage_seconds_total metric show no data. Known issues: Calico-node Pods sometimes use an excessive amount of CPU in large-scale clusters.

Secret Manager - The europe-central2 region is now available.

Anthos clusters on bare metal - 1.7. Anthos on bare metal 1.7.0 is now available. Functionality changes: Added support for configuring the number of pods per node. Known issues: Under rare circumstances, bmctl upgrade may become stuck at the Moving resources to upgraded cluster stage after finishing upgrading all nodes in the cluster.

GKE on-prem 1.5 - Anthos clusters on VMware 1.7.0-gke.16 is now available. Cluster lifecycle improvements The cluster upgrade process has changed. Platform enhancements GA: Node auto repair is now generally available and enabled by default for newly created clusters. Security enhancements: Preview: You can run Container-Optimized OS on your user cluster worker nodes. Simplify Day-2 operations: GA: Support for vSphere folders is now generally available. Functionality changes: gkectl diagnose cluster now includes validation load balancing, including F5, Seesaw, and manual mode. Breaking changes: The following Google Cloud API endpoints must be allowlisted in network proxies and firewalls. Fixes: Fixed a bug where Grafana dashboards based on the container_cpu_usage_seconds_total metric show no data. Known issues: Calico-node Pods sometimes use an excessive amount of CPU in large-scale clusters.

VMware Engine - Added support for using NetApp Cloud Volumes Service for Google Cloud.

Anthos clusters on VMware 1.7 - Anthos clusters on VMware 1.7.0-gke.16 is now available. Cluster lifecycle improvements The cluster upgrade process has changed. Platform enhancements GA: Node auto repair is now generally available and enabled by default for newly created clusters. Security enhancements: Preview: You can run Container-Optimized OS on your user cluster worker nodes. Simplify Day-2 operations: GA: Support for vSphere folders is now generally available. Functionality changes: gkectl diagnose cluster now includes validation load balancing, including F5, Seesaw, and manual mode. Breaking changes: The following Google Cloud API endpoints must be allowlisted in network proxies and firewalls. Fixes: Fixed a bug where Grafana dashboards based on the container_cpu_usage_seconds_total metric show no data. Known issues: Calico-node Pods sometimes use an excessive amount of CPU in large-scale clusters.

Pub/Sub Lite - Pub/Sub Lite now supports increasing the number of partitions in a topic.

Anthos clusters on VMware 1.6 - Anthos clusters on VMware 1.7.0-gke.16 is now available. Cluster lifecycle improvements The cluster upgrade process has changed. Platform enhancements GA: Node auto repair is now generally available and enabled by default for newly created clusters. Security enhancements: Preview: You can run Container-Optimized OS on your user cluster worker nodes. Simplify Day-2 operations: GA: Support for vSphere folders is now generally available. Functionality changes: gkectl diagnose cluster now includes validation load balancing, including F5, Seesaw, and manual mode. Breaking changes: The following Google Cloud API endpoints must be allowlisted in network proxies and firewalls. Fixes: Fixed a bug where Grafana dashboards based on the container_cpu_usage_seconds_total metric show no data. Known issues: Calico-node Pods sometimes use an excessive amount of CPU in large-scale clusters.

Cloud VPN - Cloud VPN is now available in region europe-central2 (Warsaw, Poland).

GKE - Starting tomorrow, March 24, 2021, the mechanism we use to create GKE release notes will change.

Access Approval - v1. Access Transparency logs contain a new field called accessApprovals.

AI Platform - Notebooks - v1. Cross Project Service Account support.

AppEngine Standard Python3 - App Engine standard environment provides a new metric, CPU Utilization, which indicates the CPU utilization average over all active instances.

BI Engine - BigQuery BI Engine is now available in the Warsaw (europe-central2) region.

BigQuery - BigQuery is now available in the Warsaw (europe-central2) region.

BigQuery ML - BigQuery ML is now available in the Warsaw (europe-central2) region.

BigQuery Transfer - BigQuery Data Transfer Service is now available in the Warsaw (europe-central2) region.

BigTable - Cloud Bigtable is now available in the europe-central2 (Warsaw) region. Cloud Bigtable's Cloud Console navigation has been improved.

Cloud Composer - Cloud Composer 1.15.1 release started on March 23, 2021.

Compute Engine - Generally available: Start and stop virtual machine (VM) instances automatically using instance schedules. General-purpose E2 and N1 machines are available in Warsaw, Poland europe-central2 in all three zones. Disks, snapshots, and images are available in Warsaw, Poland europe-central2 in all three zones. Support for OS Login in VPC Service Controls is now Generally Available.

Config Connector - Config Connector version 1.44.0 is now available. Added support for the ContainerAnalysisNote resource (no config-connector CLI support). Added mtu field to ComputeInterconnectAttachment. Added nodeConfig.ephemeralStorageConfig field to ContainerCluster and ContainerNodePool. Added settings.backupConfiguration.backupRetentionSettings and settings.backupConfiguration.transactionLogRetentionDays fields to SQLInstance. Made materializedView.query field in BigQueryTable immutable. Deprecated nicType field in ComputeInstanceTemplate. Added support for acquisitions of Folder using displayName and folderRef/organizationRef. Fixed incorrect file extension for Terraform files output by the config-connector CLI. Config Connector version 1.43.0 is now available. config-connector CLI now supports a flag to filter out deleted IAM members. Added support for IAPBrand (no config-connector CLI support). Added support for IAPIdentityAwareProxyClient (no config-connector CLI support). Conflict Prevention is now turned off by default. Webhook certificates that do not contain a SAN are now re-created on upgrade of the Config Connector operator. Added support for folderRef and organizationRef in Project and Folder.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko