Welcome to issue #196 June 29th, 2020

News

Your Next ‘20 OnAir journey starts here: Resources and session guides available now - Google Cloud Next ‘20 OnAir, running from Jul 14 to Sep 8, offers nine full weeks of programming to help you solve your toughest business challenges in the cloud.

The new Google Cloud region in Jakarta is now open

More value for less with Cloud SQL committed use discounts - Cloud SQL on Google Cloud database instances now have a committed use discount, applicable to MySQL, PostgreSQL, and SQL Server.

Increase visibility into Cloud Spanner performance with transaction stats - Cloud Spanner now offers transaction statistics, so you can run SQL queries to retrieve these stats for your database over several time periods.

Introducing Pub/Sub as a new notification channel in Cloud Monitoring - Cloud Monitoring now supports Pub/Sub as a notification channel.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How maintenance windows affect your error budget — SRE tips - See how maintenance windows can impact your error budget when using SRE practices, and get tips on how and when to use them.

Zero-Trust Security on GCP With Context-Aware Access - Identity Aware Proxy for serverless products on GCP.

Using Google-Managed Certificates and Identity-Aware Proxy With GKE - Setting up Identity Aware Proxy for GKE.

Migrating from Cloud Endpoints to DB-less Kong - Deploying serverless infrastructure with Terraform and solving problems.

App Development, Serverless, Databases, DevOps

Detecting and responding to Cloud Logging events in real-time - Learn how to automate responses based on changes to Cloud Logging events.

Domain Mapping in GCP - Detail guide on how to set up a custom domain for Cloud Run application.

Managing background jobs with Cloud Tasks - Use Cloud Tasks to manage your background jobs at scale, without drowning your worker.

How many COVID-19 cases near me? A chatbot solution to scale the one of the most asked questions. - Dialogflow application which is using public BigQuery dataset to provide info about COVID-19 based on postal code.

Redis with Shared VPC and Private Service Access - Imagine that we wish a Memorystore redis managed instance to be available to a number of our projects. How might we achieve this?

How to Build a Serverless Daily Weather SMS Alert with Twilio and Google Cloud Platform - A quick dive into serverless functions, third party APIs, as well as cloud cron jobs!

7 gcloud Tricks You Probably Didn’t Know - Tips and tricks when using gcloud command.

DML and Mutations - a tale of two data altering techniques in Cloud Spanner - In the article are described two APIs in Cloud Spanner that you can use to modify data: Data Manipulation Language (DML) and Mutations.

Online payments for data science apps (DSaaS) using R, Shiny, Firebase, Paddle and Google Cloud Functions - A template for R users to create paid subscription services for Shiny Apps on GCP.

Setup and Invoke Cloud Functions using Python - This articles describes the process of development, deployment and setting access for Cloud Functions (in Python).

Big Data, Analytics, ML&AI

Building production-ready data pipelines using Dataflow: Overview - The production guide for Dataflow, including sections on architecture, development process, CI/CD etc.

I’m your father… Data Lineage with Cloud Data Fusion - How to use data lineage with Cloud Data Fusion, the fully managed, cloud-native, enterprise data integration service for data integration.

Custom cohort size using Range Bucket in SQL. - Using RANGE_BUCKET command in BigQuery.

BigQuery: Creating Nested Data with SQL - Working with SQL on nested data in BigQuery can be very performant. But what if your data comes in flat tables like CSV’s?

BigQuery Streaming API with Postman - How to create and execute BigQuery streaming requests from Postman.

Firebase Event Analytics with Google BigQuery - An extensive guide to help you get started with app analytics.

Easy pivot() in BigQuery, finally - Using dynamic SQL and stored procedures to pivot in BigQuery.

Enhance your TensorFlow Lite deployment with Firebase - Blog explains how to leverage Firebase to enhance your deployment of TensorFlow Lite models in production.

Predict workload failures before they happen with AutoML Tables - How to predict whether HPC workloads will succeed or fail with the help of AutoML.

Various

Bayer Crop Science seeds the future with 15000-node GKE clusters - Learn how GKE’s support of up to 15,000 nodes per cluster benefits a wide range of use cases, including helping Bayer Crop Science rapidly process new information arriving from its genotyping labs.

Google Cloud Professional Data Engineer Certification — 2020 Mini-Guide - This article is intended to serve as a mini-guide for people taking GCP Data Engineer the exam.

How we scaled our retail operations with cloud and app platforms during the lockdown - Using GCP products for retail and hospitality business.

Slides, Videos, Audio

GCP Podcast - #224 Solutions Engineering with Grace Mollison and Ann Wallace.

Kubernetes Podcast - #109 Kubermatic, with Sebastian Scheele.

Releases

AI Platform - Deep Learning VMs - M50 release Miscellaneous bug fixes.

Anthos Config Management - 1.4.0. Anthos Config Management is now Generally Available on AKS (Kubernetes v1.16 or higher) and EKS (Kubernetes v1.16 or higher). Config Connector is not currently supported on EKS or AKS, as it is unable to run on these providers. The following Policy Controller constraint templates have been added to the Default Template Library: allowedserviceportname destinationruletlsenabled disallowedauthzprefix policystrictonly sourcenotallauthz The following constraint templates have been updated: k8sblockprocessnamespacesharing k8sdisallowedrolebindingsubjects k8semptydirhassizelimit k8slocalstoragerequiresafetoevict k8smemoryrequestequalslimit k8snoexternalservices k8spspallowedusers k8spspallowprivilegeescalationcontainer k8spspapparmor k8spspcapabilities k8spspflexvolumes k8spspforbiddensysctls k8spspfsgroup k8spsphostfilesystem k8spsphostnamespace k8spsphostnetworkingports k8spspprivilegedcontainer k8spspprocmount k8spspreadonlyrootfilesystem k8spspseccomp k8spspselinux k8spspvolumetypes See the Default Template Library documentation for more information. Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 25ca799). The nomos CLI tool now supports the KUBECONFIG environment variable in a way that matches the kubectl behavior with multiple delimited configuration files. Anthos Config Management no longer gets into a continuous PATCH loop when encountering unmanaged resources with config-management annotations and a missing last-applied-configuration annotation. Anthos Config Management is not issuing errors when it encounters certain types of malformed configurations in a resource definition. Policy Controller may fail to start successfully when synced resources are marked for deletion. This release includes several logging and performance improvements.

AppEngine Standard Go - The Go 1.14 runtime Beta for the App Engine standard environment is now available.

Google Cloud Armor - Promotional pricing for Google Cloud Armor is extended to July 31, 2020.

BigQuery - Region qualified INFORMATION_SCHEMA views are now in beta.

BigQuery ML - BigQuery ML now supports time series models as a beta release.

Billing - Committed use discounts (CUDs) are now available to purchase for Cloud SQL.

Cloud Build - Cloud Build's substitution variables can now refer to other substitution variables, manipulate them using bash-style string operations, and pull information from a trigger event payload.

Cloud Composer - New versions of Cloud Composer images: composer-1.10.5-airflow-1.10.2, composer-1.10.5-airflow-1.10.3 and composer-1.10.5-airflow-1.10.6. Composer now uses the Kubernetes v1 API, and is compatible with GKE 1.16 An updated haproxy configuration for Composer increases the maximum number of connections to 2000, and changes load balancing to be based on the number of connections. Error messages for TP_APP_ENGINE_CREATING timeout and RPC delivery issues have been expanded.

Compute Engine - To support a wide variety of BYOL scenarios, you can now configure VMs to live migrate within a sole-tenant node group during host maintenance events. N2D machine types are now available in Belgium, europe-west1, in all three zones.

Config Connector - Add an option, iam-format, to config-connector to control IAM output, options are policy, policymember, or none. ComputeForwardingRule's target field now supports referencing a ComputeTargetSSLProxy and ComputeTargetTCPProxy. DataFlowJob's serviceAccountEmail, network, subnetwork, machineType, and ipConfiguration fields now support updates. Fix an issue where config-connector would error on a Project resource.

Dataproc - New subminor image versions: 1.2.100-debian9, 1.3.60-debian9, 1.4.31-debian9, 1.3.60-debian10, 1.4.31-debian10, 1.5.6-debian10, 1.3.60-ubuntu18, 1.4.31-ubuntu18, 1.5.6-ubuntu18, preview 2.0.0-RC2-debian10, and preview 2.0.0-RC2-ubuntu18. Image 2.0 preview: SPARK-22404: set spark.yarn.unmanagedAM.enabled property to true on clusters where Kerberos is not enabled to run Spark Application Master in driver (not managed in YARN) to improve job execution time. Fixed a quota validation bug where accelerator counts were squared before validation -- for example, previously if you requested 8 GPUs, Dataproc validated whether your project had quota for 8^2=64 GPUs.

Cloud Firestore - The Google Cloud console now includes a Firestore usage dashboard.

GKE on Prem - Anthos GKE on-prem 1.4.0-gke.13 is now available. Updated to Kubernetes 1.16: Please note that Kubernetes 1.16 has deprecated some of its APIs. Simplified upgrade: This release provides a simplified upgrade experience via the following changes: Automatically migrate information from the previous version of admin workstation using gkeadm. Improved installation and cluster configuration: The user cluster node pools feature is now generally available. Improved disaster recovery capabilities: This release provides enhanced disaster recovery functionality to support backup and restore HA user cluster with etcd. Enhanced monitoring with Cloud Monitoring (formerly Stackdriver): This release provides better product monitoring and resource usage management via the following changes: Introduces a default monitoring dashboard. Functionality changes: Enabled Horizontal Pod Autoscaler (HPA) for the Istio ingress gateway. Support for a vSphere folder (Preview): This release allows customers to install GKE on-prem in a vSphere folder, reducing the scope of the permission required for the vSphere user. Improved scale: This release improves the cluster scalability by supporting a maximum of 10 instead of 5 user clusters for each admin cluster. Fixes: Fixed the issue of the user cluster's Kubernetes API server not being able to connect to kube-etcd after admin nodes and user cluster master reboot. Known issues: If a user cluster is created without any node pool named the same as the cluster, managing the node pools using gkectl update cluster would fail. If your vSphere environment has fewer than three hosts, user cluster upgrade might fail.

IAM - Using the Cloud IAM API to sign JSON Web Tokens (JWTs) or binary blobs is now deprecated.

KMS - Keys hosted by Thales are now supported in Cloud EKM.

Google Kubernetes Engine - There is a known that may cause multiple Pods on the same node to be allocated with the same IPv4 address leading to possible service disruption. Ensure your cluster(s) are subscribed to a release channel, or you have node auto-upgrade enabled. If you are experiencing any issues or do not want to use auto-upgrade you can manually initiate an upgrade at your earliest convenience. If you are experiencing issues and wish to update proactively: Follow the steps in the Manually upgrading a cluster page to upgrade the cluster master. Upgrade your node pool by applying the latest patch available for your node version. Consider using surge upgrade for your nodepool upgrade. Use the following table to determine which patch version is applicable for your cluster(s): GKE cluster versions have been updated. Masters and nodes with auto-upgrade enabled will be upgraded: Rollouts are phased across multiple weeks, to ensure cluster and fleet stability. The following Kubernetes versions are now available for new clusters and for opt-in master upgrades and node upgrades for existing clusters. No channel Note: Your clusters might not have these versions available. The COS image for GKE 1.14.10-gke-43 clusters and is cos-73-11647-459-0.

Load Balancing - The introductory period during which you can use Internal HTTP(S) Load Balancing without charge is coming to an end.

Cloud SQL MySQL - Committed use discounts (CUDs) are now available to purchase for Cloud SQL.

Cloud SQL Postgres - Committed use discounts (CUDs) are now available to purchase for Cloud SQL.

Cloud SQL SQL Server - Committed use discounts (CUDs) are now available to purchase for Cloud SQL.

Deep Learning VM - M50 release Miscellaneous bug fixes.

Anthos GKE deployed on-prem - Anthos GKE on-prem 1.4.0-gke.13 is now available. Updated to Kubernetes 1.16: Please note that Kubernetes 1.16 has deprecated some of its APIs. Simplified upgrade: This release provides a simplified upgrade experience via the following changes: Automatically migrate information from the previous version of admin workstation using gkeadm. Improved installation and cluster configuration: The user cluster node pools feature is now generally available. Improved disaster recovery capabilities: This release provides enhanced disaster recovery functionality to support backup and restore HA user cluster with etcd. Enhanced monitoring with Cloud Monitoring (formerly Stackdriver): This release provides better product monitoring and resource usage management via the following changes: Introduces a default monitoring dashboard. Functionality changes: Enabled Horizontal Pod Autoscaler (HPA) for the Istio ingress gateway. Support for a vSphere folder (Preview): This release allows customers to install GKE on-prem in a vSphere folder, reducing the scope of the permission required for the vSphere user. Improved scale: This release improves the cluster scalability by supporting a maximum of 10 instead of 5 user clusters for each admin cluster. Fixes: Fixed the issue of the user cluster's Kubernetes API server not being able to connect to kube-etcd after admin nodes and user cluster master reboot. Known issues: If a user cluster is created without any node pool named the same as the cluster, managing the node pools using gkectl update cluster would fail. If your vSphere environment has fewer than three hosts, user cluster upgrade might fail.

Service Mesh - 1.5.6-asm.0 and 1.4.10.asm.2 Contains the same fixes as OSS Istio 1.5.6.

Anthos GKE on-prem - Anthos GKE on-prem 1.4.0-gke.13 is now available. Updated to Kubernetes 1.16: Please note that Kubernetes 1.16 has deprecated some of its APIs. Simplified upgrade: This release provides a simplified upgrade experience via the following changes: Automatically migrate information from the previous version of admin workstation using gkeadm. Improved installation and cluster configuration: The user cluster node pools feature is now generally available. Improved disaster recovery capabilities: This release provides enhanced disaster recovery functionality to support backup and restore HA user cluster with etcd. Enhanced monitoring with Cloud Monitoring (formerly Stackdriver): This release provides better product monitoring and resource usage management via the following changes: Introduces a default monitoring dashboard. Functionality changes: Enabled Horizontal Pod Autoscaler (HPA) for the Istio ingress gateway. Support for a vSphere folder (Preview): This release allows customers to install GKE on-prem in a vSphere folder, reducing the scope of the permission required for the vSphere user. Improved scale: This release improves the cluster scalability by supporting a maximum of 10 instead of 5 user clusters for each admin cluster. Fixes: Fixed the issue of the user cluster's Kubernetes API server not being able to connect to kube-etcd after admin nodes and user cluster master reboot. Known issues: If a user cluster is created without any node pool named the same as the cluster, managing the node pools using gkectl update cluster would fail. If your vSphere environment has fewer than three hosts, user cluster upgrade might fail.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko