Welcome to issue #262 October 4th, 2021

The number of releases and functionalities is rising, having Cloud Next just around the corner.

Minor personal milestone, this issue marks 5 years anniversary of GCP Weekly. Thank you for your support and interest. 

News

Simplifying App Engine updates: extending support for bundled services - Announcing the launch of support for App Engine bundled services in second-generation runtimes for Java, Python, and Go.

Introducing Workflows callbacks - Introducing Workflows callbacks. Thanks to callbacks, you can put a human being or autonomous system into the loop. If your processes require human validation, or an external system to come back to you to go further, callbacks are the solution.

Speaker ID unlocks Machine Learning Speech Identification capability for Contact Centers - Google Cloud announces Speaker ID; brings Google’s speech identification technology directly to customers and contact center partners.

Built-in transparency, automation, and interoperability for Cloud KMS - New features bring increased transparency, improved interoperability, and greater automation to Google Cloud KMS.

N2 VMs with Intel processor refresh enables over 30% better price-performance - New N2 VMs powered by the 3rd Generation Intel Xeon Scalable Processor (code-named Ice Lake) further improve our Compute Engine offerings.

Protect your apps from bots with Cloud Armor and reCAPTCHA Enterprise - Protect your apps from bots with Cloud Armor and reCAPTCHA Enterprise.

Do more with less: Introducing Cloud SQL Cost optimization recommendations with Active Assist - Get AI-powered recommendations for methods to reduce cost and optimize instance sizing in Cloud SQL.

Supercharge your Cloud NAT: Introducing new Cloud NAT features - Introducing new Google Cloud NAT features that improve scalability and flexibility for Compute Engine and Kubernetes Engine workloads.

Improve your security posture with new Overly Permissive Firewall Rule Insights - Improve your security posture with the new Overly Permissive Firewall Rule Insights module, based on firewall log analysis.

Announcing Apigee Integration: An API-first approach for connecting data and applications - Google Cloud’s Apigee API management platform adds powerful new integration capabilities.

Manage Capacity with Pub/Sub Lite Reservations. It’s easier and cheaper. - Inexpensive and simpler messaging for large scale streaming analytics and data integration. Like Apache Kafka, Pub/Sub Lite a partitioned, ordered log.

Tales from the (en)crypt: What's new for Cloud Storage security - Encryption enhancements include Cloud Key Management performance improvements and support for customer-managed encryption keys (CMEK) for object composition.

Google named a leader in the 2021 Gartner® Magic Quadrant® for Full Life Cycle API Management - Gartner has recognized Google Cloud’s Apigee as a Leader in the 2021 Magic Quadrant for Full Life Cycle API Management, marking the sixth time in a row that earned this recognition.

Building and expanding network services for a smart and connected world - In the past year, we’ve expanded the portfolio of networking services that you can use on top of our planet-scale infrastructure.

VMware and Google Cloud: The next chapter - VMware and Google Cloud summarize the past year of of our partnership success with Google Cloud VMware Engine.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Cloud CISO Perspectives: September 2021 - Google Cloud CISO Phil Venables shares his thoughts on what to expect for security at Google Cloud Next ‘21, digital sovereignty, global compliance updates and more.

Cloud NAT explained! - For security, it is a best practice to limit the number of public IP addresses in your network. In Google Cloud, Cloud NAT (network address translation) lets certain resources without external IP addresses create outbound connections to the internet.

What is Cloud Load Balancing? - Cloud Load Balancing is a fully managed software-defined load balancing solution in the cloud.

Simplify traffic steering with Cloud DNS routing policies - Cloud DNS routing policies (geo-location and weighted round robin) helps you define custom ways to steer private and Internet traffic using DNS.

Deploy Anthos on GKE with Terraform Part 2: Guardrails via Policy Controller - Part two in a series on Anthos Config Management (ACM) with Terraform, this shares how to enable Policy Controller to audit and enforce fully programmable policies for your Kubernetes clusters.

Precautions for Termination of Pod of Container Native Load balancing of GKE - An overview of Container Native Load Balancing and how to properly configure Pods to prevent downtimes.

How to pass secrets to Terraform from Google Cloud Build - This tutorial explains how to pass secretes from Secret Manager to Terraform in Cloud Build.

Anthos Blog Series (Part 1) — Anthos Service Mesh - An overview of Anthos Service Mesh.

Cloud Bursting from On-Premise Kubernetes Clusters to Google Cloud Compute Engine - An example of the Hybrid cloud environment to illustrate Cloud Bursting from an on-premise Kubernetes cluster into Google Cloud GCE instances.

VPC Service Controls in Plain English - VPC Service Controls explained for technical and non-technical stakeholders alike.

App Development, Serverless, Databases, DevOps

N2D VMs with latest AMD EPYC CPUs enable on average over 30% better price-performance - Compute Engine N2D VMs with 3rd Generation AMD EPYC processors deliver, on average, over 30% better price-performance compared to prior generations.

Build and run a Discord bot on top of Google Cloud - Essential first steps to making your Hello, World bot with Discord, and hosting it on Google Cloud, so it persists even when you and your computer are asleep.

Analytics Labels for Messaging Campaigns - Using labels in Firebase Cloud Messaging for better analytics.

Migrating table schemas from Apache HBase to Cloud Bigtable - Cloud Bigtable Schema Translation Tool: A new tool to seamlessly create new tables in Cloud Bigtable from an existing Apache HBase table’s schema.

Learn how to create alerts based on your database logs in Cloud SQL - How to inspect database logs in Cloud SQL and create metrics and alerts based on those logs.

Utilizing GCP’s Identity Aware Proxy to SSH into Internal-IP only VM’s - A demo in this article consists of creating virtual machine, enabling the Identity Aware Proxy API to tunnel through to a VM that doesn’t have an external IP address.

Google Cloud Shell — Resources to get started - Let’s learn about Google Cloud Shell and how it can help you as a developer to get more productive working with Google Cloud Applications.

Kaniko and Google Container Registry without gcloud sdk and Docker - Learn how to use Kaniko with gcr.io without Docker or gcloud present.

App Engine to Google Kubernetes Engine - a journey - An overview of migrating from App Engine to GKE, issues and challenges, lessons learned.

Building Uber’s Fulfillment Platform for Planet-Scale using Google Cloud Spanner - An overview of how Uber's Fulfillment Platform is built using Cloud Spanner.

Big Data, Analytics, ML&AI

Building the data science driven organisation from the first principles - Learn about the basics and best practices on how to build a data science driven culture depending on what type of organization you are.

Managing a BigQuery data warehouse at scale - How Teads manages BigQuery data warehouse and monitors slow queries, slots usage, and table & field sizes.

Mathematical Functions you should know in BigQuery - How to Work with Numbers in BigQuery.

BigQuery Geospatial query tricks - Two useful tricks on how to implement some complex GIS queries in BigQuery.

Testable BigQuery SQL - Espresso — A framework for writing testable BigQuery queries.

Campaign Comparison Dashboard - Comparing different campaigns in the same dashboard using BigQuery and Data Studio.

Google Cloud Vertex AI + Battlesnake: Using practical reinforcement learning to defeat your friends - Learn how a small team applied Google Cloud services, Vertex AI and Reinforcement Learning to rapidly develop a game playing snake app.

People and planet AI: How to build a Time Series Model to classify fishing activities in the sea - In this episode of People & Planet AI, we share how to build a time series classification app that includes latitude and longitudinal fishing data, in order to enable fair and sustainable use of our oceans.

QuillBot cuts writing time for over 10 million users using Google Cloud - QuillBot gains scalability and cost savings for natural language processing platform with Google Cloud Compute Engine and NVIDIA GPUs.

Monitoring feature attributions: How Google saved one of the largest ML services in trouble - Feature Attributions and Explainable AI can be sometimes a critical factor of MLOps. Learn how Google saved one of its largest ML services in trouble with the methodology.

Various

Your guide to all things AI & ML at Google Cloud Next - A comprehensive guide to all things AI & ML at Next 2021. The list covers AI topics from Conversational AI to Document Understanding to Data plus AI.

Stephanie Wong’s guide to #GoogleCloudNext 2021 - A cross-product guide featuring Stephanie Wong’s top Google Cloud Next 2021 sessions for developers, data scientists, security engineers, cloud architects, and more.

Google Cloud President Rob Enslin: Touching Billions of People’s Lives - An inverview with Rob Enslin, Google Cloud president.

Mentoring GDG Cloud Abidjan — a collaboration between Europe and Africa - For one year now, Julien Landuré — a Cloud GDE, GDG Cloud Nantes leader and DevFest organizer — has been mentoring the GDG Cloud in….

Slides, Videos, Audio

GCP Podcast - #278 Managing non-REST APIs like GraphQL and gRPC with Nandan Sridhar and David Feuer.

Security Podcast - #33 Cloud Migrations: Security Perspectives from The Field.

Releases

Anthos clusters on AWS - Anthos Clusters on AWS aws-1.9.0-gke.2 is now available. You can now launch Kubernetes 1.21 clusters. Anthos Identity Service is available on Kubernetes clusters version 1.21 and above. Kubernetes 1.21 clusters now support the Kubernetes Konnectivity tool for communication between nodes and the control plane. You can now update the OIDC configuration on a running cluster. You can now specify a Cloud Storage Bucket name where Anthos clusters on AWS stores configuration data. You can now launch node pools with AWS R5 instances. The VolumeSnapshot resource API version v1beta1 is deprecated in Kubernetes 1.21 clusters. A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server. You cannot create new 1.16 clusters. Error messages when upgrading or downgrading your clusters have been clarified.

Anthos clusters on bare metal - 1.9. Release 1.9.0 Anthos clusters on bare metal 1.9.0 is now available for download. Improved cluster lifecycle functionalities: Preview: Added ability to reset individual nodes with the bmctl reset node command. Functionality changes: Changed default container runtime to containerd, containerRuntime: containerd for new clusters. Fixes: Added missing registry mirror package required for Cloud Audit Logs to the Registry Mirror. Known issues: Control group v2 (cgroup v2) is not officially supported in Anthos clusters on bare metal release 1.9.0 and later.

GKE on-prem 1.6 - Anthos clusters on VMware 1.9.0-gke.8 is now available. Features: Cluster lifecycle Improvements: GA: You can register an admin cluster during its creation by filling in the gkeConnect section in the admin cluster configuration file, similar to user cluster registration. Breaking changes: User cluster registration is now required and enforced. Changes: There is now a checkpoint file for the admin cluster, located in the same datastore folder as the admin cluster data disk, with the name DATA_DISK_NAME-checkpoint.yaml, or DATA_DISK_NAME.yaml if the length of DATA_DISK_NAME is greater than the filename length limit. Fixes: Fixed the issue of gkeadm trying to set permissions for the component access service account when --auto-create-service-accounts=false. Restoring an admin cluster from a backup using gkectl repair admin-master –restore-from-backup fails when using a private registry.

GKE on-prem 1.7 - Anthos clusters on VMware 1.9.0-gke.8 is now available. Features: Cluster lifecycle Improvements: GA: You can register an admin cluster during its creation by filling in the gkeConnect section in the admin cluster configuration file, similar to user cluster registration. Breaking changes: User cluster registration is now required and enforced. Changes: There is now a checkpoint file for the admin cluster, located in the same datastore folder as the admin cluster data disk, with the name DATA_DISK_NAME-checkpoint.yaml, or DATA_DISK_NAME.yaml if the length of DATA_DISK_NAME is greater than the filename length limit. Fixes: Fixed the issue of gkeadm trying to set permissions for the component access service account when --auto-create-service-accounts=false. Restoring an admin cluster from a backup using gkectl repair admin-master –restore-from-backup fails when using a private registry.

Anthos GKE on AWS, Anthos clusters on VMware 1.7 - Anthos Clusters on AWS aws-1.9.0-gke.2 is now available. You can now launch Kubernetes 1.21 clusters. Anthos Identity Service is available on Kubernetes clusters version 1.21 and above. Kubernetes 1.21 clusters now support the Kubernetes Konnectivity tool for communication between nodes and the control plane. You can now update the OIDC configuration on a running cluster. You can now specify a Cloud Storage Bucket name where Anthos clusters on AWS stores configuration data. You can now launch node pools with AWS R5 instances. The VolumeSnapshot resource API version v1beta1 is deprecated in Kubernetes 1.21 clusters. A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server. You cannot create new 1.16 clusters. Error messages when upgrading or downgrading your clusters have been clarified.

AppEngine Standard - Many legacy App Engine APIs are now available to select second-generation runtimes.

Google Cloud Armor - Google Cloud Armor Adaptive Protection is now in General Availability.

BigQuery - BigQuery pricing has changed as follows: BigQuery Storage Read API has moved from a single regional SKU to a set of regional SKUs for bytes scanned. BigQuery now supports the following geospatial data functions: ST_BUFFER: Returns a GEOGRAPHY that represents the buffer around the input GEOGRAPHY. Table functions are now generally available (GA). BigQuery now supports the following geospatial data functions: ST_BOUNDINGBOX: Returns a STRUCT that represents the bounding box for a geography.

Bigtable - Storage limits for Cloud Bigtable nodes have been doubled.

Binary Authorization - Binary Authorization for Cloud Run is now generally available (GA).

Chronicle - Uppercase has been rebranded as Google Cloud Threat Intelligence (GCTI).

Cloud Composer - Cloud Composer 1.17.2 release started on September 29, 2021. Cloud Composer supports the IP Masquerade agent in Preview. Changes in the preinstalled apache-airflow-backport-providers-google package for Airflow 1.10.15: Dataflow job operators can be run in async mode. New versions of Cloud Composer images: composer-1.17.2-airflow-2.1.2 composer-1.17.2-airflow-2.0.2 composer-1.17.2-airflow-1.10.15 (default) composer-1.17.2-airflow-1.10.14 composer-1.17.2-airflow-1.10.12 composer-2.0.0-preview.3-airflow-2.1.2 (default) composer-2.0.0-preview.3-airflow-2.0.2. Cloud Composer 1.12.1 has reached its end of full support period.

Compute Engine - Preview: Enable automatic renewal on your resource commitments.

Data Fusion - Preview: You can now use SAP as a source for batch-based and delta-based data extraction in Cloud Data Fusion through Operational Data Provisioning (ODP).

Dataproc Metastore - v1. CMEK integration with Dataproc Metastore is generally available (GA).

Dataproc - New sub-minor versions of Dataproc images: 1.4.73-debian10, 1.4.72-ubuntu18, 1.5.48-centos8, 1.5.48-debian10, 1.5.48-ubuntu18, 2.0.22-centos8, 2.0.22-debian10, 2.0.22-ubuntu18. Fixed an issue where complete YARN container logs were not visible in 1.5 and 2.0 Images. HADOOP-15129: Fixed in 2.0 Images: Datanode cached namenode DNS lookup failure and could not startup on.

Cloud Filestore - You can now use Customer-Managed Encryption Keys (CMEK) to protect all data at rest in Filestore's Enterprise tier instances. Filestore's Enterprise tier now supports snapshots.

IAM - IAM role recommendations for folder- and organization-level roles are now generally available.

Google Kubernetes Engine - (2021-R30) Version updates GKE cluster versions have been updated. 1.20 clusters with legacy ABAC authorization enabled should not upgrade to 1.21 until 1.21.4-gke.2500+ is available. 1.21 is now generally available Kubernetes version 1.21 is now generally available. The following features are introduced in version 1.21: CronJob (GA) The CronJob API has graduated to General Availability (GA), bringing performance improvements and allowing scheduled jobs to be run using a stable API. In Kubernetes 1.21, newly provisioned PersistentVolumes by gce-pd will use the topology.kubernetes.io/zone GA label instead of the failure-domain.beta.kubernetes.io/zone beta label. New Beta and Stable APIs The following Stable APIs are new in 1.21: batch/v1 CronJob policy/v1 PodDisruptionBudget discovery.k8s.io/v1 EndpointSlice The following Beta APIs are new in 1.21: storage.k8s.io/v1beta1 CSIStorageCapacity. Deprecated APIs The following APIs are deprecated in the 1.21 release: PodSecurityPolicy policy/v1beta1 PodSecurityPolicy Deprecated in 1.21 with removal targeted for version 1.25. 1.22 is now available in the Rapid channel Kubernetes 1.22 is now available in the Rapid channel. Removed API versions in 1.22 The following Beta versions of previously graduated APIs are removed in 1.22 in favor of the GA versions. Deprecated API versions These APIs are still served in version 1.22 but are in a deprecation period, and will be removed in 1.25: PodSecurityPolicy policy/v1beta1 PodSecurityPolicy Deprecated in 1.21 with removal targeted for version 1.25. New API versions in 1.22 The pods/eviction subresource now accepts policy/v1 eviction requests in addition to policy/v1beta1 eviction requests (#100724). Notable features in 1.22 GA: Server-side Apply Server-side Apply is a new object merge algorithm, as well as tracking of field ownership, running on the Kubernetes API server. Notable changes and bug fixes in 1.22 The terminationGracePeriodSeconds field on pod specs and container probes should not be negative. A security issue was discovered in Kubernetes, CVE-2021-25741, where a user may be able to create a container with subpath volume mounts to access files & directories outside of the volume, including on the host filesystem. A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server. There is a known issue where updating a BackendConfig resource using the v1beta1 API that removes an active Google Cloud Armor security policy from its service. Now you can see how effectively your GKE clusters and workloads are utilizing your available compute resources.

GKE - (2021-R30) Version updates Version 1.20.10-gke.301 is now the default version.

Google Kubernetes Engine Rapid - (2021-R30) Version updates Version 1.21.4-gke.1801 is now the default version in the Rapid channel. 1.22 is now available in the Rapid channel Kubernetes 1.22 is now available in the Rapid channel. Removed API versions in 1.22 The following Beta versions of previously graduated APIs are removed in 1.22 in favor of the GA versions. Deprecated API versions These APIs are still served in version 1.22 but are in a deprecation period, and will be removed in 1.25: PodSecurityPolicy policy/v1beta1 PodSecurityPolicy Deprecated in 1.21 with removal targeted for version 1.25. New API versions in 1.22 The pods/eviction subresource now accepts policy/v1 eviction requests in addition to policy/v1beta1 eviction requests (#100724). Notable features in 1.22 GA: Server-side Apply Server-side Apply is a new object merge algorithm, as well as tracking of field ownership, running on the Kubernetes API server. Notable changes and bug fixes in 1.22 The terminationGracePeriodSeconds field on pod specs and container probes should not be negative.

Google Kubernetes Engine Regular - (2021-R30) Version updates Version 1.20.10-gke.301 is now the default version in the Regular channel.

Google Kubernetes Engine Stable - (2021-R30) Version updates Version 1.19.13-gke.1200 is now the default version.

Load Balancing - External HTTP(S) Load Balancing is now available in a regional mode.

Marketplace Partners - You can now use Producer Portal's new guided configuration option to create deployment packages for your VM products directly in the Cloud Console.

KF - 2.5.4. Removed downstream lifecycle dependency for v2 buildpacks that could result in kf push failing.

Cloud Monitoring - Cloud Monitoring dashboards now support displays of data in tabular form. You can now install the Ops Agent on one or more Compute Engine VMs from the Inventory tab of the Monitoring VM Instances dashboard.

Network Connectivity Center - Cloud DNS forwarding services and Private Google Access cannot be accessed through Router appliance spokes. Previously, if you used a Router appliance spoke to connect more than 1,000 VMs, you might have experienced problems establishing BGP sessions between the router appliance instance and the Cloud Router. Network Connectivity Center includes new limits on the number of underlying resources that can be associated with a spoke.

Cloud Run - Customer managed encryption keys are now at general availability (GA).

SAP Solutions - SAP HANA certification: 12 TB m2-ultramem-416 VMs certified for OLAP scale out SAP has certified the Compute Engine 12 TB m2-ultramem-416 machine type for SAP HANA OLAP workloads in scale-out configurations with up to 16 nodes.

Cloud SQL - Cloud SQL supports the preview version of two recommenders that help you optimize your database costs: Idle database instance recommender: Identifies idle database instances in your project and provides recommendations about the savings that you can make by shutting them down. When a database instance is nearly out of storage capacity, it's automatically stopped to prevent the loss of information.

Cloud Storage - Cloud Storage now more effectively batches Cloud KMS requests.

Transcoder API - Transcoder API is GA: The Transcoder API has graduated out of beta and has reached v1. Added Troubleshooting guide. Added guidance on job limits.

VPC Service Controls - Preview stage support for the following integration: Speaker ID. General availability for the following integration: Security Token Service.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko