Welcome to issue #266 November 1st, 2021

News

BigQuery Omni now available for AWS and Azure, for cross cloud data analytics - BigQuery Omni helps teams break down silos by securely and cost-effectively analyzing data across clouds.

Quickly, easily and affordably back up your data with BigQuery table snapshots - Learn how to use BigQuery table snapshots to quickly, easily and affordably back up your data.

Django ORM support for Cloud Spanner is now Generally Available - Today we're happy to announce GA support for Google Cloud Spanner in the Django ORM.

Cloud Domains, now GA, makes it easy to register and manage custom domains - Cloud Domains, now generally available, makes performing domain-related tasks in Google Cloud simple.

Run your fault-tolerant workloads cost-effectively with Google Cloud Spot VMs - Google Cloud Spot VM allows customers to run their fault tolerant workloads at the lowest cost per VM of any leading cloud by providing more predictable pricing, graceful termination, and integration with automation.

Cloud CISO Perspectives: October 2021 - Security recap from Next ‘21, including product updates that deliver “secure products” not just “security products” and important industry momentum for tackling open source software security and ransomware.

Meet the next generation of mobile-optimized maps - Two updates are generally available—a new Maps SDK for Android and the extension of Cloud-based maps styling features to mobile.

Enhanced map style rolling out as default basemap in November

Road to an open and flexible cloud network with new Network Connectivity Center partners - Google Cloud is announcing six new networking partnerships for Network Connectivity Center for enterprises with on-prem and hybrid connectivity.

Google Cloud Next Rollup for Data Analytics - Google Cloud Data Analytics - Summary of launches, announcements and customer stories from Next.

Advance your future with learning sessions at the Government and Education Summit - Highlights the Learning Day programming at the upcoming Government and Education Summit and the many opportunities for students, educators, and professionals.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

reCAPTCHA Enterprise puts users first - reCAPTCHA Enterprise has evolved from requiring engagement from end users to being frictionless while still providing best-in-class security.

Decoding Cloud FinOps to accelerate digital transformation - Measuring business value metrics through cost efficiency, resiliency, velocity, innovation, and sustainability.

9 things I freakin’ love about Google Cloud identity and environments - Newcomers to Google Cloud will immediately notice the intentionality with which identity and environments were designed for the platform.

Cloud Blaster: How to Clean Up Your Google Cloud Project Easily - Learn how to quickly and easily declutter your development and testing GCP environments.

Accessing GKE private clusters through IAP - This article shows how to connect to the control plane of a GKE private cluster, leveraging a proxy and an IAP tunnel.

Google Cloud Networking overview - An overview of Google Cloud Networking.

App Development, Serverless, Databases, DevOps

How to get better insight into push notification delivery - Using Firebase Cloud Messaging to log notification delivery data and export to BigQuery.

Avoiding GCF anti-patterns part 2: How to reuse Cloud Function instances for future invocations - This post explores what global scope is for a Cloud Function, when to use, and what issues to look out for when used incorrectly.

How Pokémon GO scales to millions of requests? - This blog is a behind-the-scenes look into how the Pokémon GO engineering team manages and maintains the scale.

A closer look at locations in Eventarc - An Eventarc location usually refers to the single region that the Eventarc trigger gets created in. However, depending on the trigger type, the location can be more than a single region.

Cloud Run and Cloud Functions: Does the region change the performances? - Serverless abstract the infrastructure and the CPU choice. But does the performances equals between regions when you choose nothing?

GCP Workflows visual editor - Github repo of a Javascript application with which you can draw diagrams to create Workflows pipelines.

Extend disk on GCP instance - An example of using multiple Persistent Disks with Compute Engine VM.

Big Data, Analytics, ML&AI

Google Cloud's data ingestion principles - Google Cloud's six principles for planning your data ingestion work.

Open data lakehouse on Google Cloud - Exploring a new architecture choice has emerged: the data lakehouse, which combines key benefits of data lakes and data warehouses. Discuss the data lakehouse architecture and its key benefits.

5 Ways to Importing Marketing Data into BigQuery - Learn how to start working with your data in BigQuery. Discover different data import options and additional Google Cloud tools like functions, Dataflow, data prep, and data transfer.

Bike Share Chicago, Case study - The purpose of the exercise is to analyze the usage of the bike sharing data in Chicago and to increase annual memberships.

Faster distributed GPU training with Reduction Server on Vertex AI - With the help of Vertex AI’s Reduction Server, you can train your machine learning models faster on Google Cloud.

Model training as a CI/CD system: Part II - This blog post explains how to tackle maintaining a schedule to trigger the pipeline runs and creating a system such that during the experimentation phase whenever a new architecture is published as a Pub/Sub topic the same pipeline is executed with different hyperparameters.

How to implement CI/CD for your Vertex AI Pipeline - How to automatically deploy and run your machine learning pipeline. A real deep dive, step by step.

Vertex AI custom training jobs in GitLab CI - How to set up containers for MLOps pipelines.

Slides, Videos, Audio

Video walkthrough: Set up a multiplayer game server with Google Cloud - A step-by-step guide to using Google Cloud’s Compute Engine to host a multiplayer instance of Valheim from Iron Gate Studio and Coffee Stain Studios.

GCP Podcast - #282 Geospatial Cloud and Earth Engine with Chad Jennings and Joel Conkling.

Security Podcast - #40 2021: Phishing is Solved?

Releases

Anthos Config Management - 1.9.1. Config Sync will not block deletion requests if the object has non-nil metadata.deletionTimestamp. Increased git-importer memory limit to 500Mi. Fixed the issue causing nomos hydrate not to generate the configurations for clusters selected by the configsync.gke.io/cluster-name-selector annotation. Fixed the issue causing nomos hydrate to incorrectly require cluster objects to exist in the clusterregistry directory for unstructured repositories. Fixed the issue causing the namespace to be only synced to one of the clusters when the config for a namespace is defined multiple times with different configmanagement.gke.io/cluster-selector or configsync.gke.io/cluster-name-selector annotations.

Anthos clusters on AWS - Anthos Clusters on AWS aws-1.9.1-gke.0 is now available. Release aws-1.9.1-gke.0 fixes an issue in release 1.9.0 in which authorization with AWS IAM assumed roles failed. Release aws-1.9.1-gke.0 of Anthos Clusters on AWS fixes the following security issues: CVE-2021-38160 CVE-2021-3612 CVE-2021-37576 CVE-2018-5729 CVE-2018-5730 CVE-2020-16119 CVE-2021-20305 CVE-2021-22543 CVE-2021-33624 CVE-2021-33910 CVE-2021-3580 CVE-2021-3653 CVE-2021-3712 CVE-2021-37159 CVE-2021-37750 CVE-2021-40490 CVE-2021-42008 For more information, click on the CVE or search for details at https://nvd.nist.gov. The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem. A security issue was discovered in the Kubernetes ingress-nginx controller, CVE-2021-25742.

Anthos clusters on bare metal - 1.6 & 1.7 & 1.8 & 1.9. Security bulletin (all minor versions) The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem. 1.9. Release 1.9.1 Anthos clusters on bare metal 1.9.1 is now available for download. Fixes: Fixed bmctl to eliminate stack trace from error output. Functionality changes: Updated the bmctl reset cluster command to prevent you from resetting an admin cluster if the admin cluster is managing user clusters. Known issues: For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

GKE on-prem 1.7 - The security community recently disclosed a new security vulnerability CVE-2021-30465 found in runc that has the potential to allow full access to a node filesystem. Anthos clusters on VMware 1.8.4-gke.1 is now available. Fixes for version 1.8.4: Fixed high-severity CVE-2021-3711. Known issue in version 1.8.4: If you have already installed your own cert-manager in your cluster, read the suggested mitigation before upgrading to a version >=1.8.2 in order to avoid an installation conflict with the cert-manager deployed by Anthos clusters on VMware. Anthos clusters on VMware 1.7.5-gke.0 is now available. Fixes for version 1.7.5: Fixed gkectl check-config failure when Anthos clusters are configured with a proxy whose url contains special characters.

BigQuery - The table snapshots feature is now generally available (GA). SQL column-level encryption using Cloud Key Management Service (KMS) is now generally available (GA), letting you encrypt keysets within AEAD encryption functions. BigQuery Omni, a multi-cloud analytics solution, is now generally available.

Bigtable - The guidance on migrating data from HBase to Cloud Bigtable has been updated.

Billing - Recommendations for spend-based committed use discounts (CUDs) are now Generally Available.

Compute Engine - Generally available: Schedule-based autoscaling for managed instance groups now lets you configure schedules without having another autoscaling signal.

Config Connector - Config Connector 1.64.0 is now available. Added support for ComputeFirewallPolicyRule resource. Added support for FilestoreBackup and FilestoreInstance resources. Added connectionTrackingPolicy field to ComputeBackendService. Added ipv6AccessConfig, ipv6AccessType and stackType fields to ComputeInstance. Added ipv6AccessConfig, ipv6AccessType and stackType fields to ComputeInstanceTemplate. Added ipv6AccessType, stackType, externalIpv6Prefix, ipv6CidrRange fields to ComputeSubnetwork. Added nodeConfig.workloadMetadataConfig.mode; deprecated nodeConfig.workloadMetadataConfig.nodeMetadata in ContainerCluster. Added serviceAccountRef field to CloudBuildTrigger. Added monitoringConfig, dnsConfig and loggingConfig fields to ContainerCluster. Added importOnly field to KMSCryptoKey. Added disabled field to IAMServiceAccount. Added gcsDataSink.path and gcsDataSource.path fields to StorageTransferJob. Moved version field to status in DataprocWorkflowTemplate. In DNSRecordSet, ttl field is no longer required. Handle the lifecycle of ConfigConnectorContext objects in a separate controller for better isolation and scalability. Fixed the issue of changing BigTableInstance node size.

Data Catalog - Data Catalog is now available in two new North Virginia regions (aws-us-east-1 and azure-eastus2).

Deep Learning VM, Deep Learning Containers - M83 release PyTorch 1.10 is now available. M82 Release The Vertex SDK for Python is available across all deep learning environment products; it was previously available only in TensorFlow images. Theia IDE (experimental) images were refreshed.

Eventarc - Support for VPC Service Controls is now generally available (GA).

IAM - For Credential Access Boundaries, you can now use updated authentication libraries for Go, Java, Node.js, and Python to automatically exchange OAuth 2.0 access tokens for downscoped tokens.

Google Kubernetes Engine - (2021-R32) Version updates GKE cluster versions have been updated. GKE public clusters versions 1.22 and later created on or after October 28, 2021, will move to using Private Service Connect (PSC) for private control plane communication. In clusters running GKE version 1.21.0-gke.1000 and later, the destination IP address and port of the GKE metadata server has changed. In GKE version 1.22 and later, GKE cluster autoscaler and node auto-provisioning will support working on empty (zero node) clusters, and will support scaling down nodes with pods requesting local storage.

GKE - (2021-R32) Version updates The following control plane and node versions are now available: 1.19.15-gke.1801 1.20.11-gke.1801 1.21.3-gke.2003 1.21.4-gke.2302 1.21.5-gke.1302 1.21.5-gke.1802 The following control plane versions are no longer available: 1.19.13-gke.1200 1.20.9-gke.1001 1.20.9-gke.2100 Control planes and nodes with auto-upgrade enabled will be upgraded from version 1.19 to 1.19.14-gke.1900 with this release.

Google Kubernetes Engine Rapid - (2021-R32) Version updates Version 1.21.5-gke.1302 is now the default version in the Rapid channel.

Google Kubernetes Engine Regular - (2021-R32) Version updates Version 1.20.10-gke.1600 is now the default version in the Regular channel.

Google Kubernetes Engine Stable - (2021-R32) Version updates Version 1.20.10-gke.1600 is now the default version in the Stable channel.

Memorystore for Memcached - Newly created Memorystore for Memcached instances now have a default Reserved Memory percentage of 10%.

Migrate for Compute Engine 4.8 - 5.0. Migrate VMs using UEFI firmware. 5.0. #199379063 Windows migrated VMs have GooGet installed with a wrong root directory Windows VMs migrated before October 7th 2021 may have GooGet (Google package manager) installed with the wrong root directory (C:\Windows\System32\%ProgramData%\GooGet instead of C:\ProgramData\GooGet).

KF - 2.6.1. VCAP_APPLICATION route string no longer includes trailing slash. Resolved a scenario that could result in extra reconciliation loops and logs. Addressed a v2 buildpack condition that could prevent SIGTERM signals from propagating.

Cloud Router - Bidirectional Forwarding Detection (BFD) for Cloud Router is available in Preview.

Notebooks - Vertex ML Metadata is generally available (GA).

Cloud Router - Bidirectional Forwarding Detection (BFD) for Cloud Router is available in Preview.

Cloud Run - Cloud Run now supports network file systems such as NSF, NDB, 9P, CIFS/Samba, and Ceph, as well as Cloud Filestore and Cloud Storage FUSE. Cloud Run now supports a new second generation execution environment that provides full Linux compatibility rather than system call emulation.

Security Command Center - An issue that resulted in Security Command Center incorrectly reporting findings for some monitoring vulnerability detectors has been fixed.

SAP Solutions - New SAP certifications: SAP has certified the following operating systems for SAP HANA on Google Cloud: Red Hat Enterprise Linux 8.2 Red Hat Enterprise Linux 8.4 SUSE Linux Enterprise Server 15 SP3 See Certified operating systems for SAP HANA.

Cloud SQL Postgres - The following PostgreSQL minor versions and extension versions are now available. Cloud SQL now supports the max_pred_locks_per_page and max_pred_locks_per_relation flags.

Vertex AI - Vertex ML Metadata is generally available (GA).

VMware Engine - Generally available: VMware Engine integration with Google Cloud's operations suite using a standalone metrics and logs agent.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko