Welcome to issue #272 December 13th, 2021

News

Enabling keyless authentication from GitHub Actions - Authenticate from GitHub Actions to create and manage Google Cloud resources using Workload Identity Federation.

Store more and worry less with 31 day retention in Pub/Sub - Cloud Pub/Sub now supports 31 days of message retention for easier reprocessing, cache initialization, and event sourcing.

Scale your Ruby applications with Active Record support for Cloud Spanner - Enabling Ruby applications to use Google Cloud Spanner as a database provider via Object-Relational Mapping.

Google Cloud Spanner Dialect for SQLAlchemy - Enabling Python SQLAlchemy applications to use Google Cloud Spanner as a database.

Cloud IDS for network-based threat detection is now generally available - Google Cloud IDS for network-based threat detection is now generally available.

Join Google Cloud Research Innovators to accelerate scientific projects - Researchers using Google Cloud are invited to apply for the second cohort of the Research Innovators Program.

Google Cloud Easy as Pie Serverless Hackathon - A virtual hackathon to build and deploy serverless apps with Cloud Run, Cloud Functions and Workflows. Submission Deadline: February 4, 2022.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Google Cloud IDS signature updates to help detect CVE-2021-44228 Apache Log4j vulnerability - Cloud IDS to help detect CVE-2021-44228 Apache Log4j vulnerability.

Google Cloud Armor WAF rule to help mitigate CVE-2021-44228 Apache Log4j vulnerability - Cloud Armor WAF rule to help address CVE-2021-44228 Apache Log4j vulnerability.

5 key metrics to measure Cloud FinOps impact in your organization in 2022 and beyond - Measuring impact of Cloud FinOps across the organization including cloud enablement, cost allocation, financial forecasting, cost optimization, and automation.

Postmortems at Loon: a guiding force for rapid development - Discover how Loon Site Reliability Engineers used postmortems to iterate on their stratospheric software-defined network.

How Vuclip safeguards its cloud environment across 100+ projects with Security Command Center - Learn how Security Command Center enables Vuclip to manage security and risk for their cloud environment.

Let Kubernetes automate your configs and policies with Anthos Config Management - Let Kubernetes automate your configs and policies with Anthos Config Management.

DevOps and CI/CD on Google Cloud explained - Continuous Integration (CI), at its core, is about getting feedback early and often, which makes it possible to identify and correct problems early in the development process. With CI, you integrate your work frequently, often multiple times a day, instead of waiting for one large integration later on.

The past, present, and future of Kubernetes with Eric Brewer - Find out what the last decade of building cloud computing at Google was like, including the rise of Kubernetes and importance of open source security.

Deploy a Go API to Google Kubernetes Engine via Terraform - How to create and deploy a simple Golang API to GKE Cluster using only Terraform.

Egressing from Google Bare Metal Solution - Some options for accessing the internet from Cloud Bare Metal solutions.

App Development, Serverless, Databases, DevOps

EZ-Jobs — the Not-So-Easy Process of Making Job Applications Easy - Google Cloud Application deployed on GKE to extract contextual resume information using Cloud Vision API.

Mocking Events in Pub/Sub Triggered Cloud Functions (GCP, Python) - Mocking Pub/Sub data when writing tests for Cloud Functions which are triggered by Pub/Sub message.

Understand and Compare Cloud Functions vs Cloud Run - Are you confused about which GCP components would be more suitable?

Build a Twitter dashboard with bubble / python / cloud function - Discover this simple stack that uses NoCode, Python, Cloud, and Serverless computing to build great web apps.

Big Data, Analytics, ML&AI

How Pub/Sub eliminates boring meetings and makes your systems scale - What is Cloud Pub/Sub? A messaging service for application and data integration!

Google BigQuery: An Introduction to Big Data Analytics Platform. - An overview of BigQuery.

Retrieve your BigQuery query history with NodeJS SDK - Retrieving BigQuery history logs to understand which queries are taking the most of the billing account using BigQuery NodeJS SDK.

Incremental models to track user activity with dbt - This post describes how to create a historical table and track user activity using incremental models and dbt in BigQuery.

How to create a graph that automatically updates data on the web using Firestore × BigQuery × Data Portal. - Mirroring data from Cloud Firestore to BigQuery and displaying in Data Studio.

A Migration Misstep: From Redshift to BigQuery - Handling some of the issues when migrating from AWS Redshift to BigQuery.

Tokopedia’s journey to creating a Customer Data Platform (CDP) on Google Cloud Platform - Using tools like Big Query, and Data Flow, Tokopedia can better personalize search results and product recommendations for customers.

AI for all humans: A course to delight and inspire! - Making Friends with Machine Learning is a Google course specially created to inspire beginners and amuse experts. Today, it is available to everyone!

Various

Christmas wish list to Google Cloud - Top open feature requests on Google Cloud Issue Tracker.

Cloud Security podcast by Google turns 46 - Reflections and lessons! - The team behind Cloud Security Podcast by Google reflects on the year of fun episodes and cloud security challenges solved.

Slides, Videos, Audio

GCP Podcast - #287 Imposter Syndrome with Carter Morgan.

Kubernetes Podcast - #167 Kubernetes 1.23, with Rey Lejano.

Security Podcast - #46 Products and Solutions: Helping Our Customers Precipitate Change.

Releases

Anthos Migrate - Replatform Tomcat applications to containers Version 1.10 introduces a new public offering for replatforming VMs based Tomcat applications into containers using Apache Tomcat OSS community images. Migrate to GKE Autopilot clusters and Cloud Run now in GA Simplified Linux service manager, which lets you deploy containers to GKE Autopilot clusters and to Cloud Run, is now the default service manager for any migrations performed with Migrate for Anthos and GKE. Assessment of workloads for Shift to Google Compute Engine Added support for assessing Lift & Shift migrations to Google Google Compute Engine. Fit assessment of AWS EC2 workloads The fit assessment tool now supports assessments of AWS EC2 workloads by running the collection scripts directly on the assessed AWS EC2 VM, or through a remote SSH from the CLI. Fit assessment of Google Compute Engine VM workloads The fit assessment tool now supports assessment of Google Compute Engine VM workloads by running the collection scripts directly on the assessed Google Compute Engine VM, or through a remote SSH from the CLI. Source platform indication and VM path on Fit Assessment reports The fit assessment reports in HTML and Cloud Console include information on the source platform of the assessed VM, and a unique ID per platform. Assessment for containerization on Cloud Run The fit assessment tool now supports assessments of workloads for containerization to Google Cloud Run - A Google cloud fully managed serverless platform. Assessment for containerization on GKE Auto Pilot The fit assessment tool now supports assessments of workloads for containerization to GKE Auto Pilot - A new mode of operation in Google Kubernetes Engine (GKE) that is designed to reduce the operational cost of GKE clusters. Using RVTools output as a data source for fit assessment The fit assessment tool now supports analyzing the RVTools .xlsx report file from a single VMware vCenter by running $./mfit discover rvtools name.xlsx. Fit assessment automatic version checks The fit assessment tool now checks for the availability of a new version by probing a version check Google Cloud Storage resource. 190704603: Change to mFIT CLI Help text - 'Import collector script artifacts'. 190575888: Design updates to mFIT HTML report, fonts changes, layout bugs and graphs position on report. 206772515: Fixed a bug where ** in a v2kServiceManager log path was not supported. 205159324: Fixed a bug where services-config.yaml was not created even when the migration completed successfully in the new Linux system container runtime. 199382909: Data migration plans will not have comments when using the UI. 205159086: On newer Ubuntu versions migrated workloads will fail. 208040681: Operating system field 'disappears' after running guest level discovery. 194186514: Migration done in Anthos on AWS might succeed even though the files were not uploaded. Uninstall might be stuck when a sourcesnapshot CRD cannot be deleted. 204879458: If your image repository permissions are invalid, migration will get stuck in ExtractImage instead of the UploadImage step.

Network Intelligence Center - It is now possible to export Firewall Insights data in CSV format.

Cloud PubSub - Extended topic retention allows you to retain published messages for a maximum of 31 days.

Security Command Center - To facilitate the flow of information between Security Command Center and third-party systems, a resource called ExternalSystems was added under the Finding object. Event Threat Detection, a built-in service of Security Command Center, released the Exfiltration: BigQuery Data Extraction rule.

Service Mesh - 1.12.x. 1.12.0-asm.3 is now available. Managed Anthos Service Mesh isn't rolling out to the rapid release channel at this time. Anthos Service Mesh now supports installations and upgrades on Microsoft Azure Kubernetes Service (AKS) clusters. Anthos Service Mesh now supports the Certificate Authority Service integration on on-premises platforms (both Anthos on VMware and bare metal). Anthos Service Mesh now supports deploying a proxy built on the distroless base image. For unmanaged Anthos Service Mesh installations, the installer will automatically set up the default tag (the istio-revision-tag-default and istio-default-validator webhooks).

Anthos Service Mesh - 1.12.x. 1.12.0-asm.3 is now available. Managed Anthos Service Mesh isn't rolling out to the rapid release channel at this time. Anthos Service Mesh now supports installations and upgrades on Microsoft Azure Kubernetes Service (AKS) clusters. Anthos Service Mesh now supports the Certificate Authority Service integration on on-premises platforms (both Anthos on VMware and bare metal). Anthos Service Mesh now supports deploying a proxy built on the distroless base image. For unmanaged Anthos Service Mesh installations, the installer will automatically set up the default tag (the istio-revision-tag-default and istio-default-validator webhooks).

Cloud SQL Postgres - Cloud SQL now limits the rate for backup and restore operations on the data disk.

Cloud Storage Transfer - Storage Transfer Service now offers Preview support for detailed logging for objects copied between AWS S3, Azure Blob, ADLS Gen 2, and Cloud Storage.

VPC Service Controls - Beta stage support for the following integration: Firebase Security Rules.

Workflows - Dynamic keys are now supported. Resource limits for variable memory and argument size have been increased to 256 KB. A Workflows Service Level Agreement (SLA) is now available and applicable.

AI Platform Training - Runtime version 2.7 is available.

Anthos Config Management - 1.10.0. Config Sync admission webhook is disabled by default. Policy Controller has deprecated the K8sPSPSELinux (v1) ConstraintTemplate. The Config Sync feature to render Kustomize configurations and Helm charts is generally available (GA). The Policy Controller feature to support mutation is generally available (GA). Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: c36e3d8). Use nomos migrate to easily enable the RootSync and RepoSync APIs in the cluster. Added a new metric pipeline_error_observed to capture if there is any error from different stages: rendering, sync, source, readiness. nomos status surfaces messages from resource conditions when the managed resources are not ready or healthy. Increased memory request for git-sync container to 200Mi. Fixed the issue causing nomos hydrate not to render Kustomize configs if it references files in parent directories. Fixed the issue causing nomos vet --namespace to fail because it incorrectly defaults --source-format to hierarchy. Reduced the latency to sync a root repository in the multi-repo mode by reducing GET calls to the API server. Fixed the issue causing some resources not to be applied when the status updates of all the resources in a Git repository take longer than 1 minute. Fixed the issue in RootSync and RepoSync APIs causing proxy to incorrectly fail validation when auth is set to cookiefile or none.

Anthos clusters on bare metal - Anthos clusters on bare metal 1.10.0 is now available for download. Improved cluster lifecycle functionalities: GA: Enabled Node Problem Detector to run by default on all nodes. Breaking changes: The gateway capability used by the egress NAT gateway and Bundled load balancing with BGP Preview features have changed in this release. Functionality changes: Version 1.10.0 admin clusters aren't visible from the Cloud Console or when performing gcloud container hub memberships list operations. Fixed cluster lifecycle functionalities: Outputs from all bmctl commands except bmctl version are now written to log files. Known issues: For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section. 1.7. Release 1.7.7 Anthos clusters on bare metal 1.7.7 is now available for download. Fixes: The 1.7.6 release has a known issue that blocks upgrades of 1.7.5 clusters. Known issues: For information about the latest known issues, see Anthos on bare metal known issues in the Troubleshooting section.

AppEngine Standard Go - Updated Go SDK to version 1.9.72.

BigQuery ML - Anomaly detection in BigQuery ML is now generally available (GA).

BigTable - Autoscaling for Cloud Bigtable is now generally available (GA). You can now use customer managed encryption keys (CMEK) in Cloud Bigtable instances that are replicated across multiple regions.

Access Transparency - You can view Access Transparency logs for Google Workspace services in the Google Cloud Console.

Cloud Composer - We delay switching Python 3.6 to Python 3.8 in Cloud Composer images with Airflow 1.10.15. Cloud Composer 1.17.7 release started on December 9, 2021. Fixed the issue with continuous web server reloading when syncing plugins. (Cloud Composer 2) Messages in web server logs now have correctly assigned severity. (Airflow 2.1.4) The apache-airflow-providers-hashicorp package is added to images with Airflow 2.1.4. (Airflow 2.1.4) Backported the fix for the SerializedDagNotFound: DAG not found in serialized_dag table Airflow bug. (Airflow 2.1.4) Users with the Admin role can now create users in the Airflow UI. Cloud Composer 1.17.7 and 2.0.0-preview.7 images are available: composer-1.17.7-airflow-1.10.15 (default) composer-1.17.7-airflow-2.0.2 composer-1.17.7-airflow-2.1.4 composer-2.0.0-preview.7-airflow-2.0.2 composer-2.0.0-preview.7-airflow-2.1.4. Cloud Composer 1.13.2 has reached its end of full support period.

Compute Engine - The n2-node-128-864 sole-tenant node type is now available in Preview.

Deep Learning Containers - M87 Release Added Artifact Registry's Python keyring authentication library to deep learning vm environments. TensorFlow 2.x container image names are available in two formats: the current standard, which includes a tf- prefix, and the previous standard, which includes a tf2- prefix.

Deep Learning VM - M87 Release Added Artifact Registry's Python keyring authentication library to deep learning vm environments. The M87 release is the last release in which TensorFlow 2.x image names are available in two formats: the current standard, tf-xxx-2-y-zzz and the previous standard, tf2-xxx-2-y-zzz.

Dialogflow - The Dialogflow CX simulator now provides page lifecycle navigation to help you understand the execution steps taken for each conversational turn.

Cloud Functions - Cloud Functions has added support for customer-managed encryption keys, available at the Preview release level. Cloud Functions support for setting a minimum number of instances is now at the General Availability release level.

Google Kubernetes Engine - GKE version 1.22.3-gke.1500 and later support user impersonation for all user-defined users and groups. (2021-R34) Version updates GKE cluster versions have been updated. PodSecurityPolicy (beta) was deprecated in Kubernetes 1.21 and is scheduled for shutdown in 1.25. The following GKE versions fix Calico issue #4710 and Calico issue #4518, related to Pod graceful termination, in GKE clusters with Calico Network Policy enabled: 1.19.16-gke.100 and later 1.20.11-gke.1300 and later 1.21.4-gke.1500 and later For more information about the resolved issue, see the known issues page.

GKE - (2021-R34) Version updates The following control plane versions are no longer available: 1.19.13-gke.1900, 1.19.14-gke.301, 1.19.14-gke.1900, 1.19.14-gke.2300, 1.19.15-gke.500 1.21.3-gke.2003, 1.21.4-gke.2300, 1.21.4-gke.2302, 1.21.5-gke.1300 The following control planes and nodes with auto-upgrade enabled will be upgraded with this release: From version 1.18 to 1.19.15-gke.1300 From version 1.19 to 1.20.11-gke.1300 From version 1.20 to 1.20.11-gke.1300 From version 1.21 to 1.21.5-gke.1302.

Google Kubernetes Engine Stable - (2021-R34) Version updates Version 1.20.11-gke.1300 is now the default version in the Stable channel.

Google Kubernetes Engine Regular - (2021-R34) Version updates The following control plane and node versions are now available in the Regular channel: 1.20.12-gke.1500 1.21.5-gke.1802 The following versions are no longer available in the Regular channel: 1.20.10-gke.2100, 1.21.3-gke.2003 Control planes and nodes with auto-upgrade enabled in the Regular channel will be upgraded from version 1.19 to 1.20.11-gke.1801 with this release.

Google Kubernetes Engine Rapid - (2021-R34) Version updates Version 1.22.3-gke.700 is now the default version in the Rapid channel.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko