Welcome to issue #227 February 1st, 2021

News

Eventarc brings eventing to Cloud Run and is now GA - Eventarc lets you route events to Cloud Run, and is now generally available with several new features.

Opening up Google's Windows management tools - Large scale Windows fleet management is possible with strong open source tools and good security practices during WFH.

Bare Metal Solution: new regions, new servers, and new certifications - Check out new regions and a smaller 8-core server, plus HIPAA and PCI-DSS compliance, for Bare Metal Solution to move your Oracle workloads easily to Google Cloud.

Introducing VM Manager: Operate large Compute Engine fleets with ease - The new VM Manager simplifies infrastructure and compliance management for the largest of Compute Engine VM fleets.

Database observability for developers: introducing Cloud SQL Insights - New Insights tool helps developers quickly understand and resolve database performance issues on Cloud SQL.

Introducing Sqlcommenter: An open source ORM auto-instrumentation library - Sqlcommenter is an open source library that enables ORMs to augment SQL statements with comments before execution.

New Cloud DNS response policies simplify access to Google APIs - Cloud DNS response policies is a new way to provide access to services in a restricted VPC.

Respond to customers faster and more accurately with Dialogflow CX - New Dialogflow CX Virtual Agents can jumpstart your contact center operational efficiency goals, drive CSAT up and take care of your human agents.

BeyondCorp Enterprise: Introducing a safer era of computing - The GA of Google’s comprehensive zero trust product offering, BeyondCorp Enterprise, brings this modern, proven technology to organizations so they can get started on their own zero trust journey in a simple experience for users and administrators.

Firebase Experimental Extensions - Firebase now supports Experimental Extensions, to get more feedback from you and to better understand which use cases and extension features people are most interested in.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

New white paper: Strengthening operational resilience in financial services by migrating to Google Cloud - Learn how migrating to Google Cloud can play a critical role in strengthening operational resilience in the financial services sector.

Auto-Scaling Your Long Running Workloads on Kubernetes With KEDA v2 - Using KEDA Kubernetes-based Event Driven Autoscaler to ingest Pub/Sub messages.

Storing Helm charts in Google Artifact Registry - With the release of Helm 3, there is support to store Helm charts in an OCI compliant registry and this is demonstrated with Cloud Artifact Registry.

Say goodbye to improvised HTTPS Redirection Workarounds! - Native redirect support for Google Kubernetes Engine’s (GKE) Ingress has arrived.

GKE Structured log with in go with glog and logsrus - JSON logging on GKE with various Golang logging libraries.

Securing access to Google Service Accounts from Gitlab CI

App Development, Serverless, Databases, DevOps

Boost your query performance troubleshooting skills with Cloud SQL Insights - Learn how developers can use Cloud SQL Insights for application-centric monitoring and diagnosis through a step-by-step walkthrough.

Spoiled for choice: Deploying to 3 serverless platforms - I made a webapp called Hot Maze and deployed it to App Engine, to Cloud Functions, and to Cloud Run. This is what I learned.

Lifecycle of a container on Cloud Run - Cloud Run runs and autoscales your container-based application. You can make the most of the platform when you understand the full container lifecycle.

VPC Network Peering: GCP and MongoDB - Configuring GCP to route Cloud Run egress traffic through a static IP address.

Crashlytics and the Great Crash Detective Comic - This comics explains how Firebase Crashlytics service works and can be used.

How Cloud Storage delivers 11 nines of durability—and how you can help - Learn how Google Cloud achieves 11 nines of durability for Cloud Storage objects, and steps you can take to reduce your risk of data loss.

Check, please! Billing in Cloud Storage - Explaining pricing for Cloud Storage.

Memorystore Redis access through VPC peering - Accessing Cloud Memorystore from VPC in another GCP project.

Continuous Deployment on Firebase using Cloud Build - Focus on deploying a Vue application in Firebase using Cloud Build to create the Continuous Deployment pipeline.

Spring Boot RESTful Application with Cloud SQL for MySQL - Learn how to configure Cloud SQL MySQL Database in Java Spring Boot RESTful application.

Production-Scale IoT Best Practices: Implementation with GCP (Part 2/3) - How to properly store, analyze, and visualize production-scale telemetry data streaming from millions of IoT devices.

Operation Suite GCP (Part2) — Tracer, Debugger, and Profiler - Operation Suite — Cloud Trace, Cloud Debugger, Cloud Profiler.

Detect text on image using Google Cloud Vision API (python) - Extracting IMEI number from images using Cloud Vision API and Python.

Call a Workflow from a Function! - Learn how to combine Cloud Workflows + Cloud Functions to build automated serverless workload pipelines!

Guide to integrate Google Cloud Workflows with other Google services. - Example of how some of GCP services can be orchestrated and automated using Cloud Worflows.

Big Data, Analytics, ML&AI

The evolution of data architecture at The New York Times - See how the New York Times uses Google Cloud’s end-to-end data platform for fast streaming pipelines, advanced analytics, and record-high site traffic during the 2020 election.

We had an incident, and it was great - Sharing experience of error when designing ingesting pipeline with Pub/Sub, Dataflow and BigQuery.

Cache reuse across DoFn’s in Beam - This article covers LifeCycle of a DoFn, caching data for reuse across DoFn instances and refreshing cache via an external trigger.

Data operation with Cloud Spanner using Mercari Dataflow Template - Mercari Dataflow Template is an OSS tool for easy data processing using GCP’s distributed data processing service, Cloud Dataflow. In this article are examples of moving data between BigQuery and Cloud Spanner.

Creating dynamic Composer Airflow dags from JSON template. - How to manage dynamic dags creation in Google Cloud Composer from JSON template: the declarative way.

Google Pub/Sub Lite for Kafka Users - How to run your current Kafka ingestion workloads using Google Pub/Sub Lite.

Generating Unique Keys In BigQuery - The Ideal Primary Key For Data Warehousing.

Update BigQuery columns description from Google Spreadsheet - Handling BigQuery column descriptions in spreadsheets.

Invert polygons for fun and new functionality - Creating an aggregate version of ST_Intersect or STIntersection in BigQuery GIS.

How Do Column Types Effect Join Speeds In Data Warehouses? - A Controlled Experiment.

How our customers modernize business intelligence with BigQuery and Looker - BigQuery and Looker customers use modern business intelligence (BI) to find data insights and allow self-service discovery for all teams.

Retailers find flexible demand forecasting models in BigQuery ML - Try BigQuery’s design pattern for demand forecasting to create predictive analytics models for retail use cases.

How to build demand forecasting models with BigQuery ML - With BigQuery ML, you can train and deploy machine learning models using SQL. With the fully managed, scalable infrastructure of BigQuery, this means reducing complexity while accelerating time to production, so you can spend more time using the forecasts to improve your business.

Training Julia ML model in GCP - Running Julia training module on AI Platform.

Various

My Favorite Free Google Cloud Platform (GCP) Professional Cloud Developer Certification Courses - A collection of free online training courses for Professional Cloud Developer Certification from Google Cloud Platform.

Meet Lak, The Brain Behind Google Cloud’s Pathbreaking AI Solutions - An interview with Valliappa Lakshmanan, Director for Data Analytics and AI Solutions at Google Cloud.

Slides, Videos, Audio

GCP Podcast - #244 Intel with Rebecca Weekly.

Kubernetes Podcast - #135 Siri, Storage and Solutions, with Josh Bernstein.

How to Migrate 100 Clusters from On-Prem to Google Cloud Without Downtime

Building serverless applications with Google Cloud Platform & Firebase

Releases

AI Platform - Deep Learning VMs - Python 2 Python 2 is no longer supported in Deep Learning VM Image. M62 release Upgraded TensorFlow 2.3 to 2.3.2 Upgraded TensorFlow 2.1 to 2.1.3 Miscellaneous bug fixes and updates.

Anthos Config Management - 1.6.1. Hierarchy Controller is upgraded to include HNC v0.7.0. The nomos status output has been fixed for multi-repo clusters to show git.syncBranch when git.syncRev is not specified (git.syncRev defaults to HEAD) to provide a consistent experience with mono-repo clusters. The nomos status output has been fixed for multi-repo clusters to distinctly show status of multiple namespace repos synced to the clusters.

Anthos - Anthos 1.6.1 is now available.

Cloud Asset Inventory - New resource types now available. New resource types now available. New searchable fields now available.

BigQuery - Clustered tables now support the DATETIME type for clustering columns.

Billing - Pricing report access updated to allow Billing Account Users to view a version of the report The Pricing report provides SKU prices for Google's cloud services, including Google Cloud, Google Maps Platform, and Google Workspace. The Cloud Billing Committed Use Discounts (CUD) Analysis report has been updated to include spend-based CUDs, allowing you to easily visualize the effectiveness and financial impact of discounts you have purchased.

Compute Engine - Manage your operating system environments by using VM Manager.

Config Connector - Config Connector version 1.37.0 is now available. Added a column Status Age showing the last transition time for the value in Status, and added the column Age back to the default output of kubectl get for all Config Connector resources. Added resourceID support for ArtifactRegistryRepository, Bigtable resources, DataflowJob, DNS resources, Monitoring resources, RedisInstance, ResourceManagerLien, SecretManagerSecret, Spanner resources, StorageTransferJob. Fixed the issue with the legacy Common Name field on x509 certificate.

Cloud Dataflow - Dataflow snapshots are now available in Preview. GPU support on Dataflow is currently available in Preview.

Data Fusion - Cloud Data Fusion Beta instances (versions 6.1.0.2 and lower that were created before November 21, 2019) will be turned down on March 1, 2021.

Dataproc - Dataproc 2.0 image version will become a default Dataproc image version in 4 weeks on February 22, 2021.

Dialogflow - Dialogflow CX beta launch of Experiments to compare the performance of flow versions to a control version while handling live traffic.

GKE on Prem - Anthos clusters on VMware (GKE on-prem) 1.6.1-gke.1 is now available. Fixes: Fixed a bug where the user cluster upgrade is blocked if the vcenter resource pool is neither directly nor indirectly specified (that is, if the vcenter resource pool is inherited and is the one used by the admin cluster) in the configs.

Cloud Healthcare API - v1. The Cloud Healthcare API offers single-region support in the us-east4 (North Virginia) region.

Cloud Logging - Cloud Logging has increased the number of entries.write API calls from 60,000 to 120,000 per minute.

Anthos Migrate - On January 25, 2021 we released Migrate for Anthos 1.6.0. Previous releases of Migrate for Anthos required that you used Google Container Registry (GCR) and Google Cloud Storage for data repositories. In many on-prem environments, outbound internet access is tightly controlled through the use of an HTTPS proxy server. Migrate for Anthos now includes the deployment_spec.yaml file in artifacts.zip for Windows migrations. Support added for using Anthos clusters on AWS as processing clusters to perform migrations of AWS workloads. Removed support for the --password option to the migctl command when creating a migration source on Anthos clusters on VMware: migctl source create local-vmware local-vmware-src --vc '1.2.3.4' --username 'admin' --password 'pass1' You are now prompted to enter the password. 172414359: Exporting multiple cloned VMs simultaneously from the same source might fail. 174655315: A migration might hang when generating artifacts and remain in the retrying state. 170286188: A Windows pod gets stuck terminating indefinitely. 175000470: When adding a source when using a service account without the compute.disks.create permission, the source becomes ready but the migration will fail to create disks. 174299021: When creating a migration source or executing a migration, you might see this error: "Error: Internal error occurred: failed calling webhook "vmigration.kb.io": Post https://controllers-webhook-service.v2k-system.svc:443/validate-anthos-migrate-cloud-google-com-v1beta2-migration?timeout=30s: unexpected EOF" Workaround: Recreate the source or migration. 171686793: The migctl setup upgrade --gkeop command might create a new ImageRepositiry or ArtifactRepository object that lacked Google Cloud access credentials. If you try to mount a secret on a deployed pod you will not be able to access it in /run/secrets. 178469863: Running migctl setup install with either the --node-selector or --tolerations flag returns an error.

Cloud Speech-to-Text - Speech-to-Text now supports regional EU and US endpoints.

Cloud SQL Postgres - Query Insights is now generally available.

VPC Service Controls - Preview for the following integration: AI Platform (Unified).

Deep Learning VM - Python 2 Python 2 is no longer supported in Deep Learning VM Image. M62 release Upgraded TensorFlow 2.3 to 2.3.2 Upgraded TensorFlow 2.1 to 2.1.3 Miscellaneous bug fixes and updates.

Anthos GKE deployed on-prem - Anthos clusters on VMware (GKE on-prem) 1.6.1-gke.1 is now available. Fixes: Fixed a bug where the user cluster upgrade is blocked if the vcenter resource pool is neither directly nor indirectly specified (that is, if the vcenter resource pool is inherited and is the one used by the admin cluster) in the configs.

Dialogflow Enterprise - Dialogflow CX beta launch of Experiments to compare the performance of flow versions to a control version while handling live traffic.

Secret Manager - Secret Manager Expiration is available for all customers via public preview.

AI Platform Training - You can now use NVIDIA A100 GPUs and several accelerator-optimized (A2) machine types for training.

AI Platform - Deep Learning Containers - General Availability AI Platform Deep Learning Containers is now generally available. Python 2 Python 2 is no longer supported in Deep Learning Containers. M62 release Upgraded TensorFlow 2.3 to 2.3.2 Upgraded TensorFlow 2.1 to 2.1.3 Miscellaneous bug fixes and updates.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko