Welcome to issue #209 September 28th, 2020

News

Better monitoring and logging for Compute Engine VMs - New features in Cloud Monitoring and Cloud Logging simplify operations of your Compute Engine instances.

AI Platform Prediction goes GA with improved reliability & ML workflow integration - AI Platform Prediction goes GA with enhanced reliability & ML workflow integration.

All together now: Fleet-wide monitoring for your Compute Engine VMs - Cloud Monitoring now lets you manage an entire fleet of Compute Engine VMs.

Cloud Run for Anthos brings eventing to your Kubernetes microservices - New eventing capabilities in Cloud Run for Anthos make it easy to build event-driven applications on top of your GKE environment.

Better together: Google Cloud Load Balancing, Cloud CDN, and Google Cloud Armor - By adding Google Cloud Armor and Cloud CDN to your Global Load Balancer deployment, you can benefit from better security and reduced latency.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

SRE Classroom: exercises for non-abstract large systems design - Learn how to apply SRE principles in this series of workshops on non-abstract large systems design (NALSD) with Google engineers.

Anthos in depth: Easy load balancing for your on-prem workloads - Your Anthos subscription includes an external load balancer that can help you connect data to your on-prem Anthos cluster.

Are you an Elite DevOps performer? Find out with the Four Keys Project - Learn how the Four Keys open source project lets you gauge your DevOps performance according to DORA metrics.

Cloud migration: What you need to know (and where to find it) - Google Cloud offers a rich set of solutions and documentation to help guide your cloud migration. Here’s where to find what you need.

Create a MongoDB cluster in 2 command lines (with GCP + Terraform + Ansible) - This is a tutorial to create a MongoDB cluster on GCP using Terraform and Ansible in 2 command lines.

Realizing Multi-Cloud Kubernetes with Google Anthos and AWS (part 2) - Part 2 of 3 — step-by-step installation instructions to deploy Google Anthos GKE on AWS, create a cluster, and connect it to GCP console.

App Development, Serverless, Databases, DevOps

GCP API Gateway Demo with Terraform / Go / Cloud Run - An example of using API Gateway connecting 2 microservices.

Cloud Tasks or Pub/Sub ? - Comparing Cloud Tasks and Cloud PubSub use cases.

SAP Data Intelligence and Google Cloud - Setting up SAP Data Intelligence on GCP.

Access CloudSQL PostgreSQL FDW From Different VPC Or Public - Accessing GCP CloudSQL instances from different VPC or different projects.

Programmatic Authentication under IAP on GCP - Experience of using Identity Aware Proxy with JWT tokens.

Big Data, Analytics, ML&AI

BigQuery Explained: Querying your Data - This post dives into querying data with BigQuery, lifecycle of a SQL query, standard & materialized views, saving and sharing queries.

How to move from on-demand pricing to reservations in BigQuery - Translating the slot usage graph into workload management.

Monitoring your BigQuery costs and reports usage with Data Studio - Simple and effective dashboard. With actual report names, users and labels. Handy template included.

Pushing the Last Frontier of Data Analysis Democratization With BigQuery Data QnA - Using Data QnA (at the moment in private alpha) to query data in BigQuery.

How to select data from columns contain a substring from BigQuery - How to deal with the requests that ask for data from columns that contain a user-defined substring in the column name from BigQuery.

Managing Costs Efficiently With BigQuery - Handling the growth of data and users in BigQuery.

Dynamic Partition Filtering on BigQuery Using Cloud Functions - Using table partition in BigQuery to limit data usage in queries.

Using BigQuery to Track and Estimate Home Heating Oil Deliveries - Google Sheets, Big Query, and Public Data Sets to calculate Degree Days and K-Factor.

How to setup auto scalable Google Dataproc cluster? - Creating a Dataproc cluster with auto scaling policy.

How Google Dataproc cluster auto scales? - An example of an autoscaling Dataproc cluster.

Accessing Cloud SQL Data from AI Platform using Python - This article talks about a workaround to access data in Cloud SQL DB from the AI Platform.

How to create a numeric calculation web service that autoscales on GPUs - Numeric-calculations-as-a-service by leveraging ML Infrastructure.

Migrate your custom ML models to Google Cloud in 3 steps - How to migrate your custom ML models to Google Cloud in 3 steps using Kubeflow Pipelines.

Various

Google Data Engineer certification — the definitive guide - Resources and topics to study when preparing for the Data Engineer exam.

Slides, Videos, Audio

GCP Podcast - #237 NVIDIA with Bryan Catanzaro.

Kubernetes Podcast - #122 Grafana, with Torkel Ödegaard.

Releases

AI Platform - Deep Learning VMs - M56 release Bug fixes for TensorFlow 2.3 add-ons Fixes bug affecting BigQuery magic commands in some environments Adds a diagnostics tool for AI Platform Notebooks.

AI Platform - Notebooks - AI Platform Notebooks API is now Generally Available. Cloud Audit Logging for AI Platform Notebooks is now Generally Available.

Anthos Config Management - 1.5.0. Anthos Config Management now includes Config Connector v1.19.1. Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 15d56e3). Binary Authorization can now be enabled through the Anthos Config Management Operator. The syncer and importer Containers now both run in the git-importer Pod in the importer Container. The nomos CLI tool is now available via gcloud. This release includes several logging and performance improvements.

Anthos - Anthos 1.5.0 is now available.

AppEngine - If you use Cloud Load Balancing, you can use network ingress controls so your app only receives requests that are routed through the load balancer.

BigQuery - The following time-unit partitioning features are now Generally Available (GA): Creating partitions using hourly, monthly, and yearly time-unit granularities.

Billing - We've added .NET and Go client libraries for the Cloud Billing Budget API (Beta).

Compute Engine - On July 28, 2020, we announced that improved validation checks will be introduced on API calls to the Compute Engine API. NVIDIA® T4 GPUs are now available in the following additional regions and zones: Sydney, Australia: australia-southeast1-a For information about using T4 GPUs on Compute Engine, see GPUs on Compute Engine.

Cloud Filestore - Filestore backups has launched to beta.

GKE on Prem - Anthos GKE on-prem 1.5.0-gke.27 is now available. Improved upgrade and installation: Preflight checks are now blocking with v1 configs for installation and upgrades. Improved HA and failure recovery: The user cluster control plane HA feature is now generally available. Improved support for Day-2 operations: The gkectl update cluster command is now generally available. Enhanced monitoring with Cloud Monitoring: Introduces out-of-the-box alerts for critical cluster metrics and events in preview. Functionality changes: Preflight check failures now block gkectl create loadbalancer for the bundled load balancer with Seesaw. Breaking changes: Starting with version 1.5, instead of using kubectl patch machinedeployment to resize the user cluster and kubectl edit cluster to add static IPs to user clusters, use gkectl update cluster to resize the worker node in user clusters and to add static IPs to user clusters. Fixes: Fixed an issue that caused approximately 50 seconds of downtime for the user cluster API service during cluster upgrade or update. Due to a 1.17 kubernetes issue, kube-apiserver and kube-scheduler don't expose kubernetes_build_info on the /metrics endpoint in the 1.5 release.

IAM - You can now use workload identity federation, available in beta, to grant access to Google Cloud resources from on-premises and multi-cloud workloads.

Cloud Logging - The histogram panel in the Logs Viewer (Preview) now contains a viewport to help you quickly understand the time range of the log entries you're viewing within the Query results pane. Logs Buckets are now generally available.

Anthos Migrate - On September 24, 2020 we updated Migrate for Anthos 1.4. Changed the default settings on the Cloud processing cluster for migrating Linux workloads: You no longer have to specify the --scopes "cloud-platform" option when creating Cloud processing clusters for migrating Linux workloads. You can now use the Google Cloud Console to: Install Migrate for Anthos on a processing cluster Create a migration source for a Compute Engine VM See Installing Migrate for Anthos and Adding a migration source for more.

Resource Manager - The Organization policy constraints for Direct Path disablement have launched into beta.

Cloud Run for Anthos - Cloud Run for Anthos on Google Cloud version 0.17.2-gke.1 is now available for following GKE minor version: 1.18.

Cloud SQL MySQL - MySQL 5.6.42 is upgraded to 5.6.47. Cloud SQL has upgraded Replicating from an external server to version 1.1.

Cloud SQL SQL Server - Cloud SQL for SQL Server enables you to set a default collation value for databases in an instance.

Network Intelligence Center - Running Connectivity Tests to and from Google-managed services is available in Alpha.

Deep Learning VM - M56 release Bug fixes for TensorFlow 2.3 add-ons Fixes bug affecting BigQuery magic commands in some environments Adds a diagnostics tool for AI Platform Notebooks.

AI Platform Training - AI Platform Training runtime version 2.2 now supports training with TPUs using TensorFlow 2.2.

Anthos GKE on-prem - Anthos GKE on-prem 1.5.0-gke.27 is now available. Improved upgrade and installation: Preflight checks are now blocking with v1 configs for installation and upgrades. Improved HA and failure recovery: The user cluster control plane HA feature is now generally available. Improved support for Day-2 operations: The gkectl update cluster command is now generally available. Enhanced monitoring with Cloud Monitoring: Introduces out-of-the-box alerts for critical cluster metrics and events in preview. Functionality changes: Preflight check failures now block gkectl create loadbalancer for the bundled load balancer with Seesaw. Breaking changes: Starting with version 1.5, instead of using kubectl patch machinedeployment to resize the user cluster and kubectl edit cluster to add static IPs to user clusters, use gkectl update cluster to resize the worker node in user clusters and to add static IPs to user clusters. Fixes: Fixed an issue that caused approximately 50 seconds of downtime for the user cluster API service during cluster upgrade or update. Due to a 1.17 kubernetes issue, kube-apiserver and kube-scheduler don't expose kubernetes_build_info on the /metrics endpoint in the 1.5 release.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko