Welcome to issue #193 June 8th, 2020

Cloud SQL gets a cross-region replication, dbt is used more and more with BigQuery and examples of how to use the simplicity of serverless products to integrate with IM platforms. 

News

Introducing cross-region replica for Cloud SQL - Cross-region replication from Cloud SQL lets you ensure business continuity across Google Cloud regions in case of an outage or failure.

Introducing skill badges, certification preparation learning paths, and online proctored certification testing - Google Cloud skill badges help employers identify candidates with demonstrated Google Cloud technology skills, alongside new six-week learning paths and opportunities to take certification exams remotely.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How to find—and use—your GKE logs with Cloud Logging - An overview of how logging works in GKE, and how to configure, find and interact effectively with the GKE logs stored in Cloud Logging.

How Migrate for Anthos streamlines legacy Java app modernization - Latest Migrate for Anthos streamlines modernization of legacy Java apps.

Serving multi-backend application on HTTP(s) Load Balancer with Ingress on GKE - This demo aims at running a web application with multiple backends or routes behind a public facing Cloud Load Balancer by configuring an Ingress object in Google Kubernetes Engine.

Multi-Cluster Istio 1.5 with Private GKE Clusters and Google Cloud Internal Load Balancer - This tutorial shows you how to build a private multicluster service mesh solution, with Istio 1.5 and Service Mesh Hub for mesh Federation. This is achieved by using an Internal Load Balancer to connect Istio workloads running in multi-region Private Google Kubernetes Engine clusters.

Exploring GKE — VSCode + CloudCode - Using CloudCode plugin for VSCode to utilize GKE.

GCP — Monitoring Quotas with Prometheus - Monitoring GCP resource quotas in Prometheus.

Practical Monitoring with Prometheus & Grafana (Part IV) - Securing Grafana deployed on GKE with Identity-Award Proxy.

How to Configure the GCP Backend for Terraform - This tutorial goes through setting Cloud Storage as a remote Terraform backend.

The Arts of GCP Folder Structure - A few concepts of the GCP folder structure it's good to be aware of.

Security Checkpoints for deploying app on GCP - 7 step security guidelines for the application owners, system administrator, and developers wishing to deploy the application on Google Cloud Platform.

GCPinstances.info - Easy GCP Compute Engine Instance Comparison.

App Development, Serverless, Databases, DevOps

Getting Started with Artifact Registry: Deploying to Cloud Run - The article describes how to deploy an Artifact Registry container to Cloud Run.

Alert google chat room on error logs in GCP - Sending message to chat room whenever there is a log with Error severity.

Google Cloud Platform — Serverless Slack Bot - GCP Serverless Python/Go Function Template for a Slack bot.

Sending logs directly to GCS using Cloud Logging fluentd - Sending logs from Compute Engine instance to Cloud Storage using fluentd plugin (but without Cloud Logging).

Debugging a Strange Kubernetes & Firebase Connection Reset Issue - Debugging a networking issue involving Kubernetes, Firebase, and Cloud NAT networking.

Dialogflow Twilio Integration using Google Cloud - Deploying the Opensource Integration of Twilio using Google Cloud Run for a simple FAQ Chatbot which provides the frequently asked questions about Google Cloud.

Uploading a Docker image to GCR using Github Actions - Setting Github Actions to build a container image and push it into the Container Registry.

Using Spring Boot with Google App Engine in 4 steps - Setting up and deploying Spring Boot web application on App Engine.

How to automate financial data collection with Python using Tiingo API and Google Cloud Platform - A step-by-step tutorial for Python programmers looking to anayze stock prices and automate data collection through GCP.

API versioning using Cloud Endpoints - Deploying multiple API versions using Cloud Endpoints and Kubernetes Engine.

Big Data, Analytics, ML&AI

How to Automate a Cloud Dataprep Pipeline When a File Arrives - With a better mastery of Cloud Functions, you can trigger a Dataprep job via API when a file lands in a Cloud Storage bucket.

Loading and transforming data into BigQuery using dbt - A data engineering tool to build Data Lakes, Data Warehouses, Data Marts, and Business Intelligence semantic layers in BigQuery.

How Rittman Analytics Centralizes SaaS Data Sources using dbt and Google BigQuery - Using dbt to model, transform and load data into BigQuery.

Zero to Differential Privacy in 5 minutes on Google BigQuery - Differential Privacy presents a framework for asking statistical questions about a dataset while provably maintaining the privacy of the entities within that dataset.

Fibonacci series with user-defined functions in BigQuery - BigQuery can do user-defined functions written in JavaScript so let’s do some Fibonacci series!

Select partitioning and clustering keys for BigQuery tables when migrating from Netezza - When migrating a data warehouse to BigQuery one of the key parts is to select partitioning and clustering strategy for your tables.

BigQuery, PII, and Cloud Data Loss Prevention (DLP): Take it to the next level with Data Catalog - A fully automated solution to discover sensitive data across all your Big Query assets, by using Data Loss Prevention and Data Catalog.

Slides, Videos, Audio

Kubernetes Podcast - #106 CoreDNS, with John Belamaric.

Releases

Access Context Manager - General availability of custom access levels.

BigQuery - BigQuery Table ACL is now available as a beta release. You can now purchase BigQuery slots using the bq command-line tool. A new GIS function, ST_Simplify, is available. Standard SQL view definition bodies can now contain references without project qualifiers, as long as the view is created by the tables.insert REST API or is in the same project used to run the CREATE VIEW DDL query. Standard SQL user-defined function definition bodies can now contain references to tables and views.

Compute Engine - NVIDIA® Tesla® T4 GPUs are now available in the following additional regions and zones: Changua County, Taiwan asia-east1-c For information about using T4 GPUs on Compute Engine, see GPUs on Compute Engine.

Config Connector - Miscellaneous bug fixes and improvements.

Dataprep - Flow parameters: Create flow parameters that you can reference in the recipes of your flow. Introducing new Flow View: The Flow View page has been redesigned to improve the user experience and overall productivity. Redesigned Settings and Help menus: See Home Page. Report issue: If you are experiencing an issue with Cloud Dataprep by TRIFACTA INC., you can gather useful information from the application to deliver to Support. Transformer page: Join steps are now created in a larger window for more workspace. Transformer page performance: Improved performance when loading the Transformer page and when navigating between the Flow View and Transformer pages. PDF profiles: When visual profiling is enabled for a job, you can now download your visual profile in PDF format. New functions: New aggregation functions for Datetime values: MINDATE Function MAXDATE Function MODEDATE Function New parsing by data type functions: PARSEBOOL Function PARSEFLOAT Function PARSEINT Function PARSEDATE Function New functions for calculating working days between two valid dates: NETWORKDAYS Function NETWORKDAYSINTL Function WORKDAY Function WORKDAYINTL Function New time zone conversion functions: CONVERTFROMUTC Function CONVERTTOUTC Function CONVERTTIMEZONE Function New statistical functions: MEDIAN Function PERCENTILE Function QUARTILE Function CORREL Function COVAR Function COVARSAMP Function Ignore case parameter added to string functions: STARTSWITH Function ENDSWITH Function EXACT Function MATCHES Function STRINGGREATERTHAN Function STRINGGREATERTHANEQUAL Function STRINGLESSTHAN Function STRINGLESSTHANEQUAL Function SUBSTITUTE Function. Parameter overrides: If you have upgraded to Release 7.1 or later, any parameter overrides that you have specified in your flows must be re-applied. Language: All MODE functions return the lowest value in a set of values if there is a tie in the evaluation. API Documentation: API reference documentation is now available directly through the application. Send a Copy: You can no longer send a copy of a flow to another user. Re-run jobs using Cloud Dataflow templates: This feature is no longer available. TD-49559: Cannot select and apply custom data types through column Type menu. TD-47473: Uploaded files (CSV, XLS, PDF) that contain a space in the filename fail to be converted.

Dialogflow - The shutdown of 7 integrations announced in January is now extended to July 6th, 2020.

Cloud Functions - The Node.js 8 runtime is deprecated as of 2020-06-05.

Cloud Healthcare API - v1. It is now possible to import DICOM data to and export DICOM data from BigQuery using the Healthcare Browser in the Cloud Console.

Google Kubernetes Engine - As part of ensuring better representation of available resources on the node for e2 burstable node types, GKE has decided to reduce the allocatable CPU resources available to schedule user workloads (known as the node allocatable resources) on e2-micro, e2-small, and e2-medium machine types. Today, e2-micro, e2-small, and e2-medium have 1930 mCPU of allocatable resources for Kubernetes to schedule Pods on per node, and following this change it will be 940m CPU. We are making this change in order to more accurately represent the resources available in these machine types. When your cluster is upgraded to 1.16.8-gke.17 and 1.17.5-gke.5 or later (whether you perform this manually or you are automatically upgraded), your workloads may become unscheduled if there are not enough allocatable resources in the cluster. Take a moment to review your Pod resource requests. Alternatively, from the command line: If you have nodes of type e2-micro, e2-small, and e2-medium where more than 940mCPU is requested, Pods will be rescheduled onto other nodes after upgrade. To ensure you have enough allocatable capacity, you can: Enable auto-scaling on your node pool. Increase the number of nodes in the cluster, or add larger node types if you have Pods that make CPU requests which exceed the capacity of existing nodes. Decrease the resource requests made by your workloads on these nodes so that they will still fit after the upgrade, by modifying the CPU resource requests of the PodSpec. Review the status of your Pods by running kubectl get pods. If any are indicated as Pending, it may indicate that there were not enough resources available to schedule them. GKE cluster versions have been updated. Rollouts are phased across multiple weeks, to ensure cluster and fleet stability. The following Kubernetes versions are now available for new clusters and for opt-in master upgrades and node upgrades for existing clusters. No channel Note: Your clusters might not have these versions available. The Ubuntu image for GKE 1.14.10-gke.40 clusters is ubuntu-gke-1804-1-14-v20200219.

Load Balancing - HTTP(S) Load Balancing logging is now available in General Availability. You can now use a custom filter when you list endpoints in a zonal network endpoint group.

Cloud Logging - Custom retention is now generally available (GA). In the Logs Viewer (Preview), you can now save your queries, which can then be viewed and run from the Saved queries tab.

Cloud Memorystore - The Version Upgrade and Redis version 5.0 features are now Generally Available on Memorystore for Redis.

Cloud Run - The Cloud Run user interface now allows you to edit the service YAML.

Cloud Spanner - Cloud Spanner SQL now supports the following statistical aggregate functions - STDDEV, VARIANCE.

Cloud AutoML Vision - v1beta1 endpoint end-of-life After June 4, 2020, the v1beta1 version of AutoML API will deny increasing numbers of API requests from AutoML Vision users.

Cloud AutoML Vision Object Detection - v1beta1 endpoint end-of-life After June 4, 2020, the v1beta1 version of AutoML API will deny increasing numbers of API requests from AutoML Vision users.

Cloud Vision API - Access Transparency GA Access Transparency logging is now Generally Available.

Virtual Private Cloud - Hierarchical firewall policies are now available in Beta.

VPC Service Controls - The VPC accessible services feature is now generally available.

Dialogflow Enterprise - The shutdown of 7 integrations announced in January is now extended to July 6th, 2020.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko