Welcome to issue #191 May 25th, 2020

Cloud Next '20 is announced to be an online multi-week event with over 200 sessions. BigQuery turned 10 and Istio 3.

News

Announcing Google Cloud Next ’20: OnAir, a multi-week, digital event series: July 14 - Sept 8 - Starting July 14, Next OnAir will offer fresh content each week with over 200 sessions ranging from keynotes from industry luminaries to advanced learning opportunities with top Google developers.

Burst data lake processing to Dataproc using on-prem Hadoop data - Use Dataproc and Alluxio to burst workload processing to cloud from Hadoop on-prem data stores.

Zero-trust remote admin access for Windows VMs on Compute Engine - A new open-source tool to help Windows users and administrators to access and manage Windows VMs running in Compute Engine.

Dell Technologies Cloud OneFS for Google Cloud, now generally available - Migrate high-scale workloads easily to Google Cloud for flexibility and performance with Dell Technologies OneFS for Google Cloud.

Say hello to the helpful Firebase Emulator - a local first UI to boost your productivity - The Emulator UI is a local web app that allows you to manage local emulators that make up the Firebase Local Emulator Suite.

Cloud Run: Long Running Operations (upto 60 mins) - One of the top feature requests for Cloud Run (https://cloud.google.com/run) has been having long request times. Hence, we are excited to announce that we have increased the Cloud Run request time to 60 mins.

Announcing Istio 1.6

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How to Structure Your Enterprise on Google Cloud Platform - Step-by-step tips from the trenches for enterprises looking to start in Google Cloud with the right foot forward.

Anthos in depth: exploring a bare-metal deployment option - Running Anthos on bare metal may provide better performance and lower costs for some workloads.

Deploying an app to a decentralized service mesh with Anthos and Istio - Automating cross-network service-to-service communication.

What is Google Cloud Anthos? Kubernetes everywhere - Google’s Anthos software promises a single, consistent way of managing Kubernetes workloads across on-prem and public cloud environments.

How to Install Istio in Kubernetes Cluster - How to install Istio in GCP Kubernetes cluster using Helm.

GCP GitOps Style CICD with Cloud Build to Deploy Helm Chart to GKE - Implement a CI pipeline using Cloud Build to build a containerized application and upload it into GCR and implementing a CD pipeline using Cloud Build to deploy helm chart in GKE.

How to run SAP on Google Cloud if high availability is high priority - Availability can mean different things to different customers, depending on their business needs, budgets, SAP application use cases, and other factors. Google Cloud looks at the SAP high availability (HA) landscape in terms of three levels, each with its own costs, benefits, and trade-offs to consider within an overall availability strategy.

How to Create a Service Account for Terraform in GCP - Creating a Service Account for which will be used in Terraform.

Terraform Deployments with Google Cloud Build - Ready to move away from using Terraform locally? Take a look at how to deploy using Google Cloud Build.

Distributed tracing setup in GKE — Jaeger / Zipkin — Google Cloud Platform - Setting tracing for microservices deployed on GKE.

SLIM: Hydrating cloud native CI/CD pipelines to securely access GCP projects - Secret-less-identity-management system for Gitlab & Kubernetes Engine.

VPN :: AWS ↔️ GCP - Setting up a VPN connection between GCP and AWS servers.

Google Cloud — IAM users extraction across all projects in a GCP org - A simple script to get all users for organisation in GCP.

A Comparison of Secrets Managers for Google Cloud Platform - A Comparison of popular secrets management solutions for GCP by features, security concerns, and cost.

Red Hat OpenShift Container Platform UPI on GCP - Deploying Red Hat OpenShift Container Platform on GCP series.

App Development, Serverless, Databases, DevOps

GCP: Spring To Production With App Engine, Cloud Build And GitHub - How to get a spring app through a CI/CD pipeline on GCP…..

Deploying a NodeJS Application on GCP with App Engine & CloudBuild (Part 1) - This three-part tutorial will walk you through all the steps you need to deploy your NodeJS application to Google App Engine,.

Helidon applications on Google App Engine - This article details building an example Java web application using Helidon MP libraries and deploying it on Google App Engine.

Running Wordpress website on Google Cloud Run — simple and cheap - Deploying a Wordpress website to Cloud Run.

Connecting to mongodb atlas/external service from Google cloud functions via static IP address - How to allow access from an external application to Cloud Function by specifying IP.

My definitive guide to getting datastore emulator to work with python - Setting local datastore emulator to be used in App Engine.

Firebase Emulators User Interface - Overview of new Firebase tooling for local development.

Firebase Storage And Firestore With Node.js For Absolute Beginners - Storing files and data to Firebase using NodeJS from scratch.

How we moved 6 Million Users from Auth0 to Firebase - The article explains how 6 million of user accounts where migrated from Auth0 to Firebase, how it it was achieved and what were the challenges.

How Does Spanner Avoid Single Point of Failures in Writes? - Explanation of how Cloud Spanner provides both high availability and high consistency in writes.

Big Data, Analytics, ML&AI

Predicting the cost of a Dataflow job - Estimate the cost of batch and streaming analytics service jobs in Google Cloud’s Dataflow.

Apache Spark BigQuery Connector — Optimization tips & example Jupyter Notebooks - Learn how to use the BigQuery Storage API with Apache Spark on Cloud Dataproc.

Airflow with Twitter Scraper, Google Cloud Storage, Big Query — tweets relating to Covid19 - Part Two of a Four-part Data Engineering Pipeline.

Apache Airflow and Kubernetes — Pain Points and Plugins to the Rescue - Some of the Airflow pain points and how they were solved when deployed on Kubernetes Engine.

Celebrating a decade of data: BigQuery turns 10 - BigQuery, Google Cloud’s data analytics platform, turns 10 in 2020. Here’s a look back on big data trends in the past decade.

How to use Dynamic SQL in BigQuery - Format a string, and use EXECUTE IMMEDIATE.

ODBC Driver for Google BigQuery — A New Connectivity Solution from Devart - Devart has announced the release of a new connectivity solution, ODBC Driver for Google BigQuery, which allows access to Google BigQuery.

Call PostgreSQL from BigQuery for extra GIS powers - Combining PostgreSQL and BigQuery queries to validate GIS data.

Google Cloud Data Catalog — Keep Up With Your On-Prem Hive Server - Code samples with a practical approach on how to ingest metadata from an on-premise Hive server into Google Cloud Data Catalog.

15 advanced Data Studio hacks to turn you into a reporting pro - Collections of hacks and tips for working with Data Studio.

AI Workshop Experiments - AI Workshop offers customers, partners, researchers, and developers the opportunity to experiment with cutting-edge AI innovations.

Various

Audiobahn: Use this AI pipeline to categorize audio content–fast - How to create a processing pipeline to analyze audio content and a UI to view the results.

Why I think GCP is better than AWS - Personal opinion based on experience with both platforms.

How to Pass the Associate Cloud Engineering Exam on the First Try: Exam Resources and Tips - Preparing for the Associate Cloud Engineering exam.

Google Anthos To Speed Up Pentagon’s Multi-Cloud Efforts - Google Cloud on Wednesday announced that the Defense Innovation Unit (DIU) – an organization within the Department of Defense (DoD) – selects Google Cloud to build secure Cloud Management Solution to detect, protect against, respond to cyber threats.

Slides, Videos, Audio

GCP Podcast - #221 BeyondCorp with Robert Sadowski.

Kubernetes Podcast - #104 Ingress and the Service APIs, with Bowei Du.

DevOps II (Google Cloud Talks by DevRel) - Session 1: Go is the language of the Cloud (Jaana Dogan) Session 2: Securing container build pipelines (Don McCasland).

Using TensorFlow Extended (TFX) on AI Platform Pipelines

Releases

Anthos Config Management - 1.3.2. This release includes several performance and memory improvements. Error documentation has been updated to add more information on error codes. Anthos Config Management now supports a GKE-only authentication mechanism based on the service account of the cluster's node pool. Anthos Config Management now includes Config Connector v1.8.0. Anthos Config Management will now attempt to detect when resources that it manages are also managed by other controllers. Policy Controller has been upgraded to include a newer version of Open Policy Agent Gatekeeper.

Anthos - Anthos 1.3.2 is now available.

BigQuery - The BigQuery Storage API now supports reading small anonymous (cached) tables without any limitations. Happy 10th birthday, BigQuery!. Cloud SQL federated queries are now generally available (GA). Hourly partitioned tables are now in beta. Dynamic SQL is now available as a beta release in all BigQuery regions. BigQuery Trial slots are now available in US and EU multi-regions.

BigTable - The Cloud Bigtable Monitoring page in the Cloud Console has been redesigned.

Billing - Cloud Billing budgets emails: ensure your budget alert emails are seen by the right people using Cloud Monitoring notifications on your Cloud Billing budgets. New information is now available on your Cloud Billing account Overview page in the Cloud Console, featuring at-a-glance summaries of the top five spending projects and top five spending products over the last 12 months.

Compute Engine - E2 shared-core machine types now support committed use discounts in all regions. You can now SSH to your VMs using hardware-backed SSH key pairs. If your managed instance group encountered errors - for example, if a VM could not be created - you can view those errors to diagnose and mitigate the cause. Troubleshoot VMs by capturing screenshots.

Config Connector - Bug fixes and reliability improvements. Improving handling of scenarios when version field on ContainerNodePool is updated externally.

Cloud Debugger - Cloud Debugger now lets you canary snapshots and logpoints on your Java applications.

Cloud Networking Products - DNS forwarding to a non-RFC 1918 address is available in General Availability.

Cloud Filestore - Learn how to create low disk space alerts for your Filestore instances.

Cloud Functions - Cloud Functions now supports Node.js 10 at the General Availability release level.

GKE on Prem - Workload Identity is now available in Alpha for GKE on-prem. Preflight check for VM internet and Docker Registry access validation is updated. Preflight check for internet validation is updated to not follow redirect. The Ubuntu image is upgraded to include the newest packages. Upgraded the Istio image to version 1.4.7 to fix a security vulnerability. Some ConfigMaps in the admin cluster were refactored to Secrets to allow for more granular access control of sensitive configuration data.

IAM - Recommendations from the Cloud IAM recommender can now include suggestions to create custom roles.

Google Kubernetes Engine - GKE cluster versions have been updated. Nodes with auto-upgrade enabled will be upgraded: Rollouts are phased across multiple weeks, to ensure cluster and fleet stability. The following Kubernetes versions are now available for new clusters and for opt-in master upgrades and node upgrades for existing clusters. No channel Note: Your clusters might not have these versions available. The COS image for GKE 1.17 clusters is now cos-81-12871-96-0. In the Rapid release channel, all GKE clusters running 1.17.3-gke.3 and up will have etcd upgraded to 3.4.7-0-gke.1. Google Kubernetes Engine now supports the use of non-RFC 1918 private address ranges and the private reuse of public IP addresses in VPC-native clusters.

Google Kubernetes Engine Rapid - 1.17.5-gke.6 is now available in the Rapid release channel. All GKE clusters running 1.17.3-gke.3 and up will have etcd upgraded to 3.4.7-0-gke.1.

Load Balancing - For internal TCP/UDP load balancers, you can create multiple forwarding rules with the same IP address.

Cloud Logging - Logs Viewer now contains the Logs field explorer panel, which lets you view aggregation-based results for your project's log fields and makes it more efficient to refine queries.

Cloud Monitoring - Cloud Monitoring introduces an improved experience for viewing and managing incidents. Alert notifications delivered by email now come from "alerting-noreply@google.com" instead of "alerts@stackdriver.com".

Cloud Run - The Cloud Run container instance metadata server now exposes the unique identifier of the container instance and the region of the Cloud Run service.

Cloud Spanner - You can now run SQL queries to retrieve transaction statistics for your database over recent one-minute, 10-minute, and one-hour time periods.

Cloud SQL MySQL - MySQL 5.6 minor version is upgraded to 5.6.42.

Cloud SQL Postgres - PostgreSQL version 12 is now generally available.

Cloud Storage - The V4 signing process is now in GA.

Cloud TPU - Cloud TPU now supports TensorFlow 2.1.1 with Keras support.

Cloud Video Intelligence API - The following features are available in the Video Intelligence API version v1p3beta1: Face detection: Locate faces within a video, and identify attributes such as glasses being worn.

Virtual Private Cloud - Subnets in VPC networks now support IP addresses other than RFC 1918 addresses.

VPC Service Controls - Beta stage support for the following integration: Service Directory.

Anthos GKE deployed on-prem - Workload Identity is now available in Alpha for GKE on-prem. Preflight check for VM internet and Docker Registry access validation is updated. Preflight check for internet validation is updated to not follow redirect. The Ubuntu image is upgraded to include the newest packages. Upgraded the Istio image to version 1.4.7 to fix a security vulnerability. Some ConfigMaps in the admin cluster were refactored to Secrets to allow for more granular access control of sensitive configuration data.

Service Mesh - 1.5.4-asm.2 1.5.4-asm.2 is now available. Security fixes 1.5.4-asm.2 contains all the same security fixes that are in Anthos Service Mesh 1.4. Beta release of the Anthos CLI The Anthos CLI simplifies the installation of Anthos Service Mesh. Port change for automatic sidecar injection If you are installing Anthos Service Mesh on a private cluster, you must add a firewall rule to open port 15017 if you want to use automatic sidecar injection. The alpha authentication policy is deprecated See Updating to the beta security policies for more information. IstioOperator API replaces IstioControlPlane API The alpha IstioControlPlane API has been replaced by the IstioOperator API. Istio CNI plugin is supported By default Anthos Service Mesh injects an initContainer, istio-init, in pods deployed in the mesh. Enabling pod security policies no longer needed SDS security was improved by merging Node Agent with Pilot Agent as Istio Agent and removing cross-pod UDS, which no longer requires users to deploy Kubernetes pod security policies for UDS connections.

AI Platform Training - You can now use TPUs with TensorFlow 2.1 when you create a training job with runtime version 2.1.

Anthos GKE on-prem - Workload Identity is now available in Alpha for GKE on-prem. Preflight check for VM internet and Docker Registry access validation is updated. Preflight check for internet validation is updated to not follow redirect. The Ubuntu image is upgraded to include the newest packages. Upgraded the Istio image to version 1.4.7 to fix a security vulnerability. Some ConfigMaps in the admin cluster were refactored to Secrets to allow for more granular access control of sensitive configuration data.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko