Welcome to issue #200 July 27th, 2020

News

What happened week 2 of Google Cloud Next ’20: OnAir - In the second week at Next OnAir, a spotlight on collaboration and productivity solutions to help organizations work together and get things done. Here are the highlights.

Giving you better cost analytics capabilities—and a simpler invoice - Google Cloud Console features cost management tools to help financial operations (FinOps) teams analyze and predict their organization’s cloud costs.

Online shopping gets more personal with Recommendations AI - Create a more personalized online shopping experience with Recommendations AI.

Traffic Director and gRPC—proxyless services for your service mesh - With the addition of xDS API support, you can now use Traffic Director with proxyless gRPC services.

Product news and other highlights from Firebase Live - Recent Firebase releases.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Use Google Managed Certificates on a Google Cloud Load Balancer - A stress-free way to manage HTTPS certificates in the cloud.

Why should you treat infrastructure like software? - A brief intro into principles of immutable infrastructures.

Infrastructure as Code — Exploring GCP Deployment Manager - Intro to creating infrastructure with Deployment Manager.

Automate Elasticsearch deployment in GCP with Terraform - Setting up your Elasticsearch, Kibana, and Logstash (ELK) environment in one command.

Vault Secrets for GCP Credential Access Boundary and Impersonation - Vault plugin that exchanges a VAULT_TOKEN for a GCP access_token that as attenuated permissions.

GCP and AWS Networking — Concepts and Services - What is different and what is not.

Create and deploy a Scala HTTP server to Google Kubernetes Engine (GKE) - Creating basic Scala web app and deploying to GKE.

Anthos + Apigee: 4 Lessons Learned - Experience of integrating Anthos and Apigee hybrid.

App Development, Serverless, Databases, DevOps

Using new traffic control features in External HTTP(S) load balancer - Examples of how to use the new traffic control features in Cloud Load Balancing HTTP(S) load balancers.

Using Memorystore with Cloud Run - Setting Cloud Memorystore to be used in the Cloud Run web app.

The Easiest Docker & Docker-compose setup on Compute Engine - How to install Docker & docker-compose on Compute Engine with ease.

Triggering Cloud Storage With Cloud Functions in Java - Example of using Java Cloud Function.

The hidden costs of Datastore - Datastore export costs don’t show up in Monitoring. Here’s how to set up real-time alerts to better keep track of export costs.

How to Trigger Cloud Function From Logs on Google Cloud platform - Triggering Cloud Function based on log type.

Big Data, Analytics, ML&AI

Building a Data Platform to Enable Analytics and AI-Driven Innovation - Build a Data Mesh & Set up MLOps.

ETL Google Spreadsheet Using Pandas To BigQuery - Creating the ETL process on a google spreadsheet and store it into BigQuery.

Migrate and modernize your on-prem data lake with managed Kafka - Make sure your data lake is working for your business and see how to use tools like Apache Kafka to migrate from on-prem.

Google Cloud Dataflow template generation using Gradle and FAT JAR - Generating a Dataflow template using Gradle and single uber/fat jar.

How to create PubSub subscriptions across two different GCP projects - How to configure PubSub and Cloud Function to be triggered by event from a different GCP project.

ETL Pipeline for creating TF-Records using Apache Beam Python SDK on Google Cloud Dataflow - An example of scaling the process of creating TF records for a computer vision dataset in Beam pipeline deployed on Cloud Dataflow.

Getting Started with Jupyter + Spark on the Cloud in 2020 - Spinning up Spark clusters with Jupyter on Cloud Dataproc.

BigQuery: the unlikely birth of a cloud juggernaut - Story about how 10 engineers transformed cloud data analytics.

Why BigQuery Omni is a Big Deal - Google Cloud’s bet on an open platform is starting to materialize with Anthos and BigQuery Omni.

Synchronize Google Sheets with BigQuery Tables - Keep your spreadsheets synchronized with BigQuery using Sheet and Apps Script templates.

Data Science 101 for Startups- Aggregation in SQL — Part 2 - Using aggregation SQL functions on BigQuery public dataset.

BigQuery HMAC-SHA256 User-defined Function - Google Cloud BigQuery User-defined function to perform HMAC with SHA256.

5 steps for data analytics using Google Cloud and Marketing tools - How to leverage CSV, Cloud Storage, and Data Studio to support fast data-driven decision making.

Improved customer feedback management with Google Cloud AutoML - How you can use AutoML to make your customer feedback management more efficient.

Model Training using Google Cloud AI Platform — Custom Containers - Creating a custom container to train ML model on the AI platform.

Create AI Platform Notebooks Instance in Shared VPC without Service Availability - Handling issue with connection AI platform notebook instance and shared VPC.

Various

Google Cloud Booms: Thomas Kurian’s Vision for World’s Fastest-Growing Cloud - Analysis of GCP CEO Thomas Kurian's keynote for CloudNext '20.

Helping A Business Incrementally Migrate From AWS and Cloudflare to GCP - Example of multi-phased migration from AWS to GCP.

What to really expect from the latest Google Cloud Platform Data Engineering Certification… - (…and how to pass it successfully without getting stressed out).

The Cognitive Ride while taking a Google Certification Exam — journey to success!! - Describing experience of preparing and taking the Associate Cloud Engineer exam.

How I became Google Cloud Certified Professional Cloud Architect - Getting Cloud Architect certification and comparing with AWS and Azure experience.

My Journey of Google Cloud Certified Associate Cloud Engineer Exam and Guidance & Tips - A guide for those who are thinking about taking the Associate Cloud Engineer exam.

Slides, Videos, Audio

GCP Podcast - #228 Fastly with Tyler McMullen.

Kubernetes Podcast - #113 Instrumentation and cAdvisor, with David Ashpole.

Episode 14: Digital Native Podcast - Jordan Tigani, Director of Product Mgmt for Google BigQuery

Releases

Anthos Config Management - 1.4.1. Config Connector has been updated in Anthos Config Management to version 1.13.1. Anthos Config Management now includes Hierarchy Controller as a beta feature. Policy Controller users may now enable --log-denies to log all denies and dryrun failures. This release includes several logging and performance improvements. This release includes several fixes and improvements for the nomos command line utility. The use of unsecured HTTP for GitHub repo connections or in an http_proxy is now discouraged, and support for unsecured HTTP will be removed in a future release. This release improves the handling of GitHub repositories with very large histories. Prior to this release, Config Sync and kubectl controllers and processes used the same annotation (kubectl.kubernetes.io/last-applied-configuration) to calculate three-way merge patches. In error messages, links to error docs are now more concise.

Anthos - Anthos 1.4.1 is now available.

BigQuery - BigQuery Data Transfer Service is now available in the following regions: Montréal (northamerica-northeast1), Frankfurt (europe-west3), Mumbai (asia-south1), and Seoul (asia-northeast3).

BigQuery Transfer - BigQuery Data Transfer Service is now available in the following regions: Montréal (northamerica-northeast1), Frankfurt (europe-west3), Mumbai (asia-south1), and Seoul (asia-northeast3).

BigTable - Cloud Bigtable's fully integrated backups feature is now generally available.

Billing - Export your Cloud Billing account SKU prices to BigQuery.

Cloud Composer - New versions of Cloud Composer images: composer-1.11.0-airflow-1.10.2, composer-1.11.0-airflow-1.10.3, composer-1.11.0-airflow-1.10.6, and composer-1.11.0-airflow-1.10.9. Airflow 1.10.9 is now supported. New PyPi packages have been added for Composer version composer-1.11.0-airflow-1.10.6. Composer versions 1.6.1, 1.7.0, and 1.7.1 are now deprecated.

Compute Engine - NVIDIA® Tesla® T4 GPUs are now available in the following additional regions and zones: Ashburn, Northern Virginia, USA: us-east4-b For information about using T4 GPUs on Compute Engine, see GPUs on Compute Engine. N2 machines are now available in Northern Virginia us-east4-c. You can now create balanced persistent disks , in addition to standard and SSD persistent disks.

Config Connector - bug fixes and performance improvements.

Dataproc - Terminals started in Jupyter and JupyterLab now use login shells. Upgraded the jupyter-gcs-contents-manager package to the latest version. New sub-minor versions of Dataproc images: 1.3.64-debian10, 1.3.64-ubuntu18, 1.4.35-debian10, 1.4.35-ubuntu18, 1.5.10-debian10, 1.5.10-ubuntu18, 2.0.0-RC6-debian10, and 2.0.0-RC6-ubuntu18. Fixed a bug in which the HDFS DataNode daemon was enabled on secondary workers but not started (except on VM reboot if started automatically by systemd). Fixed a bug in which StartLimitIntervalSec=0 appeared in the Service section instead of the Unit section for systemd services, which disabled rate limiting for retries when systemd restarted a service.

Dialogflow - Amazon Alexa importer and exporter are no longer supported.

GKE on Prem - Anthos GKE on-prem 1.4.1-gke.1 is now available. Anthos Identity Service LDAP authentication is now available in Alpha for GKE on-prem Contact support if you are interested in a trial of the LDAP authentication feature in GKE on-prem. Support for F5 BIG-IP load balancer credentials update This preview release enables customers to manage and update the F5 BIG-IP load balancer credentials by using the gkectl update credentials f5bigip command. Functionality changes: The Ubuntu image is upgraded to include the newest packages. Fixes: Removed the static IP used by admin workstation after upgrade from ~/.ssh/known_hosts to avoid manual workaround. Known issues: During reboots, the data disk is not remounted on the admin workstation when using GKE on-prem 1.4.0 or 1.4.1 because the startup script is not run after the initial creation.

IAM - We are delaying the upcoming changes for deleted members that are bound to a role.

Istio on GKE - 1.4.x. Istio 1.4.10-gke.4 Fixes known security issues with the same fixes as OSS Istio 1.4.10.

Recommendations AI - Recommendations AI public beta Recommendations AI is now in public beta. New pricing available Pricing for Recommendations AI has been updated for public beta. UI redesign The Recommendations AI console has a new look. New support resources We have new support resources available: File bugs and feature requests on our public issue tracker. New FAQ page A Frequently Asked Questions page is now available.

Resource Manager - The Organization Policy for enabling detailed Cloud Audit Logs has launched into general availability.

Cloud Run - Cloud Run resources are now available in Cloud Asset Inventory.

Cloud Storage - Detailed audit logging mode launched.

Traffic Director - Traffic Director supports proxyless gRPC applications in General Availability.

AutoML Video Intelligence - In April 2020, a model upgrade for the AutoML Video Object Tracking feature was released.

Virtual Private Cloud - Serverless VPC Access support for Shared VPC is now available in Beta.

VPC Service Controls - General availability for the following integration: AI Platform Training.

Network Intelligence Center - Network Topology includes two new metrics for connections between entities: packet loss and latency.

Anthos GKE deployed on-prem - Anthos GKE on-prem 1.4.1-gke.1 is now available. Anthos Identity Service LDAP authentication is now available in Alpha for GKE on-prem Contact support if you are interested in a trial of the LDAP authentication feature in GKE on-prem. Support for F5 BIG-IP load balancer credentials update This preview release enables customers to manage and update the F5 BIG-IP load balancer credentials by using the gkectl update credentials f5bigip command. Functionality changes: The Ubuntu image is upgraded to include the newest packages. Fixes: Removed the static IP used by admin workstation after upgrade from ~/.ssh/known_hosts to avoid manual workaround. Known issues: During reboots, the data disk is not remounted on the admin workstation when using GKE on-prem 1.4.0 or 1.4.1 because the startup script is not run after the initial creation.

Dialogflow Enterprise - Amazon Alexa importer and exporter are no longer supported.

Service Mesh - Anthos Service Mesh on GKE on AWS is supported. 1.6.5-asm.7, 1.5.8-asm.7, and 1.4.10-asm.15 are now available This release provides these features and fixes: Builds Istiod (Pilot), Citadel Agent, Pilot Agent, Galley, and Sidecar Injector with Go+BoringCrypto.

Secret Manager - Secret Manager adds support for the following curated Cloud IAM roles: Secret Manager Secret Version Adder (roles/secretmanager.secretVersionAdder ) Secret Manager Secret Version Manager (roles/secretmanager.secretVersionManager) To learn more, see IAM and access control.

AI Platform Training - Customer-managed encryption keys (CMEK) for AI Platform Training is now generally available. The VPC Service Controls integration with AI Platform Training is now generally available. You can now train a PyTorch model on AI Platform Training by using a pre-built PyTorch container.

Anthos GKE on-prem - Anthos GKE on-prem 1.4.1-gke.1 is now available. Anthos Identity Service LDAP authentication is now available in Alpha for GKE on-prem Contact support if you are interested in a trial of the LDAP authentication feature in GKE on-prem. Support for F5 BIG-IP load balancer credentials update This preview release enables customers to manage and update the F5 BIG-IP load balancer credentials by using the gkectl update credentials f5bigip command. Functionality changes: The Ubuntu image is upgraded to include the newest packages. Fixes: Removed the static IP used by admin workstation after upgrade from ~/.ssh/known_hosts to avoid manual workaround. Known issues: During reboots, the data disk is not remounted on the admin workstation when using GKE on-prem 1.4.0 or 1.4.1 because the startup script is not run after the initial creation.

Anthos GKE on AWS - Anthos GKE on AWS is now generally available. Clusters support in-place upgrades, with the ability to upgrade the control plane and node pools separately. Clusters can be deployed in a high availability (HA) configuration, where control plane instances and node pools are spread across multiple availability zones. Clusters have been validated to support up to 200 nodes and 6000 pods. Allows the number of nodes to be scaled dynamically based on traffic volume to increase utilization and reduce cost, and improve performance. Anthos can be deployed within existing AWS VPCs, leveraging existing security groups to secure those clusters. With Anthos Config Management enterprises can set policies on their AWS workloads and with Anthos Service Mesh, they can monitor, manage, and secure them. Kubernetes settings (flags and sysctl settings) have been updated to match GKE. Upgrades from beta versions are not supported.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko