Welcome to issue #231 March 1st, 2021

News

Introducing GKE Autopilot: a revolution in managed Kubernetes - GKE Autopilot gives you a fully managed, hardened Kubernetes cluster out of the box, for true hands-free operations.

BigQuery delivers a modern view of materialized views - BigQuery materialized views is now GA and provides query cost-optimization and increased performance.

New in BigQuery BI Engine: faster insights across popular BI tools - BigQuery BI Engine ensures significantly faster queries across popular BI tools.

MongoDB and Google Extend Partnership To Drive Enterprise Cloud Modernization - MongoDB and Google Partner To Drive Enterprise Cloud Modernization.

A look at the new Google Cloud Marketplace Private Catalog, now with Terraform support - The latest version of Private Catalog simplifies management for the products you use from Google Cloud Marketplace.

At your service! With schedule-based autoscaling, VMs are at the ready - Schedule-based autoscaling for Compute Engine lets you improve the availability of your workloads by scheduling capacity ahead of anticipated load.

Increasing limits for three key Cloud Monitoring features - Cloud Monitoring now supports more projects and custom metric descriptors, and retains metrics from Compute Engine agents for longer.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

With SRE, failing to plan is planning to fail - The process of becoming a successful Site Reliability Engineering shop starts well before you take your first class or read your first manual.

Troubleshooting services on Google Kubernetes Engine by example - Learn how to deploy a sample app to your cluster and configure an alerting policy that will notify you if there are any container restarts observed.

Stay informed by customizing your budgets - TL;DR - By default, budget alerts only go to Billing Account Administrators and Billing Account Users. You can easily add up to 5 more custom emails by using a Cloud Monitoring channel. Try to send to groups, not individuals!

What’s in a name? Understanding the Google Cloud network “edge” - The network edge is in the eye of the beholder. Learn about Google Cloud’s various network edge points and how traffic handoffs occur for each one.

Identify Unused Service Accounts in GCP - Easily find and identify unused service accounts across your entire GCP organization.

Anthos lightweight developer sandbox - A quick way to get started using some features of GCP Anthos.

Google Kubernetes Engine Autopilot - Autopilot overview.

Google Kubernetes Engine — Autopilot - Autopilot overview with a demo.

How to build an accountability data lake on Google Cloud Platform in 30 minutes - How to activate Audit Logs for an organization.

Using Public and Private Docker Images from GCP Artifact Registry - Using selective access control in Artifact Registry.

App Development, Serverless, Databases, DevOps

Securing Load balanced Authenticated Cloud Run Services using IAP - When designing your cloud run services, you should always consider following points.

Conversational AI with Apigee API Management for enhancing customer experiences - How Google Cloud’s Dialogflow and Apigee make it easy to deploy interactive virtual agents.

App monetization made easy: Using Firebase, AdMob, and Google Analytics to uncover your optimal ads strategy

3 common serverless patterns to build with Workflows - Learn how to build powerful serverless applications using these common architectural patterns that are now possible with Workflows and Connectors.

Google Cloud SDK with Service Account on Raspberry Pi - Using a Raspberry Pi to interact with your Google Cloud Platform projects without having to expose your user credentials.

Crypto updates with Cloud Functions (API’s, Serverless, and Twilio) - Serverless notifications about cryptocurrency data.

Parallel executions with Google Workflows - Example of doing parallel tasks in Cloud Workflows.

Big Data, Analytics, ML&AI

Framework for building a configuration driven data lake using Data Fusion and Composer - This article provides a framework for building a configuration driven data lake on Google Cloud using Data Fusion and Composer.

Accelerate data science workflows with Looker, plus free training to help get started - Learn how data analysts and scientists can use Looker to help with data governance, join our training to walk through working examples of how Looker can be leveraged to help automate and productionalize data science workflows.

How to use a machine learning model from a Google Sheet using BigQuery ML - By combining Sheets with BigQuery ML, you can bring the power of machine learning to data in your spreadsheet.

Migrating Apache Hadoop to Dataproc: A decision tree - Are you using the Apache Hadoop and Spark ecosystem? Are you looking to simplify the management of resources while continuing to use the same tools? If yes, then Dataproc is the tool to check out. In this blog post, we will briefly cover Dataproc and then highlight the four scenarios to migrate Apache Hadoop cluster to Google Cloud.

Run data science at scale with Dataproc and Apache Spark - Dataproc Hub, a feature now generally available for Dataproc users, provides an easier way to scale processing for common data science libraries and notebooks, govern custom open source clusters, and manage costs so that enterprises can maximize their existing skills and software investments.

How To Load Data Into Google BigQuery on Dataproc and AutoML - Loading data into BigQuery using StreamSets.

Scale model training in minutes with RAPIDS + Dask and NVIDIA GPUs on AI Platform - Scale model training from 10 GB to 640 GB in minutes with RAPIDS + Dask and NVIDIA GPUs on AI Platform.

Full relational diagram for Ethereum public data on Google BigQuery - Full relational diagram for Ethereum data with SQL examples covering Internal Transactions, Ethereum Trace, Tokens and Smart Contracts.

BigQuery Hack: Flexible Queries For Any Number of Columns - How can we use BigQuery to handle tables with many columns? Here’s how using scripting and table metadata.

Training PyTorch Transformers on GCP AI Platform - How to use highly popular ML framework PyTorch on AI Platform Training.

Monitoring your Dataflow pipelines - This article gives an overview of the different metrics and logs you can use on Google Cloud Platform to monitor your Dataflow jobs.

From cURL to Automated Workflow - Creating a pipeline to load financial data into BigQuery and do Analytics.

XGBoost deployment on GCP with Google AI Platform - Reducing the gap between a PoC and an integrated solution for binary classification.

Various

Google Cloud vs AWS Onboarding Comparison

How to become certified Google Professional Data Engineer - Path to success and study materials for Data Engineer certification.

Google cloud professional security engineer study guide - Topics covered for Security Engineer certification exam.

Slides, Videos, Audio

GCP Podcast - #248 Cloud Spanner Revisited with Dilraj Kaur and Christoph Bussler.

Kubernetes Podcast - #139 GKE Autopilot, with Yochay Kiriaty.

Releases

Anthos Config Management - 1.6.2. Hierarchy Controller now includes a preview of Hierarchical Resource Quotas (HRQs). Config Sync now takes advantage of server-side apply when possible, so will be able to handle custom resources larger than 256 kB on Kubernetes clusters with a version of 1.16 or later. The Anthos Config Management Operator Deployment now specifies resources.limits for config-management-operator:manager.

Anthos - Anthos 1.6.2 is now available.

BI Engine - BigQuery BI Engine now interacts with popular BI tools such as Looker, Tableau, and more, by means of an SQL interface.

BigQuery - The INFORMATION_SCHEMA.TABLES view now includes a DDL column. BigQuery materialized views are now generally available (GA).

BigQuery Transfer - The BigQuery Data Transfer Service's 1-hour minimum file age requirement for transfers from Cloud Storage has been eliminated. The BigQuery Data Transfer Service's minimum interval time between recurring transfers from Cloud Storage has been reduced from one hour to 15 minutes.

Billing - Optimal Recommendations for Compute Engine committed use discounts are now Generally Available.

Cloud Composer - New versions of Cloud Composer images: composer-1.14.4-airflow-1.10.14 composer-1.14.4-airflow-1.10.12 (default) composer-1.14.4-airflow-1.10.10. When an environment update operation cannot start, an error message that lists possible causes for the error is generated. Improved the syncing of DAGs and plugins to the Airflow web server. Added FreeTDS system package to Cloud Composer images. Updated apache-beam package version to 2.24.0 in Airflow 1.10.10 so that Dataflow jobs now correctly create partitioned BigQuery tables. Upgraded apache-airflow-backport-providers-google package to version 2021.2.5 in Airflow 1.10.12 and 1.10.14. The google-cloud-pubsublite package is installed by default in Composer images for Airflow 1.10.12 and 1.10.14. Improved the reliability of environment upgrade operations. Fixed the cause of several update-related errors. When an environment deletion operation fails, a correct GKE error is displayed. The Airflow scheduler liveness checker is now compatible with google-cloud-logging==2.2.0. Fixed an error when some MsSQL operators were not working with Azure instances. Fixed an error in GKE cluster builds. GA: Support for Customer Managed Encryption Keys (CMEK) is now generally available.

Compute Engine - Preview: You can now use the gcloud command-line tool to import images from AWS into Google Cloud.

Data Fusion - Cloud Data Fusion Beta instances (versions 6.1.0.2 and lower that were created before November 21, 2019) will be turned down on March 1, 2021.

Dataproc - New sub-minor versions of Dataproc images: 1.3.85-debian10, 1.3.85-ubuntu18, 1.4.56-debian10, 1.4.56-ubuntu18, 1.5.31-centos8, 1.5.31-debian10, 1.5.31-ubuntu18, 2.0.4-debian10, and 2.0.4-ubuntu18. Image 2.0: Upgraded Spark to 3.1.1 RC2 version. Allow stopping clusters that have autoscaling enabled, and allow enabling autoscaling on clusters that are STOPPED, STOPPING, or STARTING. Deactivated mysql and hive-metastore components for clusters created with a Dataproc Metastore service on an image that has the DISABLE_COMPONENT_HIVE_METASTORE and DISABLE_COMPONENT_MYSQL capabilities. Image 1.3 - 1.5: HIVE-18871: hive on Tez execution error due to set hive.aux.jars.path to hdfs://. Dataproc 2.0 image version will become a default Dataproc image version in 3 weeks on March 15, 2021.

Cloud Firestore - A Firestore connector for Workflows is now available in public preview.

GKE on Prem - Anthos clusters on VMware (GKE on-prem) 1.6.2-gke.0 is now available. Fixes: Fixed a kubelet restarting issue that was found when running workloads that rely on kubectl exec/port-forward/attach, such as Jenkins.

IAM - You can now use Policy Simulator to simulate policy changes before you apply them.

Anthos Migrate - On February 23, 2021, we released Migrate for Anthos 1.6.2. 180576558: Fixed an issue where the Linux discovery tool calculated an incorrect score. Fixed an issue where using an Envoy proxy sidecar, not as part of Istio or Anthos Service Mesh, created networking issues with the migrated workload.

Private Catalog - Private Catalog supports Terraform.

Resource Manager - Project migration between organizations is now a self-serve process in public preview.

Cloud SQL MySQL - Cloud SQL now offers faster maintenance, with average connectivity loss lasting 90 seconds or less on average.

Cloud SQL Postgres - Cloud SQL now offers faster maintenance, with average connectivity loss lasting 90 seconds or less on average. 

Virtual Private Cloud - Hierarchical firewall policies are now available in General Availability. The ability to connect VM interfaces other than nic0 to a Shared VPC is now available in General Availability.

Anthos GKE deployed on-prem - Anthos clusters on VMware (GKE on-prem) 1.6.2-gke.0 is now available. Fixes: Fixed a kubelet restarting issue that was found when running workloads that rely on kubectl exec/port-forward/attach, such as Jenkins.

Service Mesh - 1.8.x. 1.8.3-asm.2 is now available.

Secret Manager - Event notifications is now available in Preview.

Artifact Registry - Support for Python packages in private PyPI repositories is now in alpha.

Memorystore for Memcached - v1. General Availability release of Memorystore for Memcached.

Anthos GKE on AWS - Anthos clusters on AWS 1.6.2-gke.0 is now available. This release fixes an issue where the management service fails to start when provided with a KMS alias. Bug fixes and security improvements.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko