Welcome to issue #249 July 5th, 2021

News

What's new with BigQuery ML: Unsupervised anomaly detection for time series and non-time series data

Mercury Rising in BigQuery with Multistatement Transactions - Multistatement transactions in BigQuery now in Preview supports COMMIT and ROLLBACK operations.

AutoML Tables is now generally available in BigQuery ML - AutoML Tables model type is now generally available as supported ML model within BigQuery ML.

Shine on with user-friendly SQL capabilities in BigQuery - We are pleased to announce new capabilities in three themes: Powerful Analytics Features, Flexible Columns, and Secure with SQL.

Let Google Cloud’s predictive services autoscale your infrastructure - A new predictive autoscaling capability lets you add additional Compute Engine VMs in anticipation of forecasted demand.

Implementing Application-Consistent Data Protection for Compute Engine Workloads - This blog announces the availability of application consistent snapshot hooks for Linux, available for snapshots and machine images running in Compute Engine.

New in Google Cloud VMware Engine: autoscaling, Mumbai expansion, etc. - A review of the latest updates to Google Cloud VMware Engine.

Announcing the 2021 State of DevOps Report Sponsors

Three security and scalability improvements for Cloud SQL for SQL Server - Three new feature releases create further functionality for Cloud SQL for SQL Server.

Top 25 Google Search terms, now in BigQuery - Google Trends datasets for the Top 25 terms and Top 25 Rising terms now available in BigQuery to enhance your business analyses.

The Cloud Storage for Firebase Emulator: The Final Piece of the Puzzle - The Cloud Storage for Firebase emulator integrates with the rest of the Firebase Emulator Suite, including the Firebase Auth emulator and the Cloud Functions emulator, unlocking the ability to locally test your app end-to-end like never before.

Google joins the O-RAN ALLIANCE to advance telecommunication networks - Google Cloud joins O-RAN ALLIANCE to drive transformative change in telecommunications.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

It’s about “Time”: A proactive approach to ransomware recovery - Ransomware is a pervasive, ever-evolving threat impacting organizations globally, regardless of size, geographic location, or industry. Taking a proactive approach to cyber resilience, including implementation of a robust ransomware recovery strategy, has emerged as a fundamental aspect of security preparedness and business continuity planning.

Dashboards on Cloud Monitoring made easier with samples - Creating custom dashboards can be complicated if you don't have a good place to start. Google Cloud Monitoring provides more than 60 open source dashboard samples across many categories that customers can customize to their needs.

Build a platform with KRM: Part 4 - Administering a multi-cluster environment - Manage the complexity of a multi-cluster platform with powerful tools from Google.

Devices and Zero Trust security - GCP Comics #7 Device security.

Google Cloud Platform and API Call Attribution - Analysis of how API calls are attributed to the GCP project.

How proper default settings can save money and trees - Investigating and solving issues with Terraform GKE module.

How to Restore From GKE Persistent Disk Snapshot? - This article explains how to restore it in a new persistent volume and volume claim in GKE.

Run Oracle on Kubernetes with El Carro - El Carro helps you with the deployment and management of Oracle database software in Kubernetes.

App Development, Serverless, Databases, DevOps

Top 13 Google Cloud reference Architectures - Summary of #BuildonGCP architecture series.

Building Firebase Quickdraw - This blog post explains how Firebase Quickdraw app has been built.

How to create a Firebase Cloud Function with static outbound IP - Assigning static IP to the Cloud Function.

Authenticate Firebase Users to consume your GCP backend - Integrating Firebase with the rest of GCP products.

The ultimate App Engine cheat sheet - App Engine is a fully managed serverless compute option in Google Cloud that you can use to build and deploy low-latency, highly scalable applications. App Engine makes it easy to host and run your applications.

Cloud SQL for SQL Server: Database administration best practices - Cloud SQL for SQL Server is a fully-managed relational database service that makes it easy to set up, maintain, manage, and administer SQL Server databases on Google Cloud Platform. While the Cloud SQL platform is fully-managed, SQL Server databases require administration in order to perform optimally. This best practices post is designed to help database administrators manage SQL Server databases running on Cloud SQL.

Running older SQL Server (< 2017) databases on Cloud SQL for SQL Server - If you have databases running on SQL Server instances with versions older than 2017, you may be wondering if you can run those databases on Cloud SQL and how.

Google Cloud CDN Custom Dashboard - An example of a custom Dashboard in Cloud Monitoring for Cloud CDN.

Big Data, Analytics, ML&AI

Dataflow, the backbone of data analytics - An overview of Cloud Dataflow.

Building a simple Google Cloud Dataflow pipeline: PubSub to Google Cloud Storage - This article examines building a streaming pipeline with Dataflow templates to feed downstream systems.

Quick Steps to Build & Deploy Dataflow Flex Templates (Python & Java) - Flex Templates package the Dataflow pipeline artefacts as a Docker image and stage these images on Google Container Registry.

BigQuery admin reference guide: Tables & routines - Understand the differences between table types and routines in BigQuery, and how to choose the right one for your needs.

Optimizing BigQuery Table Partitions - Example of migrating BigQuery table from one partition type to another.

3 ways to query BigQuery in Python - SQLAlchemy, Python Client for Google BigQuery, and bq command-line tool.

Build Robust Google BigQuery Pipelines with Python: Part II - BigQuery STRUCT in Python.

Creating a unified analytics platform for digital natives - Learn how to leverage Firebase, BigQuery and Looker to create a central platform for cross-functional analytics.

SAP Data Analytics in the Google Cloud - How to combine SAP with the Google Cloud Platform.

Choosing the right machine learning approach for your application - Choosing the right machine learning model for your application.

Google demonstrates leading performance in latest MLPerf Benchmarks - TPU v4 Pods will soon be available on Google Cloud, providing the most powerful publicly available computing platform for machine learning training.

Use Vertex Pipelines to build an AutoML classification end-to-end workflow - How you can use Vertex Pipelines to build an end-to-end ML workflow for training a custom model using AutoML.

Everything You Wanted to Know about Serving Language Models on GCP but Were Afraid to Ask - In this article, several fully-managed options provided by Google Cloud Platform are examined.

Various

A year of carbon-free energy at our data centers - Google's new data visualization shows every hour of carbon-free energy in 2020 for all its data centers.

My Journey to become Google Cloud Certified Cloud Digital Leader (Beta) - Topics and resources to study to pass the Cloud Digital Leader certification exam.

Book: The Definitive Guide to Conversational AI With Dialogflow & Google Cloud. for Building Complex Chatbots, Voicebots and Telephony Agents.

Book: Machine Learning with BigQuery ML: Create, execute, and improve machine learning models in BigQuery using standard SQL queries

Slides, Videos, Audio

GCP Podcast - #265 Sharkmob Games with Jan Harasym.

Releases

Anthos Config Management - 1.8.0. This release note contains information about 1.8.0 features that are now more widely available. Config Sync now supports accessing Cloud Source Repositories through a Google service account when Workload Identity is enabled in your cluster. Config Management is now available on GKE. The following commands have been promoted to beta: gcloud container hub config-management apply gcloud container hub config-management disable gcloud container hub config-management enable gcloud container hub config-management status gcloud container hub config-management unmanage gcloud container hub config-management upgrade gcloud container hub config-management version. The config file format for the gcloud apply command has changed. You can now configure your cluster with the same settings used by another cluster by using gcloud fetch-for-apply. Config Sync cluster selectors now support CustomResourceDefinitions.

Anthos clusters on AWS - An issue has been discovered with Anthos clusters on AWS 1.8.0. Anthos clusters on AWS aws-1.8.0-gke.7 is now available. You can now launch Kubernetes 1.20 clusters. Workload identity to authenticate to Google Cloud services from your user clusters is now available. You can now update the security groups associated with user clusters and node pools. You can now modify proxy settings on a running cluster. Anthos clusters on Azure now supports Cloud Logging and Cloud Monitoring of user cluster control planes.

Anthos clusters on bare metal - 1.8. Release 1.8.1 Anthos clusters on bare metal release 1.8.1 is now available. Fixes: Fixed CVE-2021-34824 that could expose private keys and certificates from Kubernetes secrets through the credentialName field when using Gateway or DestinationRule. 1.8. Security bulletin (1.8) The Istio project recently announced a security vulnerability (CVE-2021-34824) where credentials specified in the credentialName field for Gateway or DestinationRule can be accessed from different namespaces.

Anthos GKE on AWS - An issue has been discovered with Anthos clusters on AWS 1.8.0. Anthos clusters on AWS aws-1.8.0-gke.7 is now available. You can now launch Kubernetes 1.20 clusters. Workload identity to authenticate to Google Cloud services from your user clusters is now available. You can now update the security groups associated with user clusters and node pools. You can now modify proxy settings on a running cluster. Anthos clusters on Azure now supports Cloud Logging and Cloud Monitoring of user cluster control planes.

AppEngine Standard - Specifying a user-managed service account for each App Engine version during deployment is now available in preview. Requests from newly created or updated App Engine Cron jobs sent to the App Engine standard environment now come from 0.1.0.2.

Google Cloud Armor - Google Cloud Armor now supports parsing of the JSON content of POST bodies when preconfigured WAF rules are evaluated.

BI Engine - BigQuery BI Engine is now available in the Delhi (asia-south2) region.

BigQuery ML - BigQuery ML is now available in the Delhi (asia-south2) region.

BigQuery Transfer - BigQuery Data Transfer Service is now available in the Delhi (asia-south2) region. Audit logging, Cloud Logging, and Cloud Monitoring for the BigQuery Data Transfer Service are now generally available (GA).

BigQuery - BigQuery is now available in the Delhi (asia-south2) region. BigQuery now supports multi-statement transactions. BigQuery now supports access management data control language (DCL) statements and corresponding views: GRANT REVOKE INFORMATION_SCHEMA.OBJECT_PRIVILEGES view GRANT and REVOKE statements are generally available (GA). BigQuery now supports the following casting features: PARSE_BIGNUMERIC PARSE_NUMERIC Format clause for CAST available for the following data types: String type Date type Datetime type Time type Timestamp type Numeric types Bytes type Numeric type INT64 aliases (INT, SMALLINT, INTEGER, BIGINT, TINYINT, BYTEINT) ST_GEOGFROM These features are generally available (GA). BigQuery now supports the ALTER COLUMN SET OPTIONS data definition language (DDL) statement. Table functions are now available in Preview. The Google Trends dataset is now available in Preview and available in the Google Cloud Marketplace.

BigTable - Cloud Bigtable is now available in the asia-south2 (Delhi) region.

Billing - (Customers in India only) Starting on July 1, 2021, the first page of your invoice shows a Unified Payment Interface (UPI) QR code. (Customers in India only) We have updated information about Google's tax compliance in India, for tax deducted at source (TDS). Summary bar now available in the Cost Table report To provide additional flexibility when analyzing your data in the cost table report, we've added the summary bar as another analysis tool.

Certificate Authority Service - v1. Certificate Authority Service is now generally available with the following new features.

Chronicle - Asset Namespaces The asset namespaces feature enables you to classify categories of assets sharing a common network environment, or namespace, and then perform searches for those assets within the Chronicle user interface based on that namespace. Linux Forwarder Updates The Linux Forwarder has been enhanced with the following additional capabilities: Disk Buffering—Disk buffering enables you to buffer backlogged messages to disk as opposed to memory. Downloading Events You can download large numbers of the events associated with each threat detection as a CSV file, enabling you to search across a broad set of the data stored in your Chronicle account to hunt for security issues. Detection Engine API The VerifyRule method has been added to the Detection Engine API.

Cloud Composer - Cloud Composer 1.16.8 release started on June 30, 2021. New versions of Cloud Composer images: composer-1.17.0-preview.4-airflow-2.0.1 composer-1.16.8-airflow-1.10.15 (default) composer-1.16.8-airflow-1.10.14 composer-1.16.8-airflow-1.10.12. Fixed remaining memory issues that occurred while syncing files on machine types with more than 8 vCPUs. Cloud Composer 1.10.5 has reached its end of full support period.

Compute Engine - Preview: You can now configure N2D VMs with up to 100 Gbps of network bandwidth. The Machine types documentation has been renamed to Machine families. Preview: You can now autoscale both regional and zonal managed instance groups based on a Cloud Monitoring metric that provides an aggregated value for the group. Delhi, India asia-south2-a,b,c region has launched with E2, N2, N1, and C2 virtual machine (VM) instances in all three zones. Generally available: Compute Engine's VM instance details page has a new Observability tab, which replaces the Monitoring tab. General-purpose N2D VMs are now available in us-west4-b Las Vegas, NV.

Cloud Dataflow - GPU support on Dataflow is now in General Availability. Dataflow is now able to use workers, Dataflow Shuffle, Streaming Engine, FlexRS, and regional endpoints in zones in Delhi (asia-south2).

Dataproc - Dataproc is now available in the asia-south2 region (Delhi). The following previously released sub-minor versions of Dataproc images have been rolled back and can only be used when updating existing clusters that already use them: 1.3.91-debian10, 1.3.91-ubuntu18 1.4.62-debian10, 1.4.62-ubuntu18 1.5.37-centos8, 1.5.37-debian10, 1.5.37-ubuntu18 2.0.11-centos8, 2.0.11-debian10, and 2.0.11-ubuntu18. Added support for Dataproc Metastore in three new recently turned up regions: europe-west3, us-west1, and us-east1. Introduced a new ERROR_DUE_TO_UPDATE state, which indicates a cluster has encountered an irrecoverable error while scaling. Fixed an issue where a spurious unrecognized property warning was generated when the dataproc:jupyter.listen.all.interfaces cluster property is set.

Dialogflow - In Dialogflow CX, you can now use the Search feature (Preview launch) to search, filter, and access the core resources within an agent. In Dialogflow CX, you can now use the sys.long-utterance built-in event to handle user queries exceeding the maximum length (256 characters). The Dialogflow ES API now provides methods for managing versions and environments. Dialogflow CX now supports the asia-south1 (Mumbai) region.

Cloud Filestore - Filestore is available in the europe-central2 (Warsaw) region.

Cloud Functions - Cloud Functions now supports .NET at the General Availability release level. Cloud Functions is now available in the following region: asia-east1 (Taiwan) See Cloud Functions Locations for details.

Google Kubernetes Engine - The Istio project recently disclosed a new security vulnerability, CVE-2021-34824, affecting Istio. Config Management is now available on GKE. The asia-south2 region in Delhi is now available. In GKE node version 1.21.1-gke.2200 and later, Containerd is available as a runtime for Windows Server LTSC and SAC node images.

Cloud Logging - Cloud Logging lets you copy logs from a Cloud Logging bucket to a Cloud Storage bucket. The Ops Agent is now Generally Available as version 2.0.0.

Memorystore for Memcached - v1. Added new Memorystore for Memcached region: Delhi (asia-south2).

Cloud Memorystore - Added new Cloud Memorystore for Redis region: Delhi (asia-south2).

Anthos Migrate - On June 29, 2021, we released Migrate for Anthos and GKE 1.8. Enhanced runtime support added which lets you deploy containers to GKE Autopilot clusters and to Cloud Run, and simplifies the process of deploying containers to Anthos clusters on AWS that use workload identity. Added support for the preview release of the fit assessment tool that is intended to eventually replace the existing Linux discovery tool. 179976237: You can now create a Docker image file registry configuration with the name of a previously deleted configuration. 187922406: A migration might fail due to a LVM (Logical Volume Manager) failure. 166014117 : If you are using Migrate for Compute Engine with Migrate for Anthos and GKE to migrate Linux workloads, after you complete a successful migration, delete the migration to free up the source VM.

KF - 2.4.1. Kf Cloud Service Broker for Google Cloud for Google managed services. Prevent creating a GSA policy when the GoogleProjectId field is not set. Allow customization of external-gateway in kfsystem.yaml.

Cloud Monitoring - You can now display summaries of single-condition alerting policies on a custom dashboard. Dashboard-wide filters now apply to all charts on a dashboard. The Monitoring dashboards page in the Cloud Console now includes a collection of sample dashboards. The Ops Agent is now Generally Available as version 2.0.0. The Incidents page now provides an option to permanently close incidents if no data for that incident has arrived in the most recent alerting period. Cloud Monitoring is launching a new Observability tab on Compute Engine's VM instance details page.

Network Connectivity Center - It's no longer possible to add or remove router appliance instances to or from an existing spoke.

Cloud VPN - Cloud VPN is now available in region asia-south2 (Delhi, India).

Cloud PubSub - Pub/Sub message schemas are now GA.

Cloud Run - Cloud Run is now available in the following region: asia-south2 (Delhi, India).

Secret Manager - Secret Manager now offers a limited number of free resources as part of the Google Cloud Free program. Secret Manager now has a guide for rotating secrets and binding a secret version to your application. Secret Manager is now available in asia-south2 (Delhi). Secret Manager now has a guide for using Cloud Asset Inventory to identify and audit secrets.

Anthos Service Mesh - 1.10.x. Anthos Service Mesh user authentication is now generally available (GA). 1.10.2-asm.3 is now available and includes a fix for the known issue with control plane metric reporting reported on June 25, 2021. 1.10.x. There is a breaking change in 1.10 with inbound forwarding that affects applications that bind solely to the localhost interface.

SAP Solutions - SAP HANA certification: 12 TB m2-ultramem-416 machine type for OLAP workloads SAP has certified the Compute Engine 12 TB m2-ultramem-416 machine type for SAP HANA with OLAP workloads in an SAP HANA scale-up configuration that must be sized by using SAP workload-based sizing methods. SAP NetWeaver is supported on Bare Metal Solution with more database types In addition to SAP HANA, you can now run SAP NetWeaver on Bare Metal Solution servers for production workloads with other SAP NetWeaver supported database types, such as Oracle databases.

Cloud Spanner - Cloud Spanner now supports Cloud External Key Manager (Cloud EKM) when using customer-managed encryption keys. Cloud Spanner regional instances can now be created in Delhi (asia-south2).

Cloud Speech-to-Text - The Speech-to-Text now supports multi-region endpoints as a GA feature.

Cloud SQL MySQL - Cloud SQL for MySQL now offers stored procedures that you can execute on your instances. Cloud SQL for MySQL now supports the innodb_flush_log_at_trx_commit flag. Support for asia-south2 (Delhi) region.

Cloud SQL SQL Server - Support for asia-south2 (Delhi) region. Integration of SQL Server with Managed Service for Microsoft Active Directory is generally available.

Cloud Storage Transfer - Transfer service for on-premises data support for delete from source is now Generally Available. Storage Transfer Service offers Preview for Integration with AWS Security Token Service.

Cloud Storage - Public access preventionPreview launched. Delhi region (asia-south2) launched.

Cloud Translation - Cloud Translation - Advanced (v3) support for a regional EU endpoint is now in Preview.

VPC Service Controls - General availability for the following integration: Certificate Authority Service. Preview stage support for the following integration: Filestore. General availability for the following integration: Traffic Director.

Virtual Private Cloud - Deleting a private services access connection now also removes configurations created by the service producer, if Google is the service producer (for example, Cloud SQL). The billing issue for non-RFC 1918 addresses for Private Service Connect endpoints that you use to access Google APIs and services has been fixed. For auto mode VPC networks, added a new subnet 10.190.0.0/20 for the Delhi asia-south2 region.

Cloud VPN - Cloud VPN is now available in region asia-south2 (Delhi, India).

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko