Welcome to issue #257 August 30th, 2021

News

Save messages, money, and time with Pub/Sub topic retention - With new topic retention functionality, you can back up and replay messages that are published to Pub/Sub topic.

Six new features in Dialogflow CX - Discover 6 new features that make Dialogflow CX the best chatbot virtual agent for enterprises.

Cloud CISO Perspectives: August 2021 - Google Cloud CISO Phil Venables shares his thoughts on JCDC, Whitehouse Cybersecurity Summit, and other cloud security developments.

New features to better secure your Google App Engine apps - Announcing new features to further extend the security already provided by App Engine: Egress Controls and User-managed service accounts.

Shift security left with on-demand vulnerability scanning - Use on-demand vulnerability scanning to detect issues early and help prevent downstream problems.

Introducing Prediction Private Endpoints for fast and secure serving on Vertex AI - Learn the basics of VPC peering and how to use Private Endpoints on Vertex AI.

What's the key to a more secure Cloud Function? It's a secret! - The Google Secret Manager native integration with Cloud Functions makes it easier to access secrets for authenticating to upstream APIs and services.

Firebase SDK for Apple now fully supports Swift Package Manager - As of Firebase 8.6.0 for iOS, Firebase fully supports Swift Package Manager. This means you can now add Firebase to your iOS project without leaving Xcode.

The new Firebase JS SDK is now GA - Version 9 of the Firebase SDK is now generally available. This new version adopts a module first format that is optimized for elimination of unused code. The result is a potential significant reduction of Firebase library code in JavaScript bundles, up to 80% in some scenarios.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

One click deploy Triton Inference Server in Google Kubernetes Engine - Accelerate your GPU-enabled ML inference projects with one click using the Google Kubernetes Engine (GKE) on NVIDIA’s Triton Inference Server.

A unified and proven Zero Trust system with BeyondCorp and BeyondProd - How Google applies Zero Trust concepts to secure end-user access and running production systems at scale.

Best practices using Web Risk API to help stop phishing and more - Check out Web Risk API’s best practices to see how you can help use all of Web Risk API’s together to stop attacks targeting your end users.

Deploy Nomad and Consul Using Ansible on GCP

App Development, Serverless, Databases, DevOps

Replicating from Cloud Spanner to BigQuery at scale - Cloud Spanner is Google Cloud's native and distributed Online Transaction Processing System (OLTP). Due to its distributed nature, it can scale horizontally and therefore is suitable for workloads with very high throughputs containing large volumes of data. This invites a huge opportunity to do analytics on top of it.

All you need to know about Cloud Storage - Cloud Storage is a global, secure, and scalable object store for immutable data such as images, text, videos, and other file formats. You can add data to it or retrieve data from it as often as your application needs.

Your Google Cloud database options, explained - Picking the right database for your application is not easy. This post covers the different database options available within Google Cloud across relational (SQL) and non-relational (NoSQL) databases and explains which use cases are best suited for each database option.

How to reduce Firestore costs? - A simple solution for reducing the costs in Firestore through database refactoring.

Static Website Publishing to PubSub - Publishing messages to Pub/Sub via web page.

GCP Data Transfer Options - This article talks about various transfer option offerings by Google Cloud, there are various options given by Google Cloud to efficiently transfer the data.

Getting Started with Spring Cloud GCP: Cloud SQL - This article explains how to use Cloud SQL (MySQL) using Spring Cloud GCP.

Monitoring Diabetes with Google Cloud Platform - Monitoring blood glucose levels, generating graphs, and creating alerts using Google Cloud services.

Big Data, Analytics, ML&AI

Managing Scripts on AI Platform with GCP Cloud Source Repository - A tutorial to share the steps to manage and share scripts via GCP Cloud Source Repository.

Handling duplicate data in streaming pipelines using Dataflow and Pub/Sub - How to handle duplicate data in your streaming data pipeline using Pub/Sub and Dataflow.

Real-time analytics made easy with Cloud Spanner federated queries with BigQuery - Create real-time insights by bridging the gap between operational data and analytics.

BigQuery Admin reference guide: Monitoring - This blog aims to simplify monitoring and best practices related to BigQuery, with a focus on slots and automation.

Getting started on the BigQuery GA4 and Firebase datas - Working with Google Analytics 4 public dataset in BigQuery.

Best Practices when working with Google’s BigQuery - How to optimize Usage and Costs.

Working with Strings in BigQuery - What you have to know when working with String Functions.

Converging architectures: Bringing data lakes and data warehouses together - The convergence of data lake and data warehouse means that users can now access the data regardless of the infrastructure limitations, in where or how data is stored.

Optimize training performance with Reduction Server on Vertex AI - Learn how to configure Vertex Training jobs that utilize Reduction Server to optimize bandwidth and latency of distributed training for synchronous data parallel algorithms.

Anomaly detection with TensorFlow Probability and Vertex AI - Get a glimpse into the kinds of hard problems Google interns are working on, learn more about TensorFlow Probability’s Structural Time Series APIs, and learn how to run jobs on Vertex Pipelines.

Single-cell genomic analysis accelerated by NVIDIA on Google Cloud - Learn about single-cell genomic analysis on Google Cloud using NVIDIA and Dataproc.

kubeflow pipelines train recommender engine - Introduction to Kubeflow SDK by building a simple pipeline that learns the recommendation engine from BigQuery data.

Various

The top 5 launches of 2021 (so far) - Find out the top 5 Google Cloud launches from 2021 so far.

3 Steps to Pass Google Cloud Professional Data Engineer exam - Pass the exam within 1 month and improve your technical expertise in Google Cloud Platform.

Celebrating Women’s Equality Day with Google Cloud - In honor of Women’s Equality Day 2021 Google Cloud celebrates women in cloud and business technology (at Google and beyond).

How Renault solved scaling and cost challenges on its Industrial Data platform using BigQuery and Dataflow - Renault worked with Google Cloud’s professional services to design and build a new architecture for its data management platform.

Slides, Videos, Audio

GCP Podcast - #273 Working with Kubernetes and KRM with Megan O'Keefe.

Kubernetes Podcast - #160 KEDA, with Tom Kerkhove.

Building a Data Cloud to enable Analytics & AI-Driven Innovation - Learn how Google Cloud addresses the key challenges when building an Agile Data & AI platform.

Releases

AI Platform Unified - The following tools for creating embeddings to use with Vertex Matching Engine are available in Preview: the Two Tower built-in algorithm the Swivel pipeline template.

Anthos Config Management - 1.8.2. kube-rbac-proxy has been removed since Hierarchy Controller does not expose any sensitive metrics, and kube-rbac-proxy is no longer actively maintained. Fixed the issue causing a root or namespace reconciler to fail to be updated after switching from the auth type from none|gcenode|gcpserviceaccount to other types. Fixed the issue causing Config Sync not to report sync errors when it fails to access Git repos. Config Sync supports storing HTTPS/HTTP proxy credentials inside the git-creds Secret, using https_proxy or http_proxy as a key, to avoid exposing these credentials as plaintext. Fixed the issue causing Config Sync not to pick up the latest schema of the CustomResourceDefinitions synced from Git repos. Config Sync provides a way for users to override some system values: Use the spec.override.resources field of a RootSync or RepoSync object to override the resource limits for the reconciler container and the git-sync container. Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: 07e2fd0).

Anthos clusters on bare metal - 1.8. Release 1.8.3 Anthos clusters on bare metal 1.8.3 is now available. Breaking changes: In Anthos clusters on bare metal release 1.8.0, we added a kernel version requirement for Ubuntu 18.04. Features: Preview: Anthos Identity Service now works with Anthos clusters on bare metal to support LDAP authentication methods in addition to OIDC. Fixes: Fixed the following container image security vulnerabilities: CVE-2021-3520 CVE-2021-20305 CVE-2021-22924 CVE-2021-22925 CVE-2021-31535 CVE-2021-33560 CVE-2021-33910 Fixed cluster creation and cluster update failures for nodes running CentOS or Red Hat Enterprise Linux (RHEL) with both SELinux and Cloud Audit Logs enabled. Changes: The Kubevirt version used for working with VM-based workloads is now v0.43.0-gke.3.

Google Cloud Armor - Google Cloud Armor now has rate-based throttling and ban rules that enable you to limit requests from clients.

Cloud Asset Inventory - The Policy Analyzer page in the Cloud Console has been updated. New resource types are now available.

BigQuery - BigQuery Admin Resource Charts are now generally available (GA) for reservation users, enabling administrators to more easily monitor and troubleshoot their BigQuery environment. BigQuery Slot Estimator is now in Preview for reservation users.

Billing - The detailed usage cost data export to BigQuery is now generally available (GA). Proportional attribution for spend-based committed use discounts is now generally available (GA).

Cloud Composer - The default value for the [core]parallelism Airflow configuration option is set to match the maximum worker concurrency in Airflow. Non-error output of CloudSQL Proxy instances is routed to stdout instead of stderr. (Airflow 2.1.1) Airflow configuration options that are deprecated in Airflow 2 are now blocked. (Airflow 2) The airflow_db connection used by Airflow to connect to the Airflow database now correctly uses PostgreSQL connection type for Airflow 2 environments. New versions of Cloud Composer images: composer-1.16.15-airflow-1.10.12 composer-1.16.15-airflow-1.10.14 composer-1.16.15-airflow-1.10.15 (default) composer-1.17.0-preview.11-airflow-2.0.2 composer-1.17.0-preview.11-airflow-2.1.1. Cloud Composer 1.11.3 has reached its end of full support period.

Compute Engine - Generally available: You can now collect core dumps for uses such as debugging of unresponsive VMs.

Dataproc Metastore - v1beta1. Fixed the issue causing metadata changes introduced through imports and backups to not be reflected in Data Catalog due to broken batch sync.

Dataproc - New sub-minor versions of Dataproc images: 1.4.69-debian10, 1.4.69-ubuntu18, 1.5.44-centos8, 1.5.44-debian10, 1.5.44-ubuntu18, 2.0.18-centos8, 2.0.18-debian10, and 2.0.18-ubuntu18. Configured YARN ResourceManager to use port 8554 and Druid to use port 17071 for JMX Remote RMI port. Backported the following Trino (PrestoSQL) BigQuery connector patches in image 2.0: Make BigQuery views cache ttl configurable Move DestinationTableBuilder to BigQueryClient Fix wrong result due to column position mismatch in BigQuery Escape single quote in BigQuery string condition Fix information_schema query failures in BigQuery connector Add support for CREATE and DROP SCHEMA in BigQuery Fasten listing tables in BigQuery connector Implement case insensitive name matching for BigQuery Add BigQuerySqlExecutor and refactor tests to use it Add view_definition system table for BigQuery view Prefer ImmutableMap to Map Add test for aggregating BigQuery view more than once Upgrade Google Could library bom to 16.3.0 Fix TestBigQueryIntegrationSmokeTest.testShowCreateTable Enable views in BigQuery tests Fix incorrect result when aggregating count BigQuery view Remove dependency on presto-tests from Druid, BigQuery Refactor unused var in presto-bigquery Fix projection pushdown in BigQuery connector Add test for yearly partitioned table in BigQuery Upgrade BigQuery library to 11.0.0 Add BigQuery smoke test for HOUR-ly partitioned table Redact bigquery.credentials-key config Update BigQuery dependencies to support HOURLY partitioning of tables.

Dialogflow - During the week of September 6, 2021, two new Dialogflow IAM permissions will become effective: dialogflow.fulfillments.get and dialogflow.fulfillments.update.

Cloud Functions - Cloud Functions has added support for a new runtime, Node 16, at the Preview release level. Cloud Functions offers a native integration with Secret Manager, available at the Preview release level.

IAM - Managing Google Groups from the Cloud Console is now generally available.

Istio on GKE - 1.4.x. 1.4.10-gke.17 is now available. 1.6.x. 1.6.14-gke.5 is now available. 1.4.x & 1.6.x. The Istio project recently disclosed a series of CVEs that can expose Istio on GKE to remotely exploitable vulnerabilities.

Google Kubernetes Engine - Identity Service for GKE (Preview) is available. You can now enable Google Virtual NIC in a new GKE cluster on GPU nodes.

Load Balancing - Added Terraform examples to automate load balancer configuration: External HTTP(S) Load Balancing Internal HTTP(S) Load Balancing Internal TCP/UDP Load Balancing.

Network Connectivity Center - The Spoke Admin role now includes the following permissions: networkconnectivity.hubs.get networkconnectivity.hubs.list For full details about Network Connectivity Center permissions, see Access control.

Cloud Run - Deploying to Cloud Run from source code is now at General Availability (GA).

Secret Manager - Cloud Audit Logs and Platform Logs are now available directly in the Secret Manager UI.

Service Mesh - 1.10.x. 1.10.4-asm.6 is now available. The asmcli script is now available in preview. Google-managed data plane is now available in preview as a part of managed Anthos Service Mesh. Anthos Service Mesh for Compute Engine VMs now uses gcloud commands and supports Google-managed control planes. 1.7.x & 1.8.x & 1.9.x & 1.10.x. The Istio project recently disclosed a series of CVEs that can expose Anthos Service Mesh to remotely exploitable vulnerabilities. Anthos Service Mesh now supports skip-version upgrades for single-project clusters on GKE running versions 1.7 and higher. 1.9.x. 1.9.8-asm.1 is now available. 1.8.x. 1.8.6-asm.8 is now available. 1.7.x. 1.7.8-asm.10 is now available.

Cloud SQL Postgres - Cloud SQL for PostgreSQL now supports the following flags: huge_pages shared_buffers wal_buffers For more information about these flags, see the Cloud SQL for PostgreSQL flags documentation.

Cloud TPU - Cloud TPU team just released TF-2.3.4, TF-2.4.3 and TF-2.5.1 on Cloud TPUs.

Vertex AI - The following tools for creating embeddings to use with Vertex Matching Engine are available in Preview: the Two Tower built-in algorithm the Swivel pipeline template.

Virtual Private Cloud - Private Service Connect service attachment deletions are now logged in Cloud Logging. Using Private Service Connect with consumer HTTP(S) service controls to access supported regional service endpoints is now available in Preview. Converting a single-region legacy network to a VPC network is now available in Preview.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko