Welcome to issue #237 April 12th, 2021

News

Introducing request priorities for Cloud Spanner APIs - You can now specify request priorities for some Cloud Spanner APIs. By assigning a HIGH, MEDIUM, or LOW priority to a specific request, you can now convey the relative importance of workloads, to better align resource usage with performance objectives.

New flexibility: Run your Dataprep jobs with BigQuery or Dataflow - Dataprep can now run using either BigQuery or Dataflow.

Google Cloud and AVEVA’s OSIsoft serve the industrial sector a new flavor of PI - Check out how OSIsoft’s fully functioning PI system on Google Cloud can help industrial companies modernize their data and start getting insights quickly.

Google Cloud EMEA Retail & Consumer Goods Summit: The Future of Retail - April 22, 2021 - Google Cloud Retail & Consumer Goods Summit and learn how combining technology and business insights can solve retail challenges.

Google Cloud joins the FinOps Foundation - The FinOps Foundation is a relatively new open-source foundation, hosted by the Linux Foundation, that launched last year. It aims to bring together companies in the “cloud financial management” space to establish best practices and standards.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Making access to SaaS applications more secure with BeyondCorp Enterprise - Transitioning to a zero-trust security strategy can be difficult; Google has released BeyondCorp Enterprise to make it easier.

Building global momentum with government and security compliance certifications - Operating virtually has heightened the importance of security and compliance for public sector agencies around the world.

IDC: A multicloud strategy can mitigate regulatory, business risks - A new IDC whitepaper finds that a multicloud strategy can help regulated industries manage the risk of using cloud.

How to deploy Certificate Authority Service - Google CAS can help manage certifications at scale. Here’s how to put it to work.

Keyless API authentication—Better cloud security through workload identity federation, no service account keys necessary - With workload Identity federation, you can securely operate your workloads and no longer have to worry about managing service account keys.

Let’s Talk: Containers and Kubernetes In Google Cloud Platform — Part 2 - Explanation of basic Kubernetes concepts.

Let’s Talk: Containers and Kubernetes In Google Cloud Platform — Part 3 - Explanation of basic Kubernetes concepts.

Automation: Deploying an app in GKE using Ansible - Ansible infrastructure-as-code to automate Nginx deployment in Google Kubernetes Cluster.

Create a private GCP Kubernetes cluster using Terraform - Tutorial to create a private GKE cluster.

First look at GKE Autopilot - A hands-on review of GKE Autopilot.

Introduction to service accounts on Google Cloud Platform - A short introduction to what service accounts are and how they should be used.

Have budget notifications come to your favorite comms channels - TL;DR - Rather than wait for budget alert emails, you can use programmatic budget notifications to send budget updates to Slack (as well as anything else you can code).

Custom Roles in IAM Google Cloud - A brief overview of IAM Custom Roles.

Brief synopsis of Google IAP (Identity-Aware Proxy) - A brief overview of Identity Aware Proxy concepts.

App Development, Serverless, Databases, DevOps

Lighter lift-and-shifts with the new Database Migration Service - Learn about the latest Database Migration Service GA launch and how to run through a full database migration.

Track changes in SQL Server on Google Cloud using Change Data Capture - This blog post will walk through the steps required to enable (CDC) Change Data Capture on SQL Serve along with an example query to view captured changes.

Creating a SQL Server instance integrated with Active Directory using Google Cloud SQL - SQL Server instances in Google Cloud SQL now integrate with Microsoft Active Directory (AD) as a pre-GA feature that you can try out for yourself right now. This post describes the basic steps required to create a SQL Server instance with this new functionality.

Supercharge your mobile games business with Firebase - Firebase tool and insights that help game developers.

Building a Bot on GCP Cloud in 10 minutes - A comprehensive step-by-step guide to build and deploy FAQ bot.

Using Object Versioning for Google Cloud Storage! - Explanation of object versioning in Cloud Storage.

Firestore: The New Database - An overview of Cloud Firestore with code samples in Java.

Capturing the Elusive World of Sound: Fetch and Transcribe Google Podcasts with Python - Transcribing in bulk podcast audio files.

Big Data, Analytics, ML&AI

Creating a serverless pipeline for real-time market data - See how to create a real-time data analytics pipeline for use with market data, using serverless technology for ingestion and storage.

Easy postponed / scheduled messages through Google Pub/Sub - Event-scheduler is a high available strongly consistent high-performance application designed to receive messages from the source queue and release them to the target queue at the requested time, specified in the message attribute.

How to migrate your on-premise pyspark jobs to GCP using Dataproc Workflow Templates using Dataproc Workflow Templates with Production-Grade Best Practices Standards - Complete pattern example of how to migrate (or create from scratch) pyspark jobs to GCP with Dataproc Workflow Templates.

Export Datastore to BigQuery using Google Dataflow - How to employ Google Dataflow to export Datastore to BigQuery with additional filtering of entities.

Exploring hail reports with BigQuery and Data Studio - Using the new geospatial capabilities in Data Studio.

How to build efficient and perfomant Data Structures in BigQuery - Ways of using Denormalization and Nested Data.

Fintech startup, Branch makes data analytics easy with BigQuery - FinTech startup, Branch, built its data platform with BigQuery making data accessible to its analysts and scientists.

Super-fast Machine Learning to Production with BigQuery ML - How to use Bigquery ML to deploy your models in no time, and focus on what really matters.

Cook up your own ML recipes with AI Platform - Learn about Mars Wrigley’s new ML-inspired recipe experiment on Google Cloud and how you can get started with your own.

A recipe for building a conversational AI team - Learn how the right team and Google Contact Center AI can help you build better bots with a user-centered perspective.

How to use PyTorch Lightning's built-in TPU support - How to start training ML models with Pytorch Lightning on TPUs.

Recovering global wildlife populations using ML - Google partnered with several leading conservation organizations to build a project known as Wildlife Insights. This blog will teach you how to build a basic image classifier inspired from this wildlife project!

MLops: Kubeflow with TensorFlow TFX pipelines seamlessly and at scale - Running Machine Learning pipelines with Kubeflow and TensorFlow Extended.

Machine Learning GDEs - Highlights of the ML GDE’s hard work during the Q1 2021 which contributed to the global ML ecosystem.

Various

AI in Retail: Google Cloud transforms Cartier's product search technology - With Google Cloud, Cartier developed an application to identify any watch ever designed in its 174-year history using visual recognition technology.

OpenX is serving over 150 billion ad requests per day with cloud databases - See how OpenX, an advertising technology company, migrated its on-premises open source databases to Bigtable and Memorystore for speed and global scale.

How we’re working with governments on climate goals - Google’s Sustainability Officer Kate Brandt shares how technology companies and governments can work together to address climate change.

Slides, Videos, Audio

GCP Podcast - #254 Workflows with Kris Braun and Guillaume Laforge.

Kubernetes Podcast - #145 Weaveworks (part 2), with Alexis Richardson.

This is my Architecture to prevent Cloud Bill Shock

Releases

Anthos Config Management - 1.7.0. Anthos Config Management images are no longer included in Anthos on VMWare clusters. The ability to sync from multiple Git repositories is now a generally-available feature. A memory leak in the Anthos Config Management Operator Pod that led to high memory utilization or Pod restarts due to out-of-memory errors has been corrected. Preview versions of multi-repo occasionally used excessive CPU usage and sent unnecessary queries to the apiserver master node, resulting in an unhealthy cluster. Config Sync configured with sourceFormat: unstructured will have errors during syncing if the Git repository includes a "Repo" resource. Config Sync configured with sourceFormat: unstructured will have errors during syncing if the Git repository specifies a ClusterSelector with an invalid metadata.name field. Customers using Anthos Policy Controller who have upgraded since Anthos Config Management 1.5.1 need to update the timeoutSeconds in their ValidatingWebhookConfigurations from "5" to "3" to avoid issues with Kubernetes leader elections.

BigQuery - BigQuery now has better support for loading ENUM and LIST types in Parquet files. Beginning in early Q3 2021, BigQuery Storage Read API will start charging for network egress. The BigQuery Storage Write API is now in Preview.

BigTable - Cloud Bigtable support for customer-managed encryption keys (CMEK) is now generally available. Data Access audit logging for Cloud Bigtable is now generally available.

CDN - Serve stale, bypassing cache, and negative caching are now Generally Available. Cloud CDN now supports configuring negative caching for HTTP 302 (Found) and HTTP 307 (Temporary Redirect) status codes.

Compute Engine - Generally available: Predictive autoscaling for managed instance groups lets you improve the availability of your workloads by using Machine Learning to predict future demand and create virtual machines ahead of forecasted load. N2D machines are now available in the following regions and zones: us-central1-b - Iowa asia-northeast1-a,b - Tokyo See VM instance pricing for details. Generally available: You can now use instance schedules from the Google Cloud Console.

Config Connector - Config Connector version 1.45.0 is now available. Added support for OSConfigGuestPolicy, IdentityPlatformTenant, IdentityPlatformOAuthIDPConfig and IdentityPlatformTenantOauthIDPConfig. Added proxyBind field to ComputeTargetHTTPProxy, ComputeTargeHTTPSProxy, and ComputeTargetTCPProxy. Added enableStreamingEngine field to DataflowJob. Fixed issue where folderRef/organizationRef could not be defaulted from folder-id/organization-id annotations when creating Project/Folder resources with server-side apply. Supported a viewer cluster role so that resources can be referenced cross namespaces in namespaced mode. Updated the structs' name of any field FooBar to be KindFooBar in Go Client resources. Fixed the ListMeta type in Go Client (Issue #422).

Dialogflow - The following languages are now supported by Dialogflow CX: Arabic Bengali Filipino Malay Marathi Romanian Sinhala Tamil Telugu Vietnamese.

IAM - Workload identity federation is now generally available. You can now get recommendations for folder- and organization-level role bindings using the gcloud command-line tool and REST API.

Google Kubernetes Engine - (2021-R11) Version updates GKE cluster versions have been updated. Versions no longer available The following versions are no longer available for new clusters or upgrades: Versions 1.15 and earlier.

Google Kubernetes Engine Rapid - (2021-R11) Version updates Version 1.19.8-gke.2000 is now the default version.

Google Kubernetes Engine Regular - (2021-R11) Version updates Version 1.18.16-gke.502 is now the default version.

Google Kubernetes Engine Stable - (2021-R11) Version updates The following versions are now available in the Stable channel: 1.17.17-gke.3000 1.18.16-gke.302.

Cloud Life Sciences - Cloud Life Sciences has preview support for integrating with VPC Service Controls.

Cloud Logging - Cloud Logging now supports 22 regions in which you can create a log bucket so that you can meet compliance and audit requirements when storing your logs.

Anthos Migrate - On April 6, 2021, we released Migrate for Anthos 1.7.0. This release updates the Linux discovery tool to replace the CSV output with an HTML and JSON output. By default, Migrate for Anthos automatically disables unneeded services on a VM when you migrate it to a container. 175000470: The issue caused by adding a source when using a service account without the compute.disks.create permission has been fixed. 178469863: Running migctl setup install with either the --node-selector or --tolerations flag no longer returns an error. 183321483: If you are using a CRD file to create a migration source, and you include a secret in the CRD, then deleting the migration source might also delete the secret. 162275866: When generating migration artifacts, you might see the following error: Error: failed to update vgenerateartifactsflow.kb.io: Post https://controllers-webhook-service.v2k-system.svc:443/validate-anthos-migrate-cloud-google-com-v1beta2-generateartifactsflow?timeout=30s: no endpoints available for service "controllers-webhook-service" Workaround: Try generating artifacts again. 179028669: migctl doctor crashes if a GKE cluster is currently repairing. 183564181: When you call migctl setup upgrade on Anthos clusters on VMware or Anthos clusters on AWS without the necessary platform arguemnt (--gkeop or --gke-on-aws), migctl uninstalls Migrate for Anthos and fails on the install. 179860971: The status error returned when you enter an incorrect VM ID in the console is not helpful: googleapi: got HTTP response code 404 with body: Workaround: View the error in the Events tab in the console for more information: Click the Migrations tab to display a table containing the available migrations. 182208300: When building a Docker image for a Windows container, examine the logs. 156207267: When using Anthos clusters on VMware for your processing cluster, the migration might get stuck at the image extraction step, and the relevant migration pod shows volume attachment errors similar to the following: Warning FailedAttachVolume 5s (x10 over 4m15s) attachdetach-controller AttachVolume.Attach failed for volume migration-b849e1-dd: Invalid configuration for device 0. 182450827: If you installed Migrate for Compute Engine on your processing cluster to work with Migrate for Anthos, and want to perform four or more concurrent migrations, you cannot use GKE 1.18. 179171930: This release of Migrate for Anthos does not support GKE 1.20. 183082390: The collection script used by the Linux discovery tool uses service --status-all to query system V services.

Cloud Monitoring - Cloud Monitoring has changed the default behavior for when notifications are sent.

Security Command Center - Security Command Center Legacy, previously known as Cloud Security Command Center, and Event Threat Detection Legacy are being permanently disabled for all customers on June 7, 2021.

Cloud SQL Postgres - Cloud SQL for PostgreSQL now lets you use IAM database authentication with the Cloud SQL Auth proxy.

Cloud SQL SQL Server - Cloud SQL for SQL Server enables you to perform change data capture (CDC) operations for your Cloud SQL instances. Cloud SQL for SQL Server enables you to perform common operations on a tempdb database.

Cloud Text-to-Speech - Text-to-Speech now offers voices in the following new languages. Text-to-Speech now supports MULAW and ALAW audio encodings.

Network Intelligence Center - Connectivity Tests now evaluates hierarchical firewall policy rules as part of its configuration analysis.

Dialogflow Enterprise - The following languages are now supported by Dialogflow CX: Arabic Bengali Filipino Malay Marathi Romanian Sinhala Tamil Telugu Vietnamese.

Document AI - v1. Procurement DocAI General availability (GA) release Procurement DocAI (PDAI) solution is now available in private General Availability (GA). Human in the Loop (HITL) support for Procurement DocAI processors Procurement DocAI processors now support Human in the Loop (HITL) AI platform functionality supporting human revisions of predictions. Invoice parser behavior update The invoice parser behavior has been updated to include the following features: Offers extended support for the following languages (in addition to English): French Dutch German Spanish Improves supplier parsing accuracy with Knowledge Graph support. Expense parser (Receipt parser) behavior update The expense parser behavior has been updated to include the following features: Renamed Receipt parser to Expense parser. Human in the Loop (HITL) AI General Availability (GA) released HITL AI is now available in Private General Availability (GA) for human review of Invoice, Expense, and Utility parser predictions.

Anthos GKE on AWS - Anthos clusters on AWS 1.7.0-gke.12 is now available. This release fixes an issue mentioned in the entry on April 2, 2021.

VMware Engine - Added global quota limits for VMware Engine nodes so users have more flexibility in distributing resources across regions. Updated the display name of VMware Engine quota entries to reflect the resource type and assignment level.

Anthos clusters on AWS - Anthos clusters on AWS 1.7.0-gke.12 is now available. This release fixes an issue mentioned in the entry on April 2, 2021.

GKE - (2021-R11) Version updates The following versions are now available: 1.16.15-gke.14800 1.17.17-gke.4900 1.18.17-gke.100 Control planes and nodes with auto-upgrade enabled will be upgraded from version 1.18 to version 1.18.16-gke.502 with this release.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko