Welcome to issue #213 October 26th, 2020

News

Preparing for serverless big data open source software - Serverless capabilities at Google Cloud continue to develop, and serverless is now meeting open source as tools like Dataproc let you build on your open foundation in the cloud.

Improving security and governance in PostgreSQL with Cloud SQL - Managed cloud databases need security and governance, and Cloud SQL just added pgAudit and Cloud IAM integrations to make security easier.

Scale in at your own pace with Compute Engine autoscaler controls - With new scale-in controls for Compute Engine managed instance groups, you can control the rate at which VMs are turned down when load decreases.

Improving security, compliance, and governance with cloud-based DLP data discovery - Data discovery, a key component of DLP technology, has never been more important. Here’s why.

Strengthen zero trust access with the Google Cloud CA service - Certificate Authority Service, a highly scalable and available service that simplifies and automates the management and deployment of private CAs, is now available in public preview.

Lending DocAI fast tracks the home loan process - Lending DocAI fast tracks the home loan process for borrowers and lenders.

Announcing the Google Maps Platform On-demand Rides & Deliveries solution - On-demand Rides & Deliveries solution helps businesses improve operations as well as transform the driver and customer journey from booking to arrival or delivery–all with predictable pricing per completed trip.

AppSheet and Apps Script empower Google Workspace users to build solutions they need - AppSheet and Apps Script bring together a lot of powerful customization and functionality to Google Workspace, but you may be wondering when to use which platform. Here’s the answer.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

Preparing for peak holiday shopping in 2020: War rooms go virtual - Preparing for peak holiday season when your entire staff is working from home.

How I used Google Cloud Storage to host my own Terraform providers registry - Setting Terraform registry in Cloud Storage.

Quick Fix: Sharing Persistent Disks on Multiple Nodes in Kubernetes - How to use a single-write, multi-read persistent block storage(ie: Google Persistent Disk or Amazon Elastic Block Store Volume) to store data on multiple nodes in a Kubernetes cluster using NFS.

Monitoring Google Config Connector & Config Sync — Prometheus - Break down the details in setting up monitoring for Google Config Connector and Config Sync.

App Development, Serverless, Databases, DevOps

APIs 101: Everything you need to know about API design - API design best practices maximize value and efficiency.

Dynamic On-Demand Image Resizing Using Firebase Hosting and Google Cloud Functions to Make a Cheap Cloudinary Alternative - Resizing images using Cloud Functions and caching content with Firebase hosting.

Google API Gateway and Load Balancer + CDN - Using API Gateway with Load Balancer.

Making API Calls to a Salesforce server using a Static IP from a serverless environment in GCP. - This article illustrates how we set-up a GCP infrastructure that enabled our application deployed on App Engine to make API calls to a server that requested a Static IP for whitelisting.

Graceful shutdowns on Cloud Run: Deep dive - Learn how to perform various "graceful shutdown" tasks in your application code.

Hashicorp Waypoint — Cloud Run - Deploying Cloud Run application with Hashicorp's Waypoint.

3 Ways to Optimize Cloud Run Response Times - The article gives 3 suggestions to reduce cold starts for applications deployed on Cloud Run.

Automatically label Google Cloud Compute Engine instances and disks upon creation - Add automatically labels with a user who created Compute Engine resources.

Continuous Integration and Deployment using Google Cloudbuild - CI/CD Cloud Build pipeline to test and deploy App Engine application.

Big Data, Analytics, ML&AI

How To Create a Streaming Job On GCP Dataflow - A step by step guide with an example project.

Basic Streaming Data Enrichment on Google Cloud with Dataflow SQL - Learn the basics of Streaming and Batch Data Enrichment with Dataflow SQL.

Dataflow and Apache Beam, the Result of a Learning Process Since MapReduce - An overview of Apache Beam and Cloud Dataflow.

BigQuery — Almost all you need to know - All the important features of a serverless, modern multi-cloud data warehouse with best practices in one go!

Merge, clean, transform your CSVs easily with BigQuery - Using BigQuery to load, transform, and export CSV files.

BigQuery Detailed Dollar Costs Report - Extracting BigQuery query costs.

Column Level Security in BigQuery. - Exploring GCP Data Catalog Policy Tags.

Google Cloud Pub/Sub Ordered Delivery - This article discusses the details of how the Pub/Sub ordering works and talks about some common gotchas when trying to process messages in order in distributed systems.

Google Cloud Pub/Sub Reliability User Guide: Part 1 Publishing - The first in a series of posts that will help users of Google Cloud Pub/Sub to write reliable applications that use this service.

Google Cloud Pub/Sub Reliability Guide: Part 2 Subscribing - All important about subscribing in Cloud Pub/Sub.

Google Cloud Pub/Sub Reliability Guide: Part 3 Administrative Operations - Cloud Pub/Sub from the context of Administrative operations.

5 Machine Learning Models You Can Deploy Using BigQuery - Overview of machine learning models supported by BigQuery ML.

How AI uncovers important contract data - How AI uncovers important contract data.

A Beginner’s Guide to Painless ML on Google Cloud - Tips and tricks to get the most out of GCP AI tools.

Meeting the need for speed with a cloud data warehouse - See how e-commerce company Trendyol migrated their data warehouse to BigQuery to meet SLAs and get better performance.

MLB uses Google Cloud Smart Analytics platform to scale data insights - Learn how Google Cloud tools were applied to Statcast data to derive insights that enable MLB broadcasters and content generators to determine relevant storylines and add richer context to games.

Various

A giant list of Google Cloud resources - The growth of Google Cloud has been staggering. I decided to invest some time in building you a comprehensive list of resources.

Prepare for Google Cloud certification with one free month of new Professional Certificates on Coursera - Train for Google Cloud certifications with one free month of Professional Certificates on Coursera.

How I aced Google’s Associate Cloud Engineer Exam in two months - Topics to study for an Associate Cloud Engineer exam.

How I passed the GCP Professional Data Engineer Certification(In 1st attempt) - Preparing for exam and tips to pass Data Engineer exam.

Slides, Videos, Audio

GCP Podcast - #241 HPC with Senanu Aggor and Ilias Katsardis + Deloitte Cyber Analytics with Eric Dull.

Kubernetes Podcast - #126 Research, Steering and Honking, with Bob Killen.

Releases

Access Approval - Cloud IAM and BigQuery are have reached General Availability (GA) support in Access Approval API. History view is now available in the cloud console Access Approval page. The UpdateAccessApprovalSettings API now accepts product names (as listed here) in enrolledServices.cloudProduct, and the return values from GetAccessApprovalSettings and UpdateAccessApprovalSettings will contain product names as well.

Access Context Manager - Access levels now support checking the Storage encryption (allowedEncryptionStatuses), Require admin approval (requireAdminApproval) and Require corp owned device (requireCorpOwned) attributes of requests originating from mobile devices.

Anthos - Anthos 1.5.1 is now available.

BigQuery - The ST_GEOGFROMGEOJSON and ST_GEOGFROMTEXT geographic functions support a new make_valid parameter. Updated version of Magnitude Simba ODBC driver includes improvements to performance, logging, OpenSSL support, and bug fixes. BigQuery Audit Logs stopped using the following checks for redacting resource names for cross-project access and caller identities: The bigquery.jobs.create permission check and the internal setting for a project domain.

Compute Engine - Memory-optimized M1 machine types are available in Frankfurt europe-west3-a,b,c.

Data Fusion - In Cloud Data Fusion versions before 6.2, there is a known issue where pipelines get stuck during execution.

Dataproc - Decreased the minimum allowed value of Dataproc Scheduled Deletion LifecycleConfig.idleDeleteTtl (Dataproc API) and --max-idle flag (gcloud command-line tool) from 10 minutes to 5 minutes. New sub-minor versions of Dataproc images: 1.3.74-debian10, 1.3.74-ubuntu18, 1.4.45-debian10, 1.4.45-ubuntu18, 1.5.20-debian10, 1.5.20-ubuntu18, 2.0.0-RC16-debian10, and 2.0.0-RC16-ubuntu18. 2.0 preview image versions: Upgraded Apache Arrow version used by Spark to Apache Arrow 1.0.1. Sole-tenant node cluster create or update requests to use preemptible secondary workers or attach autoscaling policies that create preemptible secondary workers are now correctly rejected. All image versions: Fixed a bug where files uploaded to Cloud Storage through the JupyterLab UI were incorrectly base64 encoded. 1.4 and 1.5 image versions: SPARK-32708: Fixed SparkSQL query optimization failure to reuse exchange with DataSourceV2. Announcing the Alpha release of the Dataproc Persistent History Server, which provides a UI to view job history for jobs run on active and deleted Dataproc clusters. Announcing the GA (General Availability) release of the Dataproc Ranger Optional Component and the Dataproc Solr Optional Component.

Cloud Healthcare API - v1alpha2. It is now possible to use `` to escape special characters in FHIR resources.

Istio on GKE - 1.4.x. A fix for a known issue where custom resources created in the istio-system namespace were deleted when upgrading from GKE 1.16 to 1.17 and 1.18 is available in R33.

Google Kubernetes Engine - A fix for the issue reported on September 16, 2020 where custom resources in the istio-system namespace were deleted when upgrading from GKE 1.16 to 1.17 and 1.18 is now available.

Load Balancing - For HTTP requests, the httpRequest.remoteIp and httpRequest.serverIp fields can include port information.

Cloud Logging - Cloud Logging now calculates logs-based metrics from both ingested and excluded logs. Logs Views are now available in Preview. Recent queries is now generally available (GA). In the Logs Explorer you can now download your logs in JSON and CSV to your computer, Google Drive, or view them in a new tab.

Anthos Migrate - On October 21, 2020 we released Migrate for Anthos 1.5. Support for migrating Windows VM workloads has moved from the Beta stage to general availability. Migrate for Anthos offers new tools that you run on a Linux or Windows VMs to determine the workload's fit for migration to a container. Custom Services Blocklist is a new feature that lets you modify the default set of services to disable in a migrated container. The image field value of the GenerateArtifactsFlow CRD defines the names and locations of two images created from a migrated VM. When you deploy your migrated Windows containers to a cluster, you can now use a Group Managed Service Account (gMSA) to execute the container under a specific service account identity. Changed the default settings on the Cloud processing cluster for migrating Linux workloads: You no longer have to specify the --scopes "cloud-platform" option when creating Cloud processing clusters for migrating Linux workloads. 171123825: In some cases, migration process might fail, and Cloud Logging indicate errors such as: "failed to load map, error 6" or: "failed in domap for addition of new path sdd" Workaround: Delete the migration and restart it. 170706786: The Linux Discovery Tool might return exit code 0 even when not all information was collected successfully. 170627229: Migrated workload of a JBoss application might fail at startup. 167656057: Installation on a GKE cluster with ACM might fail. 157062328: In some cases, adding a service to the blocklist using a configmap will not actually stop that service from running on the deployed workload. 163800225: kubectl port-forward might not work properly for a deployed workload. 171173082: Mistakenly creating a local VMware source on a Cloud-based cluster, normally used only in an on-prem migration, results in the source being in PROCESSING state forever. 170604382: Running migctl when not connected to a cluster results in a panic error such as the one below, followed by a stack-trace: migctl setup install panic: Cannot create kubernetes client Workaround: Connect a cluster, and re-run migctl.

Cloud PubSub - Pub/Sub message ordering is now available in GA.

Cloud Run - Eventarc is now available in public preview. Cloud Run is now available in the following regions: asia-east2 (Hong Kong) asia-northeast3 (Seoul, South Korea) asia-southeast2 (Jakarta) asia-south1 (Mumbai, India) europe-west2 (London, UK) europe-west3 (Frankfurt, Germany) europe-west6 (Zurich, Switzerland) southamerica-east1 (Sao Paulo, Brazil). You can now purchase a custom domain via Cloud Domains using the Cloud Run user interface.

Cloud Spanner - A new multi-region instance configuration is now available in North America - nam7 (Iowa/North Virginia/Oklahoma).

Anthos GKE deployed on-prem - Anthos GKE on-prem 1.5.1-gke.8 is now available. Binary Authorization for GKE on-prem Preview is now available: Binary Authorization for GKE on-prem extends centralized Binary Authorization enforcement policies to GKE on-prem user clusters. This release enables customers to generate credential configuration templates by using the gkectl create-config credential command. Published the best practices for how to set up GKE on-prem components for high availability and how to recover from disasters. Published the best practices for creating, configuring, and operating GKE on-prem clusters at large scale. Fixes: Fixed cluster creation issue when Cloud Run is enabled.

Anthos GKE on-prem - Anthos GKE on-prem 1.5.1-gke.8 is now available. Binary Authorization for GKE on-prem Preview is now available: Binary Authorization for GKE on-prem extends centralized Binary Authorization enforcement policies to GKE on-prem user clusters. This release enables customers to generate credential configuration templates by using the gkectl create-config credential command. Published the best practices for how to set up GKE on-prem components for high availability and how to recover from disasters. Published the best practices for creating, configuring, and operating GKE on-prem clusters at large scale. Fixes: Fixed cluster creation issue when Cloud Run is enabled.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko