Welcome to issue #220 December 14th, 2020

Probably some had not so smooth start of the week due to Google's outage (or something else). Hopefully, you will relax a bit more by reading the latest news and articles regarding GCP.

News

Dataform is joining Google Cloud: Deploy data transformations with SQL in BigQuery - With our of Dataform, you can now leverage software development best practices to define, document, test and deploy data transformations using SQL executed within BigQuery.

Dataproc Hub makes notebooks easier to use for machine learning - Dataproc Hub, now generally available, makes it easy to use open source, notebook-based machine learning on Google Cloud, powered by Spark.

Faster machine learning on Dataproc with new initialization action - Setting up environments for Machine Learning is tricky. This blog post explains how to do so with a new initialization action.

Improve the data science experience using scalable Python data processing - Announcement of the Dask support for Dataproc, Google Cloud’s fully managed Apache Hadoop and Apache Spark service, via a new initialization action.

Cloud Run min instances: Minimize your serverless cold starts - With Cloud Run’s new min instances feature, you can ensure your application never scales entirely to zero if you don’t want it to.

Google Cloud named a leader in latest Forrester Research IaaS Platform Native Security Wave - Forrester evaluated seven public cloud platforms against 29 criteria. Google Cloud scored highest overall on current offering, and received the highest score possible in its strategic plans for security posture management, hypervisor security, guest OS and container protection, and network security.

Google Cloud initiatives offer researchers critical support during the pandemic - New initiatives offer crucial support to overburdened researchers in these difficult times.

Architecture Center - Discover reference architectures, guidance, and best practices for building or migrating your workloads on Google Cloud.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

What is zero trust identity security? - GCP Comis explains what is zero trust indentity security.

Enabling Microsoft-based workloads with file storage options on Google Cloud - Check out the range of SMB file services in Google Cloud that meet the varying needs of Microsoft shops, from self-managed to fully managed.

App Development, Serverless, Databases, DevOps

Preparing your MySQL database for migration with Database Migration Service - Database Migration Service is an easy-to-use, serverless migration tool that provides minimal downtime database migration to Cloud SQL for MySQL (Preview) and Cloud SQL for PostgreSQL (available in Preview by request).

How to develop secure and scalable serverless APIs - Building your RESTful APIs on top of a serverless platform like Cloud Run or Cloud Functions can improve performance and reduce costs.

Deploying your Spring Boot application in Cloud Run - Discover how to easily deploy a Spring Boot application on Cloud Run.

Continuous Deployment pipeline with Cloud Build on Cloud Run - Building CI pipeline for Cloud Run.

Workflow for Serverless — Cloud Functions + Workflow [Basic] - GCP Workflow serverless orchestration of Cloud functions.

Day #6 with Cloud Workflows: arrays and dictionaries - Continuation of series of articles on Cloud Workflows.

Run shell commands and orchestrate Compute Engine VMs with Cloud Workflows - Automate the execution of shell commands in a fully serverless and secure way without managing private keys.

Integrating Dialogflow with Google Chat - A walk through a bot that was inspired by a real use case here at Google.

Overview of Google Cloud Function Identities - Properly scope your Cloud Functions to limit security risks on Google Cloud Platform.

Big Data, Analytics, ML&AI

Pub/Sub makes scalable real-time analytics more accessible than ever - Here’s how the move from daily batch processing to real-time processing for data analytics gets easier with Pub/Sub to scale fast.

StarThinker On Airflow / Composer - StarThinker is a Google gTech built python framework for creating and sharing re-usable workflow components.

Things I wish I knew about Google Cloud Pub/Sub: Part 2 - This is a continuation of our three part series in providing useful tidbits in using Pub/Sub effectively.

BigTips: INFORMATION_SCHEMA Views in BigQuery, Part 2, with extra Scripts and Procedures! - Making the INFORMATION_SCHEMA a little easier to use!

Building SQL pipelines in BigQuery with Dataform (Part 1) - Collaboratively transform, document, schedule datasets using SQL.

BigTips: Random Numbers and Random Dates - Generating random numbers in a range, and random dates in BigQuery.

Time series analytics with BigQuery part 2 - The second in a series of posts on implementing time series analytics in BigQuery, this time defining sliding windows and session windows.

5 Bigquery SQL performance tips for modern data scientists - SQL tuning tips and advice to help reduce BigQuery costs. Start 2021 off on the right foot!

Using Document AI to automate procurement workflows - Shine a light on all your "dark" data with Google's Document AI. Turn unstructured pdfs into fully automated workflows with machine learning powered parsers.

Getting Started with Distributed TensorFlow on GCP - In this tutorial-style article, you’ll get hands-on experience with GCP data science tools and train a TensorFlow model across multiple GPUs. You’ll also learn key terminology in the field of distributed training, such as data parallelism, synchronous training, and AllReduce.

Various

We Burnt $72K testing Firebase + Cloud Run and almost went Bankrupt [Part 1] - Story of what led to an unexpected bill.

We Burnt $72K testing Firebase - Cloud Run and almost went Bankrupt [Part 2] - Story of what led to an unexpected bill.

MakerBot implements an innovative autoscaling solution with Cloud SQL - See how 3D printing company MakerBot migrated its MySQL instances to Google Cloud.

Google Cloud fuels new discoveries in astronomy - Google Cloud contributes to high-performance computing and machine learning research in the science of astronomy.

Keeping students, universities and employers connected with Cloud SQL - Startup company Handshake uses Google’s managed database service, Cloud SQL, to run their web apps with high performance.

Ensuring financial stability starts with database stability - From the Google Cloud blog, how one financial company migrated 180 MySQL database instances from Rackspace to Cloud SQL to save time and money.

Samsung Electronics supercharges Bixby with Cloud TPUs & TensorFlow - Samsung improves Bixby voice recognition model training speeds 18x with Cloud TPUs.

5 Best GCP Associate Cloud Engineer Certification Courses in 2021 - These are the best online courses to crack the Google Cloud Platform Associate Cloud Engineer Certification in 2021.

Why is the Google Cloud UI so slow? - Analysis of Cloud Console from web app point of view.

ML Engineer certification - Preparing and taking exam for ML Engineer Certification exam.

Slides, Videos, Audio

Kubernetes Podcast - #131 Kubernetes 1.20, with Jeremy Rickard.

Pluralsight Course - Cloud Migrations: Executive Briefing - This course will teach you the why and what of cloud migration, and give the I.T. leader a strong foundation of principles and strategies for success.

Releases

Anthos Config Management - 1.6.0. Anthos Policy Controller now includes additional policies covering many of the CIS Kubernetes Benchmark 1.5.1 controls. Anthos Policy Controller has been updated to include a more recent build of OPA Gatekeeper (hash: acd32ec7). Support for Git submodules has been fixed in this version.

Anthos - Anthos 1.6.0 is now available.

AppEngine Standard Java - Updated Java SDK to version 1.9.84.

BigQuery - Updated version of Magnitude Simba ODBC driver includes bug fixes and enhancements such as support for BigNumeric data and improved driver logic. Updated version of Magnitude Simba JDBC driver includes bug and security fixes and enhancements such as support for Java 11, SSL trust store, BigNumeric data, and version-agnostic fully-qualified class names.

Cloud Composer - Support for VPC Security Controls is now generally available (GA).

Compute Engine - Preview: You can configure how your regional managed instance group distributes instances across zones by using capacity-aware distribution shapes, which can automatically deploy instances to zones where capacity is available and optionally prioritize the use of reservations. You can migrate a VM instance from one network to another. Preview: Schedule-based autoscaling for managed instance groups lets you improve the availability of your workloads by scheduling capacity ahead of anticipated load. GA: You can now access OS inventory data from Cloud Asset Inventory. GA: Per-group metrics let you autoscale a zonal managed instance group based on any Cloud Monitoring metric—for example, a Pub/Sub queue size or custom metrics from your application.

Config Connector - Config Connector version 1.33.0 is now available. Added support for the ComputeProjectMetadata resource. Added resourceID field to ServiceUsageService and StorageNotification. Added computeResponseHeaders field to ComputeBackendService. Added maintenancePolicy.maintenanceExclusion field to ContainerCluster. Added description and disabled fields to LoggingLogSink. DataflowJobs can now be acquired via name. Added IAM support to BigtableTable.

Dataproc - Restartable jobs: Added the ability for users to specify the maximum number of total failures when a job is submitted. Image 2.0 preview Using the n1-standard-1 machine type is no longer supported. The Dataproc Metastore Service is now available in the us-east4, europe-west2, asia-northeast1, and australia-southeast1 regions in addition to the existing us-central1 region. New sub-minor versions of Dataproc images: 1.3.78-debian10, 1.3.78-ubuntu18, 1.4.49-debian10, 1.4.49-ubuntu18, 1.5.24-debian10, 1.5.24-ubuntu18, 2.0.0-RC20-debian10, and 2.0.0-RC20-ubuntu18. Image 1.5: Upgraded Hadoop to version 2.10.1.

Cloud Data Loss Prevention - Added whole document classification support with the following infoType detectors: DOCUMENT_TYPE/FINANCE/REGULATORY DOCUMENT_TYPE/FINANCE/SEC_FILING DOCUMENT_TYPE/HR/RESUME DOCUMENT_TYPE/LEGAL/BLANK_FORM DOCUMENT_TYPE/LEGAL/BRIEF DOCUMENT_TYPE/LEGAL/COURT_ORDER DOCUMENT_TYPE/LEGAL/LAW DOCUMENT_TYPE/LEGAL/PLEADING DOCUMENT_TYPE/R&D/PATENT DOCUMENT_TYPE/R&D/SOURCE_CODE DOCUMENT_TYPE/R&D/SYSTEM_LOG DOCUMENT_TYPE/R&D/DATABASE_BACKUP.

Google Kubernetes Engine - With the release of GKE node version 1.19, the Container-Optimized OS with Docker (cos) variant is deprecated. GKE cluster versions have been updated. The following Kubernetes versions are now available for new clusters and for opt-in master upgrades and node upgrades for existing clusters. No channel Note: Your clusters might not have these versions available. GKE cluster versions have been updated. The following Kubernetes versions are now available for new clusters and for opt-in master upgrades and node upgrades for existing clusters. No channel Note: Your clusters might not have these versions available.

Load Balancing - Health check logging is now available in General Availability.

Cloud Logging - Cloud Logging calculates the system logs-based metrics byte_count and log_entry_count on stored logs only, unlike user-defined logs-based metrics which are calculated on both stored and excluded logs. In the Logs Explorer, you can now stream your log entries in real time as Cloud Logging ingests them.

Cloud Memorystore - Support for Redis AUTH on Memorystore for Redis is now Generally Available.

Private Catalog - Private Catalog launches an updated Cloud Console experience for cloud admins.

Cloud Spanner - Cloud Spanner supports a new statement hint, LOCK_SCANNED_RANGES, allowing you to request an exclusive lock on a set of ranges scanned by a transaction.

Virtual Private Cloud - Packet Mirroring direction control is now available in General Availability. DNS peering for private services access is now available in Preview.

VPC Service Controls - Preview support for the following integration: Identity-Aware Proxy for TCP forwarding.

AI Platform Prediction - Runtime version 2.3 is now available.

Anthos GKE deployed on-prem - Anthos GKE on-prem 1.6.0-gke.7 is now available. Users can use a credential configuration file with gkeadm (credential.yaml), which is generated during running the gkeadm create config command, to improve security by removing credentials from admin-ws-config.yaml. Node Problem Detector and Node Auto Repair automatically detect and repair additional failures, such as Kubelet-API server connection loss (an OSS issue) and long-lasting DiskPressure conditions. Preview: Repair administrator master VM failures by using the new command, gkectl repair admin-master. Preview: Secrets Encryption for user clusters using Thales Luna Network HSM Devices. Preview: Service Account Key Rotation in gkectl for Usage Metering, Cloud Audit Logs, and Google Cloud's operations suite service accounts. Anthos Identity Service enables dynamic configuration changes for OpenID Connect (OIDC) configuration without needing to recreate user clusters. Added support for CIDR in IP block file for static IP. Google Cloud's operations suite support for bundled Seesaw load balancing: Metrics and logs of bundled Seesaw load balancers are now uploaded to Google Cloud through Google Cloud's operations suite to provide the best observability experience. Cloud Audit Logs Offline buffer for Cloud Audit Logs: Audit logs are now buffered on disk if not able to reach Cloud Audit Logs and can withstand at least 4 hours of network outage. CSI volume snapshots The CSI snapshot controllers are now automatically deployed in user clusters, enabling the users to create snapshots of persistent volumes and restore the volumes' data by provisioning new volumes from these snapshots. Functionality changes: Gkectl diagnose cluster and snapshot enhancements: Added a --log-since flag to gkectl diagnose snapshot. Breaking changes: kubernetes.io/anthos/apiserver_request_total is deprecated; instead, use kubernetes.io/anthos/apiserver_aggregated_request_total. Fixes: Security fix: Resolve credential file references when only a subset of credentials are specified by reference. Known issues: gkectl update reverts your edits on clientconfig CR in 1.6.0.

AI Platform Training - Runtime version 2.3 is now available.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko