Welcome to issue #263 October 11th, 2021

News

Dataflow Pipelines, deploy and manage data pipelines at scale - Dataflow Pipelines is a new feature in Dataflow that enables users to deploy and manage data pipelines at scale.

Unlocking the hidden value of data: Launching Dataflow Templates for Elastic Cloud - Unlocking data potential: Launching Dataflow Templates for Elastic Cloud.

BigQuery migrations made easy - Announcing BigQuery Migration Service, a set of free-to-use tools to help you with your end-to-end migrations to BigQuery.

Better Kubernetes application monitoring with GKE workload metrics - GKE workload metrics is a fully managed, highly configurable metric collection pipeline. Configure which metrics to collect, and GKE does the rest.

A new way to explore what’s possible with Google Maps Platform - Improvements for Maps Platform website.

Artifact Registry for language packages now generally available - Artifact Registry lets you store Java, Node and Python artifacts, as well as popular Linux packages, for a more secure software supply chain.

Apache Cassandra on Anthos: Scaling applications for a global market - With Apache Cassandra on Anthos, users can experience higher efficiency and better quality in their cloud native hybrid applications.

Google Cloud Spanner Provider for Entity Framework Core - The general availability of the Google Cloud Spanner provider for Entity Framework Core. This post covers how to get started with the provider and highlights the supported features.

New data sovereignty controls for EU customers - New sovereign controls can help Cloud customers in the EU meet digital sovereignty requirements.

Register now for Firebase Summit 2021 - Firebase Summit will be returning as a virtual event on November 10th, 2021 at 9:30am PST.

Training more than 40 million new people on Google Cloud skills - To help more than 40 million people build cloud skills, Google Cloud is offering limited time no-cost access to all training content.

Articles, Tutorials

Infrastructure, Networking, Security, Kubernetes

How much can you save by migrating legacy apps to Google Cloud? Forrester does the math - Migrating expensive operating systems and workloads to Google Cloud can help organizations reduce costs while improving performance and uptime.

What to expect from Active Assist at Google Cloud Next’21 - Learn what’s the latest with Google ML-Powered intelligence service #ActiveAssist and how to learn more about it at Next’21.

Accelerating the journey to the cloud with a product mindset - Product Mindset emphasizes rapid feedback, failing quickly, and creating business value. This methodology encourages leaders not to focus on creating milestones at the beginning and checking off tasks until completion. Instead, the Product Mindset recommends frequent recalibration to ensure all efforts are efficiently translating into value for the customer.

Deploy Anthos on GKE with Terraform Part 3: Enabling Cloud Resources Provisioning - Part three in a series on Anthos Config Management (ACM) with Terraform, showing how Config Connector can be enabled on a GKE cluster, extending Kubernetes approach to the management of cloud resources.

10 ways Google Cloud IaaS stands out - Across compute, networking and storage, Google Cloud has a multitude of features that make it the best choice.

Automated onboarding: How USAA’s security team onboards users to GCP - How USAA provisions access for developer teams.

Org Policies by default - A list of the most important organization policies based on the work with customers.

App Development, Serverless, Databases, DevOps

Using the Cloud Spanner Emulator in CI/CD pipelines - This post covers how to use the Cloud Spanner emulator in your Continuous Integration and Continuous Delivery/Deployment (CI/CD) pipelines.

Analyzing Twitter sentiment with new Workflows processing capabilities - The combination of iteration syntax and connectors enables you to implement robust batch processing use cases. Let’s take a look at a concrete sample. In this example, you will create a workflow to analyze sentiments of the latest tweets for a Twitter handle.

Service orchestration on Google Cloud - Business problems are often solved by coordinating multiple microservices. This coordination is based on event-driven architectures, which can be implemented via two approaches: choreography and orchestration.

Secure Google Cloud Functions with API Gateway - Configuring API Gateway for Cloud Functions.

Rate limit Google Cloud Functions with API Gateway - Configuring rate limits and quotas in API Gateway.

Rowy.io - Rowy is an open-source platform that lets you manage Firestore data in a spreadsheet-like UI, write Cloud Functions effortlessly in the browser, and connect to your favorite third-party platforms.

PostgreSQL extension turned Cloud microservce - Converting a PostgreSQL extension into a Google Cloud microservice.

LOVOO’s love affair with Spanner - The largest German-speaking dating app uses Cloud Spanner and Cloud SQL for PostgreSQL to solve for speed and scale.

How Cherre transforms real estate data with Cloud SQL for PostgreSQL - Cloud SQL for PostgreSQL allows Cherre to consolidate thousands of real estate data sources into an easy-to-use API for their customers.

Big Data, Analytics, ML&AI

Building the data analyst driven organization from the first principles - Learn about the basics and best practices on how to build a data analyst driven culture depending on what type of organization you are.

Liberating your mainframe data with Confluent and Google Cloud - Google Cloud and Confluent have teamed up to provide an end-to-end solution for connecting your mainframe application data with the advanced analytics capabilities of Google.

Out of the data Tar Pit: How loveholidays escaped from a legacy reporting system and kept the data flowing - How Loveholiday improved their reporting by investing in technology simplification and building a brand new ETL system using GCP products.

Cromwell “Hello GCP” - This post goes through steps to set Cromwell, one of the most popular Workflow engines for Bioinformatians in a secure way.

Migrating your Data Warehouse to BigQuery? Make sure to unit test your BigQuery UDFs with Dataform - Learn how to set up UDF unit testing with the Dataform CLI tool for your data warehouse migration to BigQuery.

PubSub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java - Step by step tutorial on how to create pipeline in Cloud Dataflow.

BigQuery fetching + multiprocessing - Does multiprocessing improve the fetching speed of BigQuery API requests?

Google Vertex AI: The Easiest Way to Run ML Pipelines - This article covers the steps needed to implement a reliable, reproducible and automated machine learning pipeline with Google Vertex AI.

Debugging Vertex AI training jobs with the interactive shell - Introducing the interactive shell, a new tool available to users of Vertex AI custom training jobs. This feature gives you direct shell-like access to the VM that’s running your code, giving you the ability to run arbitrary commands to profile or debug issues that can’t be resolved through logs or monitoring metrics.

Model training as a CI/CD system: Part I - A machine learning system is essentially a software system. So, to operate with such systems scalably we need CI/CD practices in place to facilitate rapid experimentation, integration, and deployment. In this post we explore some scenarios.

Serving predictions & evaluating Recommendations AI - In this post we'll show how to use Recommendations AI to display predictions on a live website and set up A/B experiments to measure performance.

Various

Next ’21 is on: Top five things to do at Google Cloud Next - Engage with experts. Dive into demos. Check out our top picks for five things to do at Google Cloud Next.

Next’21 - Your guide to Google Cloud Security sessions - Guide to security topics and updates at Google Cloud Next 2021.

Get started, build and grow your Startup on Google Cloud - Announcing the launch of Google Cloud Technical Guides for Startups, a video series for technical enablement aimed at helping startups to start, build and grow their businesses.

Slides, Videos, Audio

GCP Podcast - #279 MLB with Perry Pierce and JoAnn Brereton.

Security Podcast - #34 Instrumenting Modern Application Stack for Detection and Response.

Releases

Access Approval - Access Approval supports the following services in GA stage: Cloud SQL Google Kubernetes Engine Speaker ID.

AI Platform Training - Runtime version 2.6 is available. You can use runtime version 2.6 to train with TensorFlow 2.6, scikit-learn 0.24.2, or XGBoost 1.4.2. Runtime version 2.6 supports training with CPUs, GPUs, or TPUs.

Anthos clusters on bare metal - 1.6 & 1.7 & 1.8 & 1.9. Security bulletin (all minor versions) A security vulnerability, CVE-2020-8561, has been discovered in Kubernetes where certain webhooks can be made to redirect kube-apiserver requests to private networks of that API server.

Artifact Registry - v1beta2. You can now specify a release or snapshot version policy for Maven repositories when you create them.

Cloud Asset Inventory - The following resource types are now publicly available through the resource search API (SearchAllResources), policy search API (SearchAllIamPolicies), and Analyze Policy APIs (AnalyzeIamPolicy and AnalyzeIamPolicyLongrunning) : + Eventarc + eventarc.googleapis.com/Trigger.

BigQuery - BigQuery Migration Service is now in Preview.

BigTable - Cloud Bigtable provides a CPU utilization by app profile, method, and table metric that gives you more granular observability into the cluster's CPU usage.

Cloud Composer - Python Client for Cloud Composer version 1.0.0 is released.

Dataproc Metastore - Fixed the issue causing Dataproc Metastore service creations with CMEK enabled to fail if a service without CMEK enabled has never been created before in the project.

Dataproc - In a future announcement (on approximately October 22, 2021), Dataproc will announce that Cluster Scheduled Deletion by default will consider YARN activity, in addition to Dataproc Jobs API activity, when determining cluster idle time.

Document AI - Document AI is now generally available (GA) in the following new locations: europe-west2 northamerica-northeast1 You must request access to use the new locations.

Cloud Filestore - You can now get support for preview features for Filestore.

Google Kubernetes Engine - GKE version 1.20.8-gke.2100 or later offers a Preview of a fully managed metric collection pipeline to scrape Prometheus-style metrics exposed by any GKE workload and send those metrics to Cloud Monitoring for dashboards, alerts, and SLOs.

Cloud Logging - You can now collect Apache httpd logs from the Ops Agent, starting with version 2.4.0. The Ops Agent now supports collecting logs from the systemd-journald service, starting with Ops Agent version 2.4.0.

Anthos Migrate - v1.9.0. Migrate to GKE Autopilot clusters and Cloud Run now in GA Support for the simplified Linux service manager, which lets you deploy containers to GKE Autopilot clusters and to Cloud Run, has moved from the Public Preview stage to General Availability. Fit assessment tool now in GA The migration fit assessment tool has moved from the Public Preview to General Availability. 194605214 Use controller storage by default for pod log collection for logging migration tasks. 187922406 Fixed LVM mount failure caused from broken device mapper devices. 198092293 [MFIT] vSphere level <-> guest level data correlation failure with certain NIC configurations. 197432816 [MFIT] More granular assessment of supported Windows versions. 197206783 [MFIT] Fixed failure to run guest collect script via SSH with a non-root remote user. 196712456, 201610944 [MFIT] Minor html report UI improvements.

Cloud Monitoring - GKE version 1.20.8-gke.2100 or later offers a Preview of a fully managed metric collection pipeline to scrape Prometheus-style metrics exposed by any GKE workload and send those metrics to Cloud Monitoring for dashboards, alerts, and SLOs.

Cloud VPN - Classic VPN partial deprecation update Starting on March 31, 2022, you will no longer be able to create new Classic VPN tunnels that use dynamic routing (BGP) unless you are creating a specifically supported configuration.

Security Command Center - Security Health Analytics, a built-in service of Security Command Center, released new detectors in general availability.

Anthos Service Mesh - 1.11.x. 1.11.2-asm.17 is now available. Managed Anthos Service Mesh isn't rolling out to the rapid release channel at this time. asmcliis generally available for new installations and upgrades of Anthos Service Mesh. Using install_asm and istioctl install is deprecated and support for these tools for installations and upgrades of Anthos Service Mesh will be removed when Anthos Service Mesh 1.12 is released. The Anthos Service Mesh integration with Certificate Authority Service (CA Service) is generally available. Anthos Service Mesh uses a proxy that is based on OSS Envoy.

Cloud Spanner - You can now specify the statistics package for the query optimizer to use, to ensure predictability in your query plans.

Cloud SQL - Access Approval is now GA for Cloud SQL. Cloud SQL now supports the ability for you to specify IP CIDR ranges from your VPC network for your Cloud SQL instances allowing you to manage your IP address space better.

Cloud Storage - Turbo replication is a premium feature designed to provide inter-region replication for newly written objects within 15 minutes.

Vertex AI - Vertex Feature Store is generally available (GA).

Cloud Video Intelligence API - The SHOT_CHANGE_DETECTION model will undergo an upgrade over the next 90 days to a newer version.

Virtual Private Cloud - The number of Private Service Connect endpoints that are connected to a service attachment is now correctly adjusted when an endpoint is deleted. If you are using Private Service Connect endpoints to access services in another VPC network, and you create more endpoints than are allowed by the limit set by the service producer, any endpoints created after the limit is reached have a status of Pending, as expected.

If you have suggestion, feedback or link you want to share feel free to email me at zdenko@gcpweekly.com

Have a great week,

Zdenko