In this issue:

1. Introducing the federated learning series.

2. Google’s original federated learning paper.

3. TensorFlow Federated framework.

💡 ML Concept of the Day: A New Series About Federated Learning

Today we are starting a new series about federated learning(FL), one of the most popular deep learning architectures used in distributed systems. Pioneered by Google in 2017, FL has become one of the fundamental techniques to train systems that operate in a decentralized manner. In that category, we can place applications running on mobile or internet of things(IOT) devices and the entire field of edge computing applications.

But how does FL really works?

In a setting in which an ML model is running in multiple silos. A centralized approach to ML training would require training to be collected in devices and send to the cloud. This imposes some privacy concerns but is also brutally hard to scale in architectures with millions of devices. . In a  FL architecture, each host running the model will first download it and will improve it using local training data. After that, the FL method will push a small focused update to the cloud which is aggregated with the updates from millions of other hosts and distributed back to all of the m. Throughout this process, the training data remains locally in the host infrastructure and only the updates are distributed to cloud environments. The general principle of FL consists in training models on local, in-device data samples and exchanging parameters such as weights and biases between different devices at some frequency to generate a global model shared by all devices.

FL is far from being a single architecture and has evolved into different variants such as: data-centric, model-centric, vertical, horizontal, federated transfer learning and many others. We will deep dive into all those areas in the next editions of this newsletter.

🔎 ML Research You Should Know About: The Original Federated Learning Paper

Communication-Efficient Learning of Deep Networks from Decentralized Data was published in by Google Research in 2017. The paper introduced the notion of federated learning that has become so popular in mobile machine learning architectures.

The objective: The paper introduces a method to enable the training of mobile machine learning models while preserving privacy. Google terms this new method as federated learning.

Why it is so important? Federated learning has become the most popular technique for highly scalable, private machine learning models. Most modern deep learning frameworks already include federated learning libraries.    

Diving Deeper: Most machine learning practitioners associate model training with access to the raw training data. Think about the scenario of a mobile app deployed across millions of devices. The traditional way to model training would be to collect data from all those users, aggregate it in a centralized server and train a new version of the model. By now you should be worrying about privacy in your mobile apps and you definitely should. However, federated learning offers a brilliant alternative to the traditional approach.

In Communication-Efficient Learning of Deep Networks from Decentralized Data, Google researchers introduced the notion of federated learning as a method to decouple the training of machine learning models from the access to the raw training dataset. Essentially, federated learning leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates.

The concept of federated learning is not hard to comprehend but the devil is in the details. What are the exact updates published by the mobile devices? How are those updates aggregated to improve the training of the model? In a traditional machine learning scenario, the model will produce a set of parameters such as weights and biases which then will be processed by an optimization algorithm such as stochastic gradient descent(SGD). To adapt that model to a scenario with millions of parameter sets, federated learning introduces a new optimization algorithm known as FederatedAveraging which takes fewer iterations to produced an optimized model. The FederatedAveraging algorithm can be seen as a federated, high performance version of SGD that can receive parameter from millions of instances of the machine learning model and produced a new version of the model.

Federated learning has discovered new frontiers and sparked a lot of innovation in the mobile machine learning space. Being familiar with this technique is essential when working in machine learning scenarios with internet of things(IOT) or mobile architectures.

🤖 ML Technology to Follow: TensorFlow Federated is an Open Source Framework for Federated Learning

Why Should I Know About This:. TensorFlow is the most popular deep learning framework in the market. TensorFlow Federated enables the implementation of federated learning models on top of TensorFlow.

What is it: TensorFlow Federated(TFF) brings federated learning to TensorFlow models. The framework was built based on the experiences scaling federated learning architectures at Google. The idea behind TFF is to abstract a series of primitives that enable machine learning computations over decentralized data.

Let’s imagine that we are building an image classification model in TensorFlow. That’s a relatively easy task as TensorFlow already includes many popular algorithms for image classification and there are plenty of high quality training datasets out there. The traditional way to do this in TensorFlow would be to build the initial model and train it across the entire dataset. However, what if we could partition the training across different nodes each one containing portions of the dataset. This could be the case in a mobile scenario in which part of the training data is hosted in the individual user’s devices. This is the type of scenario that TFF can enable.

From the architecture standpoint, TFF is based on two fundamental components:

• Federated Learning (FL) API:  The  layer offers a set of high-level interfaces that allow developers to apply the included implementations of federated training and evaluation to their existing TensorFlow models.

• Federated Core (FC) API: At the core of the system is a set of lower-level interfaces for concisely expressing novel federated algorithms by combining TensorFlow with distributed communication operators within a strongly-typed functional programming environment.

These two APIs are the core components needed to enable federated learning in a TensorFlow model. The FL API enables to create model architectures in which the training data is provided by different “writers” each one hosting a portion of the training dataset. The FC API provides a lower set of computations that can be distributed across all the different nodes in the architecture. For instance, imagine that we have a topology with many nodes, each reporting a numerical indicators(ex: the temperature in a given room). We would like to calculate the average of those computations without uploading the data to a centralized server. This is the type of federated computation that can be achieved with the TFF FC API.

The computations enabled by the FC API can be seen as the equivalent of the FederatedAveraging function outlined in the original federated learning paper.

TFF provides a very simple programming model that enables the creation of federated learning architectures in TensorFlow. The native integration with TensorFlow has made TFF one of the most popular federated learning frameworks in the deep learning community and one that we should expect to grow very fast in the next few years.

How Can I Use it: TensorFlow Federated is part of the TensorFlow project and is available athttps://www.tensorflow.org/federated/.