In this issue:

• A taxonomy for classifying federated learning methods.

• Meta AI research for building highly scalable and asynchronous federated learning pipelines

• Microsoft Research’s FLUTE framework for federated learning architectures.

💡 ML Concept of the Day: A Taxonomy to Understand Federated Learning

One of the most common mistakes around federated learning, is to think about it as a single type of architecture. Even though while the principles of federated learning are unique enough to merit its own space within deep learning, there are different architectures that implement those principle in diverse way. While there is no consistent taxonomy to study federated learning, there are some categorization schemes that are proven to be quite useful. The following taxonomy might be relevant to understand the different variations of federated learning architectures:

By ML Construct: This categorization scheme focuses divides federated learning architectures based on the ML artifacts that are being distributed.

• Model-Centric: By far the most common form of federated learning, model-Centric architectures start with a centralized version of a model that needs to be improved by the different nodes in a decentralized network.

• Data-Centric: These architectures maintain a central data repository and allow the nodes to perform training and inference request against the data.

By Federation Scheme: This criteria is based on the architecture of the federation.

• Cross-Device: This is the pattern followed by the original Google federated learning architecture in which each device in the federation was contributing to updates in the model.

• Cross-Silo: These are architectures in which multiple nodes can act as a single silo in the federation. This type of architecture is more common in federated learning scenarios that involve many organizations.

By Data Partition: This criteria focuses on the feature and data partition models used in the federation.

• Horizontal Federated Learning: This architecture in based on nodes that share the feature space but differ on the sample space.

• Vertical Federated Learning: This is a federated learning architecture in which the nodes share the same samples but a different feature space.

In the next few editions of this series, we will be diving into each several of these categories.

🔎 ML Research You Should Know About: Faster and More Scalable Federated Learning

In Federated Learning with Buffered Asynchronous Aggregation, Meta AI proposes a asynchronous method for scaling the training of federated learning models.

The objective: Meta AI method addresses some of the fundamental scalability challenges of federated learning models.  

Why it is so important? Scaling federated learning across many devices is a monumental challenge for most organizations. Meta AI’s technique showcases the advance of asynchronous methods to scale federated learning models.

Diving Deeper: Most of the mainstream methods for training federated learning models rely on synchronous processes. In the scychronous architecture, nodes will download the current mode and will cooperate to send updates to a centralized server. While synchronous communication is simpler and has strong privacy guarantee properties has major scalability limitations. Effectively, synchronous federated learning is only as fast as the slowest device/node in the network. This is known as the straggler problem. In their paper, Meta AI provides an asynchronous method that addresses the limitation of synchronous federated learning techniques.

The core idea behind Meta AI is relatively simple. The architecture uses a larger number of nodes than typically needed and regularly drops the slowest ones on each training round. For instance, a training round might start with 1500 nodes and end with 1200 as the 300 slowest nodes were remove from the federation. When a node is available for training, it downloads the model, computes the update and sends it to the server just like in a synchronous scenario. However, the server just aggregates the updates until It has met its prerequisites and distributes the aggregated update to the nodes.  The slowest nodes might still send their updates to the server but those might be based on state information.

Meta AI tested this technique with federated learning architectures consisting of 100 million Android devices. In average, the training processes were about 5x faster than synchronous alternatives.

🤖 ML Technology to Follow: FLUTE is a Platform for High Performance Federated Learning Simulations

Why Should I Know About This:. FLUTE is one of the first frameworks that allow data scientists to run multi-agent federated learning scenarios.

What is it: Federated Learning Utilities and Tools for Experimentation (FLUTE) was designed to enable researchers to perform rapid prototyping of offline simulations of federated learning scenarios without incurring in high computational costs. The framework includes a core series of capabilities that enable that goal:

• Ability to simulate millions of endpoints sampling tends of thousands instances per round.

• Multi-GPU, multi-node orchestration.

• Rich model portfolio including popular CNNs, RNNs and transformer architectures.

• Extensible programming model.

• Native integration with Azure ML.

A typical FLUTE architecture consists of a number of nodes which are physical or virtual machines that execute a number of workers. One of the nodes acts as a orchestrator distributing the model and tasks to the different workers. Each worker processes the tasks sequentially, calculates the model delta and sends the gradients back to the orchestrator which federates it into the centralized model.

Extrapolating this workflow to a large number of clients, we get something like the following architecture.

In this architecture, a federated learning workflow is based on the following steps( from the research paper):

• Send and initial global model to clients.

• Train instances of the global model with locally available data on each client.

• Send training information to the Server (e.g. adapted models, logits, pseudo-gradients).

• Combine the returned information on the server to produce a new model.

• Optionally, update the logbal model with an additional server-side rehearsal step.

• Send the updated global model back to the clients.

• Repeat steps 2–6 after sampling a new subset of clients for the next training iteration.

The initial version of FLUTE is based on PyTorch which enables it interoperability with a large number of deep learning architectures. The communication protocols are implemented using OpenMPI which guarantees high levels of performance and scalability.

How Can I Use it: FLUTE is open source and available at https://github.com/microsoft/msrflute. .