A Taxonomy to Understand Federated Learning
Was this email forwarded to you? Sign up here. You can also give it as a gift. A Taxonomy to Understand Federated LearningClassifying different types of federated learning methods, Meta AI research about highly scalable and asynchronous federated learning pipelines and Microsoft's FLUTE framework.In this issue:
💡 ML Concept of the Day: A Taxonomy to Understand Federated LearningOne of the most common mistakes around federated learning, is to think about it as a single type of architecture. Even though while the principles of federated learning are unique enough to merit its own space within deep learning, there are different architectures that implement those principle in diverse way. While there is no consistent taxonomy to study federated learning, there are some categorization schemes that are proven to be quite useful. The following taxonomy might be relevant to understand the different variations of federated learning architectures: By ML Construct: This categorization scheme focuses divides federated learning architectures based on the ML artifacts that are being distributed.
By Federation Scheme: This criteria is based on the architecture of the federation.
By Data Partition: This criteria focuses on the feature and data partition models used in the federation.
In the next few editions of this series, we will be diving into each several of these categories. 🔎 ML Research You Should Know About: Faster and More Scalable Federated LearningIn Federated Learning with Buffered Asynchronous Aggregation, Meta AI proposes a asynchronous method for scaling the training of federated learning models. The objective: Meta AI method addresses some of the fundamental scalability challenges of federated learning models. Why it is so important? Scaling federated learning across many devices is a monumental challenge for most organizations. Meta AI’s technique showcases the advance of asynchronous methods to scale federated learning models. Diving Deeper: Most of the mainstream methods for training federated learning models rely on synchronous processes. In the scychronous architecture, nodes will download the current mode and will cooperate to send updates to a centralized server. While synchronous communication is simpler and has strong privacy guarantee properties has major scalability limitations. Effectively, synchronous federated learning is only as fast as the slowest device/node in the network. This is known as the straggler problem. In their paper, Meta AI provides an asynchronous method that addresses the limitation of synchronous federated learning techniques. The core idea behind Meta AI is relatively simple. The architecture uses a larger number of nodes than typically needed and regularly drops the slowest ones on each training round. For instance, a training round might start with 1500 nodes and end with 1200 as the 300 slowest nodes were remove from the federation. When a node is available for training, it downloads the model, computes the update and sends it to the server just like in a synchronous scenario. However, the server just aggregates the updates until It has met its prerequisites and distributes the aggregated update to the nodes. The slowest nodes might still send their updates to the server but those might be based on state information. Meta AI tested this technique with federated learning architectures consisting of 100 million Android devices. In average, the training processes were about 5x faster than synchronous alternatives. 🤖 ML Technology to Follow: FLUTE is a Platform for High Performance Federated Learning SimulationsWhy Should I Know About This:. FLUTE is one of the first frameworks that allow data scientists to run multi-agent federated learning scenarios. What is it: Federated Learning Utilities and Tools for Experimentation (FLUTE) was designed to enable researchers to perform rapid prototyping of offline simulations of federated learning scenarios without incurring in high computational costs. The framework includes a core series of capabilities that enable that goal:
A typical FLUTE architecture consists of a number of nodes which are physical or virtual machines that execute a number of workers. One of the nodes acts as a orchestrator distributing the model and tasks to the different workers. Each worker processes the tasks sequentially, calculates the model delta and sends the gradients back to the orchestrator which federates it into the centralized model. Extrapolating this workflow to a large number of clients, we get something like the following architecture. In this architecture, a federated learning workflow is based on the following steps( from the research paper):
The initial version of FLUTE is based on PyTorch which enables it interoperability with a large number of deep learning architectures. The communication protocols are implemented using OpenMPI which guarantees high levels of performance and scalability. How Can I Use it: FLUTE is open source and available at https://github.com/microsoft/msrflute. . TheSequence is a summary of groundbreaking ML research papers, engaging explanations of ML concepts, and exploration of new ML frameworks and platforms. We keep you up-to-date with the main AI news, trends, and technology developments. This post is only for paying subscribers of TheSequence Edge. You can give it as a gift. |
Key phrases
Older messages
ChatGPT and Whisper APIs
Sunday, March 5, 2023
Sundays, The Sequence Scope brings a summary of the most important research papers, technology releases and VC funding deals in the artificial intelligence space.
📝 How is MLOps more than just tools?
Friday, March 3, 2023
Hi there! At TheSequence, we're exploring what MLOps culture looks like across the industry at the start of 2023. A huge variety of tools are available for ML development, but the culture and
Inside Claude: The ChatGPT Competitor that Just Raised Over $1 Billion
Thursday, March 2, 2023
Claude uses an interesting technique called Constitutional AI to enable safer content.
Edge 269: A New Series About Federated Learning
Tuesday, February 28, 2023
Intro to federated learning, the original federated learning and the TensorFlow Federated framework.
Meta and Amazon Generative AI Moves
Sunday, February 26, 2023
Sundays, The Sequence Scope brings a summary of the most important research papers, technology releases and VC funding deals in the artificial intelligence space.
You Might Also Like
Stripe changes its … stripes
Wednesday, April 24, 2024
TikTok on the president's docket and Nvidia acquires Run:ai View this email online in your browser By Christine Hall Wednesday, April 24, 2024 Good afternoon, and welcome to TechCrunch PM! Today
💪 You Can Use Copilot AI as a Personal Trainer — Why Your Laptop Needs a Docking Station
Wednesday, April 24, 2024
Also: Here's How to Make Your Apple ID Recoverable, and More! How-To Geek Logo April 24, 2024 📩 Get expert reviews, the hottest deals, how-to's, breaking news, and more delivered directly to
JSK Daily for Apr 24, 2024
Wednesday, April 24, 2024
JSK Daily for Apr 24, 2024 View this email in your browser A community curated daily e-mail of JavaScript news JSK Weekly - 24th April, 2024 React 19 has introduced many great functionalities and
Daily Coding Problem: Problem #1422 [Hard]
Wednesday, April 24, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Airbnb. Given a list of integers, write a function that returns the largest sum of non-
Charted | Artificial Intelligence Patents, by Country 🤖
Wednesday, April 24, 2024
This visualization shows which countries have been granted the most AI patents each year, from 2012 to 2022. View Online | Subscribe Presented by: New on VC+: Our Visual Briefing on the IMF's World
Save your seat: 1Password’s 2024 Security report insights webinar
Wednesday, April 24, 2024
Join us April 25th. ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏ ͏
Top Tech Deals 📱 LG Flex TV, Google Pixel 7, DJI Mini 3, and More
Wednesday, April 24, 2024
Get yourself a discounted DJI drone, save on the Pixel 7, or score some PC and phone accessories. How-To Geek Logo April 24, 2024 Top Tech Deals: LG Flex TV, Google Pixel 7, DJI Mini 3, and More Find
The Protest Song Wakes Up 🎙️
Wednesday, April 24, 2024
Is this song the future of musical protest? Here's a version for your browser. Hunting for the end of the long tail • April 24, 2024 The Protest Song Wakes Up A buzzy protest song about the
JSK Weekly - 24th April, 2024
Wednesday, April 24, 2024
React 19 has introduced many great functionalities and features, among which the useOptimistic hook stands out. The useOptimistic hook offers a seamless way to manage UI states during asynchronous
The clock’s ticking for TikTok
Wednesday, April 24, 2024
The US Senate has passed a bill that would ban TikTok if its US business is not divested by Bytedance View this email online in your browser By Alex Wilhelm Wednesday, April 24, 2024 Good morning, and