Training large language models often requires data from many different devices or institutions. However, sharing raw data raises serious privacy concerns. To solve this, scientists use an approach called federated learning. In federated learning, each device trains its own copy of the model locally and then shares only the updates with others. This keeps private data on the original devices.
Traditional federated learning has been slow and costly. Devices must exchange complete model updates, which can be many gigabytes in size. This process demands large amounts of network bandwidth and electricity. As a result, updates happen infrequently, and the shared model often stays outdated.
Researchers at Stevens have developed a new algorithm named MEERKAT to address these problems. Instead of sending full model updates, MEERKAT identifies and shares changes to just 0.1 percent of these parameters - the ones that have the biggest impact.. By focusing on a small, well-chosen subset of parameters, the method shrinks communication needs by more than 1000 times. What once required gigabytes now fits into just a few megabytes.
Key improvements in efficiency
MEERKAT also avoids backpropagation, which helps artificial intelligence (AI) models learn from mistakes but uses a great deal of memory and power. The new method makes small adjustments to the model and simply checks the results, bypassing this heavy computation entirely. Because updates are so small, devices can exchange information much more often. This keeps the shared model more current and accurate.
The approach leads to better performance with far lower energy use and computational costs. It makes advanced AI training more practical for organizations with limited resources, such as hospitals or schools, where data privacy rules prevent central data collection. The method supports fairer access to powerful AI tools across different fields.
This research was presented at the International Conference on Learning Representations.