As artificial intelligence (AI) systems become larger and more capable, their need for energy increases sharply. Recent research from the University of Massachusetts Amherst shows that advanced AI performance can be reached with far lower energy demands.
The researchwers set two main goals. One was to allow AI to keep learning in real time during use, instead of only during an initial training period. The other was to reduce the energy required for the computations. The human brain offers an example of efficient operation. Much of this efficiency arises because the brain works asynchronously, with only the needed groups of nerve cells becoming active for each task.
In contrast, today's major AI systems rely on synchronized updates, and this demands extra computation and therefore extra energy. The problem grows worse as models reach billions or trillions of parameters. Past attempts to build asynchronous spiking neural networks, which use brief signal pulses, encountered obstacles in training. The usual methods for adjusting connections, such as backpropagation that calculates changes from errors, did not perform as well with those designs.
Development of the asynchronous neural turing network architecture
To address these issues, the researchers developed asynchronous neural turing networks (ANT). This new structure eliminates the global timing signal while retaining the mathematical features that support effective training with gradient-based methods. Gradient-based methods adjust connections by measuring how small changes affect the final result. The design ensures that information stays intact during independent updates. As a result, only the units required for the current computation activate, which can lower energy use by large amounts. The approach draws from prior demonstrations that recurrent neural networks with feedback loops can equal the power of turing machines, basic models of computation.
Work continues to increase the energy savings and to strengthen the capacity for ongoing real-time learning. Such systems could benefit applications where power is limited, including robots, computing devices that process data locally without large centers, autonomous vehicles, and other forms of adaptive machine intelligence.
This research is published in Nature Communications.