For the first time, researchers have demonstrated that a single brain implant can read a paralyzed person's attempts at both speech and gesture at the same time, turning these neural signals into written text and the movements of a digital avatar, Fast Company reports. This research is published in Nature, The study marks a significant advance in brain-computer interface technology.
How the technology works
The device uses artificial intelligence (AI) models to interpret patterns of electrical activity from the brain. Unlike earlier systems that could decode only one type of output at a time, such as either speech or motor commands, this implant captures both channels of intention simultaneously. The AI translates neural signals representing attempted speech into text, while also translating signals representing attempted hand or arm movements into commands for a digital avatar.
The research team employed high-density electrocorticography implants to record neural signals from three participants with paralysis. They developed parallel speech and gesture decoders that allowed participants to control a personalized virtual avatar by attempting speech and gestures simultaneously or in isolation. The researchers found that training models on both isolated and simultaneous data improved performance across behavioral contexts, demonstrating that one cortical implant can support multi-effector control.
The combination is important because human communication normally involves both verbal and non-verbal elements. Restoring only speech leaves out gestures and expressions that carry meaning. By integrating both capabilities in one system, the researchers aim to produce more natural and complete interaction for users.
The Nature publication suggests that coordinated speech and movement decoding, once thought to require separate approaches, can be achieved with a single implanted device. This could simplify surgical procedures and reduce the hardware burden on patients. The next steps will likely involve expanding vocabulary sizes, improving real-time performance, and testing long-term reliability in everyday environments.