A research team has created AI neural network models that decoded what a mouse saw while watching a movie.

The machine-learning algorithm, CEBRA (pronounced “zebra”), predicted movie frames directly from brain signals, after an initial mouse training period.

(Credit: EPFL/Hillary Sancutary/Alain Herzog/Allen Institute/Roddy Grieves)

“This work is just one step towards the theoretically backed algorithms that are needed in neurotechnology to enable high-performance BMIs [brain-machine interfaces],” said principal investigator Mackenzie Mathis, Bertarelli Chair of Integrative Neuroscience at EPFL (École Polytechnique Fédérale de Lausanne] in a statement.

How it works

The researchers used video-decoding data from the Ho Institute in Seattle. The brain signals were obtained directly by measuring brain activity via electrode probes inserted into the visual cortex area of the mouse’s brain. The activated optogenetic mice neurons were genetically engineered to glow green. During the training period, CEBRA learned to map the brain activity to specific video frames, using less than 1% of neurons in the mouse visual cortex (which consists of about 0.5 million neurons).

CEBRA is based on “contrastive learning,” a technique that can be used to infer hidden relationships and structure in the data. It enables researchers to jointly consider neural data and behavioral labels, including measured movements, abstract labels like “reward,” or sensory features such as colors or textures of images.

According to the researchers, the broad goal of CEBRA is to uncover structures in complex systems and provide insight into how the brain processes information. It could also serve as a platform for discovering new principles in neuroscience by combining data across animals, and even species, with possible clinical applications.

Citation: Schneider, S., Lee, J.H. & Mathis, M.W. Learnable latent embeddings for joint behavioural and neural analysis. Nature (2023). https://doi.org/10.1038/s41586-023-06031-6 (open access).

Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter.