Neuroimaging produces large quantities of data that can be difficult to intuitively explore and gain insights into the biological mechanisms behind brain activity patterns. 

So David Thibodeaux and colleagues at Columbia University have now developed a toolkit to explore this data by translating it into a video, with accompanying musical sound track in real time. The goal is to help interpret what happens in the brain when the subject is performing certain behaviors.

How it works

The toolkit was applied to previously collected WFOM data. It detected both neural activity and brain blood flow changes in mice engaging in different behaviors, such as running or grooming.

Neuronal data is represented in the recording by piano sounds. The volume of each note indicates magnitude of activity, pitch indicates the location in the brain where the activity occurred, and blood flow data is represented by violin sounds. The relationship between neuronal activity and blood flow is represented by piano and violin sounds, played in real time.

Experiments

The researchers have demonstrated the new technique in three different experimental settings, showing how audiovisual representations can be prepared with data from various brain imaging approaches. These include “2D wide-field optical mapping (WFOM)” and “3D swept confocally aligned planar excitation (SCAPE) microscopy.”

They present this technique in the open-access journal PLOS ONE today (Feb. 21, 2024).

Immersive audio experience

“It is almost impossible to watch and focus on both the time-varying [brain activity] data and the behavior video at the same time. Our eyes would need to flick back and forth to see things that happen together.

“Listening to and seeing representations of [brain activity] data is an immersive experience that can tap into this capacity of ours to recognize and interpret patterns.”

Funding: National Institutes of Health grants, Columbia ROADS grant, and the Simons Collaboration on Global Brain.

Citation: Thibodeaux, D. N., Shaik, M. A., Kim, S. H., Voleti, V., Zhao, H. T., Benezra, S. E., Nwokeabia, C. J., & C. Hillman, E. M. (2024). Audiovisualization of real-time neuroimaging data. PLOS ONE, 19(2), e0297435. https://doi.org/10.1371/journal.pone.0297435

Let us know your thoughts! Sign up for a Mindplex account now, join our Telegram, or follow us on Twitter.