Researchers have developed an AI system that combines music and lyrics to assign emotional tags to short video soundtracks, including clips with noisy audio or missing words.
Published October 21 in Scientific Reports, the study introduces SALF-Net, designed for recordings lasting fifteen to sixty seconds. Conventional tagging tools can struggle when songs are truncated, degraded or stripped of reliable lyric information.
The system analyzes audio with a transformer and processes available lyrics with a language model. A fusion mechanism combines their signals, while training removes one input at times so the model can continue working when lyrics are unavailable.
It also learns to estimate valence and arousal, dimensions describing how positive or negative, and how energetic, music sounds. These estimates help organize the emotional tags.
Evaluation covered MTG-Jamendo, DEAM and a new collection of 30,000 short clips. The authors report stronger performance than comparison systems and a smaller model that retains most tagging quality with lower latency.
For platforms exploring personalized music discovery, better emotional metadata could improve search and organization. The findings concern agreement with emotion annotations, however, rather than proof of how individual listeners will feel.