Researchers at Meta, the company that runs sites like Facebook and Instagram, have developed a machine-learning system called SEAMLESSM4T, Nature News reports. This system can translate spoken words from 101 languages into spoken words in 36 languages, almost instantly. It can also convert speech to text, text to speech, and text to text.
Machine translation has improved a lot recently, thanks to neural networks. However, there's a problem: there's not enough data for many languages, especially those not commonly used online. This makes it hard to train machines to translate these languages well.
Massively Multilingual and Multimodal Machine Translation
Meta's team has worked before on translating speech to speech and on a project called No Language Left Behind, which aimed to translate text for 200 languages. They found that including more languages in the system can actually help improve translations, even for languages with little data. They gathered millions of hours of speech recordings and their translations from various sources, including the United Nations.
To train SEAMLESSM4T, they used this data to match spoken words with their text versions across different languages. This allowed them to pair about half a million hours of audio with text translations. The system can translate speech directly to speech without needing to write it down first, using a speech synthesizer to create the audio output.
The system's performance got better by adding more languages and mixing different forms of text and speech. The translation delay is just a few seconds, similar to what you'd expect from a human translator. This new technology could make communication across different languages much easier, but for now, it's available for non-commercial use only, following Meta's trend of sharing tech advancements with researchers.
The researchers have described SEAMLESSM4T (the last part of the acronym stands for Massively Multilingual and Multimodal Machine Translation) in a paper titled "Joint speech and text machine translation for up to 100 languages," published in Nature.
