Revolutionizing Automatic Speech Translation with Enhanced Expressivity and Multilingual Capabilities

By Dr Silpaja Chandrasekar, PhDReviewed by Susha Cheriyedath, M.Sc.Dec 5 2023

In a paper posted to the Meta Research website, researchers unveiled a groundbreaking series of models enhancing automatic speech translation. The latest SeamlessM4Tv2 model, built upon the UnitY2 framework, boasts expanded language coverage and incorporates improved features. It serves as the core for two new models: SeamlessExpressive, preserving vocal nuances, and SeamlessStreaming, enabling simultaneous speech-to-speech/text translation for multiple languages in real-time.

These models excel in preserving meaning, naturalness, and expressivity. Moreover, the team implemented safety measures, including toxicity detection, gender bias evaluation, and watermarking against deep fakes. Their culmination, Seamless, marks a significant leap toward real-time expressive cross-lingual communication.

Background

The quest for natural speech translation involves preserving the intricate components of human communication, encompassing vocal style, prosody, and pragmatic nuances. Friedrich Schlegel's words underline the challenge of retaining these elements in translation. Existing efforts in expressive and streaming speech-to-speech translation have made strides but still face limitations in universal coverage and comprehensive preservation of semantic and stylistic elements.

SeamlessM4Tv2: Advancements in Translation

SeamlessM4Tv2 represents a substantial advancement in cross-lingual translation, building on the strengths of the foundational SeamlessM4T model. This upgrade delivers enhanced semantic accuracy, extensive language coverage, and multitasking capabilities, enabling real-time translations across diverse languages. Leveraging non-autoregressive text-to-unit decoding, it addresses issues like input speech token length disparities, boosting translation speed. By harnessing extensive unlabeled audio data and architectural innovations such as UnitY2, the model achieves superior translation efficiency and accuracy across a wide range of languages.

Furthermore, the advanced SeamlessM4T-Largev2, a 2.3B model employing multitask-UnitY2 design, underwent comprehensive assessments across various language tasks. Its standout performance in speech-to-text translation, particularly in multilingual contexts like Fleurs, showcased significant advancements over previous models. Its multitasking capabilities notably improved ASR tasks for low and medium-resource languages, excelling in zero-shot text-to-speech translation and demonstrating adaptability across diverse language settings. Robustness and proficiency were evident across multiple applications, reinforced by ablation studies optimizing input representations.

Advancements in Expressive Speech Translation

SeamlessExpressive, a speech-to-speech translation model, emphasizes capturing subtle prosodic elements like speech rate and pauses while maintaining translation excellence. Training this model involved diverse datasets across languages, highlighting distinct styles, expressiveness, and alignment traits. Researchers tailored preprocessing steps and techniques like UnitVoicebox to craft a sophisticated model capable of understanding and expressing nuanced prosodic cues for precise and expressive speech translation. The development of the Prosody and Emotion TSS in Spoken Language Generation (PRETSSEL) model involved meticulous experimentation, spanning multilingual dataset collection, and rigorous training with advanced techniques like SpecAugment and High Fidelity Generative Adversarial Network (HiFi-GAN) vocoders.

Evaluations leveraging metrics such as Automatic Speech Recognition - Bilingual Evaluation Understudy (ASR-BLEU), Vocal Style Similarity (VSim), Automatic Prosodic Control Parameters (AutoPCP), Rate, and Pause highlighted SeamlessExpressive's superior translation and expressivity preservation compared to baselines and alternative models, revealing its strength over cascade setups in handling source speech noise.

SeamlessStreaming: Multilingual Translation Insights

SeamlessStreaming, an advanced model, enables direct, simultaneous multilingual, and multimodal translations, building upon SeamlessM4Tv2's extensive language coverage. It proficiently handles diverse source languages for both speech input and output, alongside robust support for text output and streaming ASR across numerous languages. Employing Efficient Monotonic Multihead Attention (EMMA), it integrates stable estimation techniques, policy regularization, and latency metrics to ensure precise translations.

The model's development involves fine-tuning from SeamlessM4Tv2 through two stages, emphasizing performance evaluations based on quality and latency metrics like Average Lagging (AL) and Length-Adaptive Average Lagging (LAAL). Experimental assessments across speech-to-text and speech-to-speech translations unveiled trade-offs between translation quality and latency, highlighting nuances in performance influenced by linguistic relationships, cultural contexts, and resource levels across languages. The evaluation underscores the model's efficacy in simultaneous translations while revealing its strengths and limitations in handling linguistic diversity and language family influences on translation quality and latency.

Human Evaluation of Expressivity Models

The Human Evaluation in this study focuses on expressivity models using two evaluation protocols: Mean Opinion Score (MOS) and PCP. MOS rates speech quality based on naturalness, sound quality, and clarity. For PCP, bilingual annotators assess source-target audio pairs' similarity in expressive dimensions like rhythm, emotion, and overall explicit intent, along with a semantic dimension.

Each item is evaluated by three annotators, and for PCP, five bilingual annotators rate the similarity between source and target audio pairs. The study uses a modified PCP to accommodate cross-linguistic evaluation, capturing nuances in distant language pairs like English-Mandarin. The assessment includes a set of diverse languages and models, with MOS, applied to a subset of data in the X–E direction. In contrast, PCP evaluations are ongoing in both directions.

Safety Measures in Model Development

The model development process encompassed diverse measures actively assessing and mitigating potential harms in a comprehensive safety evaluation. These included red-teaming efforts to uncover critical errors, the creation of toxicity detectors like Multi-model Toxicity Detector (MuTox), the employment of MinTox for inference-time toxicity mitigation, gender bias quantification, and the implementation of a robust watermarking system for output protection across models like SeamlessM4Tv2 and SeamlessExpressive. The red-teaming sessions involved:

• Generating specific scenarios to expose critical errors in both text and speech outputs.
• Scrutinizing outputs linguistically.
• Correcting labels to ensure accuracy.

Conclusion

To sum up, the SeamlessM4Tv2, SeamlessExpressive, and SeamlessStreaming models represent a significant leap in enabling expressive, real-time cross-lingual communication. Their potential applications span from interactive communication platforms to passive content consumption. However, their ethical deployment and further improvements require careful attention to performance variations, unintended use, and inclusivity. These models hold transformative potential, but their responsible adoption necessitates ongoing vigilance and an inclusive approach to cater to diverse user contexts.