Explore how Fugatto’s innovative design and ComposableART technique empower AI to compose sounds, blend tasks, and generate unprecedented audio outputs, from "saxophone meowing" to dynamic voice transformations.
Fugatto 1 Foundational Generative Audio Transformer Opus 1. Image Credit: Shutterstock AI
A recent article posted to the NVIDIA Blog introduced Fugatto (PDF), a versatile audio synthesis and transformation model that follows free-form text instructions. The researchers tackled the challenge of audio-only training data by developing a specialized dataset comprising over 50,000 hours of diverse audio, connecting audio and language in innovative ways.
They also introduced ComposableART, an inference-time technique that extends classifier-free guidance, improving compositional abilities. This technique allows fine-grained control over attributes, enabling users to blend tasks like speech synthesis with background sounds or compose novel instructions such as "a cello shouting." Evaluations demonstrated Fugatto's competitive performance and ability to generate novel, emergent sounds.
Related Work
Past work highlighted the limitations of specialist models in adapting to diverse tasks and data compared to generalist models that scale effectively and demonstrate emergent capabilities. Challenges included the absence of explicit instructions in audio data, making generalization to unseen tasks difficult, and the need for synthetic datasets linking text and audio to capture nuanced relationships.
Generating negation or compositional data was particularly challenging, as such examples are typically unavailable. Additionally, prior methods relied on rigid techniques requiring manual intervention or external classifiers.
Innovative Audio Generation
The approach centers on leveraging advancements in large-scale computing and datasets, mirroring methods in large language models, and using a two-stage process of pre-training and fine-tuning. It introduces new dataset generation built on five pillars: generating diverse instructions with language models, creating absolute and relative tasks, synthesizing audio captions, transmuting existing datasets for novel relationships, and leveraging audio processing tools.
Unlike traditional methods focused solely on unsupervised next-token prediction, the approach incorporates strategies for dataset diversity and proposes new techniques to control audio generation during inference.
The training framework employs template-based and free-form instructions, dynamically constructed to suit various tasks. A T5-based transformer model processes text and audio inputs with adaptive layer normalization and shared embedding spaces for text and audio. The training follows a curriculum learning paradigm, progressively increasing task complexity and employing oversampling to balance underrepresented tasks. This approach enhances the model's adaptability across diverse scenarios.
Finally, the approach introduces ComposableART to extend classifier-free guidance into new domains. This method expands compositional guidance across attributes, models, and temporal contexts, allowing for innovative combinations of tasks and attributes. These innovations push the boundaries of audio generation by combining flexibility with user-defined creativity.
Audio AI Fugatto Generates Sound from Text | NVIDIA Research
Fugatto: Versatile Audio Synthesis
The experiments conducted on Fugatto demonstrate its effectiveness across diverse tasks, showcasing its versatility in audio synthesis and transformation. An ablation study highlighted the impact of various design choices, such as uniform timestep sampling, which proved most effective for text-to-speech synthesis. Larger models exhibited superior emergent capabilities, enabling the generation of novel sounds like "saxophone meowing," emphasizing the importance of parameter scale in achieving these breakthroughs.
Fugatto excelled in text-to-voice synthesis (TTS) and singing voice synthesis (SVS). In TTS benchmarks, it achieved word error rates comparable to expert models, with high speaker similarity and competitive performance against generalist models. For example, it achieved a word error rate (WER) of 2.44 with strong speaker similarity scores. In SVS, Fugatto demonstrated its ability to synthesize singing voices with high alignment to textual prompts despite challenges like higher word error rates.
The model also outperformed many generalist systems in text-to-audio (TTA) benchmarks, particularly in generating sounds for AudioCaps and MusicCaps datasets, occasionally surpassing specialized models in fidelity and accuracy.
Fugatto performed well in speech denoising, bandwidth extension, and modulation in audio transformation tasks. Although it matched specialist models in bandwidth extension, it showed potential for improvement in speech denoising. Fugatto's ability to convert emotions in speech while preserving speaker identity highlights its transformative potential. Furthermore, the model's zero-shot capabilities in musical instrument digital interface (MIDI)-to-audio conversion demonstrated its flexibility, achieving impressive results in synthesizing audio from monophonic melodies.
Fugatto's emergent capabilities and compositionality further emphasize its artistic potential. The model demonstrated emergent tasks and sounds, such as synthesizing a cello shouting or generating monophonic MIDI-based singing voices. Its ComposableART method allowed fine-grained control over synthesized audio, illustrating the ability to steer outputs based on user-defined parameters such as pitch, tempo, and style. These findings reveal Fugatto's capacity to creatively integrate attributes and tasks, positioning it as a powerful tool for artistic and technical audio synthesis and transformation applications.
Conclusion
To sum up, Fugatto demonstrated its versatility in audio synthesis and transformation, overcoming challenges related to audio-only training by introducing a specialized dataset generation approach. The ComposableART technique enhanced its compositional abilities, enabling flexible instruction composition and generating highly customizable audio outputs.
The model performed competitively with specialized systems across diverse tasks, showcasing its potential for creative and technical audio applications. Its ability to synthesize emergent sounds and blend multiple audio events through compositional guidance underscores its innovative design.