On January 26th, Baichuan AI announced that the Baichuan-Omni-1.5 open-source full-modal model has officially launched. This model not only supports full-modal understanding of text, images, audio, and video but also has the dual-modal generation capability for text and audio.
The official claim is that in areas such as visual, speech, and multi-modal streaming processing, the performance of Baichuan-Omni-1.5 surpasses GPT-4o mini; in the field of multi-modal medical applications, it has a more prominent leading advantage.
Baichuan-Omni-1.5 can not only achieve various interactive operations at the input and output ends but also possesses powerful multi-modal reasoning capabilities and cross-modal transfer capabilities.
It adopts an end-to-end solution in the field of audio technology, which can support multilingual conversations, end-to-end audio synthesis, automatic speech recognition, text-to-speech conversion functions, and also supports real-time audio and video interaction.
According to reports, in terms of video understanding capabilities, Baichuan-Omni-1.5 has significantly surpassed GPT-4o-mini by deeply optimizing multiple key aspects such as encoders, training data, and training methods.
In terms of model structure, Baichuan-Omni-1.5 supports various modalities in the model input section through corresponding Encoders / Tokenizers into a large language model.
In the model output section, Baichuan-Omni-1.5 adopts a design of text-audio interleaved output, generating text and audio simultaneously through Text Tokenizer and Audio Decoder.
Baichuan AI has built a massive database containing 340 million high-quality image/video-text data and nearly 1 million hours of audio data, using 17 million full-modal data during the SFT phase.
SEE ALSO: Baichuan AI Releases Large-scale Model Baichuan 3 with Parameters Exceeding One Trillion
Sign up today for 5 free articles monthly!
