Qwen3.8-LiveTranslate – A Real-Time Simultaneous Interpretation Model Launched by Alibaba Tongyi

1. What is Qwen3.8-LiveTranslate

Qwen3.8-LiveTranslate is a real-time simultaneous interpretation large model launched by the Tongyi Qianwen team at Alibaba. Based on the Interleave single-stream architecture, it processes audio and text in an interleaved manner, reducing the word-level latency from 2.8 seconds to 2.3 seconds. It supports recognition in 60 languages and voice output in 29 languages. The model employs a Hybrid MoE-based Thinker-Talker dual-module design, which completes speaker separation before translation and can replicate the original speaker's voice to generate the translated speech. It also supports video/image input to assist in translation disambiguation, achieving synchronized output of source text and translated text on the same frame. This model has been made available for API calls through the Qwen AI platform, primarily targeting real-time translation scenarios such as international conferences, cross-border live streaming, and video localization.

Image source: Official article
Image source: official article

Technical positioning and domain: It belongs to the field of multimodal real-time translation and is at the forefront of real-time simultaneous interpretation technology. Unlike traditional two-stage cascaded translation systems (which first perform speech recognition and then text translation), Qwen3.8-LiveTranslate directly generates translated text and speech from the audio stream in an end-to-end manner, modeling simultaneous interpretation as a single-stream prediction task that interweaves audio and text. This gives the model a unique technical positioning in optimizing simultaneous interpretation latency.

Development background: This model was developed by the Tongyi Qianwen team at Alibaba. The Tongyi Lab has accumulated substantial technical expertise in multimodal and speech technologies on the Qwen series of large models. Previous full-modal models such as Qwen2.5-Omni and Qwen3-Omni provided the technical foundation for the architecture design of LiveTranslate. The motivation for development was to address the high latency caused by the serial processing of listening, translating, and speaking in real conferences, with specialized optimization for simultaneous interpretation scenarios, rather than a byproduct of general-purpose voice assistants.

Core value: This model solves the high latency issue caused by the serial processing of "listening," "translating," and "speaking" in traditional simultaneous interpretation solutions, compressing the word-level latency to 2.3 seconds. It integrates features such as speaker separation, voice cloning, and bilingual alignment output into the end-to-end process, significantly reducing the labor costs of human interpreters. At the same time, it provides an extensible automated translation solution for content scenarios such as cross-border conferences and live streams.

Technical features: The Interleave single-stream architecture reuses cached audio that has already been listened to and translated text that has already been generated, avoiding recomputation from scratch for each sentence. The Thinker-Talker dual-module design performs understanding and translation within a unified sequence, and then synthesizes the translated speech using the source audio to preserve the speaker's voice. Combined, these two technologies form the model's differentiated advantages in low latency and voice fidelity.

2. Key Features

• Low-latency Real-time Interpretation: Based on the Interleave single-stream architecture, the model processes audio and text as an interleaved causal sequence. Both the audio that has been heard and the translated text that has already been generated are cached and reused, avoiding recomputation from the beginning of each sentence. This reduces the latency per word to 2.3 seconds, a decrease of approximately 18% compared to previous solutions. This metric brings the real-time interpretation experience in live translation scenarios close to "translate as you listen," significantly improving the fluency of cross-lingual communication.

• Real-time Speaker Separation: In meeting scenarios with multiple speakers taking turns, the model performs Diarization (speaker logging) before translation, automatically attributing each sentence to a specific speaker without requiring pre-registered voiceprints. This capability enables translated text to be grouped by speaker, maintaining clear speaker attribution in transnational meeting minutes and multi-turn dialogues, and preventing information confusion.

• Voice Cloning and Translated Speech Synthesis: When generating the translated speech, the Talker module incorporates source audio characteristics, cloning the voice of each speaker individually. This feature allows the translated speech to be "recognized as coming from a specific speaker," preserving the speaker's identifiability in multi-speaker scenarios and overcoming the limitations of traditional translation systems that use a uniform voice output.

• Synchronized Bilingual Output in the Same Frame: The architecture aligns the source text and translated text within the same interleaved sequence, naturally supporting synchronized display in the same frame. The aligned bilingual results can be directly used in downstream applications such as subtitle generation, content organization, and retrieval, without requiring additional timeline alignment or post-processing steps, greatly simplifying the localization production workflow.

• Long Context Semantic Disambiguation: The model incorporates historical conversation into its context modeling, linking previous information across turns to make translation decisions. This resolves ambiguity in proper nouns, names, and references that often arise under single-sentence judgment. In long conversations with multiple speakers, terminology and expressions remain consistent throughout, and translation accuracy does not degrade over time.

• Broad Multilingual Support: The model supports recognition input in 60 languages and speech output in 29 languages, covering major international and regional languages. The decoupled design of recognition and output languages provides flexible combinations, such as translating rare language recognition results into common languages for output, adapting to different regional content distribution needs.

• Visual Information Assisted Disambiguation: Supports video and image input, allowing the model to use visual content to assist in translation decisions. In live streaming or video conferencing scenarios, physical objects, scenes, or text in the video can serve as additional contextual signals, helping resolve reference ambiguities that cannot be addressed with audio alone, thereby enhancing translation accuracy.

3. How to Use

1. Environmental Requirements: For the web-based online experience, no software installation is required. It is compatible with mainstream browsers such as Chrome and Edge, and requires a stable internet connection and permission to access the browser's microphone. For API calls, you need to register for an account on the Qwen AI platform and activate the corresponding model service. It is recommended that the network environment supports WebSocket long connections to ensure a stable streaming transmission experience.

2. Online Experience Process: Open the online experience URL (https://omni.qwen.ai/live-translate), and the browser will request microphone access permission. You must click "Allow." On the interface, select the source language and target language (supporting 60 recognized languages and 29 output languages), and then start speaking directly. The page will display bilingual subtitles in real time and play translated speech that retains the original speaker's voice.

3. API Integration Process: Log in to the Qwen AI platform to obtain an API Key, and call the model's real-time interface ((link to be updated after official release))

4. Key Configuration Notes: When calling the API, you need to configure the source and target language code parameters. Additionally, you can choose to enable speaker separation and decide whether to return the translated speech audio based on your scenario needs. For video translation scenarios, you can also input a sequence of video frames to activate visual-assisted disambiguation functionality. In streaming sessions, it is recommended to set a reasonable silence detection threshold to balance response speed and translation completeness.

5. Best Practices and Notes: In multi-person meeting scenarios, it is recommended to use lapel microphones or array microphones to improve the accuracy of speaker separation. In environments with background music or noticeable noise, consider adding pre-processing noise reduction in the audio chain. Pay attention to the trade-off between simultaneous interpretation latency and context length—longer historical conversations can help with disambiguation, but exceeding a certain number of turns may introduce redundant information. It is advised to set the context window according to the actual scenario.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For professional scenarios requiring real-time simultaneous interpretation and emphasizing speaker differentiation and voice retention (such as international video conferences, remote court proceedings, and international exhibitions), Qwen3.8-LiveTranslate is currently the most functionally aligned solution. While GPT-4o excels in general conversational capabilities, it lacks publicly comparable data for simultaneous interpretation metrics and does not support voice cloning or bilingual frame-synchronous output. For business scenarios where document translation and website localization are the primary needs and real-time performance is not critical, DeepSeek-Translator or Azure AI Translation offers greater advantages in language coverage and batch processing capabilities. Therefore, the choice should be based on the actual business requirements for simultaneous interpretation latency and speaker differentiation, rather than a single-dimensional comparison.

6. Editor's Summary

Qwen3.8-LiveTranslate demonstrates the deep technical exploration of the Aliyun Tongyi team in the vertical field of simultaneous interpretation. The introduction of the Interleave single-stream architecture represents a notable architectural innovation—redefining simultaneous interpretation from the traditional "recognition-translation-synthesis" serial pipeline into an audio-text interleaved single-sequence prediction. This directly results in a substantial improvement in the average word latency, reducing it from 2.8 seconds to 2.3 seconds. The design of the context caching mechanism also reflects a deep understanding of the real computational bottlenecks in simultaneous interpretation scenarios.

The Thinker-Talker dual-module Hybrid MoE structure decouples understanding and translation from speech synthesis while unifying optimization. This enables voice cloning while maintaining the advantages of an end-to-end system, and this design holds practical engineering value in multi-speaker meeting scenarios.

In terms of practical value, the model's bilingual same-frame output and speaker separation capabilities directly address pain points in meeting simultaneous interpretation and content localization processes, reducing the costs associated with timeline alignment and manual post-processing. Its coverage of 60 languages also provides a flexible combination space for different regional markets. Currently, the model is only available as an API service, with weights not open-sourced, which limits the possibility of customized deployment. As the API matures and cost models become more transparent, this model is expected to establish a stable application ecosystem in real-time scenarios such as cross-border communication and exhibition services. For developers and enterprise users, the current API solution already has the foundational conditions to integrate into real business workflows, making it suitable for verification deployment in medium-scale real-time translation scenarios.

7. Application Scenarios

• Cross-border Video Conferencing: In multi-party international meetings, the model automatically distinguishes the voices of each speaker, completes the speaker log before translation, and generates translated speech while preserving the original voice characteristics of each speaker. Attendees can simply select their target language to hear the corresponding translated speech, "able to tell who is speaking," while the screen displays bilingual synchronized subtitles, making meeting records and information attribution clear at a glance. This scenario is latency-sensitive, and the average word latency of 2.3 seconds is close to the pace of human interpretation, significantly reducing communication costs in cross-border collaboration.

• Cross-border Live Streaming and Content Globalization: For live streaming scenarios, the model generates bilingual synchronized subtitles and translated speech in real time, helping live content reach global audiences without barriers. With the aid of visual information for disambiguation, product displays or textual information in the live stream can assist in translation decisions, reducing ambiguity. The platform can directly integrate bilingual subtitles into the player or streaming pipeline, without requiring additional alignment processing, simplifying the technical workflow for live streaming localization.

• Video Localization and Subtitle Generation: Simply input a video to generate aligned bilingual subtitles and voiceover, without the need to pre-prepare subtitle tracks. Since the model natively outputs bilingual alignment results, it can be directly used for subtitle rendering or voiceover replacement, greatly simplifying the translation process for films, courses, and corporate promotional videos, and reducing the human and time costs of localization production.

• International Exhibitions and Business Negotiations: In scenarios such as international exhibitions and cross-border business negotiations, both parties can access the model's services via mobile devices to receive low-latency real-time interpretation support, reducing reliance on on-site professional interpreters. Speaker separation capabilities accurately distinguish individual speakers in multi-party discussions, while voice cloning ensures the translated speech retains the original speaker's recognizability, enhancing the naturalness of communication.

• Customer Service for Overseas Enterprises and Global Training: Supports real-time voice consultations for overseas customers and cross-lingual employee training. The customer service system can connect real-time calls to the model's API, translating customer speech into the native language of the customer service representative while preserving the customer's voice characteristics for playback. The representative responds in their native language, which is then translated back into the customer's language for output, enabling natural two-way communication. In global team training, the same session can simultaneously output subtitles and voiceovers in multiple languages, improving the efficiency of content delivery.

8. FAQ

Q: How is the 2.3-second average latency per word for Qwen3.8-LiveTranslate defined?
A: The 2.3 seconds here refers to the LAAL (Latency per Word) metric, which measures the average time difference between when a speaker utters a word and when the translated word is output. This metric evaluates the end-to-end simultaneous interpretation latency, encompassing all stages from audio input, understanding, translation, to speech synthesis. Compared to traditional cascaded approaches (three-step sequential process: speech recognition, text translation, speech synthesis), which typically have end-to-end latencies of over 3 seconds, the Interleave single-stream architecture significantly reduces this time by interleaving audio and text processing.

Q: Is speaker enrollment required before the meeting for speaker diarization?
A: No. The model automatically performs Diarization before translation, which involves clustering and attribution of multiple speakers based on audio features. This is a speaker logging technique that does not require pre-registration. The model automatically distinguishes between different speakers based on their voice characteristics and assigns each a unique identifier, maintaining consistency during the translation process. No prior enrollment of speaker templates is needed for each participant.

Q: What calling methods does the current API support?
A: The official API provides a real-time interface via WebSocket long connection, which is suitable for continuous audio streaming. Developers can also refer to the model documentation to submit short audio using HTTP. For real-time simultaneous interpretation scenarios, it is recommended to use WebSocket to reduce connection setup overhead and ensure low-latency streaming. Specific connection parameters and code examples can be found on the Qwen AI platform model detail page.

Q: Can the model be used directly for offline translation of recorded videos?
A: Yes. The model supports video and image input and can directly translate recorded videos, generating aligned bilingual subtitles and voiceovers. This is ideal for localization of video content such as films and courses. Since offline video translation is not sensitive to latency, the entire video file can be input at once, resulting in aligned bilingual outputs that can be directly used for subsequent subtitle rendering or voiceover synthesis.

Q: Which languages does the model support for recognition and speech output?
A: The model supports recognition in 60 languages, covering major international and regional languages. It supports speech output in 29 languages. The smaller number of output languages compared to input languages means that recognition results for some less common languages can only be output in text form, and cannot be synthesized into the corresponding language's speech. The specific language list can be found on the Qwen AI platform model detail page.

9. Project Links

• Product Official Website: https://www.qianwenai.com
• Online Experience Entry: https://omni.qwen.ai/live-translate
• API Interface Documentation: https://www.qianwenai.com/models/qwen3.8-livetranslate-flash-realtime