MOSS-Live – ModelThink's Full-Duplex Real-Time Interaction System

1. What is MOSS-Live

MOSS-Live is a full-duplex real-time interaction system developed by ModelThink. Its core breakthrough lies in achieving semantic-level full-duplex dialogue control, allowing AI to dynamically determine the timing for responding, waiting, or accepting interruptions while continuously listening, thereby achieving a human-like natural conversation rhythm. The system employs a front-end and back-end collaborative architecture, decoupling real-time interaction from complex tasks. The front-end model is responsible for continuous multimodal perception of voice, visuals, and ambient sounds, while the back-end model asynchronously performs deep reasoning tasks such as queries and planning, completing its work and proactively feeding back results without interrupting the conversation. This product is developed based on the technical accumulation of the Fudan MOSS team and has demonstrated leading performance in benchmarks such as FDB v1. The API is temporarily free of charge.

Technical positioning and domain: MOSS-Live belongs to the category of real-time voice interaction and multimodal perception systems. Its core direction is to address the limitations of the traditional voice assistant's "question-and-answer" interaction paradigm, pushing human-machine dialogue toward natural full-duplex parallel communication. Its technical capabilities span multiple levels, including semantic understanding, voice activity detection, multimodal fusion, dialogue state tracking, and asynchronous task scheduling, representing the forefront of the transition of real-time interaction systems from "turn-based" to "streaming parallel" models.

Development background: MOSS-Live was developed by ModelThink, leveraging the technical accumulation and R&D experience from the Fudan MOSS large model project. MOSS is one of the early open-source large language models in China, and the team has deep engineering and academic expertise in dialogue systems, instruction fine-tuning, and multimodal understanding. The launch of MOSS-Live can be seen as a significant productization of the team's efforts to extend conversational AI capabilities from text-based interaction to real-time voice full-duplex interaction.

Core value: MOSS-Live targets common user experience pain points in current voice interaction systems, such as "interrupting speech," "false triggering," and "forgetting after interruptions." Through semantic-level interruption judgment and anti-interference mechanisms, the system can distinguish between agreement sounds and actual interruptions or irrelevant ambient noises, and supports resuming speech after multiple interruptions, significantly enhancing the naturalness and continuity of the conversation. The front-end and back-end collaborative architecture design solves the engineering challenge of mutual exclusivity between complex task execution and real-time dialogue, enabling AI to perform information queries and solution planning simultaneously during the conversation.

2. Key Features

• Semantic-level Full-duplex Control: The system analyzes semantics, speaking intent, and conversation state in real-time during continuous listening, dynamically deciding when to respond, wait, or remain silent. This capability is based on a dialogue state tracking model, which performs real-time intent parsing and completion assessment on user speech, rather than relying on simple voice endpoint detection, thereby achieving a smooth parallel speaking and listening experience.

• Front-end and Back-end Collaborative Architecture: The system decouples real-time interaction from deep task processing. The front-end interaction model continuously receives voice, video, and ambient sounds while maintaining the conversation state. The back-end model handles tasks requiring long-range reasoning, such as queries and planning. Both components remain synchronized through shared memory and context states, with the front-end being able to perceive the progress of back-end tasks and proactively provide feedback at the appropriate time after task completion.

• Semantic Interruption and Termination Detection: The system can distinguish between agreement sounds and genuine interruption intent, not being interrupted by short responses such as "um" or "right," while responding promptly when the user truly interrupts. Based on semantic understanding, it determines whether the user has finished speaking, waiting for pauses or thinking gaps within a sentence rather than cutting in. This mechanism achieves a 1.44% interruption rate in the FDB v1 benchmark, outperforming GPT Realtime's 10.37%.

• Irrelevant Sound Rejection: The system has the ability to understand acoustic environments, identifying which speech is directed at the AI and ignoring irrelevant sounds such as bystander conversations, self-talk, or navigation announcements played by devices. In the FDB v1.5 benchmark, its rate of misrecognizing irrelevant speech is 2%, significantly lower than the 68% to 88% range of competing products, effectively solving the problem of false activations in real-world environments for voice assistants.

• Resume from Breakpoints and Context Memory: After multiple interruptions and frequent topic switches, the system can still remember the interruption point and the conversation context, resuming the previous topic naturally at the right time. This capability relies on a structured dialogue state memory mechanism, which persistently maintains interruption points, incomplete intents, and active topics, rather than simply storing historical text.

• Multimodal Real-time Perception: Supports video calls, screen sharing, and ambient sound understanding, the system continuously integrates voice streams, video content, and acoustic environment information for comprehensive decision-making. For example, when a user points to an object in the video, the system can identify the target using visual information; when faced with irrelevant sounds like navigation announcements, it can recognize the source and refuse to respond.

• Chat While Doing Tasks: The system can asynchronously execute multi-step tasks such as queries and planning by calling tools, without blocking the conversation process. Once the task is completed, the results are proactively returned to the conversation. The front-end dialogue state and back-end task state remain connected through shared context, allowing users to continue interacting without waiting, improving the traditional voice assistant's "one question, one freeze" user experience.

• Streaming Simultaneous Interpretation: The system continuously outputs translations with near-human-level low latency, suitable for international conferences and cross-language business negotiations. This feature is implemented using cascaded or end-to-end solutions combining streaming speech recognition and machine translation, generating translations synchronously during continuous speech rather than waiting until the end of a full sentence to translate.

3. Application Scenarios

• AI Meeting Participant: MOSS-Live can act as a virtual attendee, tracking discussions in real-time and understanding multi-turn conversations and topic transitions. After being interrupted, the system can resume the context from the breakpoint; post-meeting, it can automatically summarize discussion conclusions and action items, reducing the burden of manual note-taking.

• In-Vehicle Voice Assistant: In driving scenarios, users can plan trips and check ferry tickets and hotel information via voice while driving. The system is resistant to interference from in-car sounds like navigation announcements and will not be erroneously triggered by device playback; even if the route is interrupted and re-asked multiple times, it can continue from the breakpoint and resume previous planning content.

• Personal Travel Guide: Tourists can point to buildings in front of them using the video call feature within the scenic area, and the system will identify the target based on the image and explain its historical background. During the explanation, users can interrupt at any time to ask about nearby restaurants or hotel prices; the task is executed asynchronously in the background, and the explanation will naturally adjust its direction based on the tourist's interests.

• Cross-Language Simultaneous Interpretation: In international conferences, business negotiations, or cross-language communication scenarios, MOSS-Live continuously outputs translations with near-human-level low latency, supporting streaming translation in scenarios where speakers deliver continuous speech. It is suitable for professional cross-language communication scenarios with high real-time requirements.

• Study Companion and Reminders: During recitation practice, the system can identify errors in real-time and provide corrections, or monitor important matters during meetings and proactively intervene with reminders when user-defined conditions are met. This scenario leverages the system's proactive interaction capabilities; users can customize trigger conditions, and the system continuously monitors the conversation content in the background, automatically initiating prompts when conditions are met.

4. How to Use

1. Environment Requirements: MOSS-Live can be experienced online via the official website, without requiring local deployment. Before use, ensure that a working microphone and camera are available (required for video calling functionality). It is recommended to use modern browsers such as Chrome or Edge to access the official website, to ensure that WebRTC capabilities are properly enabled.

2. Online Experience on the Official Website: Visit the MOSS Intelligence official website at https://www.moss.cn and click on the "Experience MOSS Live" entry. After registering and logging in, the browser will request microphone (and camera) permissions, which you need to authorize by clicking the appropriate button. Once authorized, users can select their preferred voice and begin a full-duplex voice conversation.

3. API Integration: If developers wish to embed MOSS-Live into their own products, they can register for a developer account at https://platform.moss.cn. After creating an application, obtain an API key and integrate the full-duplex interface according to the official documentation. The documentation provides information on interface authentication methods, session establishment procedures, and message format specifications.

4. Feature Configuration: During the experience, users can enable or disable extended capabilities such as video calling, screen sharing, and streaming simultaneous interpretation. The system defaults to enabling semantic endpoint detection and rejecting irrelevant sounds, without requiring manual threshold settings. If users wish to customize active interaction trigger conditions, they can do so using natural language instructions during the conversation.

5. Best Practices: When using the video calling feature, it is recommended to maintain adequate lighting and ensure the camera's field of view is clear, to allow the system to accurately recognize objects in the frame. In noisy environments, the system already has the capability to reject irrelevant sounds, but it is still advisable to speak as close to the microphone as possible to ensure high-quality voice capture.

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

Selection Recommendations: In scenarios requiring strong noise resistance and natural conversation pacing, such as in-car environments, open office spaces, or home settings, MOSS-Live's semantic-level interruption judgment and irrelevant sound rejection capabilities clearly outperform the baseline, offering a better practical experience. In scenarios with high requirements for multi-turn task completion, such as deep information integration and multi-step reasoning tasks, GPT-Live-1 slightly leads in multi-turn task success rate, but at the cost of higher integration costs and weaker noise resistance.

Architecture Considerations: If the application scenario requires parallel execution of conversation and tasks, such as users chatting while the system needs to query information or plan a trip, MOSS-Live's front-end and back-end collaborative architecture is more suitable; whereas GPT-Live-1 typically requires interrupting the conversation to start a new session for long tasks, resulting in a gap in interaction continuity. Additionally, MOSS-Live currently offers time-limited free API access, making it more developer- and small team-friendly.

7. Editor's Summary

MOSS-Live has made a key technical exploration in the field of real-time voice interaction, replacing traditional voice activity detection with semantic-level understanding to drive interaction decisions. Its performance of a 1.44% interruption rate and a 2% irrelevant speech misconnection rate on the FDB v1 benchmark demonstrates, from a quantitative perspective, that the evolution of full-duplex conversations—from "being able to listen and speak" to "knowing when to speak and when not to"—has now achieved engineering feasibility. The architecture design that coordinates the front-end and back-end is also worth noting—it decouples real-time interaction from deep tasks, maintaining the continuity of the conversation and synchronization of task states through shared memory. This approach effectively resolves the contradiction between "dialogue fluency" and "task complexity," offering valuable insights for the architectural evolution of similar systems.

In terms of practical value, the product has clear advantages in scenarios with high demands for real-time performance and noise resistance, such as in-vehicle voice systems, meeting assistance, and cross-language communication. The addition of multimodal perception and streaming simultaneous interpretation further expands the product's scope of application. The limited-time free API access is developer-friendly, reducing the trial-and-error costs of integration and testing. For developers of voice interaction products, smart hardware manufacturers, and industry solution teams requiring real-time voice capabilities, MOSS-Live is worth considering in the technical evaluation process. If it continues to improve in multi-turn task success rates, pricing transparency, and ecosystem development in the future, its market competitiveness is likely to be further strengthened.

8. FAQ

Q: Is MOSS-Live currently free?
A: According to official information, MOSS-Live's API is currently available for free on a limited basis. Users can directly experience the product's features on the official website. However, the billing standards after official commercial use have not yet been announced. Developers are advised to follow official announcements for the latest pricing information.

Q: Does MOSS-Live require local deployment?
A: No. MOSS-Live is currently provided through cloud services, and users can directly access the official website to experience the product online. Developers can integrate it via API. At present, there is no localized deployment solution or model weight availability. Enterprise users with extremely high data privacy requirements should evaluate this limitation.

Q: Which languages does MOSS-Live support?
A: MOSS-Live supports Chinese conversations and has streaming simultaneous interpretation capabilities, enabling continuous low-latency translation output in cross-language communication scenarios. For the specific list of supported language pairs, please refer to the official documentation.

Q: Why isn't MOSS-Live interrupted by sounds like "um" or "ah"?
A: MOSS-Live's semantic-level interruption detection mechanism can distinguish between filler sounds and genuine interruption intent. When it identifies short responses such as "um" or "right," the system determines these are natural fillers within a conversation rather than an active interruption from the user, and thus will not interrupt the current speech. However, when it detects a clear interruption intent, the system will respond promptly and continue the conversation context.

Q: Can MOSS-Live function properly in noisy environments?
A: Yes. MOSS-Live has the capability to reject irrelevant sounds and can distinguish which speech is directed at the AI. It will not respond to idle chatter from others or sounds from devices. In the FDB v1.5 benchmark, its rate of misinterpreting irrelevant speech as a response is 2%, significantly lower than the 68% to 88% range of similar products. However, it should be noted that this data reflects performance in benchmark tests, and actual results may vary depending on the specific environment. It is recommended to conduct real-world testing in the target use scenario.

Q: How can developers enable video calling and screen sharing features in the API?
A: Developers need to create an application and obtain an API key at https://platform.moss.cn. While integrating the full-duplex interface, they should configure the parameters for video streams and screen sharing according to the official documentation. The specific transmission protocols, data formats, and session negotiation methods should be referenced from the official documentation.

9. Project Links

• Product Official Website: https://www.moss.cn/products/moss-live
• Online Experience Entry: https://www.moss.cn
• Developer Platform/API Access: https://platform.moss.cn
• Official Documentation: https://platform.moss.cn (API documentation is available within the developer platform)