In-Depth Review of Sophon-Realtime v1: A Technical Breakthrough and Engineering Practice in Open-Source Full-Duplex Speech Models

1. What is Sophon-Realtime v1

Sophon-Realtime v1 is an open-source full-duplex speech dialogue model developed by Adventists AI. Designed with a 32B parameter scale, it enables complete conversational capabilities, including real-time turn detection while listening, interruption support at any time, understanding of paralinguistic information (tone, emotion), and responding with emotionally expressive speech. The model achieved a score of 82.8 on the VoiceBench speech evaluation benchmark, ranking first among open-source full-duplex models and matching the performance of GPT-4o-mini-Audio. The project offers three versions: Tag (expressive), Flash (fast response), and E2E (research preview). It can be deployed on a single 80GB GPU and is licensed under Apache-2.0, supporting commercial use.

Technical Positioning and Domain: Sophon-Realtime v1 belongs to the intersection of natural language processing and speech interaction, focusing on full-duplex speech dialogue technology. Unlike the traditional three-stage pipeline of "speech recognition - semantic understanding - speech synthesis," this model adopts an end-to-end design approach, performing auditory comprehension, turn detection, and speech generation simultaneously within a unified 32B model. Its positioning in the open-source full-duplex speech model space is quite clear.

Development Background: This project was developed by the Adventists AI team, which has deep expertise in speech interaction. Their technology has been implemented in over 300 hardware products. The motivation for development stemmed from the need to improve the "walkie-talkie" style interaction experience of traditional voice assistants—users must wait for a clear end signal before continuing to speak, which is significantly different from the natural human dialogue pattern of "speaking while listening" and "interrupting at any time."

Core Value: This model addresses the issues of response latency and lack of interruption capability in traditional voice assistant pipeline architectures. Through single-token turn detection and a pre-drafted response mechanism, it achieves a first packet response time of 0.31 seconds and supports interruption at any time. It also features multimodal understanding (charts, screenshots), multi-step tool calling, and emotional expression capabilities, elevating voice assistants from mere "Q&A machines" to intelligent dialogue agents capable of "understanding tone and expressing emotion."

Technical Features: The core technical advantage lies in its design that enables turn state determination with a single token, eliminating the need for external VAD (Voice Activity Detection) and turn detection modules. It supports parallel streaming generation of text and speech, balancing speed and audibility. It also supports integration with backend large models via the OpenAI GPT-Live protocol. The three versions (Tag/Flash/E2E) share the same "brain," differing only in the voice models used for emotional expression. The flexible architecture offers high engineering scalability.

2. Key Features

• Full-duplex conversation: The model continuously listens during the user's speech and can perceive semantics without waiting for silence to end. When the user pauses, the model determines whether the user has finished speaking or is still talking, and can immediately interrupt the current response to answer a new question. This capability is achieved through a single token turn-state judgment mechanism and does not require independent VAD and turn detection modules.

• Paralinguistic understanding (understanding tone): The model can recognize paralinguistic information such as hesitation, emphasis, laughter, sighing, and sarcasm, not just literal content. The auditory model DuplexJev ranks first (48.6 points) across the LISTEN, StressTest, and MSPB paralinguistic benchmarks, outperforming Gemini 3.8 Flash and Qwen3-Omni. This has practical value in emotional computing and dialogue intent understanding.

• Paralinguistic expression (expressing tone): When answering, the model naturally incorporates sound effects such as laughter, sighing, clearing the throat, and breathing, rather than mechanically reading out text. During response generation, the model synchronously annotates sound tags (e.g., [laughter], [sigh], [cough]). The Tag and Flash versions share the same core, with different voice models responsible for "performing" these tags, and the tags are not vocalized.

• Multimodal understanding: Supports input of charts and screenshots, providing verbal interpretations by converting visual information into speech descriptions. Unlike simple OCR, the model understands the data relationships and information hierarchy within charts, making it suitable for scenarios requiring interpretation of data visualization.

• Tool calling: Supports multi-step tool calling, achieving a Pass@1 rate of 48% on the Full-Duplex-Bench v3 benchmark, surpassing Grok (43%). The model can invoke external tools as needed during the conversation to complete tasks and can correctly resume speaking 100% of the time when required. This makes it not only a "chatbot" but also an intelligent agent capable of performing tasks.

• Live mode: Compatible with the OpenAI GPT-Live protocol, allowing collaboration with external large model backends. Sophon handles "listening and speaking," delegating queries and task processing to the backend large model, forming a collaborative architecture of "front-end voice interaction + back-end inference."

• Quick deployment and high concurrency: The service can be launched with a single command and runs on a single 80GB GPU (H100/A100). The Flash version has a first packet latency of just 0.31 seconds and supports 40 concurrent sessions per card; the Tag version has a latency of approximately 0.6 seconds and supports 16 concurrent sessions. Real-world performance testing confirms its engineering viability in production environments.

• Fine-tuning customization: Based on the ms-swift framework, it supports fine-tuning of character traits, tone, domain knowledge, and tool calling habits using the LoRA method. Fine-tuning data is in jsonl format, with audio marked using <audio> tags and sound tags such as [laughter] added to responses. Personalization is achieved while retaining sound tag support and the ability to interject.

3. Application Scenarios

• Smart Customer Service and Call Centers: The Flash version achieves a first packet latency of 0.31 seconds and supports 40 concurrent channels per card, making it ideal for large-scale call center deployments. The model can understand customer emotions (e.g., impatience, confusion) and supports multi-step tool calling to perform real operations such as checking account balances and processing transactions, elevating customer service from "responding" to "executing."

• AI Companions and Virtual Characters: The Tag version offers the strongest expressive capabilities for tone, naturally incorporating laughter, sighs, and clearing one's throat. It is well-suited for roles requiring a "human touch," such as emotional companions, virtual lovers, and game NPCs. The paralinguistic expressions make the dialogue of virtual characters feel more natural, with a sense of "rhythm" and "emotional variation."

• In-Car Voice Assistants: The full-duplex feature allows users to interrupt at any time without needing a wake-up word, and the low response latency is suitable for driving scenarios. The model can interpret tone and help determine whether the driver is fatigued or distracted, opening up new possibilities for in-car interaction.

• AI Hardware Terminals: The team's technology has been implemented in over 300 hardware products, and the Apache-2.0 license allows commercial use, making it suitable for integration into smart speakers, AI toys, and service robots. The single-card deployment and fine-tuning capabilities reduce the customization barriers for hardware manufacturers.

• Enterprise Voice Agents: The Live mode is compatible with the GPT-Live protocol, with Sophon handling "listening and speaking," while backend large models manage queries and tasks. It is suitable for customized voice services in industries such as banking, government affairs, and aviation, enabling flexible access to enterprise knowledge bases and business systems through backend large models.

4. How to Use

1. Environment Requirements: Ensure the system is Linux + CUDA + Python 3.12, and equipped with at least one GPU with 80GB of memory (recommended H100 or A100) for model inference. The model weights are approximately 42GB for the first download (Tag version), so ensure sufficient disk space is available.

2. Installation and Startup: Run pip install sophon-realtime to install the Python package. After installation, execute sophon-realtime serve to start the service. The first launch will automatically download approximately 42GB of Tag weights, and then the service will listen on port 8000, waiting for connections.

3. API Calling: After the service is started, perform full-duplex voice interaction via the WebSocket interface ws://HOST:8000/v1/realtime, or send voice and receive "text + voice" responses via the REST API POST /v1/chat/completions. The two interfaces correspond to real-time conversation and request-response modes, respectively.

4. Test Conversation: Run python client_example.py --wav my_question.wav --lang zh --voice vivian_zh to send a voice file and listen to the response. The --lang parameter specifies the language, and the --voice parameter selects the voice (e.g., vivian_zh, chelsie, etc.).

5. Integrate with Live Mode (Optional): Run cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh alongside the engine to use the OpenAI GPT-Live protocol to connect your backend large model. Sophon handles voice interaction, while the backend handles inference processing.

6. Fine-tuning and Customization (Optional): Train using ms-swift with LoRA on an 80GB GPU. Prepare dialogue data in jsonl format, mark audio with <audio> tags, and include sound tags such as [laughter] in the responses. Execute the swift sft command to adjust the persona and domain knowledge. The fine-tuning process retains sound tags and full-duplex conversation capabilities.

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

Selection Recommendations: If the core requirement is high-performance full-duplex voice conversation, low-latency response, and emotional expression, and if cloud GPU deployment is available, Sophon-Realtime v1 is currently the best choice among open-source models. Its VoiceBench score of 82.8 and 0.31-second first packet response latency place it ahead of other open-source models, and the Apache-2.0 license provides flexibility for commercial use. If deployment is constrained by consumer-grade hardware or requires on-device support, MiniCPM-o 4.5's INT4 quantization and llama.cpp-omni on-device adaptability offer greater advantages. For scenarios requiring stronger multimodal capabilities (e.g., complex document understanding), the top performance of MiniCPM-o 4.5 on the OmniDocBench is worth considering. If you are already deeply integrated with the Google ecosystem and do not rely on open-source deployment, Gemini 3.8 Flash can be included in the evaluation as a commercial API option.

7. Editor's Summary

Sophon-Realtime v1 demonstrates comprehensive technical capabilities in the open-source full-duplex speech model domain. From the perspective of technological innovation, its single token turn-taking judgment mechanism and architecture design that pre-drafts responses significantly simplify the traditional voice assistant pipeline. By consolidating the responsibilities of modules such as VAD and turn-taking detection into a unified model, it reduces system complexity while improving response speed. The voice label mechanism on the speech expression level, such as [laughter] and [sigh], enables the model to go beyond "generating text and then synthesizing speech," instead planning paralinguistic expressions from the generation stage. This design is noteworthy from an architectural standpoint. In terms of empirical data, the 82.8 VoiceBench score, 81.8% speech inference accuracy on Big Bench Audio, and 48% Pass@1 tool calling performance on Full-Duplex-Bench v3 are all relatively high levels among open-source models. In terms of practical value, the Flash version's 0.31-second first packet latency and 40 concurrent sessions on a single GPU provide an engineering foundation for production deployment. The Live mode's compatibility with the GPT-Live protocol reduces integration costs with existing large model architectures. Objectively speaking, the hardware requirement of 80GB of VRAM presents a barrier for small and medium-sized teams, and the fixed voice scheme (no cloning support) limits its applicability in commercial scenarios requiring specific voice characteristics. The new project ecosystem also requires ongoing development. This model is suitable for teams with cloud GPU resources and a need for expressive speech interaction, as well as developers aiming to integrate speech capabilities into hardware products or customer service systems. In the future, if it continues to improve in hardware compatibility, voice customization, and on-device inference, its position within the speech agent ecosystem will become more stable.

8. FAQ

Q: What hardware configuration is required to run Sophon-Realtime v1?
A: It is recommended to use a single NVIDIA GPU with 80GB VRAM (H100 or A100). The system requirements are Linux + CUDA + Python 3.12. The Flash version has faster response times (first packet in 0.31 seconds), while the Tag version has stronger expressiveness (approximately 0.6 seconds). Both versions share the same core model. The current version does not support consumer-grade GPUs.

Q: What are the differences between the Tag, Flash, and E2E versions?
A: All three versions share the same 32B "brain" (the text generation and turn-taking judgment components), with differences lying in the audio model. The Tag version has the strongest expressiveness and can naturally demonstrate paralinguistic effects such as laughter, sighing, and clearing the throat. The Flash version has faster response times (first packet in 0.31 seconds) and higher concurrency (40 streams per GPU). The E2E version is a research preview, intended for end-to-end architecture studies.

Q: How does Sophon-Realtime v1 compare to GPT-4o-mini-Audio?
A: On the VoiceBench benchmark, Sophon-Realtime v1 scores 82.8, matching GPT-4o-mini-Audio and being the highest among open-source full-duplex models. In terms of paralinguistic understanding (DuplexJev benchmark), it outperforms Gemini 3.8 Flash and Qwen3-Omni. However, as a commercial closed-source model, GPT-4o-mini-Audio still has advantages in multimodal generalization and ecosystem integration.

Q: How can I integrate my own backend large model?
A: Sophon-Realtime v1's Live mode is compatible with the OpenAI GPT-Live protocol. To do this, run the command cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh alongside the engine. Sophon handles "listening and speaking," while query processing and task handling are delegated to the backend large model, forming a collaborative architecture that decouples voice interaction from reasoning.

Q: Can the model be fine-tuned using your own audio data?
A: Yes. Using the ms-swift framework, you can fine-tune the model on an 80GB GPU with LoRA. You can adjust the character persona, tone, domain knowledge, and tool calling habits. Training data should be in jsonl format, with audio marked using the <audio> tag and responses including sound tags such as [laughter]. The fine-tuned model retains sound tags and conversation capabilities.

Q: Does it support languages other than Chinese?
A: It supports multiple languages. The --lang parameter in the client test command can specify the language (e.g., --lang zh for Chinese), and multiple fixed voice styles are available (e.g., vivian_zh, chelsie). On the NVV-SuperBench, the Chinese score is 0.86 and the English score is 0.80, with data supporting performance in both Chinese and English.

Q: What licensing considerations should be taken for commercial use?
A: Sophon-Realtime v1 uses the Apache-2.0 license, with both code and weights open-sourced, allowing free commercial use. This license permits modification, distribution, and commercial use, but requires retaining the original copyright notice and does not provide patent protection or liability guarantees. Compared to some projects that only open-source code and not weights, its commercial friendliness is higher.

9. Project Links

• GitHub Repository: https://github.com/adventists-ai/sophon-realtime