MiDashengLM-Spatial – Xiaomi's Open-Source Spatial Audio Understanding Large Model

1. What is MiDashengLM-Spatial

MiDashengLM-Spatial is an open-source spatial audio understanding large model developed by Xiaomi's Dasheng team. It is the first industry-leading end-to-end model that unifies general audio understanding and spatial perception within a single architecture. Built upon the MiDashengLM-7B base model, it introduces the Spatial-Dasheng spatial audio encoder and the Hierarchical Semantic-to-Spatial Conditioning (HSSC) module, enabling capabilities such as sound source localization, motion trajectory tracking, and spatiotemporal relationship reasoning among multiple sound sources. It also supports frame-by-frame localization of overlapping sound sources. In terms of training data, the model was trained using 1 million simulated spatial acoustic scenarios and 6 million QA data points. It is released under the Apache 2.0 license and can be used freely for commercial purposes. It will be first deployed in smart home and intelligent vehicle cabin scenarios.

Technical positioning and domain: Belongs to the audio understanding direction within the field of multimodal large models, focusing specifically on binaural spatial audio perception. It fills the gap in the spatial auditory capabilities of general audio large models and represents a key technological milestone in the transition from "understanding content" to "understanding direction" in the audio understanding domain.

Development background: Developed by Xiaomi's Dasheng team (xiaomi-research), which has deep expertise in speech recognition and audio large models. The motivation for development stems from the need for audio understanding in real-world scenarios to not only transcribe content but also determine sound source direction and movement, meeting the auditory perception requirements of applications such as intelligent cockpits, robots, and wearable devices.

Core value: Addresses the core pain point that previous audio large models could only understand the content of sounds but could not perceive their spatial attributes. The model achieves an accuracy of 70.3% on the spatial subset of the STAR-Bench benchmark, approaching human-level performance (73.7%), while maintaining the original performance of the base model on general audio understanding tasks, achieving a win-win in both spatial perception and general understanding.

Technical features: Employs a parallel dual-branch architecture combining "semantic branch" and "spatial branch," using a hierarchical semantic-to-spatial conditioning mechanism to guide spatial modeling with semantic information. It performs frame-by-frame sound source localization based on binaural input and acoustic cues such as ITD/ILD, capable of handling complex scenarios with multiple people speaking simultaneously. Additionally, the model trained on synthetic data demonstrates robust Sim-to-Real generalization capabilities on real binaural recordings.

2. Key Features

• Sound Source Localization: The model can determine the spatial location of a sound source, including positions such as left, right, and directly in front. This capability is based on acoustic cues such as inter-aural time difference (ITD) and inter-aural level difference (ILD), and is implemented through the Spatial-Dasheng encoder for frame-by-frame detection and localization. It achieved an accuracy of 72.6% in the Channel-Swap ACR test, demonstrating that its localization ability is based on real auditory spatial cues rather than semantic guessing.

• Motion Trajectory Perception: The model can perceive the movement direction and trajectory of a sound source in space. By performing temporal modeling on binaural audio sequences, the Spatial-Dasheng encoder captures the continuous changes in the sound source's position, enabling it to infer whether the sound is moving left, right, or circumnavigating, thus providing a crucial capability for spatial understanding in dynamic scenarios.

• Group Spatiotemporal Relationship Reasoning: The model can reason about the relative spatial relationships between multiple sound sources, such as determining "who is more to the left," "whether two sources are on the same side," or "which sound is closer." This capability relies on the Hierarchical Semantic to Spatial Context (HSSC) mechanism, allowing spatial modeling to leverage semantic context for fine-grained spatial relationship judgments, such as "two people are conversing on the left."

• Spatialized Content Understanding: By jointly reasoning across the two dimensions of "what was heard" and "where the sound came from," the model can answer questions that require spatial information. For example, in a multi-speaker conversation scenario, the model can determine "which speaker ordered first," associating the transcribed speech content with the spatial location of the speaker to achieve deeper scene understanding.

• Spatial Audio Description: The model supports the generation of complete spatial descriptions for all sound sources in open-ended scenarios, including their positions, motion states, and temporal relationships between sources. This feature can generate comprehensive descriptive text such as "a person is walking from the right to the left, while another person is speaking from a distance," which is applicable to scene analysis, content annotation, and accessibility information presentation.

• Spatial Speech Transcription: The model transcribes multi-speaker speech content in the order of speaking, and annotates the spatial location of each speaker for every transcribed sentence. This feature addresses the pain point of traditional speech recognition systems that cannot distinguish "who is speaking," and has direct application value in scenarios such as intelligent meeting transcription and cabin speech interaction.

3. How to Use

1. Environment Requirements and Prerequisites: To use MiDashengLM-Spatial, you need to prepare an NVIDIA GPU environment that supports CUDA, with a recommended VRAM of no less than 16GB. The operating system supports Linux and Windows, and the Python version requirement is 3.13 or higher. Model inference relies on PyTorch and the HuggingFace Transformers library, which must be installed and configured in advance.

2. Clone the Code Repository: Clone the project code to your local machine using the Git command:

   git clone https://github.com/xiaomi-research/midashenglm-spatial.git
   

   After entering the project directory, you can view the example/inference.py example file to understand the complete inference process.

3. Create an Inference Environment and Install Dependencies: Create an isolated Python environment using Conda and activate it:

   conda create -n midashengspatial python=3.13 -y && conda activate midashengspatial
   

   Then, run pip install -r requirements.txt in the project root directory to install all required dependencies, including libraries for model loading, audio processing, and inference.

4. Obtain Model Weights: There is no need to manually download weight files separately; the code will automatically pull the mispeech/midashenglm-spatial model from HuggingFace. When running for the first time, ensure that the network can access HuggingFace services normally. The model loading process may take several minutes.

5. Prepare Binaural Audio Input: The audio input required by the model is real binaural audio with a shape of (2, T), representing the temporal signals of the left and right channels. Ordinary mono audio or stereo audio that is not binaurally recorded does not meet the model's input requirements. You must obtain such audio through binaural recording with a dummy head or binaural synthesis techniques.

6. Load the Model and Processor: Use the AutoModelForCausalLM and AutoProcessor interfaces from the Transformers library to load the model and its corresponding processor with the parameter trust_remote_code=True. This parameter allows the execution of custom code in the repository, enabling the loading of the model architecture.

7. Construct Dialogue Messages and Execute Inference: Organize messages in the "audio first, then text" training order, for example, pairing audio input with instructions such as "Write a general audio caption describing the sound source and its spatial information." After processing the input with processor.apply_chat_template, call model.generate to perform inference, and finally use batch_decode to decode the output text results.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the core requirement is spatial audio understanding, including sound source localization, motion trajectory perception, and multi-source spatial relationship reasoning, MiDashengLM-Spatial is currently the most accurate open-source option. It not only provides full spatial capabilities but also validates the authenticity of perception through Channel-Swap. Although BAT-7B has explored binaural audio processing, its spatial reasoning capability is essentially absent, making it unsuitable for spatial understanding tasks.

If focusing on the overall performance of general audio understanding and multimodal tasks: Gemini-3.6-Flash demonstrates comprehensive capabilities in general tasks, but its spatial perception is significantly weaker than specialized models. Additionally, it is a closed-source commercial service, which raises concerns about data privacy and usage costs. SALMONN is suitable for academic research scenarios requiring general audio understanding but with less emphasis on spatial capabilities. Its Apache license also allows commercial use, but it lags behind MiDashengLM-Spatial in the spatial perception dimension.

6. Editor's Summary

The release of MiDashengLM-Spatial marks a significant turning point in the transition of spatial audio understanding from academic experimentation to practical engineering implementation. From a technological innovation perspective, its dual-branch architecture design reflects a structurally rational consideration—by extending a spatial encoder in parallel on the existing language model foundation, it retains the mature capabilities of general audio understanding while achieving targeted enhancement of spatial perception. The elegance of the HSSC conditional mechanism lies in its construction of a unidirectional information flow channel from semantics to space, allowing spatial modeling to leverage semantic context without interfering with the semantic branch. This "separate yet integrated" architectural design is technically replicable and provides a new reference paradigm for the capability expansion of multimodal models.

In terms of practical value, the model's 70.3% accuracy on the STAR-Bench spatial reasoning subset and a Channel-Swap ACR of 72.6% under real binaural conditions demonstrate that its spatial perception capabilities are already practically usable. Particularly notable is the fact that its general capabilities remain unaffected, addressing a common pain point in many multimodal models where the addition of new capabilities often leads to degradation of existing ones. This reduces the migration cost for adopters. With the Apache 2.0 license, MiDashengLM-Spatial can be directly integrated into commercial products under compliance conditions, making it highly attractive for applications in intelligent cockpits, robotic perception, and assistive devices.

The target audience includes three categories: first, product managers and algorithm engineers in the field of intelligent hardware, who can quickly validate the feasibility of spatial voice interaction in cockpit and home scenarios; second, researchers in the audio and multimodal domains, who can use the dual-branch architecture and HSSC mechanism as baseline references for future studies; third, developers of assistive technologies, who can leverage its directional perception capabilities to build auxiliary applications that help hearing-impaired individuals understand their environment.

In terms of future development potential, the model is currently based on a 7B parameter scale, and scaling up to larger parameter versions is expected to further increase the upper limits of spatial and semantic capabilities. Additionally, extending spatial perception capabilities from binaural audio to arbitrary microphone array configurations will be an important direction for broadening its application scope. As the spatial audio data pipeline continues to improve, the technical approach represented by MiDashengLM-Spatial is likely to become a standard capability configuration in the next generation of audio understanding systems.

7. Application Scenarios

• Intelligent Cockpit Zonal Voice Interaction: Deploying MiDashengLM-Spatial in automotive cockpits enables the system to determine the source of sound—whether it's from the driver's seat, front passenger seat, or rear passenger seat—thereby achieving zonal voice command recognition. For example, if the driver says "Turn on the AC" and the rear passenger says "Play music," the system can automatically distinguish the command issuer based on the sound source location, significantly improving voice interaction accuracy in multi-passenger scenarios and providing a perceptual foundation for personalized cockpit experiences.

• Smart Home Acoustic Perception: By deploying the model on smart speakers or home control devices, the system can perceive the location and movement trajectory of sounds within the home, such as determining whether a sound comes from the living room or bedroom, or detecting someone moving from the kitchen to the living room. This capability can be used for elderly care (detecting fall locations or calls for help), security monitoring (identifying the location of abnormal sounds), and smart home voice control联动 (directed responses based on the speaker's location).

• AR/VR Spatial Audio Interaction: In virtual reality and augmented reality scenarios, MiDashengLM-Spatial provides spatial understanding at the auditory level within the virtual world. The system can locate the position of sound sources in the virtual environment and generate spatial description text, allowing the user's virtual avatar to react accordingly based on the sound's location, enhancing immersive interaction experiences. This is applicable to virtual meetings, immersive gaming, and spatial audio content creation.

• Hearing Aid and Accessibility Assistance Devices: Helps hearing-impaired individuals perceive the direction of sound sources, enhancing environmental awareness and travel safety. After the device captures binaural audio from the environment, the model can provide real-time directional information such as "A vehicle is approaching from the right" or "Someone is speaking behind you," transmitting spatial auditory information in visual or tactile feedback formats to users, compensating for the lack of spatial perception caused by hearing impairments.

• Intelligent Meeting and Multi-Speaker Recording: In multi-participant meeting scenarios, the model transcribes content in speaking order and labels the location of each speaker. It also infers "who spoke, when, and where." Combined with spatial trajectory perception, it can reconstruct changes in participants' positions during the meeting, providing more complete contextual information for meeting minutes. This is suitable for remote collaboration, interview recordings, and classroom recordings.

• Robot Auditory Perception System: Integrating MiDashengLM-Spatial into service robots and inspection robots enables them to locate the position of speech and sounds in the environment and track the movement trajectory of moving sound sources. Robots can autonomously turn toward the sound source direction for visual confirmation based on the sound's location, improving the naturalness of human-robot interaction and task execution efficiency.

8. FAQ

Q: What is the fundamental difference between MiDashengLM-Spatial and conventional audio understanding models?
A: Traditional audio understanding models (such as ASR systems or general-purpose audio large models) only process the "content" dimension, i.e., identifying what is being said or what sounds are occurring. MiDashengLM-Spatial adds the "spatial" dimension on this basis, enabling it to determine the location of sound sources, their movement trajectories, and the relative positions of multiple sound sources. Its core distinction lies in the introduction of the Spatial-Dasheng spatial encoder and the HSSC conditioning mechanism, forming a dual-branch architecture that processes both semantic and spatial information in parallel, thereby achieving simultaneous understanding of "what is heard" and "where the sound comes from."

Q: Does the model support Chinese audio input?
A: According to official information, the model's training data and benchmark evaluations are primarily focused on English scenarios. The model's capability for spatial audio understanding in Chinese contexts has not been disclosed. Technically, the base model MiDashengLM-7B has the potential for multilingual understanding, but if the training data for spatial capabilities is predominantly English, the spatial understanding performance in Chinese scenarios may not be as stable as in English. It is recommended that Chinese users conduct small-scale testing and validation before actual deployment.

Q: Can regular stereo audio be used directly as input for the model?
A: No. The model requires real binaural audio input in the shape of (2, T), which includes complete ITD/ILD spatial cues obtained through artificial head recording or binaural synthesis techniques. Regular stereo recordings are mixed outputs and do not contain real interaural time differences (ITD) or interaural level differences (ILD) information between the left and right channels, thus failing to provide effective spatial localization cues for the model.

Q: Can the model be used in commercial products? Are there any copyright risks in the training data?
A: Yes, it can be used commercially. The model is released under the Apache 2.0 license, which allows free use, modification, and commercial deployment without requiring payment for licensing fees. Regarding the training data, the official documentation states that the spatial scenario data was constructed through a simulation pipeline, making it synthetic data and not involving real recordings, thereby avoiding copyright issues. However, users are advised to review the full terms of the Apache 2.0 license and monitor any future updates before specific commercial deployment.

Q: How much GPU resource is required to infer a single binaural audio sample?
A: The official documentation has not disclosed precise inference resource consumption data. Based on the model's parameter scale of 7B and its dual-branch architecture, FP16 precision inference would require at least 16GB of GPU memory. It is recommended to run the model on high-end GPUs such as the RTX 4090 or A100. For low-latency real-time inference, it is advised to use acceleration frameworks such as vLLM or TensorRT, and to consider quantization techniques to reduce memory usage.

Q: How does the model perform in scenarios with overlapping sound sources?
A: The Spatial-Dasheng encoder is specifically optimized for frame-by-frame detection and localization of overlapping sound sources, which is a key differentiator from competing models. In scenarios where multiple people are speaking simultaneously, the model can independently locate and perceive the direction of each sound source. However, when the number of overlapping sources is too high or when the spatial distance between sources is too close, the localization accuracy may still be affected. The specific performance boundaries should be evaluated through practical testing.

Q: Where can I find the technical report or paper for the project?
A: The project provides an arXiv technical paper that documents the complete technical details of the model architecture, training data pipeline, and evaluation results. The paper can be accessed at https://arxiv.org/pdf/2610.11156, and is suitable for researchers and technical decision-makers who need a deep understanding of the model's principles.

9. Project Links

1. Project Website: https://jinbo-hu.github.io/midashenglm-spatial-demo/
2. GitHub Repository: https://github.com/xiaomi-research/midashenglm-spatial
3. HuggingFace Model Library: https://huggingface.co/mispeech/midashenglm-spatial
4. arXiv Technical Paper: https://arxiv.org/pdf/2610.11156