HelixWorld – Noiz AI and the Hong Kong University of Science and Technology Open Source Real-Time Interactive World Model

1. What is HelixWorld

HelixWorld is a real-time interactive audiovisual world model jointly open-sourced by Noiz AI and the Hong Kong University of Science and Technology, representing a significant advancement in generative AI within the field of spatial intelligence. Given a single static image and a textual description as input, the model can generate an immersive three-dimensional world that users can freely navigate using WASD keys and directional keys. The visuals and spatial audio effects are generated in real time, with the audio-visual elements naturally transitioning as the user's perspective changes. Built on the LTX-2.3 architecture, HelixWorld employs a "bidirectional teacher + causal distillation" training paradigm, achieving real-time generation performance at a resolution of 768×512 and 24 FPS on a single H800 GPU. The project also released HelixBench, a benchmark specifically designed to evaluate spatial acoustic consistency.

Technical positioning and domain: HelixWorld falls within the intersection of interactive world models and spatial audiovisual generation. Unlike traditional text-to-video models, it enables real-time environmental exploration under user control and is the first open-source solution to jointly generate spatial audio and visual scenes, filling a critical gap in this area.

Development background: The project was jointly developed by Noiz AI and the Hong Kong University of Science and Technology. The team has deep expertise in diffusion models, video generation, and spatial audio. The motivation for the project stemmed from two major pain points: existing world models focus only on visual aspects and lack spatial auditory feedback, and their generation speed is insufficient for real-time interaction. The goal was to build a truly explorable and audible AI world simulator.

Core value: HelixWorld addresses three major core issues with traditional world models: the inability to interact in real time, the separation of audio and visual elements, and long-range generation drift. Its innovative value lies in achieving "head-turning audio change" spatial acoustic consistency for the first time under open-source conditions. By leveraging online trajectory distillation and long-range streaming fine-tuning techniques, it ensures the stability of both visuals and sound during continuous generation lasting over 30 seconds.

Technical features: The model combines a bidirectional teacher model pre-training approach with causal distillation. It injects 6-DoF camera trajectories using PRoPE position encoding and modulates discrete action signals with AdaLN-Zero. It employs a cross-modal attention mechanism to propagate camera motion to the audio branch, ultimately completing real-time inference in just 4 denoising steps, achieving an inference speed more than 2.5 times faster than similar solutions.

2. Key Features

• Real-time Interactive World Generation: Users input an initial image and textual description, and the model generates a three-dimensional world that can be freely explored. The system renders the visuals in real-time, allowing users to move and observe within the environment using keyboard controls. All content is dynamically generated by AI, rather than pre-rendered resources, offering a new paradigm for interactive content creation.

• Keyboard/Action Control System: Supports WASD for movement direction and arrow keys for adjusting the viewing angle, with the visuals responding instantly to user input. The model injects discrete action instructions into the diffusion time modulation process, ensuring that each frame of generated content precisely aligns with user operations, achieving a near-zero latency control experience.

• Native Audio-Video Joint Generation: Video and stereo audio are generated synchronously during the process, rather than being added later through manual dubbing or splicing. The audio branch receives camera motion information via cross-modal attention mechanisms, ensuring that the spatial positioning of sound matches the visual content during generation, fundamentally eliminating audio-visual synchronization issues.

• Spatial Acoustic Consistency: The sound localization changes naturally as the camera moves, such as a sound source on the left changing direction when the user turns around. The model learns the mapping between 6-DoF camera trajectories and acoustic field distributions during training, achieving an immersive auditory experience where the sound changes as the head turns. Its spatial consistency score is several times higher than that of similar solutions.

• Three-Track Decoupled Prompt Control: Three independent sets of prompts are used to control the generation process: one for video description, one for audio description, and one for joint audio-video description. This decoupled design allows users to finely control the visual and auditory aspects separately, while the joint description ensures semantic alignment between the two, balancing flexibility and consistency.

• Real-time Smooth Experience: Real-time generation at 768×512 resolution and 24 FPS is achievable on a single NVIDIA H800 GPU, with an RTF (Real-time Factor) of 0.77. By using causal distillation to compress the teacher model into a lightweight student model, combined with ODE initialization, only 4 denoising steps are required per frame, meeting the real-time demands of interactive scenarios.

• Long-range Generation Stability: To address error accumulation in long-range autoregressive generation, the model employs a combined strategy of online trajectory distillation and long-range streaming fine-tuning. By using a sliding window KV cache to gradually expand long-sequence training, it retains anchor frames at the beginning of the sequence and truncates historical gradients, ensuring that visuals and audio remain stable for continuous generation lasting over 30 seconds, with diversity even increasing during the process.

3. How to Use

HelixWorld provides two usage options: online experience and local deployment. Users can choose the appropriate solution based on their specific needs and technical conditions.

1. Online Quick Experience: Open the official website at https://helixworld.org, select an initial image from the example image library or upload a custom image, and enter a textual description of the scene to enter the AI-generated world. Use the WASD keys to control movement and the arrow keys to adjust the perspective, allowing for real-time exploration of the generated content. Prompt words can be modified at any time during operation to dynamically adjust the generation direction of subsequent scenes, making it ideal for quickly verifying effects and product demonstrations.

2. Local Environment Setup: For local deployment, the following hardware and software requirements must be met: Linux operating system, Python 3.11 or higher, CUDA 12.x environment, and an NVIDIA GPU (recommended with 80GB VRAM, such as H800 or A100). Additionally, ffmpeg and ffprobe must be installed for video processing. If VRAM is insufficient, the resolution or frame rate can be reduced to alleviate the issue, although real-time performance will be affected accordingly.

3. Project Installation and Dependency Configuration: Clone the project code by executing git clone https://github.com/NoizAI/HelixWorld.git. After creating and activating a conda virtual environment, run pip install -r requirements.txt to install all dependencies. It is recommended to use a Python 3.11-specific environment to avoid dependency version conflicts with other projects.

4. Model Weight Download: Run the python download_models.py script to automatically download the pre-trained model weights. This script will fetch the checkpoint files for both the teacher model and the student model from the official storage location. After the download is complete, the script will verify the integrity of the files to ensure that the model files are not corrupted.

5. Generation and Output: Prepare an initial image and use the ./run.sh script to specify parameters such as the image path, visual prompt, audio prompt, audio-visual joint prompt, and action sequence. After execution, the generated video in MP4 format will be output to the outputs directory. It is recommended to first generate a short sequence to verify the parameter configuration, then gradually increase the length of the action sequence to observe long-range generation effects.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For immersive applications requiring spatial audio feedback (such as VR/AR, game prototyping, film previsualization), HelixWorld is currently the only open-source solution with full audio-visual joint generation capabilities, and its advantage in spatial acoustic consistency is significant. If the project only requires visual world exploration and demands high real-time performance, consider the technical approach of Genie 2 or World Labs, but be mindful of the integration costs associated with their non-open-source nature.

For teams with limited computing resources, HelixWorld's single-card real-time solution reduces deployment barriers. For scenarios prioritizing the highest visual quality and longest generation duration, it is recommended to consider World Labs' closed-source solution or await resolution upgrades in future versions of HelixWorld. The completeness and customizability of the open-source ecosystem make HelixWorld uniquely valuable for academic research and prototype validation scenarios.

6. Editor's Summary

The release of HelixWorld marks a significant step forward for interactive world models, transitioning them from "pure visual exploration" into a new phase of "audio-visual joint immersion." From a technological innovation perspective, its "bidirectional teacher + causal distillation" training paradigm addresses the computational bottleneck of real-time inference in diffusion models, achieving single-frame generation in just four denoising steps. This technical approach provides a framework that can be referenced for future real-time generation models.

The combination of online trajectory distillation and long-range streaming fine-tuning effectively tackles the issue of error accumulation in autoregressive generation, while enhancing stability without compromising diversity. This is a relatively rare achievement in similar works.

In terms of practical value, HelixWorld reduces the hardware requirements for world models to a single H800 GPU, offering a feasible experimental platform for small and medium-sized teams and research institutions. Its spatial acoustic consistency design not only enhances immersion but also provides new technical pathways for scenarios such as embodied intelligence training and VR content production. The three-track decoupled prompting mechanism grants creators precise control over audio-visual expression, offering high flexibility and practical value in real-world workflows.

This tool is suitable for game developers, VR/AR content creators, film pre-visualization teams, embodied intelligence researchers, and academic researchers working in the field of world models. As the community ecosystem continues to mature and future versions improve in resolution and generation duration, HelixWorld has the potential to become a key infrastructure in the interactive world model domain, promoting the widespread adoption and practical implementation of spatial intelligence technologies.

7. Application Scenarios

• Immersive Game Prototyping: Game teams can generate explorable open worlds by simply inputting concept images and scene descriptions, without the need for modeling or engine development. This enables rapid validation of gameplay mechanics, scene atmosphere, and level design, reducing the demo development cycle from weeks to days and significantly lowering early-stage validation costs.

• VR/AR Virtual Space Construction: The combination of first-person perspective and spatial audio is naturally compatible with head-mounted display devices. It allows for the quick generation of virtual exhibition halls, digital twin environments, and immersive educational scenarios. Once users wear the headset, they can experience dual immersion through both visual and auditory elements, making it suitable for industries such as culture and tourism, education, and exhibition displays.

• Film and Advertising Previsualization: Directors and art teams can "step into" scenes by uploading concept art, and experience real-time audio-visual effects. This provides a low-cost way to validate shot composition, scene atmosphere, and sound design before actual filming or CG production, reducing post-production rework and improving creative efficiency and communication quality.

• AI Video Calls and Virtual Broadcaster Backgrounds: Dynamically generate matching backgrounds and spatial sound effects in real-time based on voice content and user interaction, replacing traditional green screens and static asset libraries. This provides personalized, responsive virtual environments for broadcasters and remote meetings, enhancing audience immersion and interactive engagement.

• Embodied Intelligence Training and Simulation: As an interactive world simulator, it offers a low-cost training environment with audio-visual feedback for robots and autonomous driving systems. Agents can learn physical interaction rules, spatial navigation, and sound source localization capabilities within the generated world, addressing long-tail scenarios that are difficult to cover with real-world data.

8. FAQ

Q: What are the specific hardware requirements for HelixWorld?
A: The official recommendation is to use an NVIDIA GPU with 80GB of VRAM (such as H800 or A100), with Linux as the operating system. CUDA 12.x, Python 3.11, and ffmpeg and ffprobe must be installed. If VRAM is insufficient, the resolution or frame rate can be reduced to run the model, although the real-time interactive experience will be affected.

Q: Are the generated videos and audio rendered in real-time?
A: Yes. HelixWorld uses a causally distilled student model that requires only 4 denoising steps to generate a single frame. Real-time inference at 24 FPS can be achieved on a single H800 GPU, with visual and spatial audio content generated synchronously and responding instantly to user interactions.

Q: How is the "turn head to change voice" spatial acoustic effect achieved?
A: During the training phase, the model uses a cross-modal attention mechanism to pass 6-DoF camera trajectory information to the audio branch, allowing the audio generation process to perceive changes in viewpoint. When the user changes their perspective, the audio image position is recalculated based on the new viewpoint direction, achieving a natural transition where sound follows the visual movement.

Q: How long of continuous generation does HelixWorld support?
A: Through online trajectory distillation and long-range streaming fine-tuning technologies, the model maintains stable visuals and audio for continuous generation lasting over 30 seconds, with generation diversity even increasing. In theory, it can support longer durations, but accumulated errors in extended sessions still require further validation.

Q: Can custom images and prompts be used for generation?
A: Yes. Users can choose from example images provided by the official team or upload their own custom images as the starting frame. They can also provide three independent text descriptions for video, audio, and joint audio-visual content to control the generation process, enabling personalized scene creation.

Q: Is the project fully open source and can it be used for commercial purposes?
A: The code and model weights of HelixWorld are fully open source, and users can freely use them, including for commercial purposes, according to the terms of the open source license. For detailed license information, please refer to the LICENSE file in the project's GitHub repository.

9. Project Links

• Project Website: https://helixworld.org/
• GitHub Repository: https://github.com/NoizAI/HelixWorld
• arXiv Technical Paper: https://arxiv.org/pdf/2609.38123