Hy-Nomas – Tencent HunYuan Team's Open-Source Interactive Video World Generation Model

1. What is Hy-Nomas

Hy-Nomas is an open-source interactive video world generation model developed by the Tencent HunYuan team. It generates videos using a segmented autoregressive approach, supporting seamless switching of text prompts or camera trajectory controls between segments. The model maintains long-term consistency through a lifelong KV cache mechanism. The code and weights of this model have been open-sourced, offering both distilled and INT8 quantized versions. After optimization, it can perform local inference on a single RTX 4090 GPU, significantly lowering the barrier to entry for using video world models, and providing researchers, developers, and creators with a customizable, controllable, and interactive video generation solution.

Technical positioning and domain: Hy-Nomas falls at the intersection of video generation and world modeling. Its core task is to generate coherent video sequences in a segmented, autoregressive manner, while introducing dual control signals—text prompts and camera trajectories—to enable the construction of interactive long-form video worlds. It is positioned between traditional text-to-video models and real-time interactive world models, balancing offline high-quality generation with interactive flexibility.

Development background: This model was developed by the Tencent HunYuan team, leveraging Tencent's long-term accumulation in the field of video generation. The team has deep technical expertise in DiT architecture, diffusion model distillation, and visual encoders. The motivation behind its development was to address key pain points in existing video generation models, such as the inability to support long-term interaction, precise control over camera movement, and the difficulty of deploying on consumer-grade hardware, thus pushing video world models from cloud-based experiments toward practical local applications.

Core value: Hy-Nomas addresses three major core challenges in the video generation domain: long-term consistency, interactive controllability, and deployment cost. The lifelong KV cache mechanism ensures consistency in scenes, objects, and styles across video sequences lasting tens of seconds. The dual-modal switching between text and camera trajectory provides creators with flexible control over their content. INT8 quantization and CPU offload optimizations bring inference costs within the range of a single RTX 4090 GPU. Additionally, the Apache 2.0 license ensures full open-source availability for commercial use.

Technical features: The model employs a segmented autoregressive generation architecture, reducing denoising steps to 4/6/8 through distillation. Combined with a PixelUMM decoder fine-tuned for 32-channel DCAE-V latent variables, this significantly lowers inference latency. A LoRA segmented forced training pipeline supports fine-tuning with custom video data and merging back into the distilled model, achieving a balance between customization capabilities and inference efficiency.

2. Key Features

• Segmented Autoregressive Generation: Hy-Nomas introduces a segmented autoregressive mechanism on the DiT architecture, generating long videos in segments, with each segment conditioned on the current text prompt or camera signal. By maintaining cross-segment context using a lifelong KV cache, the scene, objects, and style remain highly consistent across the timeline, effectively avoiding drift issues commonly encountered in long video generation. The generated video resolution can reach up to 1280×704, and with the movie-style LoRA, the visual quality can be further enhanced.

• Free-form Text Control: Supports any form of text prompts, allowing users to describe desired objects, styles, scenes, and actions in natural, conversational language. The built-in Nomas-PE prompt enhancement module automatically converts natural language instructions into structured generation prompts, lowering the usage barrier and enabling non-expert users to precisely express their creative intent.

• Camera Trajectory Control: Enables conditional generation based on camera motion trajectories, allowing users to specify movements such as push, pull, pan, tilt, or circular tracking. Camera control and text prompts can be freely switched and combined between segments—for example, the first segment may use text to describe the scene content, while the next segment switches to a specific camera trajectory for exploration. This dual-modal switching mechanism is relatively rare among open-source models of this type.

• Multi-round Interactive Generation: The Web interface supports injecting new instructions at segment boundaries, allowing users to change the scene direction, object behavior, or style settings at any time during the generation process, enabling multi-round continuous generation. Instructions such as "make the camera move closer to the house door" or "change the sky from sunny to dusk" can be input at any segment boundary, and the model will continue generating subsequent content that meets the requirements based on the current context.

• Consumer-grade Local Deployment: Through INT8 quantization and CPU offloading optimization, the DiT weights and KV cache are stored and computed using INT8 while maintaining BF16 precision. Additionally, a sliding window is used to truncate historical context and control KV cache growth. The text encoder, DiT, and VAE switch between CPU and GPU in stages, keeping peak memory usage within the capacity of consumer-grade GPUs. A single RTX 4090 is sufficient to complete the full inference pipeline.

• LoRA Fine-tuning Support: Provides full LoRA fine-tuning capabilities, using a non-distilled checkpoint as the base model and employing a segment forcing training pipeline, training only the LoRA parameters (rank/alpha 256). After training, the LoRA weights can be merged back into the distilled checkpoint, resulting in an inference model that retains the speed benefits of distillation while offering customization. The official team has also released the dataset and annotation process, supporting the development of custom world models.

• Multi-backend Inference Engine: Supports various inference backend environments, meeting the needs of multiple scenarios ranging from academic research to production deployment. The distilled version supports native loading and execution, while the quantized version is suitable for low-memory environments. This multi-version strategy allows developers to flexibly choose deployment methods based on their hardware conditions.

3. How to Use

1. Environment Requirements: It is recommended to use a Linux operating system with a CUDA version no lower than 11.8. For the inference phase, the NVIDIA RTX 4090 GPU (24GB VRAM) is recommended; the quantized version enables local inference. If using the non-quantized version, a higher VRAM configuration is required. For training/fine-tuning, it is recommended to use a multi-GPU A100/H100 cluster, and PyTorch 2.0 or higher and the HuggingFace Transformers library must be installed.

2. Model Acquisition: Download the model weights from HuggingFace (https://huggingface.co/tencent/Hy-Nomas), selecting either the distilled version or the INT8 quantized version. Clone the full code repository from the GitHub warehouse (https://github.com/Tencent-Hunyuan/Hy-Nomas), and install the required dependencies as outlined in the README, including core libraries such as diffusers and accelerate.

3. Inference Deployment: Run the basic inference command to load the model and generate video from text. Key configuration options include: setting the --num-inference-steps parameter to 4/6/8 (corresponding to the distillation steps), setting the output resolution with --height 704 --width 1280, enabling KV cache quantization with --kv-cache-dtype int8, and activating CPU offload mode with --cpu-offload. If camera trajectory control is required, additional camera path parameters (e.g., spiral ascent, horizontal scanning, etc.) must be provided.

4. Web Interface Interaction: Launch the web interface provided in the repository. Enter the initial prompt in the text box to generate the first video segment. After generation is complete, you can input new text instructions or switch camera trajectory modes at the segment boundary to continue generating the next segment. The interface supports real-time preview of each generated segment and allows parameter adjustments at any stage.

5. LoRA Fine-tuning: Prepare a custom video dataset and organize the data format according to the official data annotation process. Select a non-distilled checkpoint as the base model, configure LoRA training parameters (rank/alpha set to 256), and run the training script. After training, merge the LoRA weights back into the distilled checkpoint to obtain a customized inference model.

6. Notes: When running for the first time, it is advisable to first verify the basic inference workflow, and then gradually attempt advanced features such as camera trajectory control and multi-round interactions. The quantized version may have slight differences in inference quality compared to the non-quantized version; for scenarios with high visual quality requirements, it is recommended to revert to the non-quantized version. When fine-tuning with LoRA, ensure the diversity and quality of the video dataset in the training data, as this will directly impact the generalization capability of the fine-tuning results.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: If the core requirement is a video world model that is commercially usable, customizable, and supports local deployment, Hy-Nomas is currently the pragmatic choice among open-source models. Its dual-modal control mechanism and LoRA fine-tuning capabilities provide developers with ample room for secondary development, making it particularly suitable for technical teams that need to integrate video generation capabilities into their own products. The Apache 2.0 license also eliminates legal barriers to commercialization.

If the goal is to achieve an ultimate real-time interactive experience, Genie 3's 24fps per-frame generation mode is still leading, but it is limited by its closed-source nature, 60-second experience duration, and availability only to U.S. subscribers. As a result, it is not ideal as a technical foundation. For scenarios that do not require interactivity and focus on single, high-quality video generation, CogVideoX and Wan2.2 each have their own advantages in terms of video quality and ecosystem support.

6. Editor's Summary

Hy-Nomas demonstrates a relatively complete technical path in the field of video world models. Its segmented autoregressive architecture is not a simple engineering compromise, but rather a design that balances long-range consistency and interactive flexibility through a lifelong KV cache mechanism—this is a representative feature among similar open-source models. From a technical evolution perspective, the distillation process into 4/6/8-step denoising, combined with a dedicated PixelUMM decoder for handling 32-channel DCAE-V latent variables, provides a practical reference case for efficient inference of diffusion models on consumer-grade hardware. The combination strategy of INT8 quantization, sliding window truncation, and CPU offload also reflects the team's careful consideration regarding low-memory inference. Notably, the design of the LoRA segmented forced training pipeline—training from a non-distilled checkpoint and then merging back into the distilled checkpoint—cleverly addresses the challenge of fine-tuning distilled models, expanding the space for model customization.

In terms of practical value, Hy-Nomas has real application potential in scenarios such as game prototype development, robot simulation training, and film pre-visualization. The official Robotics LoRA's ability to generate robotic arm operation environments indicates that it already provides synthetic training data support for the field of embodied intelligence; the cinematic LoRA directly serves the pre-visualization needs of the film industry. These specialized resources prevent the model from remaining at the technical demo level. For researchers, the complete training code, data annotation process, and open weights make it a suitable foundation for teaching and research in the world model domain. For developers, the Apache 2.0 license and consumer-grade deployment capabilities lower the threshold for commercial integration. For creators, the web-based interactive interface and prompt-enhancing modules offer a relatively gentle learning curve. Of course, the non-real-time segmented generation mode has a generational gap compared to Genie 3's frame-by-frame real-time interaction, and the community ecosystem is still in its early growth phase. These factors will to some extent influence its application boundaries.

7. Application Scenarios

• AI Game Prototype Development: Game teams can leverage Hy-Nomas to quickly generate interactive game worlds using text and camera instructions. For example, inputting a description such as "stylized low-poly forest with a stream running through it" to define the scene, and then using camera trajectory controls to generate exploratory paths around the environment, allows for low-cost validation of gameplay concepts and level design feasibility. The multi-round interaction mechanism enables dynamic adjustment of environmental elements during generation, accelerating the prototype iteration cycle and improving the efficiency of early-stage creative validation.

• Robotics Simulation Training: Using the official Robotics LoRA, realistic robotic arm operation environments can be generated to provide synthetic training data for embodied intelligence. Researchers can configure different object placements, lighting conditions, and camera perspectives to generate a large volume of diverse simulation scenarios, compensating for the limitations of real-world data collection. This synthetic data approach offers clear advantages in terms of cost and scale, making it suitable for expanding training data in reinforcement learning tasks such as grasping strategies and obstacle avoidance planning, thereby reducing reliance on real physical environments for data collection.

• Film Previsualization and Storyboarding: Directors and storyboard artists can use the cinematic LoRA to generate coherent long takes, aiding in camera movement design and visual previews. By switching between camera instructions such as "zoom in on the character's face" or "pan around the entire scene" between segments, they can validate shot language and pacing before actual filming, reducing the cost of trial and error during real shoots. This application is also suitable for pre-visualization stages in advertising, short videos, and other visual content creation.

• Virtual Tourism and Spatial Presentations: Cultural and tourism sites, real estate showrooms, and similar locations can use camera trajectory controls to generate navigable virtual environments. For instance, after inputting a description of the scenic area's architecture, a presentation video can be generated using push-pull, pan, and tilt trajectories, replacing the traditional 3D modeling workflow. Audiences can explore different perspectives and areas through multi-round interactions. This application can be extended to online museums, virtual city planning, and other spatial presentation scenarios, reducing production costs for display content and enhancing visual appeal.

• AI Research and Education: The complete training, fine-tuning, and data annotation code is open-sourced, serving as a foundational platform for research and teaching in world model development. University laboratories can conduct research on topics such as video generation, long-range consistency, and controllable generation based on this architecture. In educational settings, it is also well-suited as a practical project for courses on deep generative models, helping students understand core technologies such as diffusion models, autoregressive generation, and KV cache management.

8. FAQ

Q: What hardware requirements does Hy-Nomas specifically have? Can it really run on a single RTX 4090?
A: Hy-Nomas provides both distilled and INT8 quantized versions. The quantized version stores model weights and KV cache in INT8 format, while maintaining BF16 precision for computation. It also switches the text encoder, DiT, and VAE between CPU and GPU in stages, combined with a sliding window to truncate historical context. In practical tests, a single RTX 4090 (24GB VRAM) can complete the inference process. However, it should be noted that the quantized version may experience slight quality degradation in terms of visual details in extremely complex scenarios. If using the non-quantized version, it is recommended to use professional-grade GPUs with higher VRAM, such as the A100.

Q: How can trained LoRA weights be merged into the inference model?
A: The official documentation provides a specific workflow: LoRA training is based on non-distilled checkpoints, and after training, the LoRA weights are merged back into the distilled checkpoint. This approach is used because fine-tuning directly on the distilled model yields poor results. Training first and then merging allows for retaining the acceleration benefits of distillation while also achieving customization. Once merged, the model can be directly loaded for inference without any additional configuration.

Q: HuggingFace download speeds are slow, is there an alternative?
A: At present, the official model weights are mainly released through the HuggingFace platform. If download speeds are unsatisfactory, you can try the following methods: use a HuggingFace mirror site (e.g., hf-mirror.com), configure a proxy network environment, or check the GitHub repository to see if there are domestic channels such as Baidu Netdisk for distribution. It is recommended to first confirm whether the network environment is stable before choosing an appropriate download method.

Q: What types of camera movements does the camera trajectory control support? Can custom trajectories be defined?
A: The official implementation supports conditional generation of predefined camera motion trajectories, including common techniques such as zooming in, zooming out, horizontal panning, and orbiting. The current version mainly achieves control by passing standardized trajectory parameters. To define custom complex trajectories, you need to understand the model's camera conditioning mechanism and extend the code accordingly. Compared to Genie 3, which allows real-time control using WASD key bindings, Hy-Nomas is more oriented toward offline camera motion design rather than real-time exploration.

Q: What is the maximum length of the generated video? Is it related to the complexity of the scene?
A: Theoretically, the video length is limited by the capacity of the lifelong KV cache and the sliding window truncation strategy. Generating longer videos consumes more KV cache space, and once the capacity is exceeded, the model will truncate the earliest historical context. As a result, object consistency may be affected in extremely long video scenarios. Under consumer-grade GPU configurations, it is recommended to control the total duration of a single video generation or explicitly maintain descriptions of key elements through text instructions between segments to achieve more stable generation results.

9. Project Links

• Hy-Nomas Project Official Website: https://hy-nomas.hy.tencent.com/
• GitHub Repository: https://github.com/Tencent-Hunyuan/Hy-Nomas
• HuggingFace Model Library: https://huggingface.co/tencent/Hy-Nomas