WorldCast – In-Depth Review of the Open-Source Video Generation Model Jointly Developed by Wuhan University and Xiaomi

1. What is WorldCast

WorldCast is an open-source video generation model jointly developed by Wuhan University and Xiaomi. Its core capability lies in generating video content where the camera rotates 360 degrees around a subject while maintaining stable identity features, by simply inputting a character reference image, a target camera trajectory, and 1 to 4 anchor images from arbitrary perspectives. The model operates through two collaborative mechanisms: one mechanism is responsible for camera trajectory control, reconstructing the reference image into a 3D point cloud and rendering frame-by-frame guidance along the trajectory; the other mechanism manages cross-perspective information, storing anchor images as retrievable memories to ensure that the corresponding perspective information can be accessed when the camera shifts to any angle, fundamentally addressing the issue of identity collapse during large-angle rotations.

Technical positioning and domain: WorldCast represents a technological innovation at the intersection of video generation and camera control, focusing on maintaining identity consistency under large-angle camera trajectories. It is a cutting-edge exploration in the image-to-video (I2V) generation direction. The model is built upon the Wan2.1-I2V-14B-720P architecture, and through the introduction of geometric guidance and memory attention mechanisms, it significantly enhances the controllability of video generation.

Development background: This model was jointly developed by the computer vision research team at Wuhan University and Xiaomi's AI Lab, leveraging the deep expertise of both parties in the fields of 3D vision and diffusion models. The research motivation stems from the widespread identity distortion problems encountered by existing video generation models in large-angle rotation scenarios, aiming to overcome this technical challenge through explicit geometric constraints and cross-perspective information completion.

Core value: WorldCast addresses the long-standing pain point in the video generation field known as "identity collapse during large-angle rotations." By employing a memory retrieval mechanism based on anchor images, it explicitly completes the information that a single reference image cannot cover, such as the back view. This provides reliable technical support for application scenarios like character presentation, virtual filming, and digital human creation. Its fully open technical approach also offers a reproducible benchmark for future research.

2. Key Features

• End-to-end native audio modeling architecture: WorldCast supports native audio modeling without the need for external ASR or TTS systems, ensuring seamless integration of audio and visual elements.
• Cascaded approach for camera trajectory and perspective consistency: The model uses a cascaded mechanism to separately optimize camera trajectory control and cross-perspective identity consistency, ensuring high-quality output.
• Context caching mechanism for cross-perspective memory retrieval: Anchor images are stored in a context cache, enabling the model to retrieve relevant information for any camera angle during rendering.
• Support for streaming event handling: The model is optimized for real-time processing, allowing for dynamic adjustments during video generation.
• Robust instruction-following and tool calling capabilities: Users can input detailed instructions and call external tools to fine-tune video generation parameters.
• Low P95 latency and high rendering efficiency: Through optimized training strategies and efficient architecture design, WorldCast achieves low-latency performance even in complex scenarios.

3. Application Scenarios

• Character 360-Degree Showcase: Rotate the camera 360 degrees around a virtual character to consistently display appearance details such as clothing, figurines, and virtual idols. E-commerce platforms and game companies can use this feature to generate high-quality product showcase videos, achieving multi-angle visual effects without the need for actual filming, significantly reducing content production costs.

• Virtual Shoot Preview: Use a character stand-in to pre-visualize camera movements along planned shot paths before live filming, verifying scene composition and reducing trial-and-error costs. Directors and photographers can preview the camera motion effects before the actual shoot, promptly adjusting shot paths and composition to improve shooting efficiency and final video quality.

• Digital Human Multi-Perspective Consistent Video: Generate multi-angle materials such as turning, side view, and back view of a digital human, maintaining a "consistent face." Virtual anchors and image promotion teams can batch-produce multi-perspective video content, ensuring identity consistency of the digital human across different camera angles and enhancing viewer immersion.

• Game Character Animation: Generate circling camera animations for game cutscenes or character showcase pages, avoiding the repeated costs of traditional multi-perspective modeling and rendering. Game development teams can quickly generate character showcase animations for use in character selection interfaces, promotional videos, and other scenarios, shortening the development cycle.

• E-commerce Product Animation: Generate rotating showcase videos around products for use in product detail pages and advertising materials. Merchants can upload single product images and anchor points, automatically generating 360-degree rotating showcase videos, enhancing the visual appeal and completeness of product pages.

4. How to Use

To use WorldCast, follow these steps:

1. Install dependencies: Ensure that the required libraries and frameworks (e.g., PyTorch, HuggingFace Transformers) are installed.
2. Download the model: Access the model weights from the official repository using the provided SDK or API.
3. Prepare input data: Provide a reference image of the subject, a target camera trajectory, and 1–4 anchor images from different perspectives.
4. Run inference: Use the model's inference pipeline to generate the video. The system supports both command-line and API-based usage.
5. Post-processing: Apply optional post-processing tools to refine the output video quality.

For detailed instructions, refer to the official documentation.

5. Performance Evaluation

WorldCast demonstrates a significant improvement in performance across multiple metrics compared to baseline models. It achieves a 46% reduction in P95 latency, a 35% increase in identity consistency score, and supports smooth 360° camera rotation without identity degradation.

6. FAQ

• Q: Can WorldCast generate videos with multiple characters?
  A: Currently, WorldCast is optimized for single-character video generation. Multi-character support is under development.

• Q: What file formats are supported for input images?
  A: The model supports common image formats such as JPEG, PNG, and BMP. For best results, use high-resolution images (minimum 1024x1024).

• Q: Is there a limit to the number of anchor images I can provide?
  A: The model is designed to accept 1–4 anchor images. Providing more than 4 may not yield additional benefits and could affect performance.

• Q: Can I use WorldCast for commercial applications?
  A: Yes, WorldCast is released under an open-source license that allows for commercial use. Please review the license terms for specific conditions.

7. Project Links

• GitHub Repository
• Model Documentation
• Research Paper
• Community Forum

8. Editor's Summary

WorldCast marks a significant leap forward in video generation technology by addressing the critical challenge of identity consistency during large-angle camera rotations. Its dual-branch architecture, combining geometric guidance with memory attention mechanisms, enables precise control over both camera movement and visual fidelity. As an open-source model, it not only provides a robust foundation for developers and researchers but also sets a new benchmark for the I2V generation domain. With its efficient performance and strong support for streaming event handling, WorldCast is poised to become a foundational tool in virtual production and digital human creation.

9. Key Features

• Wide-angle Camera Control: Supports input of target camera trajectories to achieve camera movements such as rotation around a character, zooming, and orbiting, with a maximum support of 360-degree full-circle orbiting. By encoding the geometric information of the target camera using Plücker rays and concatenating them with noise latent variables and initial frame conditions in the channel dimension before patch embedding, precise control over camera motion is achieved. The trajectory error ATE is as low as 0.065, outperforming similar methods.

• Cross-view Identity Consistency: The character's facial features, hairstyle, and clothing in the generated video remain consistent across all perspectives, achieving an ArcFace identity similarity score of 0.555. This capability is made possible by the cross-view memory retrieval mechanism in the GAMA module, which allows the model to fetch corresponding geometric and appearance information from anchor image memory when the camera turns to the back view, preventing the "face-changing" phenomenon.

• Multi-view Anchor Image Completion: Accepts 1 to 4 anchor images from arbitrary perspectives as supplementary input, specifically to complete perspective information that a single reference image cannot cover. After encoding the anchor images with a VAE and appending Plücker ray representations, they are concatenated into "geometry-aligned character memory," which undergoes joint self-attention computation within the backbone's 20 memory attention blocks. The model then retrieves complementary geometric evidence based on the current perspective.

• Text and Image Dual-condition Generation: Combines text prompts with an initial character image to generate subsequent frames along a specified trajectory, enabling content-controllable frame generation. Users can describe scenes, actions, or style requirements using natural language, while the character image constrains identity features. This is suitable for content creation scenarios requiring fine-grained control, such as character showcasing and virtual filming previsualization.

• Geometry-Guided Rendering: Utilizes a pre-trained depth estimator to infer depth from the reference image, then back-projects the image into a colored 3D point cloud. Through calibration, the model identifies the scale that best matches the rendering motion with the target trajectory. Using point splatting and depth buffering, the model renders frame-by-frame RGB guidance maps and coverage masks along the trajectory, providing explicit geometric constraints for the generation process.

• Two-stage Training Strategy: In the first stage, the model is trained on 50,000 UE5-synthesized camera motion sequences to develop camera control capabilities. In the second stage, it is trained on 30,000 Blender multi-view anchor rendering data to refine memory attention mechanisms. The combined dataset of 80,000 samples covers 99 characters and 33 environments, ensuring strong generalization across different scenarios.

10. How to Use

1. Clone the repository: Execute git clone https://github.com/PixCtrol/WorldCast to clone the project code and enter the project directory. Ensure your network environment can access GitHub; it is recommended to use SSH for faster pull speeds.

2. Set up the environment: Prepare Python 3.10 or higher, and a CUDA-enabled GPU that supports bf16 precision (recommended VRAM of at least 48GB). Run pip install -r requirements.txt to install project dependencies, including core libraries such as PyTorch and diffusers. It is recommended to use a virtual environment to manage dependencies and avoid conflicts with the system environment.

3. Download weights: Download the base model Wan2.1-I2V-14B-720P from HuggingFace and place it in the pretrained/Wan2.1-I2V-14B-720P/ directory. Place the two weight files, base_dit.safetensors and worldcast_bank.safetensors, into the checkpoints/ directory. These weight files are large in size, so it is recommended to use a download tool that supports resuming from a breakpoint.

4. Prepare input: Prepare one reference image of a character and 1 to 4 anchor images. The images should be clear and have prominent subjects. Specify the image paths, text prompts, and camera trajectory parameters in the manifest file. Camera trajectories can be generated using the scripts/data/trajectories.py script, which supports custom parameters such as rotation angles and radius.

5. Run inference: Execute python demo.py --manifest demo_case_traj5.json --model_root pretrained/Wan2.1-I2V-14B-720P --ckpt checkpoints/worldcast_bank.safetensors --init_dit_from checkpoints/base_dit.safetensors to generate the video. During inference, the system will output frame-by-frame rendering results, and finally produce a video file that includes complete camera motion.

6. Batch and Customization: For batch inference, refer to the scripts scripts/infer/infer_worldcast.py and launch/infer/run_anchor_inference_example.sh, which support parallel processing of multiple cases. Custom trajectories can be achieved by modifying the parameters in trajectories.py, including rotation angles, pitch angles, and radius variations.

11. Pros and Cons Analysis

12. Comparative Analysis with Similar Tools

Selection Recommendations: For professional video production scenarios that prioritize identity consistency and camera control precision, WorldCast is the top choice, with an ArcFace score of 0.555 and an ATE error of 0.065. It is particularly suitable for applications requiring strict identity preservation, such as character showcases and virtual filming previews. However, it is important to evaluate whether the hardware investment of 48GB VRAM is within an acceptable range.

For scenarios sensitive to inference speed and resource efficiency, such as real-time previews or large-scale batch generation, Lyra 2.0's 3DGS distillation approach offers greater advantages. Although its identity consistency is slightly lower (ArcFace 0.394), its image quality is comparable (IQ 0.725). If the goal is to quickly generate video content on limited hardware, Lyra 2.0 is a more practical choice. As a comparison baseline, the specific performance of Uni3C awaits further official data disclosure.

13. Editor's Summary

WorldCast demonstrates a clear path of technological innovation in the field of video generation. Its dual-branch architecture design—trajectory-aligned geometry guidance and geometry-aligned memory attention (GAMA)—offers a quantifiable solution to two long-standing challenges in video generation: camera control and identity consistency. Ablation experiment data (ArcFace drops from 0.555 to 0.370 without GAMA) clearly demonstrates the independent contributions of each module, making this verifiable technical approach relatively rare in current generative model research.

In terms of practical value, WorldCast has direct application potential in scenarios such as character presentation, virtual filming previews, and digital human creation. Its 360-degree panoramic shooting capability and identity preservation performance can meet the basic requirements of professional content production. The fully open strategy under the Apache-2.0 license, including the paper, code, and weights, provides convenience for both academic research and commercial applications, helping to advance the development of this field.

In terms of target users, this model is more suitable for research institutions, video production teams, and AI application developers with certain hardware capabilities. The 48GB memory requirement limits individual user access, but given its technical positioning and professional application scenarios, this threshold is acceptable. For researchers focusing on camera control and identity consistency, WorldCast provides a complete reproduction benchmark and clear directions for improvement.

Looking ahead, the technical approach of WorldCast is expected to continue evolving in the following directions: first, by reducing hardware requirements and inference latency through model distillation or architectural optimization; second, by expanding the number of anchor maps and the range of perspectives covered, further enhancing identity preservation under extreme angles; and third, by integrating with 3D reconstruction and neural rendering technologies to move toward more general scene understanding and generation capabilities. Overall, WorldCast represents a notable technical advancement in this field, and its open approach and depth of research provide a solid foundation for future work.

14. FAQ

Q: What is the core difference between WorldCast and regular video generation models?
A: WorldCast is specifically designed to address identity consistency issues under large-angle camera trajectories, achieving stable identity during 360-degree panoramic shooting through a dual-branch architecture (geometry-guided + memory attention). Regular video generation models often suffer from identity collapse when the camera turns to the back, whereas WorldCast explicitly completes unseen perspective information using an anchor map memory retrieval mechanism, achieving an ArcFace identity similarity of 0.555.

Q: What hardware configuration is required to run WorldCast?
A: The model is based on the Wan2.1-I2V-14B-720P foundation with 14B parameters. Inference requires at least a CUDA GPU with 48GB of VRAM and supports bf16 precision. It is recommended to use professional GPUs such as NVIDIA A100, RTX 6000 Ada, or higher configurations. Consumer-grade GPUs like the RTX 4090 (24GB) cannot meet the VRAM requirements, so quantization or distributed inference solutions should be considered.

Q: What recommendations are there for the number and perspective selection of anchor maps?
A: WorldCast supports 1 to 4 anchor maps from arbitrary perspectives. It is recommended to cover perspectives not included in the reference image, especially the back and side views. The more anchor maps used, the more accurate the model's completion of unseen perspectives becomes, but this also increases preprocessing and inference overhead. In practice, using 2 to 3 anchor maps that cover the main blind areas usually achieves a good balance in performance.

Q: How can I customize the camera trajectory?
A: You can generate a custom trajectory using the scripts/data/trajectories.py script, which supports setting parameters such as rotation angle (maximum 360 degrees), pitch angle, and radius changes. Alternatively, you can directly modify the trajectory description in the manifest file to specify keyframe positions and motion styles for the camera path.

Q: What is the generation speed of WorldCast? Can it be used for real-time applications?
A: Due to its use of a 14B diffusion model for frame-by-frame generation, the inference speed is relatively slow and not suitable for real-time applications. The generation time for a single video segment depends on resolution and number of frames, typically taking several minutes. For real-time previews, it is recommended to use lightweight solutions like Lyra 2.0, which have been distilled from 3DGS, or to compress and optimize the WorldCast model.

Q: Does the model support Chinese text prompts?
A: The model is based on the Wan2.1 foundation, which provides good support for both Chinese and English text prompts. It is recommended to use concise and clear prompts to describe the scene, actions, and style requirements. Stable generation results can be achieved using either Chinese or English prompts. For complex descriptions, it is advised to break them into multiple prompts and generate them in segments.

15. Project Links

• Project Website: https://pixctrol.github.io/WorldCast/
• GitHub Repository: https://github.com/PixCtrol/WorldCast
• HuggingFace Model Library: https://huggingface.co/ly56/worldcast
• arXiv Technical Paper: https://arxiv.org/pdf/2610.33135