WBench – Meituan’s Interactive Video World Model Multi-Turn Benchmark

1. What Is WBench

WBench is Meituan LongCat’s first systematic multi-turn benchmark for interactive video world models—289 test cases, 1,058 interaction rounds, six scene types (nature, city, indoor, workspace, fantasy, sports), and seven art styles (photoreal, anime, cartoon, oil, ink, flat, sketch). A unified interaction interface evaluates 20 frontier models, mapping the gap from passive viewing to active control—navigation, manipulation, camera changes—and exposing where today’s world models fail on long-horizon interaction.

Technical positioning and domain: Evaluation for interactive video worlds vs single-clip quality metrics. Multi-turn feedback loops are first-class—industry lacked a reproducible standard here until WBench.

Development background: LongCat’s video/world-model work needed apples-to-apples comparison as Genie, Kling, Wan, etc. evolved. WBench standardizes protocols, decouples scene from action, and scores five hard dimensions.

Core value: Solves three evaluation gaps: no multi-turn framework (error accumulation invisible), heterogeneous action APIs (text vs 6DoF vs discrete keys), and scene-action coupling (can’t isolate failure modes). Think “CT scan” for world models.

Technical characteristics: World definition + instruction sets + Action Router + metric suite; autoregressive rounds feed prior frames forward—e.g., navigation scores drop ~33 points over turns on average.

2. Key Features

• Rich world definitions: Six scenes × seven styles × first/third/embodied POV—controlled diversity from realistic to stylized fantasy.

• Four interaction families: Navigation (move/turn), agent motion (run/jump/grab), event editing (spawn objects, weather), camera control (free look, target lock).

• Action Router: Maps text, 6DoF poses, and keyboard discretions to a common schema—fair compare across Kling 3.0 text mode vs HY-World 1.5 action controls.

• Five-metric suite: Video quality, spec adherence, interaction adherence, temporal consistency, physical plausibility—plus NavScore and Gated Spatial Consistency.

• Multi-turn evaluation: Default up to 8 autoregressive rounds—quantifies drift vs single-turn peaks.

• Batch cross-model runs: Same cases across models → radar charts and CSV tables for procurement/research.

3. How to Use

1. Environment: Python 3.8+, Linux/macOS; GPU with ≥16 GB VRAM for model inference; Git + pip installed.

2. Clone & install:

   git clone https://github.com/meituan-longcat/WBench.git
   pip install -r requirements.txt
   

3. Download data:

   huggingface-cli download meituan-longcat/WBench --local-dir ./data
   

4. Register models: Edit config/models.yaml with API endpoints or local inference scripts; adapt via Action Router examples.

5. Pick scenario: config/scenario.yaml—scene, style, subject, POV; or use bundled 289 cases.

6. Pick interactions: config/interaction.yaml—compose nav/motion/edit/camera sequences.

7. Single-turn run:

   python run.py --mode single --scenario config/scenario.yaml --interaction config/interaction.yaml
   

8. Multi-turn run:

   python run.py --mode multi --rounds 8 ...
   

9. Batch compare:

   python run.py --mode batch --models model1,model2,model3
   

10. Reports: Results under results/; python visualize.py --report results/report.json for charts.

4. Pros and Cons

5. Comparison with Similar Tools

Selection guidance: Interactive world models (Genie/Kling/Wan) needing nav + multi-turn decay → WBench (only option). Classic T2V quality → VBench. Unified 3D/4D/I2V/T2V worlds → WorldScore. Lightweight multi-model scoring → EvalCrafter.

6. Editor's Take

WBench moves evaluation from “pretty frame” to “can you steer the world over time.” Decoupling stage (scene) from script (actions) is the key design win—finally see which skill broke.

Findings already steer R&D: text-first models vs navigation specialists, quality decoupled from control—teams shouldn’t optimize pixels alone.

Audience: World-model researchers, embodied AI groups, interactive video/game PMs, paper reviewers needing a standard harness.

Future: Richer physics, custom world gen, adversarial action suites, public leaderboards.

— − for case count, physics depth, 8-round limit; still indispensable in its niche.

7. Use Cases

• Model iteration: Compare single- vs multi-turn scores to fix nav/consistency regressions.

• Embodied AI proxy: NavScore as virtual path-planning/memory probe.

• Product QA: Stress AI game/world products with long action chains.

• Academic baselines: Reproducible cross-paper comparisons.

• Build vs buy: Open vs API world models on identical WBench cases.

8. FAQ

Q: Which models work?
A: Any interactive video generator—text-driven (Kling, Wan) or action-native (Genie, HY-World)—via Action Router adapters.

Q: Trust scores?
A: Open cases/commands; automated metrics with human validation on NavScore etc.; rerun for means.

Q: How is multi-turn realistic?
A: Each round conditions on prior generated frames + new command—mirrors streaming user control; compare to single-turn to see drift.

Q: Custom scenes/actions?
A: Yes—JSON scene specs plus composable instruction sets thanks to decoupled design.

Q: Hardware?
A: Harness runs on CPU; model inference wants ≥16 GB GPU; parallel batch needs multi-GPU planning.

Q: vs VBench?
A: VBench = static clip quality; WBench = interactive control + multi-turn decay.

9. Project Links

• Site: https://meituan-longcat.github.io/WBench/
• GitHub: https://github.com/meituan-longcat/WBench
• Hugging Face dataset: https://huggingface.co/datasets/meituan-longcat/WBench
• Paper: https://huggingface.co/papers/2605.25874