EvaOmni – An Open-Source Multimodal Large Model Evaluation Benchmark from Shanghai Jiao Tong University, Fudan University, and Others

1. What is EvaOmni

EvaOmni is an open-source multimodal large model evaluation benchmark jointly developed by institutions such as Shanghai Jiao Tong University, Fudan University, and Tencent YouTu Lab. It aims to address the deeper flaws in current video understanding evaluations, such as "getting the answer right but being unable to provide supporting evidence." The benchmark includes 392 real video clips and 1,027 human-annotated questions and answers, covering 15 capability dimensions and 3 reasoning levels. It requires models to not only provide answers but also return time intervals of audio and video evidence, and it evaluates models comprehensively based on three metrics: answer accuracy, evidence coverage, and bimodal localization. In practical tests with 24 mainstream multimodal models, the top-performing Gemini3.7-Flash achieved an accuracy rate of 79.4%, but its bimodal localization score was only 35.1%, a gap of 59.2 percentage points compared to human experts' 94.3%. This data clearly exposes the significant shortcomings of current models in terms of evidence traceability capabilities.

Technical positioning and domain: Belongs to the field of multimodal large model evaluation and benchmarking, focusing on systematic assessment of audio-visual joint understanding capabilities. EvaOmni's unique positioning lies in incorporating "evidence localization" as a core part of the evaluation, thus promoting the industry's shift from a mere accuracy-oriented approach to a more interpretable and traceable reasoning capability verification.

Development background: Jointly developed by multiple top-tier academic and industrial institutions, including Shanghai Jiao Tong University, Fudan University, and Tencent YouTu Lab. The motivation for its development stems from the fact that existing video question-answering benchmarks (such as Video-MME) only assess the correctness of answers, failing to distinguish whether models truly understand audiovisual content or rely on shortcuts from a single modality. Therefore, a more rigorous evaluation system is needed to reveal the true capabilities of models.

Core value: Solves the critical issue in video understanding evaluations where "correct guessing" and "correct answers with supporting evidence" cannot be distinguished. By requiring models to output bimodal evidence time intervals and using dual metrics—evidence coverage and bottleneck localization—to quantify the quality of evidence, EvaOmni can precisely identify specific shortcomings of models at different depths, such as retrieval, understanding, and long-range multi-hop reasoning, providing actionable directions for model improvement.

Technical features: It constructs a four-stage data production pipeline (video screening, human annotation, cross-review, shortcut filtering), using three unimodal inputs—text, audio, and vision—to filter out questions that can be answered through shortcuts. Additionally, it designs a dual metric scoring system that evaluates based on the weaker modality, preventing models from exploiting strong modalities to compensate for weak ones.

2. Key Features

• Answer Evaluation (Acc.): Determine whether the model's answer to audio-visual multiple-choice questions is correct, and output an overall accuracy metric. This metric serves as a fundamental scoring component, providing the prerequisite for subsequent evidence quality assessment, ensuring that only samples with correct answers proceed to in-depth analysis of evidence localization.

• Evidence Annotation and Return: Each question is annotated with human-labeled audio and video time intervals that support the answer. The model must return both audio and video time intervals along with the answer. This design expands the model's output from a single option to a triplet structure of "option + audio timestamp collection + video timestamp collection," allowing the evaluation program to separately verify the correctness of the answer and the quality of evidence localization.

• Evidence Coverage Score (C.G.): Calculates the proportion of human-annotated evidence segments that are hit by the model's predicted intervals, measuring whether the model has identified all supporting clues. This metric focuses on the model's recall ability—whether it found the evidence or not—and allows for broad predictions to earn some score, without distinguishing the precision of the boundaries.

• Dual-Modal Bottleneck Localization (B.G.): Computes the temporal intersection over union (IoU) separately for audio and video, and takes the weaker of the two modalities as the final score. This design enforces the requirement for the model to accurately locate evidence in both modalities simultaneously, mechanically preventing the model from masking weaknesses in one modality with strengths in the other, ensuring the authenticity and fairness of the evaluation results.

• Three-Tier Capability Hierarchy: The dataset is organized into three levels based on reasoning and evidence complexity: Level 1 tests the alignment capability of time, events, and entities across audio and video; Level 2 requires basic understanding by integrating complementary audio-visual cues to comprehend events, emotions, and intentions; Level 3 demands complex reasoning by combining multiple pieces of evidence scattered across different time points for multi-hop judgments. As the level increases, the number of pieces of evidence per question and the coverage duration significantly increase.

• Shortcut Filtering Mechanism: The model is prompted to answer using three separate unimodal inputs: text, audio, and visual. Any question that can be answered correctly by a single modality or by textual common sense is returned for revision and re-annotation. This mechanism systematically blocks the "shortcut" of bypassing joint audio-visual reasoning at the data level, ensuring that evaluation questions must rely on genuine cross-modal understanding to be solved.

• Model Horizontal Evaluation and Leaderboard: Provides a unified benchmark and leaderboard for 24 full-modal models, including both open-weight and proprietary models. Researchers can connect their own models using standardized protocols for direct comparison with existing leaderboard results, supporting model selection and technical iteration decisions.

3. Application Scenarios

• Multimodal Model Development and Evaluation: When iterating on the Omni large model, model teams can use the Acc./C.G./B.G. metrics to precisely identify capability gaps where "answer accuracy improves but evidence localization lags." For example, if the model's accuracy improves but the B.G. score remains stagnant, it indicates structural issues in the model's temporal awareness and cross-modal alignment. This allows for targeted adjustments to training data and model architecture.

• Enterprise AI Selection and Acceptance: In industries such as finance, government, and healthcare, where models must "provide verifiable evidence," purchasers can use the B.G. bottleneck identification metric to filter out models that truly possess dual-modal reliability. Compared to looking at accuracy alone, the evidence localization metric effectively identifies models that "seem intelligent but lack substantiation," reducing the risk of business implementation.

• Video Content Moderation and Traceability: Content moderation applications require the model to clearly indicate where违规 content appears—in which frame or which audio segment. EvaOmni's time interval output format and localization metrics can directly serve as acceptance criteria for moderation systems, helping to build traceable and verifiable automated moderation workflows.

• Q&A Products for Educational/Film Content: Video-based learning assistants and film explanation products need to reference original video clips when answering questions. Developers can use EvaOmni to evaluate whether the model can accurately locate relevant scenes and dialogue, ensuring that the product provides verifiable video evidence when answering user questions, thereby enhancing user experience and content credibility.

• Academic Research and Benchmark Construction: Researchers can conduct in-depth studies on audio-visual reasoning, temporal localization, multi-hop reasoning, and other areas using the open-source dataset of 392 video clips with evidence annotations. This benchmark provides the academic community with high-quality annotated data resources, supporting reproducible experimental comparisons.

4. How to Use

1. Install the ModelScope client: Run the command pip install -U modelscope to install or upgrade the ModelScope SDK. This SDK is the fundamental tool for downloading datasets and model weights. It is recommended to use a Python 3.8 or higher environment.

2. Download the dataset: Use the dataset_snapshot_download function, specifying the dataset ID as VisionXLab/EvaOmni-Dataset, to pull all data into a local directory. The dataset includes 392 video clips, 1,027 question-answer pairs, and corresponding audio/video evidence interval annotation files.

3. Prepare model input: Preprocess the videos according to the evaluation protocol, including standardizing parameters such as resolution and frame rate, and extracting the audio content accordingly. Ensure the input format aligns with the interface specifications of the model under evaluation to avoid affecting inference results due to format issues.

4. Integrate the model under evaluation: Follow the integration instructions provided in the GitHub repository to connect your multimodal model to the evaluation framework. Use a unified prompt template to make the model output a triplet structure of "option + audio timestamp + video timestamp," ensuring the output format can be correctly parsed by the evaluation program.

5. Run the evaluation: Use the evaluation framework to perform batch inference on all 1,027 question-answer pairs. The evaluation program will automatically verify the correctness of the answers, calculate the overlap between the evidence intervals and the manually annotated ones, and output scores for each metric. It is recommended to run this in a GPU environment to reduce inference time.

6. View results and analysis: Read the three core metrics—Acc., C.G., and B.G.—as well as the scores for the three-tiered capability classification, and compare them with the results of existing models on the leaderboard. By analyzing performance differences across inference levels, identify specific weaknesses in the model's retrieval, basic understanding, and complex reasoning capabilities, providing data support for further optimization.

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

Selection Recommendations: If the evaluation goal is to verify a model's explainability and evidence traceability in real-world scenarios, EvaOmni is currently the most targeted option. Its dual-modal localization metric can effectively identify models that "get the answer right but can't justify it," making it particularly suitable for scenarios with strict traceability requirements such as finance, government affairs, and content moderation. For teams needing a comprehensive evaluation of a model's generalization capabilities across various video understanding tasks, Video-MME remains a reliable benchmark due to its larger scale and broader task coverage.

In model development and iteration scenarios, it is recommended to use EvaOmni in conjunction with Video-MME: use Video-MME to quickly screen the model's overall capability baseline, then use EvaOmni to deeply analyze shortcomings in evidence localization. This combined strategy ensures broad coverage of the evaluation while providing actionable improvement directions, especially suitable for the iteration and optimization cycles of full-modal large models.

7. Editor's Summary

The release of EvaOmni marks a paradigm shift in video understanding evaluation, moving from a single-dimensional focus on "answer accuracy" to a dual-dimensional approach encompassing "answer + evidence." This innovation holds significant value for the industry. From a technological standpoint, its dual-metric scoring system (C.G. + B.G.) and three-tiered capability stratification design offer notable differentiation compared to similar benchmarks. Particularly, the mechanism of determining scores based on weaker modalities effectively prevents models from engaging in speculative behaviors at the evaluation level, thereby enhancing the fairness of assessments. Empirical data reveals a substantial capability gap—59.2 percentage points between the strongest models and human experts in dual-modal localization—which highlights a critical bottleneck in the development of fully multimodal models.

In terms of practical value, EvaOmni provides enterprises with a more discerning evaluation tool for selecting AI models. In scenarios such as finance, government affairs, and content moderation, where traceable evidence is essential, relying solely on accuracy for selection poses considerable risks. EvaOmni's evidence localization metric effectively filters out models that genuinely possess dual-modal understanding capabilities. This benchmark is applicable to full-modal model development teams, AI procurement decision-makers, content moderation system developers, and academic researchers working on audiovisual reasoning, especially for users who prioritize model interpretability and deployment reliability.

Looking ahead, EvaOmni's open dataset and standardized evaluation protocol lay the groundwork for establishing a unified evidence localization evaluation standard across the industry. As more organizations adopt this benchmark for model iteration and selection validation, it is expected to drive continuous improvements in the evidence traceability capabilities of fully multimodal models. Expanding the dataset size and covering more Chinese-specific scenarios will be key priorities for its future development, essential for realizing its broader industry value.

8. FAQ

Q: What is the core difference between EvaOmni and Video-MME?
A: In addition to answer accuracy, EvaOmni requires the model to return the time intervals of audio and video evidence. It quantifies the quality of evidence through two metrics: Evidence Coverage (C.G.) and Bimodal Bottleneck Localization (B.G.). In contrast, Video-MME only evaluates the accuracy of multiple-choice answers and does not involve evidence localization. EvaOmni focuses more on testing the model's ability to trace evidence and provide explainability.

Q: Why does the B.G. metric take the weaker of the audio and video IoU values?
A: This is to prevent the model from exploiting the stronger modality to compensate for the weaker one. If the average or maximum value were used, the model could accurately locate the video while ignoring the audio and still achieve a high score. By using the weaker modality's score, EvaOmni enforces accurate localization in both modalities, ensuring that the evaluation results genuinely reflect the model's bimodal joint understanding capability.

Q: Are there any requirements for the timestamp format in model outputs?
A: The model must output a triplet structure consisting of "option + audio timestamp set + video timestamp set." Timestamps are expressed in interval form, and the evaluation program calculates the overlap with manually annotated intervals using temporal intersection over union (IoU). For specific format requirements, please refer to the integration documentation in the GitHub repository.

Q: How does the shortcut filtering mechanism work?
A: Each question is presented to the model in three unimodal formats: text-only, audio-only, and visual-only. If the model can answer the question correctly under any of these unimodal conditions, it indicates that the question can be solved via a shortcut, without relying on true audio-visual joint reasoning. Such questions will be revised and re-annotated or removed entirely, ensuring that the final evaluation questions can only be answered through cross-modal understanding.

Q: Can the EvaOmni dataset be used freely for commercial purposes?
A: The dataset is publicly released via HuggingFace and ModelScope, following open-source licensing agreements. Please refer to the license details provided on each platform for specific terms. Academic research and commercial applications are typically allowed, but it is recommended to confirm the exact licensing terms before formal commercial use.

9. Project Links

• Project Website: https://xiaoxing2001.github.io/EvaOmni.github.io/
• GitHub Repository: https://github.com/VisionXLab/EvaOmni
• HuggingFace Dataset: https://huggingface.co/datasets/VisionXLab/EvaOmni-Dataset
• arXiv Technical Paper: https://arxiv.org/pdf/2610.28444