Xiaomi-OCR-0: In-Depth Evaluation of Xiaomi's Open-Source 0.8B Parameter Lightweight Multimodal Document Understanding Model

1. What is Xiaomi-OCR-0

Xiaomi-OCR-0 is a lightweight multimodal document understanding model with a 0.8B parameter scale, open-sourced by Xiaomi Corporation. It is trained based on the Qwen3.5-0.8B base model. This model unifies three core capabilities—document parsing, key information extraction (KIE), and OCR visual question answering (VQA)—into a single architecture. It ranks first on the Real5-OmniDocBench benchmark with a score of 95.24 and second on OmniDocBench v1.6 with a score of 96.83. It achieves an average score of 83.2 across five OCR-VQA tasks, surpassing models of similar parameter scale comprehensively. The model has been successfully deployed in Xiaomi's "Super Xiaoai" intelligent assistant on Xiaomi smartphones, achieving a complete closed-loop from laboratory research to on-device product implementation.

Technical Positioning and Domain: Belongs to the field of multimodal document understanding (Document AI), focusing on the intersection of OCR and vision-language models. Its unique positioning lies in achieving a triple integration of document parsing, key information extraction, and OCR visual question answering with an extremely compact 0.8B parameter size, offering significant competitiveness in on-device deployment scenarios.

Development Background: Developed by Xiaomi's AI Lab (Xiaomi Research), leveraging Xiaomi's deep expertise in on-device AI and smartphone ecosystems. The goal is to address the pain points of traditional OCR systems, which require stitching together multiple specialized models and are difficult to run efficiently on mobile devices. The team has invested significant effort in building the data engine, forming a complete automated data production pipeline.

Core Value: Solves the core issue of insufficient accuracy for lightweight models in complex document understanding tasks. Through 170 million high-quality training samples and an innovative heterogeneous expert consensus annotation (ETC) mechanism, it achieves a technical approach that trades data quality for model size, enabling the 0.8B parameter model to outperform general large models with tens of billions or even hundreds of billions of parameters on multiple benchmarks.

Technical Features: Employs innovative technologies such as heterogeneous expert consensus annotation, rendering-guided verification, and a progressive three-stage training approach, constructing a complete system of "data engine + training strategy + toolchain." It also provides配套 MCP (Model Context Protocol) service and Agent Skill toolchain, supporting flexible integration with intelligent agents.

2. Key Features

• Document Structural Parsing: Identify and reconstruct the main text, complex tables, and mathematical formulas within a document, outputting structured Markdown/LaTeX results in natural reading order. This feature is based on the model's deep understanding of page layout and can handle complex formats such as multi-column layouts and nested tables. The output can be directly used in document management systems or knowledge base construction.

• Key Information Extraction (KIE): For semi-structured documents such as invoices, bank cards, ID cards, and passports, this feature precisely locates and outputs structured JSON data according to specified fields. By using a unified instruction interface to switch task types, field-level extraction can be completed without changing the model, significantly simplifying the complexity of business process integration.

• OCR Visual Question Answering (VQA): Breaking through the limitations of traditional OCR text transcription, this feature supports cross-element retrieval, numerical calculation, and semantic reasoning questions on documents and charts. For example, you can ask, "How much did revenue increase in the second quarter compared to the first quarter?" The model can automatically locate the relevant data and perform the calculation and reasoning.

• Heterogeneous Expert Consensus Annotation (ETC) Data Engine: This engine uses eight OCR expert models with different architectures, data, and strategies to independently predict document regions. It calculates pairwise consistency separately for text (NED), tables (TEDS), and formulas (CDM), selects the top three consensus triplets, and takes the median value as pseudo-labels. Only samples with a consensus level ≥ 0.9 are directly added to the training set, filtering out single-model biases from the source.

• Render-Guided Refine-and-Judge: For regions where expert opinions differ, the Qwen3.5-122B-A10B large model re-renders the candidate parsing results into images and compares them with the original region at both the pixel and structural levels to identify hidden defects such as misaligned rows and columns or missing symbols. The correction process can iterate up to three rounds; samples that still do not converge are passed to a second-stage "diagnosis—correction—validation" workflow.

• Progressive Three-Stage Training: The first stage uses Q-Mask text anchoring pre-training to establish a fine-grained alignment between textual content and spatial positions. The second stage involves multi-task OCR continuous pre-training, expanding coverage to tasks such as full-page parsing, region identification, and VQA under the same autoregressive format. The third stage concludes with DAPO-style Mix-RL reinforcement learning on mined difficult examples, routing rewards based on task type.

• MCP and Agent Skill Toolchain: Provides standardized MCP services and Agent Skill components, enabling the model to seamlessly integrate into agent systems. Developers can implement an automated workflow of "recognize screen/document—extract fields—execute next step" through standard interfaces, reducing integration barriers.

3. Application Scenarios

• Real-time Credential Management: Identify time-sensitive information such as takeout pickup codes, delivery pickup codes, and queue numbers, and display them on the "island" for instant access when needed. This scenario has been implemented in Xiaomi's "Super Xiaoi" app, showcasing the model's capability for low-latency inference on the device side.

• Smart Accounting and Bill Organization: Extract amount, merchant, transaction time, and category of consumption with one click from the detailed pages of WeChat or Alipay bills, automatically generating structured accounting records. Compared to traditional template-matching approaches, this model can adapt to various bill formats without requiring separate rule configurations for each app.

• Structured Entry of Card and Document Information: Capture key fields from ID cards, bank cards, invoices, passports, etc., via photo or screen recognition, and output them in JSON format, replacing manual entry and verification processes. In scenarios such as finance, government services, and hotels where document entry is frequent, this significantly improves work efficiency and reduces the error rate caused by human input.

• Intelligent Schedule Generation: Extract information such as time, location, and participants from group chat notifications and event posters, automatically generating calendar event cards and adding them to the user's calendar. This scenario requires a high level of semantic understanding from the model, including the ability to distinguish between irrelevant information and accurately identify time expressions and location entities.

• Travel Information Integration: Extract travel details such as flight numbers and train numbers from chat records, automatically query real-time departure and arrival information, and push reminders. The model must accurately identify travel entities within unstructured text and integrate with external APIs, demonstrating its value as the "perception layer" of an intelligent agent.

4. How to Use

1. Environment Requirements: Requires Python 3.9 or higher version, and it is recommended to use the Linux operating system. Inference requires PyTorch 2.0+, and it is recommended to equip with an NVIDIA GPU (VRAM ≥ 4GB) to fully leverage the model's performance. The model can also run in a pure CPU environment, but the inference speed will significantly decrease.

2. Get Code and Install Dependencies: Clone the GitHub repository SeerRay-Lab/Xiaomi-OCR-0, and run pip install -r requirements.txt to install the required dependencies, including core components such as transformers, accelerate, and sglang.

3. Download Model Weights: Pull the 0.8B model weight files from the Hugging Face model library SeerRay-Lab/Xiaomi-OCR-0. You can use git lfs clone or download via the huggingface-cli download command. Ensure that your network environment can access the Hugging Face service.

4. Quick Experience: Run the project's built-in script to launch a browser-based interactive demo. Upload document images to experience document parsing, key information extraction, and visual question answering features online. This method is suitable for quickly verifying the model's performance without writing any code.

5. Configure Agent Calls: Integrate the MCP service or Agent Skill from the repository into your agent system, and implement an automated document processing workflow through standard interfaces. The MCP service runs as an independent process and communicates with the main program via the JSON-RPC protocol.

6. Deploy Local Service: Deploy the model as a local inference service using SGLang or vLLM, allowing business systems to call it via HTTP API. It is recommended to use SGLang to achieve better inference performance. When deploying, configure appropriate batch size and concurrency parameters.

7. Integrate Page Layout Detection (Two-Stage Pipeline): First, use PP-DocLayoutV3 to detect text, table, and formula regions. Then, crop each region and feed it into Xiaomi-OCR-0 for recognition block by block. Finally, assemble the results into page-level output. This approach provides more stable results in complex layout scenarios.

8. Write Instructions (Prompt): Construct prompts in a unified image–instruction–response format, specifying the task type as parsing, KIE, or VQA. The instructions must clearly define the output format requirements, such as specifying "output in Markdown format" or "output a list of JSON fields."

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

Selection Recommendations: If your primary requirement is edge deployment, low-latency inference, and the need to simultaneously cover three tasks—document parsing, key information extraction, and visual question answering—Xiaomi-OCR-0 is the preferred choice in the 0.8B–1.2B parameter range. Its robustness in real-world scenarios makes it particularly suitable for uncontrolled environments such as mobile phone photography and screen captures. If your workflow focuses on complex table processing and requires extremely high accuracy in table structure restoration, TeleOCR's TEDS score of 97.05 is more impressive, although its capabilities in KIE and OCR-VQA have not been fully demonstrated. If you need to handle general visual understanding tasks rather than specialized document parsing, and are not restricted by model size or deployment costs, general multimodal large models still offer broader coverage. However, they have been surpassed by specialized small models in document-specific metrics.

7. Editor's Summary

The technological innovation of Xiaomi-OCR-0 lies in its systematic breakthroughs in the data engine and training strategy, rather than mere architectural innovation. The heterogeneous expert consensus annotation (ETC) mechanism controls pseudo-label quality from the data source through multi-model voting and consistency screening; rendering-guided validation leverages the rendering capabilities of large models to trace parsing defects, forming a closed-loop of "parsing—rendering—comparison—correction." These two technologies together establish a scalable, high-quality data production pipeline. The accumulation of 170 million samples enables a 0.8B parameter model to match or even surpass general large models with billions of parameters on multiple benchmarks, validating the feasibility of the technical approach of "replacing model scale with data quality."

In terms of practical value, the model's most notable contribution is advancing on-device document understanding from "usable" to "user-friendly." The integrated design of three capabilities eliminates the engineering burden of combining multiple models, and the 0.8B parameter count makes smooth operation on mobile devices like smartphones a reality. Xiaomi's implementation in "Super Xiaoai" demonstrates the feasibility of this technical path in real-world products, providing a reference model for on-device document intelligence in the industry.

This model is suitable for the following users: application developers who need to deploy document understanding capabilities on mobile or edge devices; enterprise users who wish to achieve high-precision OCR parsing with lower computational costs; and researchers interested in the data engine methodology and looking to adopt its data production process. For scenarios requiring extreme table accuracy, it is recommended to combine Xiaomi-OCR-0 with TeleOCR or perform targeted fine-tuning.

Looking ahead, with continuous iteration of the data engine and gradual improvement of the community ecosystem, Xiaomi-OCR-0 still has clear room for improvement in areas such as Chinese scene optimization, complex table processing, and handwritten document recognition. If its data methodology is widely adopted by the community, it has the potential to further drive the document AI field toward a "data-driven" paradigm. Xiaomi's ongoing investment in on-device AI may also enable this model to be deployed across more Xiaomi ecosystem devices, forming a large-scale on-device intelligence network.

8. FAQ

Q: What is the fundamental difference between Xiaomi-OCR-0 and general-purpose OCR tools (such as PaddleOCR)?
A: Traditional OCR tools mainly perform text detection and recognition, outputting text box coordinates and text content; Xiaomi-OCR-0 is a multimodal document understanding model that can not only recognize text but also understand document structure (tables, formulas, reading order), extract key fields, and perform cross-element semantic reasoning. For example, PaddleOCR can recognize all the text on an invoice, but Xiaomi-OCR-0 can directly output structured JSON results such as "Invoice Number: 12345678" and answer semantic understanding questions like "What is the total amount on this invoice."

Q: How can a small model with 0.8B parameters surpass general large models with hundreds of billions of parameters in terms of accuracy?
A: The key lies in the differences in data quality and training strategies. General large models are pre-trained on massive general-purpose data, with document understanding being just one of their many capabilities; Xiaomi-OCR-0, on the other hand, is trained on 170 million high-quality samples specifically tailored for document understanding tasks. These samples are filtered and validated through heterogeneous expert consensus annotation (ETC) and rendering-guided verification, ensuring data accuracy from the source. Additionally, a progressive three-stage training strategy (text-anchored pre-training → multi-task continuous pre-training → Mix-RL reinforcement learning) enables the model to achieve deep specialization in document tasks, realizing the effect of "specialization surpassing breadth."

Q: What languages does Xiaomi-OCR-0 support? How effective is it for Chinese?
A: The model is trained on Qwen3.5-0.8B and provides good support for both Chinese and English. It performs well on standard document parsing tasks in both languages. However, it is important to note that the model is primarily trained on general document data, and its optimization for specific Chinese scenarios (such as handwritten Chinese, complex Chinese table styles, and ancient text layouts) is not as thorough as for English scenarios. It scores 73 points on the ancient Chinese calligraphy evaluation, but the evaluation dataset is limited and lacks comparative benchmarks. It is recommended that Chinese users perform targeted fine-tuning for better performance in special scenarios.

Q: How can I deploy Xiaomi-OCR-0 locally? What hardware configuration is required?
A: We recommend using an NVIDIA GPU with at least 4GB of VRAM for smooth operation. There are two deployment options: one is to deploy it as a local inference service using SGLang or vLLM and call it via HTTP API; the other is to directly integrate it with the transformers library for inference. It can also run on a CPU environment, but the speed will be slower. For production environments, it is recommended to use SGLang for better inference performance and to configure an appropriate batch size to improve throughput.

Q: Compared to TeleOCR (from Alibaba), what are the respective advantages of Xiaomi-OCR-0?
A: Both models have similar parameter sizes (0.8B vs 1.2B), but they are positioned differently. Xiaomi-OCR-0's advantage lies in its three-in-one capabilities (parsing + KIE + VQA) and robustness in real-world data collection scenarios (scoring 95.24 on Real5-OmniDocBench, the highest). It also comes with a配套 MCP and Agent Skill toolchain, making it more suitable for edge deployment and agent integration. TeleOCR slightly outperforms in table structure restoration (TEDS 97.05) and Wild-OmniDocBench (88.53), but it does not emphasize KIE and OCR-VQA capabilities. The choice depends on specific needs: prioritize table accuracy with TeleOCR, or choose Xiaomi-OCR-0 for multi-task coverage and edge deployment.

Q: How can I integrate Xiaomi-OCR-0 into my agent system?
A: The project provides two standard integration methods: MCP (Model Context Protocol) service and Agent Skill. The MCP service runs as an independent process and communicates with the main program via JSON-RPC protocol, supporting an automated workflow of "recognize screen/document → extract fields → execute next step." Agent Skill packages the model as a reusable skill component that can be directly embedded into mainstream agent frameworks. For specific integration steps, refer to the MCP configuration documentation and Agent Skill example code in the repository.

Q: What input formats does the model support? Can it process PDF files?
A: The model directly accepts image input (supporting common formats such as JPG, PNG, etc.). For PDF files, they need to be first converted into images (typically page by page), and then fed into the model for processing. It is recommended to use a two-stage pipeline: first use PP-DocLayoutV3 to detect layout regions, then feed each region into the model for recognition. For multi-page PDFs, process each page individually and manually concatenate the results.

Q: What is the open-source license for Xiaomi-OCR-0? Can it be used commercially?
A: The model weights and code have been open-sourced on GitHub and Hugging Face, using the Apache 2.0 license, which allows commercial use and modification. Please refer to the LICENSE file in the repository for specific terms. It is important to note that the model is trained on Qwen3.5-0.8B, and its use must also comply with the relevant usage terms of the Qwen series of models.

9. Project Links

• GitHub Repository: https://github.com/xiaomi-research/Xiaomi-OCR-0
• Project Website (Demo Experience): https://huggingface.co/spaces/xiaomi-research/Xiaomi-OCR-0
• Hugging Face Model Library: https://huggingface.co/xiaomi-research/Xiaomi-OCR-0
• arXiv Technical Paper: https://arxiv.org/pdf/2609.36136