WeVisOne-VL Evaluation: WeChat's Vision Team Explores Unified Vision-Language Modeling with a Single Token Prediction Architecture

1. What is WeVisOne-VL

WeVisOne-VL is an open-source dual-version vision-language foundation model developed by Tencent's WeChat Vision Team, built upon the Qwen3-VL architecture, and offers two parameter scales: 2B and 4B. This model introduces a unified structured prediction paradigm known as "coordinates as tokens," enabling a single model to simultaneously perform seven visual tasks: object detection, referential localization, counting, pointing, OCR, referential segmentation, and promptable image matting. Its built-in pixel decoder contains only 9.6 million parameters and can directly output binary masks and hair-level alpha channel matting, eliminating the need for external segmentation models like SAM. This fundamentally simplifies the traditional "detector + segmenter" multi-model cascaded workflow at the architectural level.

Technical positioning and domain: Belongs to the field of multi-modal vision understanding and unified modeling of vision-language tasks, covering multiple fundamental visual capabilities such as detection, segmentation, OCR, and counting. It represents an engineering exploration in the direction of end-to-end general-purpose vision-language models. Its positioning lies between general-purpose VLMs and specialized vision models, aiming to replace multiple specialized model pipelines with a single set of weights.

Development background: Developed and open-sourced by Tencent's WeChat Vision Team, which has long-term technical experience in business scenarios such as content moderation and image understanding. The model is based on the Qwen3-VL architecture and uses the Apache 2.0 license, targeting both academic research and industrial deployment. The project's official website and GitHub repository are now publicly available.

Core value: Addresses the fragmented deployment issue of visual tasks—previously, detection, segmentation, and matting required multiple models to be chained together. Now, a single model, a unified set of weights, and a single inference pipeline can cover all seven tasks, significantly reducing deployment costs and inference latency. Its ability to output "None" when the target does not exist effectively mitigates hallucination problems in the visual perception stage, which is particularly important for sensitive scenarios such as content moderation and robotic perception.

Technical features: Replaces traditional coordinate regression heads with a structured output approach of "coordinates as tokens," unifying detection, segmentation, and OCR under the next-token prediction framework; includes a built-in 9.6 million parameter pixel decoder that directly outputs masks and alpha matting, freeing the model from reliance on external segmentation models; trained on over 100 million images and approximately 3.6 billion object bounding boxes, combined with a three-stage curriculum fine-tuning and hybrid task reinforcement learning strategy, achieving state-of-the-art performance on more than 30 benchmarks.

2. Key Features

• Structured Output Paradigm with Coordinates as Tokens: The model discretizes coordinates into 1001 specialized tokens, with one detection box occupying 4 tokens and one point occupying 2 tokens. Detection boxes, text positions, and pixel masks are all expressed using the same vocabulary. Task switching only requires changing the prompt prefix, and when the target does not exist, the model directly generates None. This unifies the output format across seven types of tasks at the generation mechanism level, eliminating the need to design dedicated output headers for each task.

• Built-in Pixel Decoder: In addition to the Qwen3-VL base model, a small decoder module with only 9.6 million parameters is integrated. The visual feature adapter extracts four layers of ViT features from before the base model's pre-merging step, which are then refined into a shared feature pyramid via deep convolution. Each <OBJ> token's hidden state, combined with Fourier-encoded box priors, forms an instance query. After being iteratively refined through six deformable cross-attention blocks, the instance condition dense head first predicts the mask, then restores a full-resolution alpha matte at hair-level detail through a three-stage cascaded process: S/4 → S/2 → S/1. This entire process does not require external segmentation models like SAM.

• Rejection Capability: During the GRPO reinforcement learning phase, the model treats two types of errors as key penalties: "outputting boxes when no target is present in the image" and "missing boxes when targets are present." This significantly improves the reliability of rejection capability. On the WeRef-Bench benchmark, the 4B version achieves a rejection accuracy of 96.4%, while the comparison model Qwen3-VL-8B only reaches 18.3%. This ability is highly valuable for scenarios with high costs of false positives, such as content moderation and robot obstacle avoidance.

• Promptable Matting: The model supports specifying targets through natural language or point selection, and outputs a continuous alpha channel instead of a binary mask. Fine details such as hair strands and semi-transparent objects are preserved naturally, enabling direct use in downstream tasks like background replacement and e-commerce white background image generation. According to official data, the MAD (Mean Absolute Difference) of matting is almost halved compared to cascaded approaches.

• OCR and Position Output: The model can not only read text content within images but also annotate the coordinates of the text location, unifying text recognition and spatial localization within the same output format. This directly supports scenarios such as document structuring and invoice information extraction.

• Knowledge Graph-Driven Data Engine: The knowledge graph, which includes category and referential expressions, serves as the basis for target-specific data collection. Candidate boxes are generated by SAM3 and an internal DINO detector, verified by Qwen3-VL for authenticity, masks are expanded by SAM2, and continuous alpha candidates are generated by BiRefNet and MatAnyone2. Samples must pass multiple checks, including IoU, coverage, and foreground accuracy, before being accepted. Errors made by the model are fed back into the next round of screening, forming a self-improving annotation loop. This has cumulatively produced over 100 million images and approximately 3.6 billion target boxes for training data.

3. Application Scenarios

• Content Moderation and Security Detection: Leverage reliable rejection recognition capabilities to accurately locate non-compliant objects in images, and clearly return None when no such objects are found, rather than forcing an output. This effectively reduces missed reviews and false positives. This capability aligns closely with WeChat's content moderation requirements, making it particularly suitable for handling existence judgment tasks such as "whether it contains a specific prohibited item."

• E-commerce Product Image Segmentation: After inputting a product image, directly output hair-fine alpha segmentation. With one click, generate a white background image or replace the background with a poster. This replaces manual image editing and the multi-model pipeline of "detection + SAM + segmentation models." The 2B version can run on consumer-grade GPUs, making it suitable for small and medium e-commerce teams to integrate at a low cost.

• Document and Invoice Structuring: Combine OCR with coordinate output capabilities to convert invoices, contracts, and forms into structured data with positional information. This data can be directly integrated with automated order entry, reconciliation, and document management systems, reducing the need for manual data entry.

• Data Annotation and Retrieval Enhancement: Utilize reference localization and segmentation capabilities to batch produce annotated data with bounding boxes and masks for downstream model training. In text-to-image retrieval scenarios, it can return precise target regions rather than the entire image, improving retrieval accuracy and the usability of results.

• Robotics and Autonomous Driving Perception: Deploy on the edge for environmental perception and object manipulation. The rejection recognition mechanism prevents hallucinations such as "identifying air as an obstacle," thereby enhancing operational safety. A single model covers detection, segmentation, and pointing tasks, simplifying the software stack complexity of robotic perception systems.

4. How to Use

1. Environment Requirements: Requires a Linux operating system and a Python 3.9 or higher environment. It is recommended to use PyTorch 2.1 or later versions. The 2B version can perform inference on a single GPU with 16GB VRAM, while the 4B version is recommended to use a GPU with 24GB VRAM or higher. The official recommendation is to use NVIDIA Ampere architecture or newer GPUs to fully leverage performance.

2. Obtaining Code and Installation: Clone the Tencent/WeVisOne-VL repository from GitHub (git clone https://github.com/Tencent/WeVisOne-VL.git), and install the required dependencies according to the instructions in the README file, including core components such as transformers and deformable-attention. It is recommended to use conda to create an isolated virtual environment to avoid dependency conflicts.

3. Preparing Model Weights: Download the WeVisOne-VL-2B or WeVisOne-VL-4B weight files from the Tencent organization collection page on Hugging Face. After downloading, place the weights in the designated directory within the repository, or specify the model path using environment variables.

4. Constructing Task Prompts: Clearly specify the task requirements in the prompt according to the task type. For detection tasks, use the format Locate all the instances of …. For object localization, describe the target characteristics directly in the prompt. For segmentation tasks, explicitly request the output of the alpha channel. Existing annotations can be passed to the model using structured tags such as <ref>, <box>, and <point>.

5. Calling the Inference Interface: Use the model.predict() interface provided in the repository to input the image and prompt text into the model. The interface internally handles the full workflow of tokenization, inference, and output parsing. The official demo is built based on this interface.

6. Parsing the Output: The model generates results token by token. Coordinates from the <0>-<1000> specialized tokens are converted back to normalized coordinates. The <OBJ> token triggers the pixel decoder to output the mask or alpha channel. If the target does not exist, the model returns a None structure.

7. Post-processing and Application: Map the bounding box, mask, or segmentation results back to the original image dimensions for use in downstream tasks such as retrieval, annotation, or background replacement. It is recommended to write custom scripts for coordinate restoration and mask scaling based on actual business requirements.

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

Selection Recommendations: If the business scenario involves four types of requirements: detection, segmentation, inpainting, and OCR, and is sensitive to deployment complexity, WeVisOne-VL's seven-in-one design can significantly reduce the number of models and the length of the inference pipeline, making it particularly suitable for content moderation and e-commerce scenarios. The 4B version has a differentiated advantage in rejection accuracy and pixel-level output capability, making it one of the few "single-model end-to-end" solutions among open-source vision-language models.

Another Set of Requirements: If lightweight deployment on edge devices is needed and the primary tasks are basic detection and captioning, the 0.23B version of Florence-2 offers better power efficiency; if there is an established SAM segmentation pipeline and only stronger grounding capability is required, Qwen3-VL-8B remains a reliable general-purpose base model. The applicable boundary for WeVisOne-VL is scenarios that require pixel-level output without the need to maintain multiple models.

7. Editor's Summary

The technical value of WeVisOne-VL lies in unifying the output format of visual tasks into a single expression framework based on token sequences, replacing the traditional model's design of independent output heads for each task with a structured next-token prediction approach. This architectural choice directly reduces the cost of task expansion—new tasks no longer require the design of new output branches or loss functions, only the adjustment of prompt prefixes is needed. The design of the 9.6 million parameter pixel decoder also demonstrates clever engineering, achieving lightweight, high-quality mask and alpha outputs through shared feature pyramids and deformable cross-attention, thereby avoiding the long inference chains and increased deployment complexity that come with using SAM externally.

From the practical benchmark results, the 4B version outperforms several comparison models with 8B parameters across multiple dimensions, including detection (LVIS 55.3), counting (PixMo-Count 89.6), and OCR (TotalText 52.0), highlighting its parameter efficiency. The significant difference in rejection capability (96.4% vs. 18.3%) is particularly notable, indicating that the GRPO reinforcement learning phase's targeted punishment for two types of errors—"fabricating information" and "missing information"—has yielded tangible results, and also validates the feasibility of a "mistake notebook"-style training strategy in visual tasks. The knowledge graph-driven data engine is also worth noting, with a scale of 100 million images and 3.6 billion bounding boxes combined with a self-improvement mechanism for error feedback, providing a replicable path for data generation for future visual models.

This model is suitable for four types of users: backend development teams requiring multi-task visual capabilities, developers of e-commerce and design tools relying on precise image segmentation, engineers of review systems with strict requirements on false positive rates, and small and medium-sized teams seeking low-cost access to unified visual capabilities. The Apache 2.0 license and dual version configurations (2B/4B) lower the entry barriers for teams of different scales.

Notable shortcomings include the fact that current public benchmarks are primarily focused on English scenes and general visual tasks, with performance on Chinese scenes and fine-grained domain-specific tasks (such as medical imaging and satellite imagery) not yet fully disclosed. Additionally, the knowledge graph data engine heavily relies on the quality of the graph during the cold start phase, and whether the open-source community can further improve the coverage of Chinese and vertical domain knowledge graphs will be a key factor in determining the model's scalability. As the community ecosystem continues to grow, WeVisOne-VL has the potential to become a representative open-source practice in the visual-language model domain for the "single model, multiple tasks" approach.

8. FAQ

Q: What is the relationship between WeVisOne-VL and Qwen3-VL?
A: WeVisOne-VL is built upon Qwen3-VL, using its visual encoder and language model as the foundation. It adds a unified token output head and a pixel decoder module on top of this, and has undergone large-scale multi-task fine-tuning. It can be seen as a derivative model that has been specialized for visual tasks based on Qwen3-VL.

Q: How to choose between the 2B and 4B versions?
A: The 2B version is suitable for edge or consumer-grade scenarios with limited resources, as it requires less VRAM and has lower inference latency. The 4B version outperforms the 2B version in terms of detection accuracy, rejection accuracy, and segmentation details, making it more appropriate for server-side deployment where output quality is critical. It is recommended to make a decision based on the required business accuracy and hardware budget.

Q: Does the model support Chinese input?
A: Qwen3-VL natively supports both Chinese and English, as well as multiple other languages. WeVisOne-VL inherits this capability, supporting Chinese reference localization and OCR. The official benchmarks are primarily based on English, so for Chinese scenarios, it is recommended to validate with a small-scale test set before deploying in production.

Q: Can inference be run on a CPU?
A: The 2B version can be run on a CPU, but the inference speed will significantly decrease, resulting in several seconds of latency for detection and segmentation tasks. For production environments, it is recommended to use an NVIDIA GPU and enable FP16 or INT8 quantization to improve throughput.

Q: How to address unsatisfactory hair detail in the segmentation edges?
A: First, confirm the input image resolution—too low a resolution can result in loss of hair detail. Second, check whether the prompt accurately specifies the target region; using point selection (via the <point> tag) can help the model more precisely identify the target area. Finally, ensure that the <OBJ> token is used to trigger the pixel decoder path, rather than relying solely on detection output.

Q: Is commercial use supported?
A: WeVisOne-VL is licensed under the Apache 2.0 License, which allows for commercial use, modification, and redistribution without requiring payment for a license. However, the original copyright notice must be retained, and the user assumes full responsibility for compliance during usage.

Q: What does "None" mean in the model's output?
A: "None" indicates that the model has determined the target described by the user does not exist in the image. This is a normal output from the rejection mechanism, demonstrating the model's ability to confidently state "there is nothing." If the business scenario requires distinguishing between "uncertain" and "definitely not present," it is recommended to perform a secondary judgment at the application layer using the model's confidence score.

9. Project Links

• Project Website: https://tencent.github.io/WeVisOne-VL/
• GitHub Repository: https://github.com/Tencent/WeVisOne-VL
• Hugging Face Model Library: https://huggingface.co/collections/tencent/wevisone-vl
• arXiv Technical Paper: https://arxiv.org/pdf/2610.31329