LingBot-Touch: Ant's Lingbo Open-Source Tactile Pretrained Foundation Model, Bridging the Critical Gap in Robot "Contact Perception"

1. What is LingBot-Touch

LingBot-Touch is an open-source tactile pretrained foundation model developed by Ant's Lingbo Robotics team. Its core positioning is to act as a "translator" between tactile sensors and robot action models. This model can unify heterogeneous sensor signals such as magnetic tactile, visual tactile, and piezoresistive into standardized tactile Tokens, outputting features such as contact position, force magnitude, and direction that action models can directly understand. It effectively addresses the industry challenge where feeding raw signals directly into policy models reduces operational success rates. Trained using the JEPA (Joint Embedding Prediction Architecture) approach, LingBot-Touch employs a shared tactile Transformer backbone, with model parameters controlled at approximately 5 million. It has been validated on 26 real-robot tasks and demonstrates zero-shot transfer capabilities across unseen sensors, filling the crucial perceptual gap between vision, which handles pre-contact information, and tactile sensing, which handles post-contact details.

Technical Positioning and Domain: LingBot-Touch belongs to the intersection of robot tactile perception and representation learning, specifically targeting the direction of tactile pretrained foundation models. In robotic manipulation tasks, visual perception addresses the question of "where the object is and how to approach it," while tactile perception addresses the "contact phase" issues such as "where the fingers touch, how much force is applied, and whether there is slippage." This model establishes a unique positioning in this field: "unifying heterogeneous tactile signals and enabling zero-shot cross-hardware transfer," providing a general solution for standardizing and reusing tactile data.

Development Background: This model was developed and open-sourced by the Lingbo Robotics team under Ant Group. The motivation for its development stems from the widespread industry pain point of tactile data being "non-interoperable"—different sensor principles and tactile point layouts generate data in vastly different formats, requiring retraining from scratch when switching to a different hand. By leveraging its technical expertise in embodied intelligence, the Lingbo team chose the JEPA route to build a general tactile representation, significantly reducing data annotation costs through glove-based data collection, and ultimately open-sourced the technical report, code, and model weights.

Core Value: LingBot-Touch solves three major practical problems: first, it enables the unified encoding of heterogeneous tactile sensor signals, ending the issue of fragmented tactile data formats; second, through the "addressed" Token mechanism and shared backbone, it achieves zero-shot transfer for unseen sensors and dexterous hand layouts, making "the experience from one hardware a starting point for the next robot"; third, as a frozen encoder that can be seamlessly integrated with mainstream action models, it only increases the parameter count by approximately 0.13%–0.16%, achieving an average success rate improvement of over 20 percentage points across 26 real-robot tasks, significantly lowering the application threshold for tactile perception.

Technical Features: The core technical advantages of LingBot-Touch are reflected in four aspects: first, it uses a standardized geometric interface and Token addressing mechanism, using time, finger number, and 3D position as "addresses" and injecting them via RoPE, thus eliminating dependency on fixed tactile point layouts; second, it adopts a "multi-input, shared-backbone" hybrid architecture, with each of the three sensing principles equipped with a lightweight Tokenizer, followed by joint learning through a shared Transformer backbone; third, it is trained using JEPA-based feature space prediction, randomly masking 90% of tactile points to force the model to learn true physical representations of contact; and fourth, it introduces tactile point perturbation strategies, artificially creating a fourth layout to ensure that the learned representations are transferable across different hardware.

2. Key Features

• Heterogeneous Signal Unified Encoding: Translates the raw signals from three mainstream tactile sensors—magnetic tactile, visual-tactile, and piezoresistive—into standardized tactile Tokens. This feature addresses the significant differences in sampling frequency, physical units, and noise characteristics across different sensing principles, establishing a universal language for tactile data from the source. As a result, downstream action models can understand contact information without needing to distinguish the signal origin.

• Contact State Perception: Parses the contact position, magnitude, and direction of force over time from tactile signals, enabling robots to clearly understand "where the finger is touching, how much force is being applied, and whether there is slippage." This feature outputs physical contact quantities rather than raw readings, directly benefiting contact-intensive, fine-grained operations such as zipping a zipper or twisting a bottle cap, and provides actionable contact feedback to policy models.

• Zero-shot Cross-hardware Transfer: Utilizes a unified coordinate space and shared backbone architecture to map contact points into a standardized fingertip coordinate space. The Token address mechanism eliminates dependency on fixed numbers and arrangements of contact points. In practical tests, this model achieved a success rate increase from 48% to 74.17% on the XHand sensor, which it had never encountered before, without any parameter adjustments, fully validating the transferability of the representation.

• Plug-and-Play Integration with Action Models: As a frozen feature encoder, it can be directly mounted onto mainstream action models such as π0.5, Diffusion Policy, and T-Rex. The parameter count increases by only approximately 0.13%–0.16%. This feature enhances existing policy models with a tactile perception channel in a minimally invasive manner, without altering the original model's network structure or training process, significantly reducing the engineering cost of adopting tactile perception.

• High-frequency, Efficient Real-time Inference: The smallest version, HaT-Tiny, contains only about 5.46 million parameters and supports high-frequency real-time inference, meeting the millisecond-level response requirements of dexterous operations. This characteristic allows tactile encoding to be embedded into real-time inference loops on actual hardware, without becoming a computational bottleneck, making it especially critical for contact tasks sensitive to real-time performance.

• Improved Success Rate for Contact-intensive Tasks: Achieves an average success rate improvement of over 20 percentage points across 26 real-robot tasks, particularly excelling in contact-intensive, fine-grained operations such as zipping a zipper, twisting a bottle cap, and inserting a straw. This feature provides clear representations of contact states, helping policy models make more accurate force control decisions during contact phases and reducing failures caused by blind force application.

3. Application Scenarios

• Retail Sorting: In scenarios such as pharmacies and convenience stores, robots need to pick up items of varying sizes and shapes. By integrating LingBot-Touch into the fingertips, robots can real-time perceive grip force and slippage, allowing them to adaptively adjust clamping force and effectively reduce product damage rates. This scenario requires high deformation perception capabilities from tactile sensing, and the standardized Token, after unified encoding, can directly drive motion control strategies.

• Logistics Sorting: When handling soft packaging and fragile items (such as bagged goods and bottled liquids), LingBot-Touch enables robots to sense package deformation and internal movement, achieving stable grasping and placement. Compared to visual solutions, the tactile modality continues to provide reliable contact feedback under conditions such as packaging bag occlusion and lighting changes, ensuring the stability of sorting operations.

• Industrial Precision Assembly: During assembly tasks such as plugging, twisting, and alignment (e.g., screwing on bottle caps, inserting keys to unlock), the camera view is often obstructed by hands or components, making tactile sensing the sole source of perception. LingBot-Touch continuously senses changes in resistance during the contact phase, providing real-time haptic feedback to the strategy model, enabling precise and controllable assembly actions even when "out of sight."

• Catering and Beverage Operations: During fine tasks such as inserting straws, squeezing toothpaste tubes, or sealing bags, robots need to perceive the sudden force change at the moment of film puncture. LingBot-Touch can capture subtle changes in contact states, helping robots apply or release force at the right time to avoid package damage or operational failure, maintaining consistent product quality during high-frequency repetitive tasks.

4. How to Use

1. Obtain model resources: Get the open-source code from the GitHub repository (https://github.com/Robbyant/lingbot-touch), download the pre-trained model weights from the Hugging Face model library (https://huggingface.co/collections/robbyant/lingbot-touch) or the ModelScope community, and access the technical paper (https://arxiv.org/pdf/2611.12806) to understand the implementation details.

2. Integrate sensor signals: Connect the raw signals from existing tactile sensors on the dexterous hand—such as magnetic tactile, visual-tactile, or piezoresistive sensors—to LingBot-Touch. The model is designed to be compatible with heterogeneous sensor inputs, requiring no hardware replacement or adjustment to the sensor installation method. Simply integrate the raw data stream according to the interface specifications provided in the code repository.

3. Encode tactile features: Before running LingBot-Touch forward inference, encode the raw tactile signals into standardized tactile Tokens. The model output includes standardized representations such as contact position, force magnitude, and force direction. These features can be directly read by downstream action models, eliminating the need for additional feature engineering.

4. Post-training of action models: When training policy models (e.g., π0.5, Diffusion Policy), include the frozen LingBot-Touch features as additional inputs during post-training. Specifically, add the tactile Token vectors to the existing observation space. The network structure of the policy model does not need to be modified, only a small number of input dimensions are added.

5. Deploy inference in production: Deploy the trained policy onto a real robot. During deployment, LingBot-Touch runs in real-time as a frozen encoder. The parameter count increases by approximately 0.13%–0.16%, having minimal impact on inference latency. It is recommended to choose the full version or the lightweight HaT-Tiny version based on the actual computational resources of the robot's main controller, to balance accuracy and real-time performance.

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

For selection recommendations, scenarios focusing on R&D and generalizabilityshould prioritize LingBot-Touch. Its zero-shot cross-hardware migration capability allows research teams to develop strategies on one hardware platform and then migrate them to the next generation of robotic platforms, significantly reducing redundant development costs. This makes it particularly suitable for laboratory environments with multiple robot models and mixed sensor usage. However, for production scenarios involving large-scale deployment on a single hardware platform, if the sensor model is fixed over the long term and the task types are relatively simple, traditional specialized solutions or FTP-1 can still be viable options. These approaches may offer better performance on specific hardware, but their limitations in scalability and cross-platform capabilities must be clearly acknowledged. Overall, for teams pursuing generalization of tactile capabilities and reuse of data assets, LingBot-Touch is currently the more forward-looking choice.

7. Editor's Summary

LingBot-Touch's core innovation in the field of tactile perception lies in successfully introducing the methodology of pre-trained foundational models into the tactile modality, addressing the long-standing issue of data generalizability that has hindered the practical application of tactile technologies. From a technical perspective, its "separate entry, shared backbone" hybrid architecture and feature space prediction strategy based on JEPA demonstrate clear rationality: the former uses a lightweight Tokenizer to handle differences in sensor hardware, while the latter employs 90% masking training to force the model to focus on actual contact information rather than background statistical patterns. The introduction of the contact point perturbation mechanism precisely corrects the hidden risk of "model memorization of layouts," and these design choices ensure the physical transferability of representations at the system level, rather than merely focusing on data-level improvements.

In terms of practical value, LingBot-Touch achieves a remarkable success rate improvement of over 20 percentage points with a lightweight parameter scale of approximately 5 million and a parameter intrusion cost of just 0.13%–0.16%, offering a compelling cost-benefit ratio in the field of tactile perception. Its zero-shot cross-hardware migration capability (48% → 74.17%) directly addresses the long-standing problem in the robotics industry of having to start over when switching hardware, providing a viable path for the asset-based reuse of tactile data.

In terms of positioning, this model is suitable for three types of users: first, research teams conducting studies on dexterous manipulation, who can use it as a standard tactile feature extractor to accelerate strategy iteration; second, robotics hardware manufacturers aiming to reduce the adaptation costs across multiple platforms through a general tactile encoding; and third, tactile sensor manufacturers who can leverage this model to unify the data interface standards across their product lines. Looking ahead, as the open-source community expands to include more sensor principles and continues to grow in data scale, the "tactile foundational model" approach represented by LingBot-Touch is likely to become the foundational infrastructure in the field of robotic tactile perception, driving tactile sensing from a "specialized accessory" toward a "general-purpose capability."

8. FAQ

Q: What types of tactile sensors does LingBot-Touch support?
A: Currently, it supports three mainstream tactile sensing principles: magnetic tactile, visual tactile, and piezoresistive. The model employs a "separate entrance, shared trunk" architecture, designing specialized Tokenizers for each of the three sensor types and mapping the Tokens uniformly to a standardized fingertip coordinate space. For sensors outside of these three categories (e.g., capacitive or ultrasonic), the official has not yet confirmed compatibility. It is recommended to follow future updates in the project repository.

Q: Does LingBot-Touch require a separate adapter for each robot?
A: No. This is one of the core design advantages of the model: during pre-training, it learns hardware-agnostic representations of contact physics, rather than specific sensor geometries. In practical testing, the model achieved zero-shot transfer on the XHand sensor, which it had never encountered before, without any parameter adjustments. However, when connecting to a completely new sensor, it is recommended to first validate performance on a small dataset to confirm the transfer effectiveness.

Q: How much computational resources are needed to integrate LingBot-Touch?
A: The full version of LingBot-Touch has approximately 5 million parameters, while the smallest version, HaT-Tiny, has about 546 thousand parameters. The model has relatively low computational requirements. When running as a frozen encoder, the additional inference overhead is minimal, enabling high-frequency real-time inference on common robotic main control devices. During deployment, the appropriate version can be selected based on the device's computational power.

Q: How can LingBot-Touch be integrated with existing action policy models?
A: The model is integrated as a frozen encoder. It encodes raw tactile signals into standardized Tokens and then concatenates the Token features as additional input into the observation space of the action model (e.g., π0.5, Diffusion Policy, T-Rex). This process does not require modifying the network structure of the action model, and only increases the parameter count by approximately 0.13%–0.16%, minimally interfering with the original training process.

9. Project Links

• Project Website: https://technology.robbyant.com/lingbot-touch
• GitHub Repository: https://github.com/Robbyant/lingbot-touch
• Hugging Face Model Library: https://huggingface.co/collections/robbyant/lingbot-touch
• Technical Paper: https://arxiv.org/pdf/2611.12806