Laya – Open-Source Non-Autoregressive AI Decision Model, an Open-Source Alternative to Jev

1. What is Laya

Laya is an open-source non-autoregressive AI decision model based on a bidirectional encoder architecture, developed by the ConvAI Innovations team. Unlike mainstream large language models, Laya does not generate text. Instead, it uses the ModernBERT/mmBERT bidirectional encoder to perform a single forward pass on a given input state and a set of typed questions, outputting structured decision results and calibrated probabilities in approximately 33ms. The model includes an in-built language routing mechanism, supporting automatic identification and distribution across 100+ languages. Compared to the closed-source model Jev, Laya offers clear advantages in accuracy (76.6% vs 72.7%), multilingual support (100+ languages), and cost (free for local deployment), providing an efficient, controllable, and fully open-source alternative for structured decision tasks.

Technical Positioning and Domain: Laya belongs to the domain of structured decision models in natural language processing, focusing on tasks such as classification, scoring, and yes/no judgments. Its uniqueness lies in completely abandoning the autoregressive text generation paradigm, instead adopting a bidirectional encoder with typed outputs, achieving order-of-magnitude improvements in inference speed for specific tasks.

Development Background: This model was developed by the ConvAI Innovations team, whose motivation was directly aimed at addressing two major pain points of existing large models in structured decision scenarios—excessive inference latency and unreliable confidence scores. The team selected ModernBERT-large and mmBERT-base as the backbone networks, combined with the RLCD reinforcement learning training method, to build a specialized model optimized for decision tasks.

Core Value: Laya addresses the efficiency and reliability issues of traditional large models in scenarios requiring fast, calibrated, and interpretable decisions. Its confidence scores are trained using strict true scoring rules, giving them statistical significance and enabling them to directly drive automated approval or manual review business processes. This feature holds practical business value in scenarios such as financial risk control, content moderation, and intelligent customer service.

Technical Features: Laya employs a bidirectional non-autoregressive architecture, assigning an independent [MASK] position to each candidate option and outputting the probability distribution of all options in a single forward pass. Through training with the RLCD strict true scoring rules, the model only achieves high scores when it outputs genuine posterior probabilities, fundamentally avoiding the issue of "blind confidence" commonly found in traditional large models.

2. Key Features

• Single Forward Decision: Given a state text input and a set of typed questions, the model can directly return structured results with just one forward pass, without any text generation. Real-world testing shows that inference on a T4 GPU takes approximately 33ms per query, and in batch scenarios, it can be as low as 7.2ms per question. This is about 7.8 times faster than the closed-source competitor Jev, providing a solid technical foundation for high-concurrency real-time decision-making scenarios.

• Three Decision Primitives: The model includes three fundamental decision types: choice, score, and noul. Choice selects one option from a set and provides probabilities for each. Score assigns a rating on an ordered scale and outputs the distribution. Noul answers yes/no questions and returns a probability between 0 and 1. These three primitives cover the majority of structured decision-making scenarios in classification, scoring, and yes/no judgment, and can be freely combined.

• Language Auto-routing: The built-in Router component detects the Unicode character set and language of the input text using pure Python before inference, taking less than 0.5ms. It automatically routes the request to the English or multilingual checkpoint. This mechanism addresses the issue of English models confidently providing incorrect answers with 0.95 confidence but 0 accuracy on non-Latin scripts.

• Confidence Threshold Gating: The probabilities output by the model, trained with RLCD, have statistical significance. Users can set a confidence threshold (e.g., 0.85), allowing high-confidence decisions to be automatically approved, while low-confidence results are forwarded for manual processing. This feature enables Laya to be integrated into automated business workflows, ensuring accuracy while controlling human labor costs.

• Predefined Workflow Templates: Offers preconfigured question templates for model routing, prompt injection protection, content safety review, and ticket triage, covering high-frequency scenarios such as Agent system entry protection and UGC platform content moderation. Users do not need to design question templates from scratch and can use them out-of-the-box.

• Local Fine-tuning Capability: Comes with a Kaggle notebook that supports a complete fine-tuning process using proprietary data, including data generation, RLCD training, temperature fitting, and evaluation. Users can customize the model based on their own business data, and after temperature fitting, the average ECE can be reduced from 0.466 to 0.081.

3. How to Use

1. Install Environment: Run pip install laya to install the PyPI package. Note that this package is still in rapid development, with the version number increasing from 0.1.0 to 0.3.4 within two days. The API may change, so it is recommended to lock the version number for use. It is recommended to use Python 3.9+ environment, and for GPU inference, it is advised to use T4 or higher models.

2. Load the Model: Use Router(preload=True) to pre-load three checkpoints (English, multilingual, and routing model), avoiding the need to rebuild the model every time the language is switched during cold start. Pre-loading increases initial memory usage but significantly reduces first-time inference latency.

3. Prepare State Input: Assemble the text to be analyzed into a dictionary structure, for example, {"from": ..., "subject": ..., "body": ...}. This structure supports any language or JSON nested formats, and users can freely define fields based on their business scenarios.

4. Define Typed Questions: Write a set of questions in JSON format, specifying the type (choice / score / noul), instructions, and criteria (option list or scale range). For example, classification questions should provide candidate options, and scoring questions should define the upper and lower bounds of the scale.

5. Perform Prediction: Call router.predict(state, questions) to perform inference, returning answers to all questions in a single forward pass. English input automatically routes through the laya checkpoint, while non-Latin languages such as Hindi are automatically directed to the multilingual checkpoint. The routing decision is completed before inference.

6. Read and Gate: Extract the choice labels, score values, or noul probabilities from res["answers"], and the res["routing"] field explains why this checkpoint was selected. Set a threshold for confidence (e.g., 0.85) to implement a workflow where high-confidence results are automatically processed and low-confidence results are forwarded to human agents — but this requires first performing temperature fitting on your own data.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

From the comparison, it is evident that Laya and Jev each have their own strengths. In scenarios where latency is critical, data privacy is a high priority, multilingual support is required, and budget is limited, Laya clearly has a significant advantage with its 33ms-level inference speed and zero deployment cost. However, in high-cardinality classification tasks (such as intent recognition with 70+ options) and scenarios requiring precise alignment with teacher distributions, Jev demonstrates greater maturity and stability.

For selection recommendations, if the business scenario primarily involves binary classification, low-cardinality classification, or scoring/ranking tasks, and strict requirements are placed on inference latency and data privacy, Laya is a more cost-effective choice. If the task involves distinguishing among a large number of fine-grained categories (such as ticket classification with 70+ categories), or if the team lacks the capability for fine-tuning, it is advisable to prioritize Jev's commercial API service.

6. Editor's Summary

Laya has made clear trade-offs in its technical approach: sacrificing general-purpose text generation capabilities in favor of focusing on the niche scenario of structured decision-making. This "less is more" design philosophy yields two direct benefits — inference speeds in the 33ms range and confidence outputs with statistically significant reliability. The former is attributable to the architecture design that allows a bidirectional encoder to output the probabilities of all options with a single forward pass, while the latter stems from the RLCD training method with its strict true scoring rules. These two technical choices form the core of Laya's competitive advantage.

In terms of practical value, Laya's confidence gating mechanism provides a reliable technical foundation for automated decision-making processes. Under the premise that ECE drops to 0.081 after temperature calibration, the probability outputs can directly drive business rule engines, enabling a closed-loop system with high-confidence automated processing and low-confidence manual intervention. This capability has direct production value in scenarios such as customer service ticket triage, content safety review, and Agent protection.

Laya is primarily suitable for: backend engineers requiring low-latency decision-making capabilities, enterprise developers concerned with data privacy, small and medium-sized teams looking to reduce inference costs, and international business developers needing multilingual support. The Apache-2.0 license and complete fine-tuning toolchain lower the barrier to entry, but the model's relatively weak zero-shot performance necessitates users to have a certain level of fine-tuning capability.

In terms of growth potential, the "specialized small model + strict probability training" approach represented by Laya offers an alternative to large models in AI decision-making scenarios. As the RLCD training method matures and the community ecosystem develops, models like Laya are expected to be deployed in more vertical domains. However, shortcomings in high-cardinality classification capabilities and interface stability still require continuous improvement. These are also the thresholds that open-source projects must overcome to transition from prototypes to production environments.

7. Application Scenarios

• Customer Service Ticket Triage Automation: In customer service systems, Laya can determine the department (billing/technical/sales) a ticket belongs to, its urgency, user frustration level, and risk of churn. High-confidence tickets are automatically routed to the corresponding department, while low-confidence tickets are forwarded to human agents for handling. With a P95 latency of 33ms, real-time triage is achieved, significantly reducing the response time for customer service teams.

• Content Safety and Moderation: Community and UGC platforms can leverage Laya's noul primitive to detect toxic, harassing, and threatening content, directly outputting the probability of violation. Moderation systems can set thresholds to automatically block content with a probability exceeding 0.9, while borderline content is forwarded for human review, balancing efficiency and accuracy.

• Prompt Injection Protection (Guardrails): Deploying Laya at the entrance of an Agent system enables detection of jailbreaking, injection, and key leakage attacks. With a P95 latency of 33ms, it can be embedded into every request chain without affecting user experience, providing a pre-processing security layer for LLM applications.

• Intelligent Model Routing: In a hybrid large model architecture, Laya can determine whether a request should be routed to a smaller model or a cutting-edge large model. Through low-cost scheduling decisions, simple requests are efficiently directed to smaller models while maintaining answer quality, effectively reducing overall inference costs.

8. FAQ

Q: What is the fundamental difference between Laya and traditional large language models (e.g., GPT, Llama)?
A: Laya does not generate text. Instead, it performs a single forward pass on the input using a bidirectional encoder and directly outputs structured decision results. Traditional LLMs generate text token by token, resulting in high latency and a risk of hallucination. Laya assigns independent [MASK] positions to each candidate option and outputs the probability distribution of all options in one forward pass, making it faster and more controllable in output format.

Q: Why can Laya's confidence scores be directly used for automated decision-making?
A: Laya is trained using RLCD reinforcement learning, with strict true scoring rules such as log scoring, spherical scoring, and ranked probability scoring as rewards. The model only receives high scores when it outputs true posterior probabilities, and confidence naturally decreases when uncertain. After temperature fitting, the average ECE can be reduced to 0.081, at which point the probabilities have statistical significance and can directly drive automated approval or manual review processes.

Q: Why is temperature fitting necessary? Can't the out-of-the-box model be used directly?
A: The out-of-the-box checkpoint suffers from overconfidence, with a probability distribution that is overly sharp. Temperature parameters must be fitted on proprietary data based on (question type, number of options) to calibrate the probabilities. After fitting, the average ECE can drop from 0.466 to 0.081. Using uncalibrated probabilities for threshold judgments may lead to a large number of misclassifications.

Q: How does Laya perform on non-Latin languages? Why is language routing needed?
A: The English checkpoint will provide answers with 0 accuracy at a confidence level of 0.95 for non-Latin scripts, and the confidence itself cannot issue warnings. Therefore, Laya includes a Router that detects the input script and language before inference and automatically routes it to the English or multilingual checkpoint. The Router supports over 100 languages, with detection time under 0.5ms.

Q: Is Laya suitable for handling high-cardinality classification tasks?
A: No. On the Banking77 high-cardinality classification task, Laya achieves only 0.425 accuracy (77 options), while Jev reaches 0.870 (72 options). If the task involves distinguishing among a large number of fine-grained categories, consider other approaches. Laya is more suitable for binary classification, low-cardinality classification, and scoring/ranking tasks.

9. Project Links

• GitHub Repository: https://github.com/NandhaKishorM/laya
• Hugging Face Model Library: https://huggingface.co/convaiinnovations/laya