SemIf: Technical Analysis of the Open-Source Semantic Decision Model Replicating the Jev Pattern

1. What is SemIf

SemIf (formerly OpenJev) is an open-source AI decision model designed to replicate the semantic decision interface pattern of TypeSafe's closed-source service, Jev, using open-source methods. This model employs a single forward pass mechanism, directly reading the logits corresponding to candidate options for scoring, and constructs structured results on the server side, thereby avoiding the complex process of traditional autoregressive generation followed by text parsing. When multiple questions share the same state, SemIf can reuse prefix calculations, increasing throughput by nearly an order of magnitude. The model can run on a single RTX 3090 GPU and provides a browser-based demo environment, including reproducible benchmark tests line by line, a temperature calibration mechanism, and a 27B model bridging solution, balancing performance, auditability, and deployment flexibility.

Technical positioning and domain: SemIf belongs to the semantic decision and reasoning models within the field of natural language processing. Its core application areas include structured decision outputs, candidate option scoring, and lightweight Agent decision-making. Its unique positioning lies in replicating the decision interface pattern of a closed-source commercial service through open-source means, filling a gap in this niche area of open-source development.

Development background: This project was initiated by independent developer TheoLeeCJ, previously known as OpenJev, with the aim of verifying and replicating Jev's semantic decision capabilities through a fully open-source approach. The project team has submitted all fixed prompts, model versions, line-by-line outputs, and known failure cases to the repository, demonstrating a strong emphasis on reproducibility and transparency.

Core value: SemIf addresses the issues of high latency, frequent formatting errors, and unstable decoding that arise from the "generation-parsing" workflow in traditional decision tasks. By directly reading logits for scoring, it reduces the time required for 21 binary decisions to approximately 1 second, which is 5.2 times faster than generating JSON, significantly improving decision efficiency and reliability.

Technical features: The model uses a frozen 4B parameter Qwen series as its base, supporting multiple backends including CUDA, Apple Silicon, CPU, and browser-based WebGPU. It includes an internal temperature calibration mechanism tailored to workload, aligning output confidence with actual accuracy, and can directly drive downstream automated decision workflows.

2. Key Features

• Direct logit scoring: A single forward pass directly reads the logits of candidate options and converts them into probabilities, completely skipping the answer text generation and parsing steps. Compared to the traditional approach of generating text autoregressively and then parsing JSON, this mechanism improves speed by approximately 5 times, while fundamentally eliminating issues such as JSON formatting errors, decoding loops, and output parsing failures.

• Shared state reuse: When multiple decisions share the same context state, the system only needs to prefill the prefix once, after which it can sequentially or parallel evaluate each decision criterion. Measured data shows that throughput increases from a baseline of 2.33 decisions per second to a maximum of 20.03 decisions per second under this mode, an improvement of nearly an order of magnitude, making it especially suitable for batch decision scenarios.

• Runtime-definable decisions: Decision criteria and option descriptions are dynamically passed with each request, eliminating the need for fine-tuning or predefined task templates for specific tasks. This feature enables the model to flexibly handle small Agent decision needs such as route selection and retry determination, significantly reducing task adaptation costs.

• Fully auditable: The project repository includes fixed prompts, model versions, prompt hashes, line-by-line output results, and known failure cases. Any user can precisely reproduce benchmark test results and audit and verify each decision individually, a feature that is relatively rare in AI decision-making projects.

• Temperature calibration: An internal temperature scaling mechanism is built-in, which is fitted to the workload and improves consistency between probability confidence and actual accuracy. Calibrated confidence outputs can be directly used in automated decision-making processes, avoiding misjudgments caused by overconfidence or underconfidence.

• Multi-backend support: Supports CUDA GPU, Apple Silicon (MPS/MLX), CPU, and browser-based WebGPU demo environments. Users can choose the most suitable runtime method based on their hardware conditions, covering everything from consumer-grade GPUs to pure CPU environments, significantly lowering the barrier to entry.

3. How to Use

1. Environment Setup: Install Python 3.10 or higher and configure the CUDA environment (if using GPU inference). The graphics card should have at least 8GB of VRAM to accommodate the frozen 4B BF16 model. It is recommended to use an RTX 3090 or a graphics card with comparable performance.

2. Install the Project: After cloning the GitHub repository, create and activate a virtual environment, then run pip install -e '.[test]' to install the project and its testing dependencies. This command will also install all libraries required for model inference.

3. Prepare Input Data: Create a decision file in JSONL format, where each line includes id (decision identifier), state (shared state context), question (description of the decision question), and options (list of candidate options). Format example: {"id": "1", "state": "User account status is normal", "question": "Which queue should this ticket be routed to?", "options": ["Account Support", "Billing Support", "Technical Support"]}.

4. Run Scoring: Execute the command semif-score --mode direct --model Qwen/Qwen3.5-4B --revision <fixed version> --input examples/decisions.jsonl --output results.jsonl. After completion, the output file will contain per-line option probabilities, inference time, and prompt hash values.

5. Shared State Mode: When all lines in the input file have exactly the same state field, use the --mode shared parameter instead. The system will pre-fill the shared prefix only once, then perform parallel evaluation of all decision criteria, resulting in a throughput improvement of nearly an order of magnitude.

6. Temperature Calibration Application: Use the project's built-in calibration script to fit temperature scaling parameters on the target workload. Once calibration is complete, the output probability confidence will align with actual accuracy and can be directly used in automated decision-making processes.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For teams prioritizing low latency, high throughput, and full control over the decision-making process, SemIf is an ideal choice, especially in scenarios where GPU resources are already available and data privacy is a concern. Its open source nature also facilitates auditing and secondary development. If a team lacks hardware resources or requires maximum decision quality, commercial hosted services like Jev can be considered, though they come with the trade-off of reduced control due to their closed-source nature. For scenarios with highly fixed tasks and ample training data, dedicated classification models may offer superior accuracy and speed, but they fall short in flexibility and generalization compared to SemIf.

6. Editor's Summary

SemIf demonstrates clear technological innovation in the field of AI decision-making models. Its core contribution lies in transforming the traditional "generate-parse" decision-making paradigm into a discriminative paradigm of "direct scoring," a structural change that brings about a speed improvement of over five times while fundamentally eliminating formatting errors and decoding instability during the text generation process. The design of the shared state prefix reuse mechanism also reflects deep engineering considerations, enabling nearly an order of magnitude increase in throughput for batch decision-making scenarios. In terms of practical value, SemIf keeps the hardware requirements low, allowing it to run on a single RTX 3090 GPU, while providing support for multiple backends including CPU, Apple Silicon, and WebGPU, covering a wide range of deployment scenarios from servers to personal computers. The project's commitment to reproducibility—fixed prompts, fixed model versions, line-by-line output submissions, and known failures—sets a high standard for transparency in open-source AI projects.

This model is primarily suitable for development teams and researchers who are sensitive to decision latency, require batch processing of decision tasks, or wish to incorporate the decision-making process into an audit system. Its runtime-defined decision-making feature makes it highly valuable in lightweight decision-making scenarios such as Agent routing, action firewalls, and evidence ranking. In terms of growth potential, as the 27B model bridging solution matures and community contributions accumulate, SemIf is expected to further narrow the gap in decision quality with closed-source services. Its ability to run on edge devices also provides opportunities for expansion in the field of on-device AI decision-making. For teams seeking decision efficiency and auditability, SemIf offers an open-source solution worth thorough evaluation.

7. Application Scenarios

• Customer Service Ticket Routing: Based on the customer's problem description and account status, directly score across queue options such as "Account Support / Billing Support / Technical Support," enabling millisecond-level automatic assignment. SemIf's logit direct scoring mechanism ensures extremely short decision-making time, effectively supporting real-time routing needs in high-concurrency customer service scenarios.

• Agent Action Firewall: Before performing high-risk operations, the Agent makes a binary judgment on candidate actions (execute/reject). SemIf achieves a composite accuracy of 0.700 in this scenario, serving as the first layer of filtering in the security framework. It can quickly intercept clearly inappropriate actions, reducing the risks associated with autonomous Agent behavior.

• Evidence Retrieval and Ranking: In codebase retrieval and enterprise knowledge base Q&A scenarios, SemIf can replace traditional rerankers by providing relevance scores for candidate documents with a single forward pass. In practical testing, it achieved an accuracy of 0.929 on knowledge base Q&A tasks, balancing ranking quality with inference speed.

• Inference Conclusion Verification: Used for NLI tasks, it determines the logical relationship between "evidence and conclusion." Confidence outputs calibrated with temperature can directly drive downstream automated processes, such as automatically flagging questionable inference conclusions for manual review, thereby enhancing the overall reliability of the decision-making pipeline.

8. FAQ

Q: What is the core difference between SemIf and Jev?
A: SemIf is an open-source implementation (MIT License), which completes decision-making by directly reading and scoring option logits, with zero output tokens throughout the process. Jev is a closed-source commercial managed service provided by TypeSafe, with its internal mechanisms not disclosed. SemIf achieves a consistency rate of 0.845 on the 102-line subset, which is lower than Jev's 0.883, but offers local deployment, full auditability, and higher decision speed.

Q: What hardware environments does SemIf support?
A: SemIf supports CUDA GPUs (recommended RTX 3090 or equivalent or higher performance GPU), Apple Silicon (accelerated via MPS/MLX), pure CPU environments, and browser-based WebGPU demonstrations. The default 4B BF16 model requires approximately 8GB of GPU memory, and users can choose the most suitable backend based on their own conditions.

Q: What are the requirements for using the shared state mode (--mode shared)?
A: The shared state mode requires that the state field in all lines of the input JSONL file be exactly the same. When this condition is met, the system pre-fills the shared prefix only once, then parallelly evaluates all decision criteria, increasing throughput from 2.33 to a maximum of 20.03 decisions per second. If the state fields vary across lines, the default direct mode must be used.

Q: How is the decision quality of SemIf ensured?
A: The project ensures decision quality through three mechanisms: first, fixed prompts and model versions ensure reproducibility; second, an internal temperature calibration feature fits and scales parameters according to the workload, aligning confidence with actual accuracy; third, known failure cases are submitted to the repository, helping users understand the model's capability boundaries. Additionally, the 27B model bridging solution can achieve a consistency rate of 0.958 on manual decision-making tasks.

Q: How can SemIf be integrated into an existing Agent system?
A: SemIf is provided as a command-line tool, outputting in JSONL format that includes per-line option probabilities and prompt hashes. Developers can integrate it into existing systems by calling it as a subprocess or encapsulating it as a microservice. Since decision criteria and options are dynamically passed with each request, no separate fine-tuning is required for each task, resulting in a low integration cost.

Q: What is the 27B model bridging in SemIf?
A: The 27B model bridging is a quality-enhancing solution provided by SemIf, which executes the same logit scoring logic on a larger-scale model, achieving a consistency rate of 0.958 on manual decision-making tasks. This solution is suitable for scenarios with higher decision quality requirements, but it requires corresponding hardware resources.

9. Project Links

• Project Website: https://openjev.com/
• GitHub Repository: https://github.com/TheoLeeCJ/SemIf-OpenJev