QwenGyre – The Online Reinforcement Learning Framework for Long-Horizon Agents Launched by Alibaba Qwen

1. What is QwenGyre

QwenGyre is an online reinforcement learning framework introduced by the Alibaba Qwen team, specifically designed for long-horizon agent tasks that involve executing approximately 700,000 tokens in a single run and can last for several hours or even longer. The framework dynamically switches idle GPU resources between sampling and training through an elastic scheduling mechanism, without interrupting the task. It also leverages trajectory processing techniques to preserve real context, evaluate partial results, and deduplicate shared prefixes to control training costs. On the Qwen3.8-2.4T flagship model, QwenGyre improved the pass rate of the NL2RepoBench benchmark from 52.48% to 58.54% in just 48 steps, achieving a 1.78x training acceleration compared to asynchronous end-to-end approaches, setting a new technical benchmark for long-horizon agent reinforcement learning.

Technical Positioning and Domain: QwenGyre belongs to the domain of large language model reinforcement learning training infrastructure, specifically focusing on the intersection of online reinforcement learning (Online RL) and agent training. Unlike traditional offline RL or short-horizon online RL, QwenGyre extends the scope of reinforcement learning training to long-horizon agent tasks involving millions of tokens per execution, filling a critical gap in the field and providing a practical training solution for complex scenarios such as code generation and software engineering automation.

Development Background: QwenGyre was developed by the Alibaba Qwen team, which has accumulated extensive experience in large-scale pre-training and alignment on the Tongyi Qwen series of models. Long-horizon agent tasks are in high demand in real-world applications, but training them presents core challenges such as GPU memory limitations, difficulties in parallelizing sampling and training, and sparse trajectory signals, which traditional frameworks struggle to address. Based on deep optimization experience with the SGLang inference engine and the Megatron training framework, the Alibaba Qwen team introduced two key technologies—elastic scheduling and trajectory tree processing—to build QwenGyre, a specialized online RL framework for long-horizon scenarios.

Core Value: QwenGyre addresses three major key challenges in online reinforcement learning for long-horizon agents: first, it achieves a high degree of overlap between sampling and training through elastic GPU scheduling, eliminating GPU idle waste; second, it ensures the consistency of real context in training signals using trajectory trees and Token-in/Token-out mechanisms, preventing model degradation caused by "false context"; third, it maximizes the utilization of effective training signals under conditions of sparse long-horizon task signals and high costs, through partial scoring and trajectory budgeting mechanisms, significantly improving the pass rate of the flagship model on complex code generation tasks.

Technical Features: The core technical advantage of QwenGyre lies in its fine-grained cell-level elastic scheduling architecture. Each cell can provide sampling services or host a training replica, and the scheduler dynamically switches roles based on "water level," with the switching process taking only a few seconds. Meanwhile, request rerouting and RDMA KV cache migration technologies ensure the complete preservation of the agent's execution state and tool state during the switch, achieving a seamless elastic training experience with zero task interruption.

2. Key Features

• Elastic GPU Scheduling: Divide the GPU cluster into multiple homogeneous computing units (cells). Each cell can provide sampling services via SGLang or host full training replicas via Megatron. The scheduler tracks the "water level" (amount of unfinished tasks) in real time. When the remaining sampling capacity is sufficient to support ongoing tasks and the training data supply is adequate, idle cells are seamlessly switched into training mode in sequence, achieving a high degree of overlap between sampling and training, and eliminating GPU idle time and waste.

• Request Re-routing and KV Cache Migration: During the GPU role switch, the proxy redirects model requests to the new inference engine and automatically retries unfinished requests, without requiring the backend processes or communication groups to be rebuilt. Meanwhile, KV cache is migrated using RDMA technology, fully preserving the agent's workspace and tool states. The switch takes only a few seconds, negligible compared to the hours-long execution time, ensuring zero interruption to tasks.

• Streaming Training and Mid-Session Node Joining: The core cell stores authoritative parameters and optimizer states. Satellite cells can pull parameter snapshots via RDMA and join ongoing batches at any time, receiving micro-batches according to progress and participating in gradient accumulation. The core cell then consolidates and updates all changes. This mechanism allows training to scale incrementally as data becomes available, achieving a high degree of pipeline integration between training and sampling.

• Trajectory Tree and Real Context Preservation: By using a Token-in Token-out mechanism, the exact input/output token IDs and pairs of behavior strategies with log probabilities are recorded for each model call. Shared prefixes are reused, and branches are created at fork points, forming a trajectory tree structure. Each output always corresponds to the original context at the time of generation, avoiding "false context" training signals caused by historical compression or concatenation of sub-agent contexts.

• Partial Scoring: For tasks that are not completed due to timeouts, if they have produced verifiable intermediate results (such as partial code files) during execution, they are still scored according to task rules and included in training. This mechanism maximizes the utilization of sparse but valuable signals from long-running executions, preventing the waste of all training data due to incomplete tasks.

• Trajectory Budget and Loss Balancing: Each execution retains a maximum of Jmax trajectories. Shared prefixes are counted only once for loss, preventing tasks with excessive branching from gaining disproportionately high weights during training. Loss calculation first averages the selected tokens within the execution and then averages across all executions, decoupling execution weight from the number of branches, ensuring training stability.

• Fine-grained Resource Allocation: Cell-level resource scheduling eliminates the long-tail waiting and idle issues of the Colocate mode, as well as the rigid fixed partitioning of the Async mode. The training process is fully hidden behind sampling, achieving maximum GPU utilization and significantly reducing training time under the same GPU budget.

3. How to Use

Currently, QwenGyre is publicly released in the form of an academic paper, and the Alibaba Qwen team has not yet opened the source code or installation packages. The following usage process is based on the technical architecture disclosed in the paper, and is intended for reference by researchers and engineers.

1. Environment Requirements: Deploying QwenGyre requires a multi-GPU cluster environment. High-performance GPUs such as NVIDIA A100 or H100 are recommended. The cluster nodes must support RDMA networks to enable KV cache migration and parameter snapshot synchronization. On the software side, SGLang inference engine and Megatron training framework must be pre-installed, along with properly configured PyTorch and CUDA environments.

2. Cluster Initialization and Cell Partitioning: Divide the GPU cluster into multiple homogeneous computing units (cells). Each cell can independently provide sampling services or host a training replica. Configure the core cell (core cell) to store authoritative weights and optimizer states, while the others serve as satellite cells (satellite cells) and are ready to join training batches at any time.

3. Scheduler Configuration: Launch the elastic scheduler (Elastic Scheduler) and set the "watermark" threshold parameter, which defines the balance condition between the amount of unfinished tasks, available sampling capacity, and training data supply. The scheduler automatically determines when to switch idle cells from sampling mode to training mode based on this configuration.

4. Task Submission and Trajectory Processing: Submit ultra-long-horizon agent tasks to the system. The agent records the exact input/output token IDs and policy log probabilities for each model call. The trajectory processor organizes trajectory trees by shared prefixes. During the training admission phase, it selects no more than Jmax trajectories for each execution based on role priority. Only tokens generated by the policy model are involved in the loss calculation.

5. Training Execution and Monitoring: The core cell aggregates gradient updates from all satellite cells and synchronizes the authoritative parameters. During training, key metrics such as GPU utilization, overlap between sampling and training, and the number of trajectory branches can be monitored in real time. Hyperparameters such as Jmax and partial scoring rules can be adjusted based on task completion status.

6. Notes: Since QwenGyre is designed for ultra-long-horizon tasks, the time span of training batches is relatively large. It is recommended to accurately estimate task execution time and token consumption before training, and to reasonably configure the number of cells and scheduling thresholds. Additionally, ensure that the cluster's network bandwidth meets the requirements for RDMA transmission to avoid KV cache migration becoming a performance bottleneck.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For research teams working on ultra-long-term intelligent agent tasks such as code generation and software engineering automation, QwenGyre's elastic scheduling and trajectory tree mechanism can significantly improve training efficiency and model performance, making it an ideal technical solution. However, it requires waiting for the code to be open-sourced and having an RDMA network environment. For short-term RL training needs at the dialogue level, mature open-source frameworks like VeRL from ByteDance offer simpler deployment and more complete community support, making them a more practical choice. For teams that already have a fixed GPU cluster and relatively stable task patterns, it may be worth considering incorporating some of QwenGyre's trajectory processing ideas into the Async approach for optimization.

6. Editor's Summary

QwenGyre demonstrates strong technical innovation in the field of online reinforcement learning for ultra-long-horizon agents. Its core contribution lies in introducing the concept of elastic scheduling into the RL training infrastructure. By implementing resource management at the cell level with fine-grained control, it breaks through the GPU utilization bottleneck caused by fixed partitioning in traditional Async approaches, as well as the problem of sampling and training blocking each other in Colocate approaches. Experimental data on the Qwen3.8-2.4T flagship model show that training with 48 steps increased the NL2RepoBench pass rate from 52.48% to 58.54%, achieving an end-to-end acceleration of 1.78 times. These figures fully validate the framework's effectiveness in real-world large-scale scenarios.

In terms of practical value, QwenGyre addresses key pain points in training ultra-long-horizon agent tasks—sparse signals, high costs, and context distortion. The trajectory tree and TITO mechanism ensure the fidelity of training signals, while the partial scoring mechanism maximizes the utilization of intermediate results from timeout tasks. These designs are of significant importance in enhancing the model's capabilities in complex engineering tasks. The primary users of the framework are research teams and AI platform engineering teams working in areas such as code generation, software engineering automation, and multi-agent collaboration.

It is worth noting that QwenGyre is currently only available in paper form, and its code has not yet been open-sourced, which to some extent limits the dissemination and application of the technology. However, considering the active participation of the Alibaba Qwen team in the open-source community, it is expected that once open-sourced, QwenGyre could become a crucial infrastructure in the field of ultra-long-horizon RL training. As the complexity of agent tasks continues to increase, the elastic scheduling and trajectory processing technical approach represented by QwenGyre provides a solution that is worthy of reference for the industry.

7. Application Scenarios

• Complete Code Repository Generation via Natural Language: Build multi-package software repositories from scratch based on natural language specifications, with a single execution handling approximately 700,000 tokens and lasting several hours. QwenGyre continuously optimizes the model's planning and execution capabilities for long-range code generation tasks through online RL training, gradually improving the compilability and functional correctness of the generated repositories.

• Long-Range Software Engineering on Existing Repositories: Perform complex tasks such as feature development, bug fixing, and cross-file refactoring within existing codebases. The model needs to continuously read, understand, and modify multiple files over several hours. QwenGyre's trajectory tree mechanism ensures that training signals align with real context, effectively enhancing the model's code comprehension and modification capabilities.

• Multi-Round Terminal Automation Operations: Intelligent agents perform hundreds of consecutive tool calls in a command-line environment to complete system configuration, service deployment, and operations tasks. QwenGyre's partial scoring mechanism allows for the evaluation of partial results from timeout tasks, enabling the model to continuously learn and improve its operational strategies through long-range terminal interactions.

• Large-Scale Codebase Refactoring and Migration: Engineering tasks requiring several hours of continuous planning and execution, such as cross-repository technology stack upgrades and overall architecture refactoring. QwenGyre's elastic scheduling ensures efficient GPU resource utilization during long sampling periods, and online RL training enables the model to progressively master the decomposition and execution of complex refactoring tasks.

• Multi-Agent Collaboration Tasks: Hierarchical execution scenarios where the main agent delegates tasks to sub-agents. QwenGyre's trajectory processing utilizes fork structures to control sub-task training costs, and the shared prefix deduplication mechanism avoids redundant computations, enabling efficient multi-agent collaboration training within controllable costs.

8. FAQ

Q: What is the fundamental difference between QwenGyre and traditional asynchronous RL approaches?
A: Traditional asynchronous RL approaches fix the partitioning of sampling and training, making GPU resources unable to be dynamically allocated. This can lead to idle time at either the sampling or training end for long-horizon tasks. QwenGyre employs cell-level elastic scheduling, allowing each cell to dynamically switch between sampling and training based on "water level." Combined with RDMA KV cache migration, this enables seamless task execution without interruption, significantly improving GPU utilization and end-to-end training efficiency.

Q: How does QwenGyre ensure the contextual authenticity of training signals in ultra-long-horizon tasks?
A: QwenGyre uses a Token-in Token-out (TITO) mechanism to record the exact input/output token IDs and policy log probabilities for each model call. These are organized into a trajectory tree structure based on shared prefixes. Each output always corresponds to its original context at the time of generation, avoiding "false context" training signals caused by historical compression or context concatenation from sub-agents.

Q: What hardware requirements does QwenGyre have for GPU clusters?
A: QwenGyre requires a multi-GPU cluster environment, with high-performance GPUs such as NVIDIA A100 or H100 recommended. The cluster nodes must support RDMA networks to enable KV cache migration and parameter snapshot synchronization. Regular Ethernet may not meet the low-latency transmission requirements.

Q: Is the code or installation package of QwenGyre currently available?
A: Currently, QwenGyre is only released in the form of an academic paper. The Qwen team at Alibaba has not yet made the code or installation package public. Researchers and developers cannot yet directly reproduce the experiments and should follow the Qwen team's future open-source release plans.

Q: How does the partial scoring mechanism work?
A: For tasks that are not completed within the time limit, if intermediate results that can be verified are produced during execution (such as partial code files, configuration scripts, etc.), the system still scores them according to the task rules and includes them in training. This mechanism maximizes the utilization of sparse yet valuable signals from ultra-long-horizon execution, avoiding the waste of all training data due to incomplete tasks.

Q: How should the trajectory budget parameter Jmax be set?
A: Jmax controls the maximum number of trajectories that can be retained for each execution, with shared prefixes counted only once for loss. It is recommended to optimize Jmax based on the complexity of task branches and GPU memory capacity. For tasks with more branches, Jmax can be increased appropriately to retain more effective trajectories, but one must balance memory usage and loss.

9. Project Links

• arXiv Technical Paper: https://arxiv.org/pdf/2609.33848 (Official technical paper introducing the architecture design and experimental results of QwenGyre)