The bottleneck sits in environment management
Agentic reinforcement learning trains a language model by letting it act over multiple turns: the model generates an action, a tool or environment returns an observation, and the loop repeats until a reward is computed. The paper's central claim is that this loop is limited less by the RL trainer than by the systems around it. Each rollout must allocate, drive and tear down external environments such as sandboxes, local model services and external APIs, and the authors state that reuse and lifecycle costs can even exceed text generation.
That framing explains the design target. The authors leave long-horizon optimization to an existing RL trainer, verl, and aim instead at the environment bottleneck: managing heterogeneous resources concurrently at high throughput without starving the trainer. AgentFly is therefore best read as a systems contribution to agentic RL, not a new RL algorithm.
One engine schedules four kinds of resource
AgentFly defines a resource as the smallest unit a tool or reward uses, and represents each kind as a distinct, typed specification resolved through a single engine. The four kinds named in the paper are containers (with image, CPU and memory limits, and an isolation backend), local model services (a vLLM server with model, parallelism and memory configuration), external model APIs (endpoint and credentials), and in-process environments such as a chess engine. All implement a common lifecycle interface of start, reset, status and release, so one scheduler manages heterogeneous resources rather than a single kind.
Two mechanisms make this more than bookkeeping. First, the engine keeps a pool per resource specification, and a repeated acquisition under the same caller-supplied identifier returns the same live instance instead of allocating a new one; a container created in a rollout's first turn is thus reused across later turns, amortizing setup cost over the trajectory. Second, each pool declares a maximum capacity, and when it is saturated further acquisitions wait asynchronously until other rollouts release instances, which bounds total usage without blocking generation. Resources are scoped either to a single rollout, released automatically when it ends, or globally, persisting across rollouts as shared services.
Four layers keep tool code free of allocation logic
The framework is split into four layers. The agent layer exposes the user-facing pieces: an agent (model, prompt, parser, memory), tools, and rewards. The rollout layer composes them into the agent loop, drives the alternation between generation and tool execution, and computes the reward on the finished trajectory. The context layer injects rollout information such as identifiers and reference answers, and brokers resources. The resource layer performs allocation and recycling. Together the context and resource layers form the unified resource system.
The practical consequence appears in the paper's code listing. A shell-command tool declares a container specification with a global cap of 24 instances and acquires it under the rollout identifier, so all turns of that rollout reuse one container; a reward function acquires an external model API with global scope, shared across rollouts. Neither function contains allocation or cleanup logic. The paper also decouples the agent module from training: the two sides communicate only through trajectories, so the agent side can evolve independently of the training infrastructure.
Asynchronous rollout removes batch-wide waiting
Rollout interleaves token generation with tool calling, and step costs vary widely across turns and tasks. In a batch-synchronous design, every phase waits for the slowest rollout, creating idle gaps. AgentFly instead makes the whole rollout asynchronous: generation, tool calls, reward computation and resource operations are each awaited independently, so a slow step in one rollout does not block the others, and resource allocation overlaps with ongoing rollouts.
The paper's schematic compares three two-turn rollouts under both schedules and shows the asynchronous version finishing earlier, though the durations there are explicitly illustrative rather than measured. The measured evidence for the resource system comes from scaling instead: on ScienceWorld, with all settings except resource count fixed, training time per step falls from 66.95 minutes with 1 container to 35.28, 18.67, 10.5, 5.08, 2.71 and 1.86 minutes as containers increase to 8, 16, 32 and 64. The authors report that once enough resources are supplied, training becomes mainly bottlenecked by generation throughput and the trainer's parameter updates.
Throughput gains grow with heavier environments
The paper reports what it calls the first controlled cross-framework throughput comparison between agentic RL frameworks. It uses WebShop and R2E-Gym, both officially supported by AgentFly and by the compared frameworks, and because AgentFly uses enroot while SkyRL defaults to Docker, the authors implement an enroot backend for SkyRL to keep the comparison fair. Trajectory throughput is reported in two settings: end-to-end training, which measures the whole system, and rollout-only, which excludes the trainer and focuses on the agent module.
On both tasks AgentFly achieves the best throughput. The paper states it reaches around 2 times that of verl-agent and 1.75 times that of RAGEN, and that the advantage is larger on R2E-Gym, where AgentFly reaches 3.8 times the throughput of SkyRL. The conclusion generalizes this to about 2–4× at equal hardware. These are trajectory-throughput ratios under the paper's own configurations, not claims about final model quality; the WebShop runs train Qwen2.5-3B-Instruct for 150–200 steps, while the R2E-Gym measurement covers the first ten training steps with Qwen3-4B at roughly 30k context and at most 20 turns.
Re-tokenized text destabilizes multi-turn training
A separate finding concerns data handling rather than scheduling. A multi-turn agent decodes each response to text so tool calls can be parsed, and a naive pipeline later re-tokenizes that text for training. Because re-tokenization is not an identity transformation for BPE tokenizers, the trainer computes log-probabilities, importance ratios and the KL penalty on tokens the policy never produced, and in multi-turn rollouts the drift also enters later prompts. AgentFly instead keeps token IDs as the unit of exchange between inference and training.
The measured effect is small in volume but large in consequence. Training Qwen3.5-4B on WebShop with chain rollout and GRPO for 150 steps, the authors find drift affects 36% of trajectories on average but only 0.07% of tokens. All three runs track each other for about 75 steps, then the re-tokenized runs diverge: KL loss peaks at 3.5 and 0.6 and gradient norms at 30 and 12, versus at most 0.12 and 2.9 when training on IDs. One re-tokenized run recovers only partially to a validation reward of 6.6, the other collapses from 6.8 to 0.8, while training on sampled IDs stays stable and ends at 7.0.
Chain rollout usually wins, but not universally
Because both rollout strategies run on the same agent, tools and resources, AgentFly can compare them with only the strategy changed. Chain rollout keeps one growing conversation per trajectory and trains on all assistant turns in a single sequence; step rollout re-prompts at every environment step with a fresh prompt and produces one training row per step, which bounds context length and gives each step its own sample for step-level credit assignment such as GiGPO.
Chain rollout matches or outperforms step rollout in nearly every setting. Its advantage is largest for GRPO, where in-distribution WebShop success rises by 5.8, 19.1 and 2.2 points at 1.5B, 3B and 7B, and carries over to held-out products at 3B and 7B (52.5 vs. 39.8 and 58.0 vs. 48.0). GiGPO's step-level credit assignment helps most under step rollout, improving on GRPO by 6.4, 15.9 and 7.6 points in distribution, but under chain rollout that gain shrinks to 1.6, 0.9 and 5.7 points. ALFWorld shows the same pattern: chain rollout improves GRPO from 75.5% to 91.0% and GiGPO from 89.3% to 98.1%, and GiGPO's gain over GRPO shrinks from 13.8 points under step rollout to 7.1 under chain rollout. At 1.5B, held-out WebShop results of all four configurations lie within about one point, inside the roughly two-point evaluation noise.
Where the resource layer matters most
The paper's own priority is clear: the heavier and more heterogeneous the environment, the more the unified resource layer buys. The throughput advantage is larger on R2E-Gym, which uses one container per task, than on WebShop, which shares a container across rollouts, and the ScienceWorld scaling curve shows per-step time flattening only once the container pool is large enough. For a team whose rollouts are dominated by sandbox setup and teardown, that is the result to weigh first.
The second decision is not a single best setting. Chain versus step rollout depends on the task, model size and estimator, which is why the framework supports both behind one interface; at 1.5B the held-out differences fall within evaluation noise, while at 3B and 7B chain rollout's held-out advantage is substantial. The token-ID finding cuts across both: a pipeline that re-tokenizes decoded text can look fine for dozens of steps before diverging, so keeping sampled token IDs is a precondition for trusting multi-turn training curves rather than an optional optimization.
Sources used in this answer
AgentFly: Extensible and Scalable Reinforcement Learning for LM Agents
AgentFly schedules containers, model services, external APIs and in-process environments as typed resources through one engine, reporting about 2–4× the training throughput of other agentic RL frameworks at equal hardware and showing that re-tokenizing decoded text destabilizes multi-turn training.