EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning

What it claims

EnvSched is an online scheduler for GRPO-style agentic RL. It treats the environment, not only the task prompt, as part of the selection model. The page also shows a comparison against two published task-selection methods, DPS and LEEPS, alongside random selection. On Qwen3-1.7B, EnvSched after 40 steps matches or beats both baselines run for 200 steps on all three benchmarks. Training uses the EnvScaler pool (191 environments, about 7K tasks) with matched per-step rollout budgets. Code is at https://github.com/Hhankyangg/envsched.

How it works

Before training, an LLM reads each task's instruction, initial-state summary, tool descriptions and verifier. It writes a structured operation description: required operations, prerequisites, dependencies and reward checks. Two fixed similarity matrices are built from this:

• Within-environment similarity is a weighted sum of cosine similarities over six TF-IDF views: instruction words, character fragments, verifier-check text, verifier code structure, initial context and the full operation description.
• Cross-environment similarity strips tool names, field names, entity IDs and target values. It maps operations to abstract categories such as lookup, eligibility checking and relation replacement.

Each node stores three statistics per rollout group of K samples: mean reward, population reward variance, and a mean-corrected dispersion. At each step, the top-K neighbors along each edge type give estimates that are shrunk toward a global reference. The two estimates are averaged equally into a graph prior, which is fused with the node's own history. Unobserved nodes rely on the prior alone. A decline in the short-window estimate against the long window lowers priority, and the result is a percentile score q.

Each batch is split: a fraction ρ (half in the reported setup) goes to coverage, the rest to exploitation. Coverage first filters candidates lexicographically by current-batch environment coverage, cumulative environment exposure, then task observation count. Exploitation ranks by q·(1 + c/c0)^(-1/2), where c is the task's selection count. Both phases pick greedily and subtract λ_sim times the maximum similarity to tasks already in the batch. Environment quotas and consecutive-selection limits also apply. A progress filter drops tasks with non-positive estimated progress from exploitation. An identifier filter uses fixed-action replay to detect suspicious reward-producing actions and removes those tasks entirely. No pilot rollouts are needed.

flowchart LR
  G[Offline graph build] --> P[Propagate history]
  P --> C[Coverage: under-seen envs]
  P --> E[Exploitation: high predicted variance]
  C --> B[Greedy batch, similarity penalty, quotas]
  E --> B
  B --> R[Rollout and GRPO update]
  R --> P

Evidence

Weighted overall scores on Qwen3-1.7B (Random, DPS and LEEPS at 200 steps):

• BFCL-v3 multi-turn: base 9.75; Random 10.46; DPS 9.92; LEEPS 11.25; EnvSched 11.75 at 40 steps and 12.42 at 200 steps.
• τ-Bench: base 11.11; Random 15.56; DPS 16.97; LEEPS 16.77; EnvSched 16.97 at 40 steps and 17.58 at 200 steps.
• ACEBench-Agent: base 30.67; Random 28.67; DPS 29.33; LEEPS 30.67; EnvSched 30.67 at 40 steps and 31.33 at 200 steps.

On ACEBench at 1.7B, random training falls below the base model. EnvSched at 40 steps only returns to the base score.

On Qwen3-4B, mean within-task reward variance of selected tasks rises from 0.03208 to 0.04290. Environments per batch rise from 36.53 to 45.96. The authors report that ablations show neither the variance component nor the coverage component is sufficient alone.

Limits

The fetched text cuts off partway through Table 1. Per-benchmark Qwen3-4B and Qwen3-8B rows, the ablation table, and the hyperparameters in Appendix D (feature weights, K_within, K_cross, c0, λ values) are not visible. Gains at 1.7B are one to two points absolute. All training uses one environment pool, EnvScaler. The method depends on an LLM pre-pass over every task and on readable verifier code. No licence is stated on the page.