AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

What it claims

AgentTune is not a new training method. It is plumbing that lets one trajectory record (the EventLog) feed the runtime, RL training, SFT/distillation, evaluation and audit, instead of each stage rebuilding its own format. It dispatches to TRL for GRPO, PPO, DPO, RLOO and BCO. The headline task result is a legal contract-search agent: GRPO improves held-out CUAD exact match from 60.0% to 69.0% and cuts searches per question from 3.00 to 2.90. Other training paths are backed only by "GPU-verified smoke tests" showing they run end to end. The authors say each capability is evaluated separately and no single experiment covers the whole lifecycle.

It is distinct from AgentTuning, which is a fine-tuning method and dataset.

How it works

flowchart LR
  R[Runtime episode] -->|light| E[EventLog]
  RL[RL rollout engine] -->|full| E
  D[DECIDE run] -->|light| E
  V[Eval trace] -->|light| E
  E -->|full only| S[SFT rows]
  E --> EV[Evaluator view]
  E --> A[Audit records]
  A --> C[Detect, classify, correct, retrain, gate]
  C -->|corrected trajectories| E

• Two tiers. light logs hold content only: tool, result, reward. full logs also carry token spans and logprobs, and only a rollout engine produces them. masked_tokens() and as_dataset_rows("sft") raise ValueError on light logs.
• Event kinds. There are eight. REWARD is scoped to step, turn or episode.
• SFT loss weights. Reasoning and text get weight 1. Tool calls get weight 2, because a malformed call is treated as worse than imprecise prose. Tool results and observations get weight 0. A multi-call turn is unwrapped so each call becomes its own weight-2 message with JSON-parsed arguments.
• DECIDE. This module compiles YAML decision workflows (24 templates, 8 stage types), appends every run to audit.jsonl, and drives the closed recovery loop shown above.
• Rewards and retrieval. There is a 26-entry reward registry and three SearchBackend implementations.

How to use it

Install from source:

git clone https://github.com/Lexsi-Labs/AgentTune.git
cd AgentTune
pip install -e .

Optional extras are [service], [docs], [dev] and [flash-attn]. The core agenttune.agentic needs no GPU, network or API key, but it imports PyTorch at import time. Any RL training needs a CUDA GPU.

This GPU-free quickstart uses a scripted policy:

from agenttune.agentic import Project, DictToolHarness, ReActStrategy
harness = DictToolHarness({"add": lambda a, b: a + b}, max_steps=4)
strategy = ReActStrategy(
lambda s: {"name": "add", "arguments": {"a": 2, "b": 3}}
if s.step == 0 else {"name": "finish", "arguments": {"answer": "5"}})
proj = Project(strategy=strategy, harness=harness)
proj.infer("what is 2+3?")# one episode -> EventLog
proj.evaluate([{"task": "what is 2+3?", "expected": "5"}])# -> mean_score 1.0
proj.evaluate_agentic(["what is 2+3?"])# programmatic metrics

Entry points for training:

• RL training goes through create_agentic_trainer, or agenttune.api.train_agentic as the public wrapper.
• SFT and distillation use Project.train(fmt='sft') and Project.distill(). These take a caller-supplied trainer_factory and bypass the RL core.

Limits

Licence. The Lexsi Labs Source Available License v1.1 is free for research, evaluation, education and audit. Commercial use and removal of safety behaviour are restricted. It is not a permissive open-source licence, despite the abstract's wording.

CLI. The CLI cannot do tool-calling training because it has no --tools flag, so agentic training is Python-API only.

TRL coupling.

• Accepted trainer kwargs are derived from the installed TRL via inspect.signature(), so behaviour shifts with the TRL version. Pin to the dependency set the repo records.
• It inherits TRL's single-node scaling limits. For cluster-scale RLHF the authors point to verl or OpenRLHF.

SFT projection gotchas.

• loss_mask polarity is inverted relative to TRL's completion_mask: True means excluded from the loss.
• The projection drops REWARD, TURN_COMPLETE and MEMORY_OP. Turn boundaries flatten, and a MemoryReActStrategy agent's memory operations never reach training.

Failure detection. Loop collapse is detected only for identical repeated actions, meaning the same tool and the same arguments.

Safety and scope (stated non-goals).

• Tool defaults favour usability over confinement and are not a sandbox, so run untrusted code remotely.
• DECIDE domain templates encode "plausible" logic, not regulator-approved logic.
• No reported number is scored by an external agentic benchmark harness.

Schema scope. The schema cannot represent several things; concurrent tool calls, for example, are serialized. The fetched text cuts off mid-list, so the remaining items were not reviewed.