CompSkill: Compiling Execution Experience for Skill Retrieval

What it claims

CompSkill reranks skill documents for LLM agents using execution outcomes. The documents and the executor stay unchanged. With one document per task, it completes 256 of 332 ToolQA tasks with GPT-5.6-luna and 260 of 332 with DeepSeek-V4.1-Flash, above BM25, SkillRet-8B, Tool2Vec and RankGPT. On AppWorld and ALFWorld it only matches the strongest baselines. The authors' main finding is that selection matters beyond relevance. A near-miss "sibling" guide from the same database hurts more than an unrelated document. Pairing the right guide with a same-database sibling, instead of another database's guide, costs 8 of 96 tasks.

How it works

1. Compact keys. A frozen encoder embeds each document at 4096-d. A residual MLP compressor refines the first 512 coordinates into a normalized 512-d key. The compressor is trained once on SkillRet and then frozen; target libraries get no relevance training. Queries use the normalized first 512 encoder coordinates. A semantic Top-10 forms the candidate pool.
2. KeyOpt. The final score adds four terms: the frozen SkillRouter cross-encoder score, a per-skill offset β, a clipped residual term (q − μ)ᵀΔk, and a weighted event reading. β is fitted with a pairwise ranking loss on same-task, same-executor, same-budget comparisons where the two skills had different outcomes. Under a logistic outcome model, these discordant pairs cancel task difficulty. Tied tasks contribute nothing. Δk is then fitted on the same pairs, with a "keep" penalty that holds it near zero on queries with no preference. It captures query-dependent suitability, for example single-day versus range queries. The cap c, weights κ and α, and whether to use Δk at all are chosen by five-fold cross-validation over support tasks grouped by source question. Ties go to the weaker correction.
3. Experience hypergraph. Each event stores the task, executor, environment, document versions and arm outcomes. Events can recall up to 4 skills that semantic Top-10 missed, alongside 6 kept candidates. They also ground an LLM "cited review" that quotes documents and cites events.

Each skill stores 1,025 floats ([k0; Δk; b]). At test time, reading the compiled experience costs one inner product and a clip per candidate, with no extra model call. Everything updates only between rounds, from support runs.

flowchart LR
  S[Support runs: paired arms, same task] --> F[Fit β, Δk, events]
  F --> K[Skill keys + hypergraph]
  Q[Test query] --> P[Semantic Top-10 + event recall]
  K --> P
  P --> R[Score: reranker + β + clipped Δk + events]
  R --> D[One original document to frozen executor]
  D -. outcomes, support only .-> S

Evidence

• SkillRet. 512-d keys reach NDCG@10 70.9 against 64.0 for the 4096-d embeddings they compress. Recall@10 is 77.5 against 70.6 (three-seed mean).
• ToolQA (332 evaluation tasks). GPT-5.6-luna Hit@1 is 82.2 for CompSkill against 46.7 for SkillRouter. Even the strongest baseline puts the labeled guide first for at most 78.3% of tasks.
• Executor transfer. The GPT fits come from GPT-5.5 support runs and transfer to GPT-5.6-luna without refitting.
• Held-out sets. On 332 newly generated ToolQA tasks, CompSkill completes 256 with GPT-5.6-luna, significantly more than four retrievers. Cited review moves the result from 250 to 256 (9 tasks rescued, 3 harmed). Across all 844 ToolQA tasks, it completes 24 more than the strongest baseline.
• Error analysis. On 180 held-out tasks, the KeyOpt index supplies a sibling guide on 16, against 37 for Vector Top-1.
• Fitted settings. ToolQA keeps the residual for both executors: c = 0.3 and α = 0 for GPT, c = 0.2 and α = 0.05 for DeepSeek.

How to use it

Code, models and support data are public at https://github.com/satoshinji2992/CompSkill. The page gives no install commands or licence. Deployment needs a frozen encoder with 4096-d outputs, the SkillRouter cross-encoder, and support tasks run with several candidate skills per task on the target executor.

Limits

• Needs contrasting support. Gains depend on support runs where skills had different outcomes. With no discordant pairs, every correction vanishes and the index falls back to the reranker. This is why AppWorld and ALFWorld show parity only.
• Where gains concentrate. The authors say improvements appear where confusable guides decide execution.
• Compressor training. The compressor was trained on SkillRet, and its behaviour on other libraries rests on that one training set.
• Truncated page. The fetched text ends partway through Section 3.3. Appendix B proofs, the full result tables and the ablations are not shown.