Skip to content

Back to the catalog

GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models
arxiv.orgpaper

GACC-Bench: Benchmarking Goal-Adaptive Context Compression Ability across Large Language Models

GACC-Bench is a benchmark for Goal-Adaptive Context Compression, testing how well LLMs retain goal-relevant information when compressing a source context under a length budget before downstream questions are known. It has 500 goals and 2,571 human-reviewed questions over 281 document and agent-trace contexts from seven collections; a fixed reader answers the withheld questions from the compressed context. Ten LLM compressors show large gaps to full-context performance, especially in answering every question tied to a goal. Target-goal conditioning improves accuracy over goal-agnostic compression, while mismatched goals hurt it.

TL;DR Benchmark of 500 goals and 2,571 questions measuring how well LLMs compress long contexts and agent traces toward a stated goal; best models still lag full-context performance.
Open original ↗
Document

What it claims

GACC-Bench tests whether an LLM compressor keeps the information needed for a stated high-level goal, such as "understand how the study's findings characterize the change in X," when the questions that will be asked later are hidden from it. The paper frames this as the setting behind compaction in coding agents such as Codex CLI and Claude Code, where the user may know the next objective but not the specific questions. It positions itself as the first benchmark in which the compressor sees a goal and the reader sees a question. Its Table 1 contrasts this with LongContextBench, SCBench and LongMemEval-V2: in those, the compressor gets the same question the reader answers, so the future task is not withheld.

How it works

flowchart LR
  C[Source context] --> K[Compressor LLM]
  G[Goal] --> K
  B[Token budget] --> K
  K --> X[Compressed context]
  X --> R[Fixed reader]
  Q[Withheld questions] --> R
  R --> J[Fixed judge]
  C --> J
  A[Reference answers] --> J

Building the questions. Questions are generated from the source using seven strategies, each used only when the source supports it:

  • information retrieval
  • multi-step derivation
  • cross-entity comparison
  • whole decomposition (rebuilding a total from its parts, a share or a residual)
  • rule application, including hypothetical input changes
  • bounded trend extrapolation
  • conflict resolution between competing or revised claims

Each question carries a reference answer and links to its evidence passages. An LLM then groups the questions by shared future task and writes a goal for each group. The goal may not list the questions or reveal answers.

Quality control. The filters are strict:

  • Three models check goal–question fit, and any valid negative judgment removes the question.
  • An LLM must match each question to its goal, and each goal to a question, among the context's alternatives. A failed match removes the item.
  • A goal is dropped if it covers a question from another group.
  • Questions must be answered correctly across repeated full-context attempts.
  • A question is removed if any of four models answers it correctly without the source.
  • Multi-fact questions need at least two contributing passages.
  • Incidental details, such as command counts or line numbers in SWE-agent traces, are removed.
  • Groups with fewer than four questions are discarded.
  • Near-duplicate questions are pruned at the information level.
  • All instances get human review.

Evidence

The fetched text contains only the motivating experiment. DeepSeek-V4-Pro served as both compressor and reader on 20 contexts from each of three sources: LongCodeBench, L-Eval, and SWE-agent trajectories on SWE-Bench Verified. Results are changes in question-level accuracy relative to goal-agnostic compression:

  • Matched goal: every cell improved, from +4.3 to +50.0 points. On LongCodeBench the gains ran from +15.7 to +45.0; on L-Eval, from +13.9 to +34.1.
  • Mismatched goal: 42 of 44 cells lost accuracy, some by as much as −49.0 (LongCodeBench) and −54.1 (L-Eval). The two exceptions were small positive spillovers of +9.0 and +6.1.

The practical takeaway for compaction design is that a wrong focus hint can do much more harm than giving no hint at all.

How to use it

The data and code are at https://github.com/weiting24/GACC-Bench. The fetched text gives no install commands, run commands or licence. To use the benchmark you need a compressor that accepts a context, a goal and a token budget, plus the paper's fixed reader and judge, which the fetched text does not name.

Limits

  • The fetched text stops partway through the methodology section. It does not include the names of the ten compressors, their scores, the budget levels, the identities of the reader and judge, the six-model goal ablation, or a full list of the seven collections; only LongCodeBench, L-Eval and SWE-Bench Verified are named.
  • The headline matched and mismatched numbers come from a 60-context pilot that uses one model as both compressor and reader.
  • Answer correctness is decided by an LLM judge, and most quality-control filters are LLM-based, with human audit on top.
  • Check the repository for the licence and the terms of the source datasets before reusing the data.

Organisations named here

  • arXiv

Technologies named here

  • GACC-Bench
  • LLM

Techniques named here

  • Goal-Adaptive Context Compression

Related in evals