CivicBench: A Benchmark and Memory Framework for Web-Based Civic Data Analytics

What it claims

CivicBench tests whether a tool-using agent's computed civic claim stays supported by the records it used. The benchmark has 400 tasks built from 2,504,840 retained 311 service-request rows across Austin, Boston, Chicago and San Francisco for 2024 and 2025. These are joined with NOAA daily station weather, city boundary GeoJSON, and ACS 2024 five-year and TIGER/Line tract data. Austin, Boston, Chicago and San Francisco contribute 98, 116, 98 and 88 tasks. San Francisco alone supplies 1,454,388 of the 311 rows. No training or development split is released, so the set is evaluation-only.

How it works

Each city receives one fixed 16-file bundle that every task for that city shares. The bundle holds:

• daily event tables indexed by date, unit and service family (195,769 rows in total)
• a deterministic point sample of at most 20 records per service-family and unit cell (15,550 rows)
• NOAA files, a boundary file, and manifests
• a shared 124-unit ACS/TIGER proxy table
• five policy cards

Municipal records are mapped to four common service families. Each city keeps its own units: Austin council districts, Boston BPDA neighborhoods, Chicago community areas and San Francisco supervisor districts.

Tasks are built contract-first. The generator picks a scope and capability and runs the operation to get the reference result. It then freezes an executable contract (operation, constraints, answerability, scorer and target) before any question text is written. The contract is rendered through one of four framings: operational handoff, planning decision, oversight review or decision deliverable. Construction emits three ID-aligned records, and only the public task reaches the agent.

flowchart LR
  C[Frozen contract] --> P[Public task: scope, schema, files]
  C --> H[Hidden ground truth + scorer]
  C --> I[Internal generation record]
  P --> A[Agent, tools on 16 files]
  A --> S[Operation-specific scorer]
  H --> S

Every contract has one of three outcomes:

• a direct answer
• a qualified answer that limits a broader interpretation
• a scope-aware refusal that names the missing category or time grain and returns the finest supported alternative

The ten capabilities are weighted toward priority ranking (82 tasks). Trend change and governance boundary have 46 each, and there are 24 missing-family refusals. The agent returns a structured_result and an analysis_brief with finding, evidence, limitations and claim type. Numeric checks use fixed tolerances. The brief is scored only for completeness and an exact claim-type match, so writing quality is not scored.

TSM stores typed assertions carrying file, field, spatial, temporal and step provenance. It has three operations:

• a deterministic Write
• a Resolve that merges only same-key, same-scope assertions and leaves conflicts visible
• a bounded Retrieve organized around unresolved constraints

Evidence

The 400 tasks reduce to 316 semantic specifications and 305 distinct structured answers. Seventy-two specifications appear as two or three framings, covering 156 tasks. Confidence intervals are therefore bootstrapped by semantic-contract cluster rather than per task. TSM's mean-score gain has an interval excluding zero for eight of nine model pairs. The fetched text does not name the models and gives no absolute scores or ablation numbers.

Limits

The fetched text stops at the evaluation-protocol example. It does not show which models were tested, per-capability results, the ablations, the failure taxonomy or the resource costs. It gives no release location or licence.

The Figure 1 city-year matrix (for example, Austin 24 and 126) does not sum to the per-city task counts given in the text. The memory method is also called "Auditable Evidence Memory (TSM)" in one place. The difficulty bands describe the reasoning required and are not human-difficulty ratings. Because the data is frozen, live-portal drift is deliberately not tested.