Skip to content

Back to the catalog

CallScreenBench: Benchmarking Small Language Models as Phone Secretaries
arxiv.orgpaper

CallScreenBench: Benchmarking Small Language Models as Phone Secretaries

CallScreenBench is a benchmark for evaluating small, quantized (4-bit, 0.6-4B parameter) language models acting as autonomous phone secretaries that screen inbound calls without a cooperative caller-assigned task. It reports five automated call-and-note measure groups tied to owner endorsement (service, recall, plausibility, triage discrimination) plus three guardedness diagnostics for toolless, credential-free proxies, deliberately avoiding a single composite score to expose trade-offs like scam-detection TPR versus legitimate-call false positives. Testing across three paired model-family checkpoints reveals that larger checkpoints score higher on some measures while triage discrimination follows a different ordering, and scripted degenerate policies (e.g., hangup-and-echo) expose measurement floors.

TL;DR A benchmark for on-device small LLMs acting as adversarial-call-screening phone secretaries, deliberately reporting split quality/guardedness metrics instead of one composite score.
Open original ↗
Document

What it claims

CallScreenBench frames the on-device call secretary as a delegated-agent problem, distinct from the task-oriented agent benchmarks it names (τ-Voice, τ²-bench). The key inversion: the caller holds the goal and may be adversarial, while the agent acts for an absent owner who never assigned a task. The evaluation question is not task success but whether the owner, reading the transcript, would endorse how their proxy handled the call. Scope is deliberately narrow: text-domain conversational decisions only — no speech recognition, audio, latency, or handset execution — and a toolless proxy holding no credentials and calling no tools, so credential exfiltration and transactions are out of scope by construction.

How it works

A fixed system prompt carries one owner profile card: identity fields, a five-item may-disclose list, and a thirteen-category never-disclose list, byte-identical across all scenarios and models. Scenarios are schema-validated records (persona, hidden goal, verbatim opening line, escalation ladder, stated entities, hangup policy, per-scenario judge criteria). The caller is simulated by MiniMax-Text-01 at temperature 0.7 (not 0, which collapses persona variation); turns alternate strictly, capped at 8 round trips, followed by a post-call probe eliciting the note left to the owner. Judging uses MiniMax-M2.5 with binary rubric-item checks rather than 1–10 scales.

Measurement is deliberately not collapsed into one score (following HELM/CheckList):

  • Primary — five call-and-note groups: Q1 triage discrimination, Q2 initial entity recall, Q3 plausibility, Q4 legitimate-call handling, Q5 engagement, each with counter-metrics and bootstrap intervals.
  • Secondary — three guardedness diagnostic counts with separate denominators: G1 contextual-information candidates, G2 prohibited-action flags, G3 scam-side callback-number acceptance.
flowchart TD
  S[Scenario record + fixed owner profile] --> C{Caller family}
  C -->|scam / Family A| L[Strict alternation, ≤8 round trips]
  C -->|legit / Family B| L
  C -->|gray / Family C| L
  L --> N[Post-call owner-note probe]
  N --> Q[Q1–Q5 profile + counter-metrics]
  N --> G[G1–G3 diagnostics, separate denominators]

Evidence

Three model families are tested as paired 4-bit checkpoints, 0.6–4B, six total, run once each under one protocol. Findings the authors stress:

  • The larger checkpoint scores higher on several service, recall, and plausibility measures, but triage discrimination follows a different ordering — capability does not track safety.
  • Bare scam-side TPR rewards universal suspicion; pairwise separation between models changes once legitimate-side false positives are included, and across judge snapshots.
  • Six scripted degenerate agents expose metric floors: notably a hangup-and-echo policy achieves entity recall = 1.000, showing why Q2 needs its counter-metric.
  • Taxonomy: 22 archetypes, 180 total slots (90 scam / 56 legit / 34 gray). Only the evaluation half (CallScreenBench-89) is generated; the held-out half is a design target. Actual runs use a compute-bounded Core-48 panel, balanced 24/15/9 across families and 16/16/16 across tiers.

Limits

  • Text-domain only — the authors note (via τ-Voice) that omitted speech/latency channels can substantially change performance elsewhere.
  • Exact external replay is currently unavailable: the provider cache is not in the repository, and regeneration is impossible because the temperature-0.7 simulator is stochastic and no seed was retained.
  • No benchmark-wide composite, leaderboard, or endorsement threshold is defined — you cannot rank models with a single number, by design.
  • Small-sample caveats: comparisons are scenario-matched, not controlled on identical dialogue; caller-name repetition is concentrated ("Marcus" in 8/48 Core cells); no per-archetype or family×tier claims are supported. Of 288 calls, 50 (17.4%) hit the 8-turn cap and are right-censored.
  • Only 0.6–4B 4-bit models tested; frontier models are noted to degrade under adversarial multi-turn pressure but are not the study target.

Organisations named here

  • CallScreenBench
  • Simiao Ren

Techniques named here

  • Toolless proxy
  • Guardedness diagnostics
  • Call-and-note measure groups

Related in evals