MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

What it claims

MAServeBench (Peking University and Georgia Tech, arXiv October 2026) argues that request-level traces cannot benchmark multi-agent serving. Arrival times and token lengths record the realized work, but not the collaboration rules that produced it. The paper splits collaboration's effect on serving into two parts. Work generation is which model calls, inputs and token lengths emerge. Work exposure is when those operations become eligible to run under their dependencies. The benchmark varies each part separately. Two tasks with similar request counts and token lengths can still differ in readiness, batching opportunities and memory demand.

How it works

A task is modeled as W = (S, D, Γ, M):

• S holds logical stage slots.
• D holds inter-stage dependencies and artifact interfaces.
• Γ handles composition: sequence, choice, parallel fork and join, bounded repetition, and handoff.
• M holds logical memory: private histories, task-shared versioned state, and optional persistent memory.

Each stage is C_s = (R_s, Ctrl_s, Ctx_s). These define the participating roles, the local control (spawn, fork, join, repeat, finish), and how context is built from inherited or fresh history and delivered artifacts. Context choices directly change token work and prefix/KV reuse. Control rules constrain readiness only; batching, scheduling and placement stay with the serving backend.

Four reusable templates compose into workflows: Spawn, Fork–Join, Refinement Loop and Debate. A task binding (inputs, role instructions, tools, evaluation criteria) is kept separate from the collaboration rules. The same task can therefore produce different workloads.

Whether a change can be tested on a fixed trace depends on whether it alters any operation's resolved input:

flowchart TD
  A[Proposed workload change] --> B{Alters any resolved input?}
  B -- No, e.g. added legal barrier --> C[Fixed-work replay on same trace]
  B -- Yes, e.g. select answer field only --> D[New native realization]
  C --> E[Compare batching, state lifetime, task SLOs]
  D --> E

A barrier counts as legal only if it keeps all artifact and state prerequisites and creates no dependency cycle.

How to use it

Code is at https://github.com/ShengxuanQiu/MAServeBench. The fetched text gives no install or run commands.

A Semantic Trace is versioned and hierarchical, with task, stage, session and operation levels. The schema fields shown include:

• task_id, root_arrival and terminal_operation_ids
• stage_instance_id, realized_control and context_construction
• session_id and runtime_role
• operation_type, control_dependencies and data_dependencies

Traces store resolved inputs, generation settings, optional native output token IDs and output-length targets, and tool results with measured elapsed times. Payloads sit in a content-addressed store.

During replay, tools return their recorded results after their recorded durations, without calling external services. A scenario binds a trace set to root arrivals and release rules.

Limits

The fetched text ends partway through Section 4. It does not include the GPU/NPU results, the specific hardware, the workloads or tasks evaluated, or numeric findings on SLO attainment, so the summary's efficiency-versus-completion claim cannot be checked here.

Persistent-store replay depends on reproducible snapshots and restoration hooks. Changing any operation's input requires a fresh native run rather than a replay.