EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures

What it claims

EdgeAgent is an inference runtime for several concurrent LLM agents on one CPU-GPU unified-memory SoC. It targets an Apple M4. The paper's starting point is that existing tools fit this workload poorly. Cloud servers (its refs 17, 30, 41) assume many requests spread over discrete GPUs. Edge frameworks such as llama.cpp and MLX assume one request at a time. The paper is accepted at ASPLOS '27 and released under CC BY 4.0. The headline results come from the abstract: a 1.29× speedup from UMA-aware execution and 1.77× for the full system, both measured against batched speculative decoding. The 1.77× figure holds "under extreme tool-use latencies", not as a general average.

How it works

The motivation section explains why each of the three mechanisms exists.

• CPU+GPU tensor parallelism only helps for small decode matrices. In the paper's M4 bandwidth microbenchmark (Table 1), CPU and GPU running together beat the GPU alone by 1.36× on a 4096×4096 matrix. That drops to 1.05× at 4096×28672 and to 0.99× at 4096×128256. At the larger sizes the GPU alone already uses over 80% of the 120 GB/s peak, and the two units fight over the shared bus. The CPU side depends on ARMv9 SME matrix extensions, for which the authors wrote custom SME micro-kernels.
• Graph compilers block zero-copy splitting. MLX's lazy graph and ExecuTorch's ahead-of-time graph both track data hazards per whole tensor. When the CPU and GPU write to non-overlapping parts of one buffer, the compiler treats this as a conflict. It then inserts fences, forces eager synchronisation or adds copies. EdgeAgent bypasses the graph compiler for these writes.
• The best draft length depends on the task. The test used DeepSeek-R1-Distill-Llama-8B on the M4. Structured output (MBPP samples) peaks at a draft length of L=16, giving 1.69×. Reasoning (LongBench samples) reaches only about 1.10× and stops improving at shorter drafts. Accepted tokens grow sub-linearly with L. Past the peak, rejected drafts make speedup fall. This is the case against one fixed draft length per batch. EdgeAgent instead sizes each agent's draft budget from how predictable its output currently is.
• Suspend-and-yield. When an agent is waiting on a tool call, it is suspended in place. Its compute slot and speculative budget go to the agents that are still active.

Evidence

The fetched text stops in the middle of §2.5, so the evaluation section is missing. Only the abstract's two figures are available. Their setting, as stated there, is one Apple M4, with batched speculative decoding as the baseline. The introduction says the evaluation uses trace-driven concurrent multi-agent workflows and measures both throughput and end-to-end makespan, but gives no figures for either. The fetched text names no target or draft models, agent counts, trace sources or makespan numbers for the main results.

Limits

• The 1.77× is reported under extreme tool latency. Expect less when tool calls are short.
• The authors' own Table 1 shows that splitting decode across CPU and GPU stops paying off for large weight matrices. LM-head-sized shapes (4096×128256) get 0.99×. On devices without SME-class CPUs the gain is likely smaller still.
• The text contradicts itself on the memory layout. The abstract says "asymmetric memory layouts" and the contributions list says "symmetric".
• Only Apple M4 results are shown. Snapdragon X is mentioned as a UMA SoC but not evaluated in the available text.
• No code, repository or release is mentioned. To adopt this you would have to reimplement it outside MLX's graph path.