HERMES: High-Performance Private Decoding for Large Language Models

What it claims

HERMES is a privacy-preserving inference (PI) system for LLMs that targets decoding, measured as time per output token (TPOT), rather than prefill and time to first token (TTFT). The authors state that prior PI systems mainly optimise TTFT, which suits single-token outputs but not multi-token responses. They describe their work as the first to generate full multi-token answers for real prompts under PI. On Llama 3.1 8B Instruct, HERMES lowers TPOT by up to 2.7× and total PI latency by up to 2.1×.

How it works

The paper names two bottlenecks in private decoding:

1. Nonlinear kernels need many interactive communication rounds. HERMES replaces them with protocols that use more computation and bandwidth but fewer rounds. Under PI, per-token latency is dominated by round trips rather than raw compute or bytes.
2. Attention protocols waste ciphertext capacity and keep moving the KV cache. Existing protocols re-encrypt or retransmit the growing key and value cache at every step. HERMES uses a homomorphic-encryption matrix-multiplication protocol that keeps the KV cache encrypted on the server, so it is not re-sent each token. It uses grouped-query attention (GQA) to pack ciphertexts more fully.

The second point explains the GQA choice. Sharing KV heads across query heads lets the protocol fill more of each ciphertext.

Evidence

The only results on the page are the headline figures: up to 2.7× lower TPOT and up to 2.1× lower total latency on Llama 3.1 8B Instruct. They hold "across the network conditions and prompts used in our evaluation". The abstract does not name the baseline system, the network bandwidth and latency settings, the prompt set, the output lengths or the absolute latencies.

Limits

The page carries only the abstract and metadata. It is a preprint on the Cryptology ePrint Archive, received 18 October 2026 and approved 19 October 2026. Both speedups are "up to" figures, so gains in a given network setting may be smaller. Only one model, Llama 3.1 8B Instruct, is reported. Because the attention protocol relies on GQA, its benefit for models without grouped-query attention is not stated. The page does not describe the threat model, security assumptions or code availability. The paper is licensed CC BY. The authors (Olyaiy, Shao, Yang, Shahrad, Lis) list University of British Columbia ECE contact addresses.