Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling

What it claims

Alibaba's Herschel targets a specific gap in post-release tuning on a model-as-a-service fleet. Persistent inefficiencies sit inside historical baselines, so anomaly detectors miss them. Always-on PyTorch Profiler or Nsight tracing costs too much to run continuously. Lighter tools such as StriaTrace or py-spy sampling drop what diagnosis needs: object state, CUDA Runtime and Driver API records, kernel shapes, and cross-process dependencies.

Herschel attaches to running processes on demand with no engine changes or restarts, records full-stack detail in bounded windows (typically 30 s), then detaches. In controlled tests, the overhead while collecting was under 0.5% for TTFT and under 7% for TPOT. The 7% TPOT cost applies only during a window, which is why the windows are kept short.

How it works

Hprof attachment. The CLI pauses one thread with ptrace and inspects its stack. It skips threads in allocator, loader or threading contexts, because those could hold locks the injected calls need. It then redirects the thread to call pthread_create, and the new thread dlopens an in-process controller library. After that one-time entry, further commands go over a socket to the controller, which turns probes on and off.

Probes capture:

• Python calls
• supported object fields
• accelerator API calls
• device activity

A fleet control plane picks the processes and windows, and per-machine daemons carry out those plans. Hprof runs on NVIDIA GPUs and on Alibaba's in-house accelerators, but only on runtimes it supports.

Analyzer rebuilds operator executions and cross-process dependencies from the records. It then checks them against a catalog of recurring inefficiency patterns, each tied to a reference fix. Examples are asynchronous execution to remove stalls, better operator dataflow, and implementations suited to the target device.

Optimizer runs engine and kernel agents in an autoresearch-style loop. They test at reduced scale, but keep the workload and dependencies that triggered the problem.

flowchart TD
  CP[Control plane picks process + window] --> H[Hprof attach, record ~30s, detach]
  H --> A[Analyzer: reconstruct deps, match patterns]
  A -->|evidence insufficient| S[Expert-approved supplementary profiling]
  S --> H
  A -->|mechanism confirmed + reference fix| O[Agents implement & test under replayed conditions]
  O -->|correctness/perf feedback| O
  O --> R[Expert review]
  R --> C[Canary + staged rollout, outside Herschel]

Evidence

Beyond the fleet totals already in the summary, the paper gives three concrete cases:

• GLM-5.2, KV-service blocking. An upstream KV-service allocation sat behind a synchronous RPC. This delayed command dispatch, so Workers waited and GPUs sat idle.
• Same cost, different priority by deployment. Outer KV preparation took over 8% of a GLM-5.2 Worker's active window under high KV reuse with short executions. In another service with fewer cache hits it took under 1%.
• Gated DeltaNet kernel. The most frequent query-token/context-length bin in one deployment reached only 14.7% of the compute/bandwidth speed-of-light (SOL) bound.

These cases support the paper's claim that the same inefficiency can matter a lot in one workload and very little in another. That is the argument for profiling in production rather than in offline tests.

Limits

• No release. The page mentions no code, release or licence. Treat this as a description of an internal Alibaba system, not something you can adopt.
• Overhead measured in controlled tests only. The tested model sizes are not given in the visible text.
• Attachment risks. Injecting through ptrace relies on excluding lock-holding contexts, and the page frames supported fields and runtimes as limited. Check that your runtime and security policy (ptrace permissions) allow this.
• Not autonomous. Supplementary profiling needs expert approval, every change needs expert review, and canary and rollout happen outside the system.
• Truncated page. The fetched text stops partway through §5. Details of session control, the pattern catalog, how agents validate correctness, and the full production results were not available.