Nehanth/pooled

What it claims

Pooled splits one open model by layers across browser tabs on different devices. It uses its own WGSL engine, not WebLLM, MLC or llama.cpp. The page states three specifics:

• Per-token traffic is small. Each token sends about 10 KB of hidden state between devices on the 27B and 4 KB on the MoE.
• Speculative output is exact. With greedy decoding, the speculative stream matches plain one-device decoding.
• It is slower than native. Native llama.cpp is about 2x faster on plain decode for the MoE and 1.3x for the 27B. Prefill is the largest gap.

The demo puts Qwen 3.6 35B MoE across a Mac and an iPhone.

How it works

• Layer placement. The host embeds the token. Workers run contiguous layer ranges on their own GPUs. The host applies the final norm and LM head, samples, and drafts the next tokens with the model's own draft layer. The planner sizes each device's share from its measured speed and memory.
• Weights. Each device downloads only its own layers, from Hugging Face or from a peer that already has them, and caches them. If a device leaves, the host reassigns its layers with one click.
• Prompt transfer. Prompts go out in pipelined 16-token frames.
• MoE disk tier. Experts are streamed from SSD (OPFS in a tab) into a per-layer GPU cache. The page gives an example with Qwen3.8-Flash, which uses 10 of 512 experts per token. On recorded traces, caching 17% of a layer's experts served about 77% of lookups.

flowchart LR
  T[Codex / Claude Code / SDK] --> B["@pooled/cli on 127.0.0.1"]
  B --> H[Room host]
  H -->|hidden state| W[Worker tabs]
  W --> H
  H -.->|UDP blocked| R[TURN relay, TCP/TLS 443]
  H -->|prompts + answers| V[Everyone in room]

The relay carries activations only. Model weights never pass through it.

Evidence

Single device. Tested on an NVIDIA GB10 (DGX Spark) in Chrome with coding prompts:

• 70–85% of drafts are accepted, which closes most of the decode gap with native.
• Time to first token on a 172-token prompt is about 1.0 s on the MoE and 2.5 s on the 27B. Native takes about 0.07 s and 0.5 s.

Simulated room. Each device was a headless Chromium on loopback WebRTC, with one-way delay added in the page:

• Plain decode pays one delay per device per token. Three devices at 50 ms add about 150 ms to every token.
• Speculative decode at 20–50 ms is 1.6–2.1x faster than plain on the MoE and 1.5–1.8x on the 27B.
• Time to first token moves at most +0.03 s at 50 ms.
• Each extra device costs about 3.5–4 ms even at 0 ms delay.
• Non-code prompts accept fewer drafts: 47–59% on an itinerary prompt.

Caveats stated on the page:

• Loopback has no bandwidth limit, packet loss or jitter.
• Only model-data frames were delayed.
• All devices shared one GPU, so the 0 ms rows are pessimistic.

Full history is in docs/bench-log.md.

How to use it

Expose a room as an API. This needs Node 22+, and no GPU on the machine running the bridge:

npx @pooled/cli serve "https://pooled.run/r/4TKG9P#k=…"   # the room's invite link, or its code

This serves the OpenAI API at http://127.0.0.1:8080/v1 (chat/completions and responses, any API key) and Anthropic messages at http://127.0.0.1:8080.

• Set POOLED_TOKEN to require a key.
• Tool calls and JSON schemas are grammar-constrained, so they always parse, even from the 1.7B.
• /v1/completions is not served.
• Images are reduced to a text note.
• Requests run one at a time in the room's queue.
• Per-tool context settings are in cli/README.md.

Self-host and test:

git clone https://github.com/Nehanth/pooled && cd pooled
npx -y serve -l 8080 .        # then open http://localhost:8080/room
npm test                      # unit tests, no GPU (Deno)
npm run test:gpu              # golden tests on Qwen3 0.6B (needs a WebGPU GPU and Deno 2)
npm run test:q38              # 27B suites, including speculative == plain

The licence is MIT.

Limits

• Browser support. Chrome on macOS is the only tested host. Safari on a Mac reloads under memory pressure when holding most of the 27B, so do not host from it. Firefox and Linux Chromium need WebGPU enabled and are untested. Devices without WebGPU join only as guests who can ask questions.
• Privacy. Prompts sent through the bridge reach the host and, by default, everyone in the room. Workers see hidden states, which "are not private against a determined peer."
• Networks. Work networks often block UDP. The relay fallback only works if the site owner has configured TURN.
• Host version. Hosts running older Pooled versions return HTTP 400 on tool calls.
• Prefill speed. Prefill is the biggest performance gap. Faster prefill is the top roadmap item.