FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

What it claims

FedRAPP is a server-side aggregation rule for federated test-time prompt tuning of a frozen CLIP. In this setting each client adapts only soft prompts on its own unlabeled, non-IID test stream, starting without any source-pretrained prompt. The authors contrast this with COTE, which initializes prompts from supervised source-domain training, and with Latte, which shares class prototypes across clients. Clients upload only the tuned prompt and two scalar entropy values per round. Under a non-IID partition with AdaPrompt as the local method, FedRAPP gains +0.09 to +3.07 top-1 points over no federation across nine CLIP datasets.

How it works

Each round, client i runs its local test-time method TTA(·) on prompt P^t-1 and an unlabeled batch. The method is pluggable; AdaPrompt is the one shown. The client records mean prediction entropy before adaptation (H^pre) and after (H^post) on the same batch.

The server then does four things with that upload:

1. Update and drift. It computes the update Δ = P̃ − P^t-1 and the drift Q = log(1 + ‖Δ‖²).
2. Reliability score. It computes r = z(H^pre − H^post) − λ_ent·z(H^post) − λ_drift·z(Q). Each z-score is taken across the clients active in that round. The score rewards entropy reduction and penalizes leftover uncertainty and large prompt movement.
3. Per-client EMA memory. It keeps M_i = μ·M_i^prev + (1 − μ)·P̃_i, with μ = μ₀ − η·r clipped to [0, 1]. Reliable clients therefore overwrite their memory faster.
4. Peer weights. It builds collaboration weights from cos(Δ_i, Δ_j) and the reliability of each source client j. Each client then receives its own personalized prompt.

No labels or validation accuracy are used anywhere in this loop.

flowchart LR
  A[Client: local TTA on unlabeled batch] --> B[Upload tuned prompt + H_pre, H_post]
  B --> C[Reliability score r]
  B --> D[Update direction Δ]
  C --> E[EMA memory, μ depends on r]
  C --> F[Peer weights]
  D --> F
  E --> G[Personalized prompt per client]
  F --> G
  G --> A

Evidence

Figure 1 compares FedRAPP against None-Fed (each client adapts alone) and FedAMP. The setting is a non-IID partition with AdaPrompt local adaptation, and the metric is top-1 accuracy. FedRAPP's gains over None-Fed range from +0.09 to +3.07 points. The largest are about +3.07, +2.16 and +1.90; two datasets gain only +0.09.

The datasets are Flower102, DTD, Caltech101, OxfordPets, Food101, UCF101, SUN397, FGVCAircraft and StanfordCars. The extracted figure text does not reliably map each gain to its dataset. FedAMP's own numbers are not legible in the extracted text.

The paper also claims a benchmark spanning several continual prompt-tuning methods, several aggregation strategies, IID and non-IID splits, and five ImageNet variants. No tables for these results appear in the fetched text.

Limits

The fetched text stops partway through Section 3.5. It ends before the full peer-aggregation formula, the hyperparameter values (λ_ent, λ_drift, μ₀, η), the results tables, the ablations and any stated limitations. No code, CLIP backbone size, client count or communication cost is given.

The reliability signal is entropy reduction on the client's own batch. The authors themselves note that test-time updates from small batches are noisy and batch-biased. Gains over isolated adaptation are below 0.7 points on four of the nine datasets in Figure 1.