Skip to content

Back to the catalog

Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration
arxiv.orgpaper

Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration

This paper introduces Quantizer-Aligned Recalibration (QuAR), a single-pass, backprop-free test-time adaptation method for quantized vision transformers that recalibrates activations at frozen quantizers to counter distribution shift without updating model parameters. On ImageNet-C with ViT-B across 3-, 4-, 6- and 8-bit weight/activation precision, QuAR beats the strongest backprop-free baselines by up to 4.00 points while cutting latency 46% and adding only 0.17 MB memory overhead, with gains holding across continual streams, label shift, seven OOD suites, and three other backbones.

TL;DR A zero-backprop, single-pass recalibration method restores accuracy lost when quantized ViTs face distribution shift, beating prior backprop-free TTA baselines at multiple bit-widths with near-zero overhead.
Open original ↗
Document

What it claims

Quantized ViTs lose more accuracy under distribution shift than full-precision ones. On ImageNet-C, ViT-B falls from 55.6 top-1 at full precision to 28.4 unadapted at W3A3, which the authors put at 1.7× amplification. The cause they identify is mechanical. Frozen per-tensor quantizers receive per-channel activations whose mean and scale have drifted, so distinct values collapse onto the same code. At W6A6, ImageNet-C severity 5, 20.6% of the 21,504 recalibrated channels lose at least half a bit of code resolution.

QuAR's fix is to undo that drift at the quantizer input, in closed form, inside the normal forward pass. Its gain over no adaptation grows as precision drops: +3.8 at W8A8, and more than +6 at every quantized setting. On a code-distribution distance (Wasserstein-1 between code histograms), it removes about 52% of the drift. That is roughly 9× the best baseline, and the other methods leave the distribution unchanged or make it worse.

How it works

  • Input model. A ViT quantized with standard PTQ (PTQ4ViT or AdaLog). Weights, scale s and zero-point z stay frozen throughout.
  • Stored statistics. Per-channel source mean and std (µ_s, σ_s) are recorded at every recalibrated interface from the clean calibration set.
  • Running statistics. At test time, running target moments (µ_t, σ_t) are initialized from the source values and updated online from the stream. No past samples are stored.
  • Correction. Each channel gets x̂ = (1−λ)x + λ[(x − µ_t)·σ_s/(σ_t+ε) + µ_s] before quantization.
  • The ratio σ_s/(σ_t+ε) is clamped to [1/3, 3] (ρ=3). This stops a collapsed channel from inflating the correction and bounds the gain over long continual streams.
  • Local branch. Applied at the inputs of attn.qkv, attn.proj, mlp.fc1 and mlp.fc2 in the last four blocks (8–11), with λ=0.5. Full matching (λ=1) over-corrects scale, most visibly on ViT-S and distilled DeiT-B. The paper frames partial matching as the bias–variance optimum when moments come from finite samples.
  • Global branch. The same affine map applied to the final CLS representation at the classifier input. The fetched text cuts off before giving its λ.
  • Implementation. Forward pre-hooks. The cost is one multiply–add per recalibrated activation, with no backprop, no parameter updates and no extra forward pass.

The idea is not new; what is new is where it is applied. The affine form is standard normalization-statistics adaptation. Placed at block outputs, as prior methods do, it recovers only about half the gain at every bit-width. Placed at the quantizer inputs, it adds 1.8–3.1 points.

Evidence

  • Accuracy vs. backprop-free baselines. On ViT-B / ImageNet-C, QuAR beats the strongest backprop-free baseline at every precision. The baselines are T3A, FOA, ZOA, NEO and FOZO, and the margin runs from +2.28 at W8A8 to +4.00 at W3A3.
  • Latency. 46% lower than ZOA and FOZO.
  • Memory. 0.17 MB overhead, or 0.01% of peak inference memory.
  • Edge hardware. On an 8 GB Jetson Orin Nano at batch 64, QuAR is the most accurate method that fits. The backprop-based methods run out of memory.
  • Robustness of the setting. One fixed configuration holds its lead across:
  • long continual streams
  • non-i.i.d. label shift
  • ImageNet-R, ImageNet-Sketch, ImageNet-3DCC, CIFAR-10-C, CIFAR-100-C and DomainNet-126
  • DeiT-B, ViT-S and ResNet-50 backbones

How to use it

The code is at chahh9808/QuAR. The page lists only that handle and gives no install commands.

To deploy you need three things:

  1. A PTQ-quantized model with per-tensor activation quantizers.
  2. Per-channel source moments captured at calibration.
  3. Pre-hooks at the listed layers.

The hyperparameters are fixed across all reported results: λ=0.5 (local), ρ=3, blocks 8–11.

Limits

  • The fetched text is truncated in the Method section. Results tables, the global-branch details and the appendices (ablations, comparison with Adaptive Quantile Recalibration) were not visible.
  • The method assumes you still hold, or can recompute, the source calibration statistics.
  • It targets small-batch operation. Adaptive Quantile Recalibration is compared only at its large-batch paper default, and only in an appendix.
  • The 46% latency figure is stated against ZOA and FOZO, not against the single-pass baselines NEO and T3A.
  • Block choice and λ were selected once. The authors report a flat plateau, but λ=1 visibly degrades the smaller and distilled models.
  • The arXiv version is v1, posted 6 Oct 2026.

Organisations named here

  • ImageNet-C

Technologies named here

  • ViT-B
  • Vision Transformer
  • Transformer

Techniques named here

  • Quantizer-Aligned Recalibration
  • Post-training quantization
  • Test-time adaptation

Related in research