Rethinking Sequence Compression with Test-time Regression

What it claims

TTR-Compressor compresses a block of T tokens into one d-dimensional vector by solving a small ridge-regression problem built from those tokens. The fitted coefficients are the compressed vector. The authors (CUHK, Alibaba Token Foundry; arXiv October 2026) argue that mean pooling, learned projections and attention-weighted pooling all weight tokens without regard to inter-token correlation. As a result, redundant tokens add up and distinctive ones are diluted. TTR-Compressor adds only 2d learnable parameters per compressor and is presented as a drop-in replacement for those operators.

How it works

• Keys: each token is L2-normalised, k_t = x_t/‖x_t‖, so KᵀK measures only directional similarity.
• Targets: each token gets a scalar target v_t = aᵀx_t from a learned vector a. Targets are deliberately not normalised. With normalised targets, the solution collapses toward the constant vector a. A local 1D convolution (language) or 2D convolution (vision) can optionally be applied before the projection.
• Regulariser: the ridge λI is replaced by a learned diagonal D = diag(softplus(ρ)). This removes ridge hyperparameter tuning during pretraining.
• Solve: because T < d in typical compression windows, it uses the dual form. Keys are rescaled K̃ = KD^(-1/2), then a T×T Cholesky factorisation of I + K̃K̃ᵀ is computed, triangular solves give α, and the output is h = D^(-1/2)K̃ᵀα. Cost is O(T²d + T³), against O(Td² + d³) for the primal form.
• Training: a and ρ are trained end-to-end by backpropagating the task loss through the closed-form solve. At inference they are fixed and only h is recomputed per block. There is no iterative inner optimisation.

The Gram matrix is what pools cannot reproduce. In the paper's toy example, three identical keys and one distinct key get α weights of 1/4, 1/4, 1/4 and 1/2. The repeated-to-distinct influence ratio becomes 3:2, against 3:1 for additive pooling. A smaller learned D strengthens this deduplication.

The authors contrast it with TTT, MesaNet and DeltaNet-style layers. Those produce fast weights through iterative or single-step updates. Here the blocks are small, disjoint and independent, so an exact closed-form solve is possible and the coefficients themselves are the output.

Evidence

The fetched text cuts off in Section 4, before the results tables. Beyond the abstract's claims (better RULER retrieval than attention-based compressors, citing Hu et al., 2026; matching the MLP visual merger with 55.6% fewer visual tokens), it gives no backbone models, model sizes, compression ratios or per-task scores.

Limits

• The efficiency argument assumes block size T is smaller than channel dimension d. Cost grows as T³, so large windows lose the advantage.
• The design depends on blocks having no cross-block dependencies.
• The available text states no code release, weights or licence. A project page is listed at sphereLab.ai/ttr-compressor.
• Alternative target parameterisations are deferred to Appendix E.2, which is not in the fetched text.