Semantically Rich, Metrically Affect-Blind: A Geometric Blind Spot in Modern Text Embedders

What it claims

Eleven frozen text encoders released between 2021 and 2026 all encode sentence sentiment well enough for a linear probe to read it. Their cosine geometry barely uses that signal, and the paper calls this a "decodability–geometry dissociation." The author, Mahdi Naser Moghadasi of BrightMind AI (arXiv, 23 October 2026), traces it to dilution. The polarity direction holds only 2.4% to 6.7% of embedding variance, so it cannot steer a cosine computed over all dimensions. Scale does not help: Qwen3-Embedding-4B (4.3%) is more dilute than Harrier-0.6B (6.6%). A small trained head raises that share to 85.5% to 96.7%. With it, Harrier-OSS-v1 0.6B reaches 91.8 SST-2 retrieval MAP at 128 dimensions, against 90.3 for Qwen3-4B with the same head.

How it works

The Attribute Alignment Gap (AAG) is the mean cosine similarity of same-label pairs minus that of different-label pairs, computed on L2-normalized embeddings. It needs only categorical labels, applies to any attribute, and costs O(n²) similarity evaluations.

The SISE head is a two-layer MLP: a 256-unit ReLU hidden layer and a 128-d L2-normalized output. It has 256d + 33,152 parameters, which is about 0.13M at d=384 and 0.69M at d=2560. It is trained with supervised contrastive loss (τ = 0.07) for 60 epochs, using Adam (lr 1e-3, weight decay 1e-5) and batch size 512. Training uses cached embeddings only, so the corpus is not re-encoded. Everything, including embedding with the 4B model, ran on a 2-core CPU with no GPU, and each head trains in seconds.

The head's output can be used in two ways, with different costs to general similarity:

flowchart LR
  A[Frozen encoder embedding] --> B[SISE head, 128-d]
  B --> C[Compact: sentiment metric only]
  A --> D[Concatenate]
  B --> D
  D --> E[Fused: keeps general semantics]

Evidence

The data are SST-2 (4,000-sentence training subsample, 872-sentence validation split used as test) and MR in its Rotten Tomatoes form (4,000 train, 1,066 test). Selected SST-2 results, base then +SISE:

• Harrier-0.6B: probe 90.8 to 93.3, MAP 73.6 to 91.8, AAG 0.034 to 0.512.
• Qwen3-4B: probe 91.3 to 92.9, MAP 64.3 to 90.3, AAG 0.029 to 0.393.
• Arctic-Embed-L: probe 82.2 to 86.1, MAP 54.4 to 79.6, AAG 0.005 to 0.264. This is the most affect-blind base space in the panel.
• MiniLM (2021): probe 77.4 to 76.0. It is the only encoder to regress on probe accuracy, and it also drops 76.3 to 73.5 on MR.

Across encoders, MAP gains range from +9.8 to +26.0 and AAG rises 7 to 61 times.

A task-conditioning control used Jina-v5-text-small. Switching it from its retrieval task to its classification task raised SST-2 MAP from 63.2 to 73.8, while SISE on retrieval mode reached 85.1. On MR the figures are 58.7, 68.4 and 77.6. Task switching therefore recovers 48% to 51% of the injection gain and leaves AAG almost unchanged (0.039 to 0.043). SISE still adds 7.7 MAP on top of classification mode.

In the few-shot setting, a probe given 2 labels per class gains up to +31.8 points (Arctic, SST-2). In that setting the head had already been trained on all 4,000 labels.

How to use it

Code, raw results and configuration are at https://github.com/mahdinaser/sent2vec-sise. The page gives no install commands or licence. The method needs a labelled training set for the target attribute (4,000 examples in the paper). The head is applied to stored embeddings, so an existing vector index can be projected without re-embedding text.

Limits

The compact 128-d head costs 38 to 41 Spearman points on STS-B. The fused variant costs 2.4 to 12.0 points. Each head is therefore specialized to one attribute.

Only binary sentiment is tested, on two movie-review datasets. The SST-2 test set is the validation split. The author states that hyperparameters were fixed in advance and that the ablation on this split was not used for selection. The objective-level explanation rests on a task switch in one model, which the author says cannot fully isolate pretraining objectives. The author also calls the probe-minus-MAP gap an informal indicator, not a calibrated measure. The fetched text ends partway through the results section, so the STS-B section and any stated limitations section were not available.