Skip to content

Back to the catalog

Embedding Models Measure in Peculiar Ways
arxiv.orgpaper

Embedding Models Measure in Peculiar Ways

This paper studies whether embedding models' notions of semantic similarity align with objective physical measurements of mass, distance, time and volume. The authors find that physical measurement is only weakly captured in embedding space, and that embedding similarity for quantities is instead heavily driven by superficial string similarity (e.g., shared digits or units) rather than true magnitude proximity, with recalibration attempts failing to fix the misalignment.

TL;DR Embedding models poorly capture true physical measurement equivalence, relying instead on superficial string similarity between numbers/units.
Open original ↗
Document

What it claims

The paper tests whether text embedding spaces encode physical measurements — mass, distance, time, and volume — quantities that have an objective, unique notion of equivalence and ordering (e.g. 1000 g equals 1 kg; 2 m is closer to 3 m than to 100 m). It finds that embeddings model physical measurement only weakly, and instead exhibit "peculiar" measurement patterns that don't track the real numeric relationships.

Evidence

The abstract gives the direction of the findings but no numbers, benchmarks, or model names. Two concrete mechanistic claims are stated:

  • Embedding representations of physical measurements are strongly influenced by superficial string similarity rather than by the underlying magnitude — i.e. two quantity strings that look alike land near each other even when the values differ, and equal-magnitude values expressed differently do not.
  • Recalibrating similarity does not substantially improve alignment with true physical distance, so the problem is not merely a scaling artefact that a post-hoc correction fixes.

Why it matters for the reader

If you rely on embedding cosine similarity anywhere quantities matter — retrieval over specs, dedup of measurement-bearing records, unit-aware search, RAG over documents with masses/volumes/durations — this is a stated failure mode: the space keys on surface form, not on the number. The negative recalibration result specifically warns against assuming a lightweight fix will recover magnitude ordering.

Limits

Only the abstract was available. It names no models, no datasets, no metrics, and no quantitative results — "weakly modeled," "peculiar," and "strongly influenced" are the authors' qualitative characterizations, not measurements you can compare against a baseline. Which embedding models were tested, how physical equivalence was operationalized, and what recalibration method was tried are all unstated here and must be read from the full PDF before acting. Submitted 17 Sep 2026 (v1); no peer-review status indicated.

Related in evals