Language Models for Text Classification: From Bag-of-Words to Jev

What it claims

Raschka ran TypeSafe AI's proprietary Jev classifier (jev-1.13.0) on the full 25,000-review IMDb test set with no fine-tuning. It reached 96.47% accuracy via the Choice API for about $0.65 in roughly 22 minutes. That matches or slightly beats a fine-tuned ModernBERT (~95%), which needed 23 minutes of training. His argument is that Jev's value is breadth: the same model also plays Tetris in real time. Hand-rolled "Jev clones" match the API shape but not that breadth.

Evidence

All results below are on the IMDb test set (25,000 reviews):

Calibration. The author trained ModernBERT on cross-entropy plus Brier loss. This gave only a "modest"/"very small" improvement over plain temperature scaling.

RLCR (2025 paper, Qwen2.5-7B), HotpotQA vs RLVR:

• Expected calibration error fell from 0.37 to 0.03.
• Accuracy was similar: 62.1% vs 63.0%.
• Across six other datasets, ECE fell from 0.46 to 0.21 and accuracy rose from 53.9% to 56.2%.

How it works

Jev's own architecture and RLCD training are not public. The author guesses it is a small ModernBERT-like model trained on fully synthetic, heavily curated data.

Retrofitting a Jev-like Choice API onto BERT, GPT or T5:

1. Replace the N-class head with a single-output scoring head, s_i = wᵀh_i + b.
2. For each candidate, feed the input text, the instructions and that candidate's description. Score each one with the same head.
3. For h_i, use the [CLS] state for BERT-style models and the last non-padding token for GPT-style models (because of causal masking).
4. Softmax across the candidate scores and train jointly with cross-entropy.

The number of options can change without changing the architecture.

flowchart LR
  T[Text + instructions] --> A[+ option A desc]
  T --> B[+ option B desc]
  T --> C[+ option C desc]
  A --> H[Shared 1-output head]
  B --> H
  C --> H
  H --> S[Softmax over N scores]

RLCR. The model emits, in order:

1. Reasoning
2. An answer
3. An uncertainty analysis
4. A confidence q

The reward is the correctness reward minus a Brier term (q − correct)². The page's examples:

• Wrong at 0.9 confidence → −0.81
• Wrong at 0.2 confidence → −0.04
• Right at 0.9 confidence → 0.99

How to use it

Endpoint: https://api.typesafe.ai/v1/systemone. There are three question types:

• choice: one label from named criteria. Returns choice, confidence and per-class probabilities. Note that confidence measures how concentrated the distribution is; it is not the winning class's probability.
• noul: a single "yes" probability. For multi-label tasks, ask one question per label; the probabilities need not sum to 1.
• score: a rubric level (e.g. 0/1/2).

curl -sS https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-1.13.0",
    "state": "The acting was excellent and the story kept me engaged throughout. I would happily watch this movie again.",
    "questions": {
      "sentiment": {
        "type": "choice",
        "instructions": "What is the overall sentiment of this movie review?",
        "criteria": {
          "negative": "An overall unfavorable opinion of the movie",
          "positive": "An overall favorable opinion of the movie"
        }
      }
    }
  }'

OpenAI announced a comparable Decisions API at DevDay on 29 September 2026. It is in limited preview.

Limits

• Contamination unknown. It is not known whether IMDb was in Jev's training data.
• Non-deterministic runs. A second Choice run gave different numbers. The same text scored 1.0 via Choice but 0.98 via Noul.
• Weak spot reported. A reader reports Jev does poorly on non-English legal reviews. The author's suggested workaround is translating first.
• Fine-tuning still wins at the extremes. For high-volume, narrow tasks, a fine-tuned specialist should still be better, faster or cheaper.
• No open weights. Jev is proprietary, so privacy-sensitive use has no open equivalent of comparable breadth yet.
• RLCD link is a guess. The connection between RLCD and RLCR is the author's assumption.