Transformers Model Integration Contracts

What it claims

Hugging Face proposes Model Integration Contracts (MIC, package transformers-mic) for catching Transformers regressions on PRs with cheap CPU runs. Each contract pairs a shrunk fixture model, a runnable Markdown usage path and human-reviewed baselines. The contract holds if the usage path on the fixture still reproduces the baselines. A 66 MB UMT5 fixture, shrunk from a 1,227 MB checkpoint, flagged a T5 SDPA bug on every replayed commit. That bug let UMT5's decoder attend to future tokens and stayed in main from 30 July to 28 September 2026. The project justifies the approach by test volume: Transformers has 750+ model types, 88,000+ CPU tests on main and 200,000+ GPU tests in a daily run, so full integration tests cannot run per PR.

How it works

A contract covers an architecture plus a usage path, not one checkpoint. Compatible checkpoints share a fixture but keep their own expected results. The SmolLM contract, for example, greedily generates 4 tokens from prompts like "The capital of France is" and compares token IDs. It also compares training loss, checks that gradients are finite and checks that an optimizer step changes the weights. A fleet dataset on the Hub pins each MIC to a frozen revision and the package versions used to record its baselines. Each CI run logs the fleet revision it used. On a PR, MICs are selected from the files the patch changes.

The shrink recipe has four steps:

1. Shrink the config: reduce hidden sizes, heads and depth, but keep every layer kind (sliding and full attention, dense and MoE). Widths that the generic rule does not recognise take their sizes from the model's own ModelTester in test_modeling_<model>.py.
2. Copy tokenizer and processor files byte for byte, preserving vocabulary, special tokens and input dimensions.
3. Build weights and validate them, as below.
4. Record baselines on a known-good build per platform and precision, review them, then publish a pinned snapshot. CI reuses the snapshot and does not rebuild it.

The original weights file format is kept so that conversion code still runs.

flowchart TD
  A[Shrunk config] --> B[Seeded random weights]
  B --> C{Outputs respond to input changes?}
  C -- yes --> E[Reload: outputs, loss, grads finite?]
  C -- no --> D[Overfit on contract's small batch]
  D --> E
  E -- no --> X[Reject build]
  E -- yes --> F[Record and review baselines, pin snapshot]

An LLM drafts the usage code and manifest for new models, and CPU and GPU runs measure candidate baselines. Humans must approve changed expectations. When agents hit edge cases, the team fixed the recipe rather than individual fixtures.

Evidence

The team replayed frozen MICs over five months of commits. The UMT5 fixture showed label log-probabilities 1.22 off the baseline and training loss about 0.20 off. A run that used only eager attention missed the bug; the contract caught it because it used SDPA, the default path. Reverting the fix on a proof branch in Transformers CI reproduced the failure. The T5 runner step took 13.4 seconds on CPU, excluding setup.

Llama 4 Scout shrinks from 108.6B to 44.7M parameters. Its vision stack goes from 34 to 2 layers and its language stack from 48 to 4, with MoE routing style kept. In a weight-initialisation study on Whisper, random fixtures caught 18 of 24 injected faults and overfit fixtures caught all 24. The overfit fixtures were trained on tones paired with words such as "red".

How to use it

Replace org/model with the source checkpoint:

pip install "transformers-mic[runtime]" "hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git"
hf auth login
tmic run org/model --local-only --fixtures-only --no-history --bill-to ''

This drafts the contract, builds the fixture and records baselines locally. It asks for review at each step and stops before uploading. The LLM draft runs through Inference Providers and is billed to the user's own account. Published contracts are in huggingface-mic, and coverage gaps are listed in the coverage space.

Limits

As of 1 November 2026, the MIC job reports failures but does not block PRs. PR catch rate, false-alarm rate and baseline-review cost are unmeasured. The Whisper faults were synthetic, and no real regression has yet been caught only by an overfit fixture. Both fixture types missed an assisted-decoding bug that only the real Whisper checkpoint exposed. Overfitting can fail to converge and adds maintenance. Coverage across model families and APIs is partial, and existing tests stay in place.