A Systematic Study of Small Language Models on Abstract Reasoning Tasks

What it claims

This is a descriptive study, not an architecture comparison. Each family (decoder-only, encoder–decoder, MoE) is represented by a single, different pretrained model. Observed differences therefore mix architecture with pretraining, tokenization, context length and capacity, and the authors restrict cross-family statements to "profiles." The main finding is that strong seen-family accuracy can coexist with narrow task-family coverage and weak robustness to shift. The attention-entropy and attention-edge-knockout analyses are explicitly exploratory and run on selected tasks only.

How it works

ARC-TGI turns each ARC transformation into a Python generator module:

1. create_input samples a valid grid.
2. transform_input applies the fixed rule.
3. create_grids assembles the demonstration and test pairs, rejecting invalid or ambiguous episodes.

Task-level variables (for example, joining direction) stay fixed within an episode. Grid-level variables (colour, position, object configuration, size) vary across demonstrations. Each episode also carries reasoning chains and partially evaluated Python programs. As a result, the same latent task can be posed three ways: direct grid transduction, executable-rule induction, or a reasoning-description prompt.

The released inventory has 461 human-validated generators:

This study uses only the ARC-AGI-1 training-family subset that passes its filters on token length, grid size and experiment condition.

Evaluation conditions:

• Held-out seen-family: new episodes from training generators, with task IDs always disjoint.
• Within-family shift: grid scale only in the reported experiments. Same generators, with grid sizes constrained on each side.
• Cross-benchmark: original ARC-AGI evaluation tasks. This changes both the generator source and the difficulty, so it is not a pure novelty test.

Adaptation is full fine-tuning or LoRA. The study varies hyperparameters, checkpoint granularity, training data, prompt strategy and evaluation shift. Each intervention changes one axis within matched blocks. Scoring is exact grid match.

Evidence

The fetched text describes the scale of the sweeps but contains no result numbers:

• Depth sweep: n ∈ {10, 20, 40, 50, 70, 80, 100} training examples per generator, evaluated on a common disjoint set of 30 examples per generator.
• Breadth sweep: {60, 80, 100, 140, 180, 193} generators, with 50 training examples each (the text cuts off here).
• Total: more than 1,000 runs.

How to use it

The protocol is the reusable artifact. If you are evaluating your own ARC fine-tune, three practices carry over:

• Resample within a family to separate memorising a task from learning its rule.
• Shift grid scale while keeping the rule fixed.
• Score both grid output and program output, since they solve different subsets.

For serialized grids, token length grows with spatial scale. That makes context-length sensitivity a confound for any scale-shift result.

Limits

• Novel families untested: the generator-disjoint split is described as protocol support but not reported as an endpoint, so transfer to unseen families is not measured here.
• Only one shift type: grid scale is the only parameter shift tested. Shifts in colour, position and other parameters are left as extensions.
• Scope of generators: conclusions apply only to the curated, filtered generator families, and the authors say generators are not equally natural or difficult.
• No causal claim: the attention diagnostics do not establish causal mechanisms.
• Missing details: the specific model names and sizes and all numeric results fall outside the fetched text. Check the full paper before citing any figure.