NVIDIA Kumo Tabular Sets a New Accuracy-Efficiency Frontier for Tabular Prediction

What it claims

Kumo Tabular is an in-context-learning tabular model with three sizes (Small/Medium/Large, 28M–215M parameters). It ranks first on TabArena with an ELO of 1950 using default settings. Against LimiX-2, the next-best model, the page reports it as "17 faster", most likely 17×, on a single RTX 6000 Pro. All three sizes sit on the accuracy–efficiency Pareto front. Classification and regression are separate models. Regression outputs 999 quantiles, so every prediction comes with an uncertainty estimate, not just a point value.

How it works

The summary names column, row and in-context attention. The page gives these specifics:

1. Cell embedding. A group of cells becomes one token. Numerical and categorical values each go through their own Fourier features (sines and cosines of learned frequencies). Missing values get special handling, so no imputation is needed. Context tokens also get a label embedding.
2. Row embedding. Column attention and row attention alternate several times. - Column attention uses induced self-attention, so its cost grows linearly with the number of rows. - Row attention uses rotary positions to tell columns apart. - Four learnable [CLS] tokens per row act as the readout. After this step, cost no longer depends on the number of columns.
3. In-context stage. Context rows attend to each other. Query rows attend to context rows only. - A prediction never depends on which other rows are scored in the same batch. - The context's keys and values are computed once and reused for follow-up predictions. - Test-GQA shrinks that cache.
4. Attention temperature. Queries are scaled by a temperature that grows with log(number of keys), with a learned coefficient per head.

flowchart LR
  A[Cells → Fourier tokens] --> B[Column ⇄ row attention ×N]
  B --> C[4 CLS tokens per row]
  C --> D[Context rows: self-attend, KV cached]
  C --> E[Query rows: attend to context only]
  D --> E
  E --> F{Task model}
  F -->|classification| G[Class probabilities]
  F -->|regression| H[999 quantiles]

Pretraining.

• Data: synthetic tables from structural causal models. Node functions are linear maps, small MLPs, trees or Gaussian processes. A tree-ensemble check discards tables with no learnable signal.
• Built-in imperfections: several missingness patterns, coarsened features that make duplicate rows disagree on their label, high-cardinality categoricals, and heavy-tailed regression targets.
• Training ran in three stages, following TabICLv2:
• 1,024 rows and up to 100 columns (the longest stage).
• 400–10,240 rows.
• Up to 60,000 rows, still at most 100 columns.
• Small/Medium/Large saw about 35M/71M/137M synthetic tables respectively.

Evidence

• TabArena: ELO 1950, first overall. The comparison included tuned GBDTs, AutoGluon and recent tabular foundation models, all on a single RTX 6000 Pro.
• BeyondArena: ELO 1418, Improvability 7.78%, first.
• TALENT: best overall ranking. Average ranks are 6.67 for classification accuracy, 3.98 for classification log-loss and 4.22 for regression RMSE.
• ScoringBench (predictive distributions): Large ranks first and Medium second on average rank.

The page gives no absolute latency or throughput figures, and no per-dataset results.

How to use it

• License: OpenMDW-1.1, which permits commercial use.
• Weights: nvidia/Kumo-Tabular on Hugging Face, downloaded automatically on first use.
• Library: NVIDIA's GPU-native structured-data-models (github.com/NVIDIA/structured-data-models), imported as sdm. It supplies the preprocessing, ensembling and many-class handling used in the evaluations, so reproducing the reported numbers presumably requires it.

The page's example, verbatim:

import sdm  # structured-data-models
# Tensorize tabular data:
table = sdm.TableTensor.from_pandas(pd.load_csv(...), device="cuda")
na_mask = table["target"].isnan()
model = sdm.models.KumoTabular(device="cuda")
pred = model(
    # In-context examples (features/targets):
    x_context=table[~na_mask].drop_columns("target"),
    y_context=table[~na_mask, "target"],
    # Prediction examples (features):
    x_query=table[na_mask].drop_column("target"),
)

The snippet won't run as written. pandas has read_csv, not load_csv, and the code calls both drop_columns and drop_column. Check the repository for the real API. The page shows no install command.

Limits

• Input types: only numerical and categorical columns go into the model. Text, images and timestamps must first be converted with the library's preprocessing recipes.
• Classes: one forward pass handles at most 10 classes. Beyond that, the library uses error-correcting output codes.
• Out of range: accuracy may degrade on tables well beyond the training ranges, which were up to 60,000 context rows and 100 columns in the final stage. It may also degrade when query rows come from a different distribution than the context rows.
• Validation: the authors advise checking accuracy and calibration on your own held-out data before deploying.
• Not yet released: the training recipe and synthetic data generators ("will be released soon"), so you cannot yet retrain or adapt the pretraining.