Beyond IID: How General Are Tabular Foundation Models, Really?
Lennart Purucker, Andrej Tschalzev, Nick Erickson, Gioia Blayer, David Holzmüller, Alan Arazi, Alexander Pfefferle, Mustafa Tajjar, Gaël Varoquaux, Frank Hutter
arXiv:2606.30410 · 42▲ · cs.LG, cs.AI
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
The paper introduces BeyondArena, a benchmark of 142 manually curated tabular datasets that spans IID, temporal, and grouped (non-IID) prediction tasks across sample sizes from 100 to 1 million rows, plus DataFoundry, a Python framework for reproducible dataset curation. It evaluates 11 models including three tabular foundation models (TabPFN-2.6, TabICLv2, TabDPT) against gradient-boosted trees and MLPs using in-context learning for the foundation models and tuning plus ensembling for the traditional ones.
Why it matters
If you were considering dropping a tabular foundation model into a production pipeline, this shows they only win on small, random-split data and lose to tuned CatBoost or RealMLP once you hit non-IID splits (time-based or grouped), large datasets, high dimensionality, or high-cardinality categorical features, which covers most real deployment scenarios. It also documents that using a random train-test split on data that should be split by time or group can badly distort model rankings and inflate measured performance.
Practical takeaway
For tiny-to-medium random-split problems you can try a tabular foundation model in zero-shot mode and skip tuning, but for anything time-ordered, grouped, over ~100k rows, or with high-cardinality categoricals, keep using tuned gradient-boosted trees or MLPs. Also watch your validation split: match it to how the model will actually be deployed rather than defaulting to a random split.
Key result
Foundation model error degrades sharply above roughly 100,000 rows: mean normalized TFM error rises from 0.183 (tiny, 100 to 1k rows) to 0.528 on large datasets, a statistically significant jump (Mann-Whitney, rank-biserial r between 0.68 and 0.79). On the full 142-dataset leaderboard, tuned+ensembled RealMLP leads at 1282 Elo versus 1224 for zero-shot TabPFN-2.6, with XGBoost calibrated to 1000 Elo.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.