MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
Alan Arazi, Eilam Shapira, Shoham Grunblat, Mor Ventura, Elad Hoffer, Gioia Blayer, David Holzmüller, Lennart Purucker, Gaël Varoquaux, Frank Hutter, Roi Reichart
arXiv:2605.10616 · 142▲ · cs.LG, cs.CL, cs.CV
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
MulTaBench is a benchmark of 40 datasets (20 image-tabular, 20 text-tabular) for multimodal tabular learning, curated by an automated pipeline that only keeps datasets where each modality adds independent predictive signal and where task-specific tuning of the encoder beats frozen off-the-shelf embeddings. The curation compares four conditions (unimodal, joint frozen, joint target-aware) across five tabular learners, using LoRA finetuning of the top 3 layers of e5 (text) and DINO-v3 (image) encoders as a preprocessing step to produce Target-Aware Representations.
Why it matters
Current tabular foundation models like TabPFN cannot handle text or images natively and instead consume frozen embeddings, which discard fine-grained task-relevant detail. This benchmark documents that tuning those embeddings to the target consistently helps and gives researchers a filtered testbed for building joint multimodal tabular models, so its immediate value is to model developers rather than to practitioners shipping today.
Practical takeaway
If you are feeding frozen image or text embeddings into a gradient-boosted or tabular model, expect measurable gains from lightly finetuning the encoder on your target label first (LoRA on the last few layers), especially for GBDTs, though this adds substantial finetuning compute and overfitting risk on small datasets.
Key result
On the PetFinder dataset, TabPFN-2.5 with full trimodal target-aware representations reached 88.0% AUC versus 84.9% for the same frozen trimodal setup and 83.7% for frozen structured-plus-image, a roughly 3 point gain; the paper reports the target-aware advantage holds across all learners, encoder sizes, and PCA dimensions but notes the curation pipeline selects for datasets where tuning helps, so absolute rankings are biased.
Jargon
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.