In-context learning

In-context learning (ICL) is the ability of a model to solve a new task from examples supplied in its input at inference time, without any gradient updates to its weights. If you know how a large language model answers a question after you paste a few labeled examples into the prompt, that is ICL. The weights stay frozen; the "learning" happens entirely in the forward pass as the model conditions on the examples you provide.

For tabular foundation models, this reframes what used to be a training step. Instead of calling model.fit(X_train, y_train) to update parameters, you pass the training rows and labels as context alongside the test rows, and the model returns predictions in one forward pass. The fit step becomes an inference step.

A concrete shape: to classify 500 test rows, you hand TabPFN a context of, say, 3,000 labeled training rows plus the 500 unlabeled rows, and it outputs all 500 predictions at once. No per-dataset tuning, no epochs.

This is why the paper above evaluates its foundation models (TabPFN-2.6, TabICLv2, TabDPT) with ICL while giving gradient-boosted trees the usual tuning-and-ensembling treatment: the comparison is between a frozen model reading your data as context versus a model retrained specifically for it.

Two practical consequences:

• Context size is bounded, so very large training sets (the benchmark goes to 1 million rows) must be subsampled or otherwise compressed to fit.
• Generalization depends on how closely your task resembles the synthetic distributions the model was pretrained on, which is exactly what a non-IID benchmark stresses.