TabPFN and TabICL Outperform Tuned XGBoost on 14 Tabular Datasets Without Training

Tabular Foundation Models Tested

A comparison was conducted between TabPFN and TabICL, which are tabular foundation models, and a tuned XGBoost model. The evaluation used fourteen datasets from the Grinsztajn benchmark, maintaining consistent data splits and processing times for all models.

Performance Results

The tabular foundation models, TabPFN and TabICL, won on all fourteen datasets. This performance advantage persisted even with up to 32,000 rows of data. The key distinction is that these models predict on new data without requiring specific training on that data.

How Tabular Foundation Models Work

Tabular foundation models are pretrained on millions of synthetically generated tables. When presented with a new table, they do not adjust their internal weights. Instead, they process the training rows as context and generate predictions in a single forward pass, similar to in-context learning in language models. This process, while internally referred to as 'fit', does not involve gradient descent but rather a data copy operation.

Implications for Machine Learning Practice

The results indicate a potential change in how machine learning is applied to tabular data. If these models consistently outperform tuned boosting methods without requiring dataset-specific training, the mandatory step of searching for hyperparameters could become less critical. This could simplify and accelerate the deployment of models for tabular prediction tasks.