A recent experiment tested the claims of TabPFN and TabICL, tabular foundation models that can make predictions on a table without training on it, against tuned XGBoost on 14 datasets from the Grinsztajn benchmark. The results showed that the model that doesn't train, TabPFN or TabICL, won in all 14 cases, with its advantage holding up to 32,000 rows, challenging the conventional practice of hyperparameter tuning. This suggests that searching for optimal hyperparameters may no longer be a necessary step, and instead, become a luxury that doesn't always pay off, potentially changing the landscape of machine learning practices.