Validation and Leakage
A validation score is only as honest as the order of the steps that produced it. The posts here measure the gap directly: on the OpenML employee salaries data, mean-encoding the categories on all rows before cross-validation reported an MAE of 4,054 USD, and the same model scored 4,636 on a held-out set.
Start with the preprocessing leak if you fit scalers or encoders yourself, then the threshold and baseline posts if your metric looks good but the decisions built on it do not.
- Target encoding leakage: what scikit-learn’s TargetEncoder cross fitting protects you from
Mean-encoding categories before cross-validation reported an MAE of 4,054 USD; the same model scored 4,636 on a holdout. Inside a pipeline, plain mean encoding was fine until an ID-like column appeared, then TargetEncoder's cross fitting kept the error flat. Tested with scikit-learn 1.9.1 on real salary data.
- How to choose a classification threshold from error costs (scikit-learn TunedThresholdClassifierCV)
On the German Credit data, a logistic regression at the default 0.5 threshold cost 0.866 per applicant, more than rejecting everyone (0.700). The threshold from the cost matrix, 1/6, cut it to 0.545, and TunedThresholdClassifierCV found the same from training data alone. Tuning on the test set looked better than it was.
- Before replacing LightGBM with TabPFN-3.5, check your evaluation split
TabPFN-3.5's reported benchmark lead is a reason to evaluate, not a universal replacement rule. A local LightGBM demo scores 1.0000 AUC on random rows and 0.4748 on unseen groups; no TabPFN model is run.
- How to use MinMaxScaler in scikit-learn: fit, transform, and inverse_transform
MinMaxScaler step by step with real output: fit on training data only, why new values go above 1, the 1D-array error, how to inverse_transform a single predicted column, NaN handling, and keeping DataFrame column names.
- Train, validation and test sets: what overfitting looks like in numbers
Why you need three data splits, measured on a real dataset: a fully grown decision tree scored 1.000 on training data and 0.912 on validation, and picking the best model on one small validation split overstated the test score.
- Why you should always build a baseline model first
Before training a real model, measure the simplest possible one. On a price-like series, "tomorrow = today" beat a random forest by more than 3×. Tested with scikit-learn, with the reason the forest failed.
- Why fit preprocessing on training data only? A data leakage demo
Fitting a scaler or a feature selector on all your data before cross-validation leaks test information. On pure noise, leaky feature selection scored 0.80 instead of 0.50. Tested with scikit-learn 1.9.1.
- Why is 99% accuracy useless? Precision, recall and F1 explained
On data with 1% fraud, a model that never predicts fraud scores 99% accuracy. Tested with scikit-learn: what precision, recall and F1 show instead, and how the decision threshold trades one for the other.
- MinMaxScaler vs StandardScaler: which scaler should you use?
StandardScaler for KNN, SVM, linear models and PCA; MinMaxScaler when you need a 0–1 range; RobustScaler when there are outliers; nothing for tree models. Tested in scikit-learn: scaling lifts KNN from 0.691 to 0.950.
