Sharing a mistake because writing it down is the only thing that stops me from making it again.
I trained a classifier on a table of events and got 97% on the held-out set. I told the team. A week later someone asked me to re-run it on newer data and it dropped to the mid 70s.
Three leaks, in order of embarrassment:
- Scaling on everything. I standardised the features using the mean and variance of the whole dataset and split afterwards. The test set had already influenced the preprocessing.
- Duplicates across the split. The same user appeared many times. A random row-level split put near-identical rows on both sides, so the model was partly remembering users.
- Time. The task was predicting the future, but my split was random. It trained on events after the ones it was tested on.
What I do now:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train) # the scaler only sees training data
print(model.score(X_test, y_test))
Split by group (user) and by time when the task is about the future, and put every learned preprocessing step inside the pipeline. A number that is too good is a reason to look for a bug, not for a celebration.