Saltar al contenido ForoHackers

I reported 97% accuracy and it was data leakage. A post-mortem.

Evaluación de IACiencia de datosAprendizaje automático
#1077 · git_blame_myself · · Denunciar

Sharing a mistake because writing it down is the only thing that stops me from making it again.

I trained a classifier on a table of events and got 97% on the held-out set. I told the team. A week later someone asked me to re-run it on newer data and it dropped to the mid 70s.

Three leaks, in order of embarrassment:

  1. Scaling on everything. I standardised the features using the mean and variance of the whole dataset and split afterwards. The test set had already influenced the preprocessing.
  2. Duplicates across the split. The same user appeared many times. A random row-level split put near-identical rows on both sides, so the model was partly remembering users.
  3. Time. The task was predicting the future, but my split was random. It trained on events after the ones it was tested on.

What I do now:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)      # the scaler only sees training data
print(model.score(X_test, y_test))

Split by group (user) and by time when the task is about the future, and put every learned preprocessing step inside the pipeline. A number that is too good is a reason to look for a bug, not for a celebration.

#1659 · two_phase_commitment · · Denunciar

The duplicates one has a database cousin: a join that quietly multiplies rows, then a "random" split that sends the copies to both sides. Check for unique keys across train and test before anything else. It is a two-line query.

#1660 · rubber_duck_supervisor · · Denunciar

Thank you for writing it up. Question for everyone, since this is the useful part: what was the first sign that made you suspicious of a number that looked good?

#1661 · mutex_mike · · Denunciar

Mine is the same shape in a different field: a benchmark where the second run was twice as fast as the first. The cache had been warmed by the previous run. A result that is great and was not predicted is a measurement of something you did not intend.

Inicia sesión o únete para responder.