Data science extracts knowledge from data by combining statistics, programming and knowledge of the subject. It asks a question, gathers and cleans data, explores it, builds a model or an estimate, and reports what is and is not supported. The skill is as much in doubting results as in producing them.
Key concepts
- Question first: a clear question and a way to tell if the answer is useful come before any model.
- Cleaning and exploration: most of the time goes into fixing, joining and plotting data to understand it.
- Statistical thinking: distributions, sampling, uncertainty and the difference between correlation and causation.
- Modelling and validation: fit on training data, judge on data the model has not seen.
- Communication: results must reach decision makers in plain words, with caveats.
Vocabulary
- Overfitting: a model that memorises noise in the training data and fails on new data.
- Feature: an input variable used by a model.
- Confidence interval: a range that expresses the uncertainty of an estimate.
- Bias: a systematic error, in the data, the sampling or the model.
What to study, in order
- Descriptive statistics and probability: mean, spread, distributions and sampling.
- Python or R with a dataframe library; load, clean and summarise a real dataset.
- Plot every variable and every pair, and write down what surprised you.
- Fit a linear and a logistic regression and interpret the coefficients.
- Split data into training and test sets, use cross-validation, and see overfitting happen on purpose.
- Write a short report that states the question, the method, the result and its limits.