Saltar al contenido ForoHackers

Data science: start here

Ciencia de datos
#403 · Ophuscado · · Denunciar

Data science extracts knowledge from data by combining statistics, programming and knowledge of the subject. It asks a question, gathers and cleans data, explores it, builds a model or an estimate, and reports what is and is not supported. The skill is as much in doubting results as in producing them.

Key concepts

  • Question first: a clear question and a way to tell if the answer is useful come before any model.
  • Cleaning and exploration: most of the time goes into fixing, joining and plotting data to understand it.
  • Statistical thinking: distributions, sampling, uncertainty and the difference between correlation and causation.
  • Modelling and validation: fit on training data, judge on data the model has not seen.
  • Communication: results must reach decision makers in plain words, with caveats.

Vocabulary

  • Overfitting: a model that memorises noise in the training data and fails on new data.
  • Feature: an input variable used by a model.
  • Confidence interval: a range that expresses the uncertainty of an estimate.
  • Bias: a systematic error, in the data, the sampling or the model.

What to study, in order

  1. Descriptive statistics and probability: mean, spread, distributions and sampling.
  2. Python or R with a dataframe library; load, clean and summarise a real dataset.
  3. Plot every variable and every pair, and write down what surprised you.
  4. Fit a linear and a logistic regression and interpret the coefficients.
  5. Split data into training and test sets, use cross-validation, and see overfitting happen on purpose.
  6. Write a short report that states the question, the method, the result and its limits.

Related tags

#probability #machine-learning #data-visualization #python

Inicia sesión o únete para responder.