Skip to the content ForoHackers
Log inJoin

Data science: start here

Data science
#2909 · · Quote · Report

Data science extracts knowledge from data by combining statistics, programming and knowledge of the subject. It asks a question, gathers and cleans data, explores it, builds a model or an estimate, and reports what is and is not supported. The skill is as much in doubting results as in producing them.

Key concepts

  • Question first: a clear question and a way to tell if the answer is useful come before any model.
  • Cleaning and exploration: most of the time goes into fixing, joining and plotting data to understand it.
  • Statistical thinking: distributions, sampling, uncertainty and the difference between correlation and causation.
  • Modelling and validation: fit on training data, judge on data the model has not seen.
  • Communication: results must reach decision makers in plain words, with caveats.

Vocabulary

  • Overfitting: a model that memorises noise in the training data and fails on new data.
  • Feature: an input variable used by a model.
  • Confidence interval: a range that expresses the uncertainty of an estimate.
  • Bias: a systematic error, in the data, the sampling or the model.

What to study, in order

  1. Descriptive statistics and probability: mean, spread, distributions and sampling.
  2. Python or R with a dataframe library; load, clean and summarise a real dataset.
  3. Plot every variable and every pair, and write down what surprised you.
  4. Fit a linear and a logistic regression and interpret the coefficients.
  5. Split data into training and test sets, use cross-validation, and see overfitting happen on purpose.
  6. Write a short report that states the question, the method, the result and its limits.

Related tags

#probability #machine-learning #data-visualization #python

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply