Skip to the content ForoHackers
Log inJoin

Data mining: start here

Data mining
#2917 · · Quote · Report

Data mining is the search for useful patterns in large datasets: groups of similar items, things that occur together, unusual cases, trends. It sits between statistics, databases and machine learning, and its output is a hypothesis to examine, not proof. Without care, a big enough dataset will always show patterns that are only chance.

Key concepts

  • Frequent patterns: sets of items that often appear together, such as products bought in one basket, and the rules that follow.
  • Clustering: grouping similar records without labels, such as k-means or hierarchical methods.
  • Classification and regression: predicting a category or a number from labelled examples.
  • Anomaly detection: finding records that do not fit, such as faults or fraud.
  • Process: select data, clean it, transform it, mine it and then interpret and validate the result.

Vocabulary

  • Support and confidence: how often an itemset occurs, and how often a rule holds when its condition does.
  • Distance measure: the definition of "similar" that clustering depends on.
  • Curse of dimensionality: with many features, distances lose meaning and data becomes sparse.
  • Spurious pattern: a pattern that arises by chance, especially after many comparisons.

What to study, in order

  1. Descriptive statistics and a dataframe library, to inspect and clean a dataset.
  2. Implement the Apriori idea on a small basket dataset and read the rules it finds.
  3. Implement k-means from scratch, then see how the choice of k and of distance changes the groups.
  4. Build a decision tree classifier and evaluate it on held-out data.
  5. Try outlier detection on data with known anomalies and count what it catches and misses.
  6. Test a discovered pattern on fresh data before trusting it, and write what the data cannot say.

Related tags

#data-science #machine-learning #databases #probability

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply