Skip to the content ForoHackers
Log inJoin

Data mining: start here

Data mining
#411 · Ophuscado · · Report

Data mining is the search for useful patterns in large datasets: groups of similar items, things that occur together, unusual cases, trends. It sits between statistics, databases and machine learning, and its output is a hypothesis to examine, not proof. Without care, a big enough dataset will always show patterns that are only chance.

Key concepts

  • Frequent patterns: sets of items that often appear together, such as products bought in one basket, and the rules that follow.
  • Clustering: grouping similar records without labels, such as k-means or hierarchical methods.
  • Classification and regression: predicting a category or a number from labelled examples.
  • Anomaly detection: finding records that do not fit, such as faults or fraud.
  • Process: select data, clean it, transform it, mine it and then interpret and validate the result.

Vocabulary

  • Support and confidence: how often an itemset occurs, and how often a rule holds when its condition does.
  • Distance measure: the definition of "similar" that clustering depends on.
  • Curse of dimensionality: with many features, distances lose meaning and data becomes sparse.
  • Spurious pattern: a pattern that arises by chance, especially after many comparisons.

What to study, in order

  1. Descriptive statistics and a dataframe library, to inspect and clean a dataset.
  2. Implement the Apriori idea on a small basket dataset and read the rules it finds.
  3. Implement k-means from scratch, then see how the choice of k and of distance changes the groups.
  4. Build a decision tree classifier and evaluate it on held-out data.
  5. Try outlier detection on data with known anomalies and count what it catches and misses.
  6. Test a discovered pattern on fresh data before trusting it, and write what the data cannot say.

Related tags

#data-science #machine-learning #databases #probability

Log in or join to reply.