Data mining is the search for useful patterns in large datasets: groups of similar items, things that occur together, unusual cases, trends. It sits between statistics, databases and machine learning, and its output is a hypothesis to examine, not proof. Without care, a big enough dataset will always show patterns that are only chance.
Key concepts
- Frequent patterns: sets of items that often appear together, such as products bought in one basket, and the rules that follow.
- Clustering: grouping similar records without labels, such as k-means or hierarchical methods.
- Classification and regression: predicting a category or a number from labelled examples.
- Anomaly detection: finding records that do not fit, such as faults or fraud.
- Process: select data, clean it, transform it, mine it and then interpret and validate the result.
Vocabulary
- Support and confidence: how often an itemset occurs, and how often a rule holds when its condition does.
- Distance measure: the definition of "similar" that clustering depends on.
- Curse of dimensionality: with many features, distances lose meaning and data becomes sparse.
- Spurious pattern: a pattern that arises by chance, especially after many comparisons.
What to study, in order
- Descriptive statistics and a dataframe library, to inspect and clean a dataset.
- Implement the Apriori idea on a small basket dataset and read the rules it finds.
- Implement k-means from scratch, then see how the choice of k and of distance changes the groups.
- Build a decision tree classifier and evaluate it on held-out data.
- Try outlier detection on data with known anomalies and count what it catches and misses.
- Test a discovered pattern on fresh data before trusting it, and write what the data cannot say.