Saltar al contenido ForoHackers

Data mining: start here

Minería de datos
#2917 · · Citar · Denunciar

Data mining is the search for useful patterns in large datasets: groups of similar items, things that occur together, unusual cases, trends. It sits between statistics, databases and machine learning, and its output is a hypothesis to examine, not proof. Without care, a big enough dataset will always show patterns that are only chance.

Key concepts

  • Frequent patterns: sets of items that often appear together, such as products bought in one basket, and the rules that follow.
  • Clustering: grouping similar records without labels, such as k-means or hierarchical methods.
  • Classification and regression: predicting a category or a number from labelled examples.
  • Anomaly detection: finding records that do not fit, such as faults or fraud.
  • Process: select data, clean it, transform it, mine it and then interpret and validate the result.

Vocabulary

  • Support and confidence: how often an itemset occurs, and how often a rule holds when its condition does.
  • Distance measure: the definition of "similar" that clustering depends on.
  • Curse of dimensionality: with many features, distances lose meaning and data becomes sparse.
  • Spurious pattern: a pattern that arises by chance, especially after many comparisons.

What to study, in order

  1. Descriptive statistics and a dataframe library, to inspect and clean a dataset.
  2. Implement the Apriori idea on a small basket dataset and read the rules it finds.
  3. Implement k-means from scratch, then see how the choice of k and of distance changes the groups.
  4. Build a decision tree classifier and evaluate it on held-out data.
  5. Try outlier detection on data with known anomalies and count what it catches and misses.
  6. Test a discovered pattern on fresh data before trusting it, and write what the data cannot say.

Related tags

#data-science #machine-learning #databases #probability

No has iniciado sesión, así que esto se publica como anónimo: no se guarda con ello ninguna cuenta, nombre ni dirección, y no puede llevar enlaces ni imágenes. Inicia sesión o únete para publicar con tu nombre. Los registros de nuestro servidor web guardan la dirección y la hora de cada petición hasta 14 días; para que la tuya no quede en ellos, usa Tor. Para escribir al equipo de forma confidencial, usa el buzón. Firmar como fuente habitual (una clave guardada solo en este navegador)

Enviar tarda unos segundos aquí: tu navegador resuelve antes una prueba de trabajo contra el spam.

Responder