Saltar al contenido ForoHackers

Big data: start here

Big data
#415 · Ophuscado · · Denunciar

Big data means data too large, too fast or too varied for one machine and ordinary tools. The answer is to spread storage and computation over many machines, and to move the code to the data rather than the data to the code. The difficulty is less the volume itself than keeping the work correct and cheap as it grows.

Key concepts

  • Distributed storage: files or tables split into blocks, copied across machines to survive failures.
  • Parallel processing: MapReduce and its successors split a job into independent tasks and combine the results.
  • Partitioning and shuffles: choose keys so work spreads evenly; moving data between machines is the costly step.
  • Columnar formats: storing each column together, compressed, so analytic queries read only what they need.
  • Batch and stream: process stored history in bulk, or events as they arrive.

Vocabulary

  • The three Vs: volume, velocity and variety, the usual description of what makes data big.
  • Data lake: cheap storage holding raw data in many formats, structured later.
  • Skew: an uneven spread of data across partitions that leaves one machine doing most of the work.
  • Scale out and scale up: add more machines, or buy a bigger one.

What to study, in order

  1. Check first whether one machine with good tools suffices; many "big" problems fit on a laptop.
  2. Implement word count in a MapReduce style by hand, with map, shuffle and reduce steps.
  3. Run the same job on a distributed engine such as Apache Spark and watch its stages.
  4. Compare a row format such as CSV with a columnar one such as Parquet on size and query time.
  5. Cause skew with a lopsided key and fix it.
  6. Build a streaming version of the job and note what changes about ordering, lateness and failure.

Related tags

#distributed-systems #data-engineering #hpc #storage

Inicia sesión o únete para responder.