Saltar al contenido ForoHackers

Measurement and evaluation: start here

Medición y evaluación
#887 · Ophuscado · · Denunciar

Performance evaluation is the science of measuring how fast and how well a system works, and of trusting the numbers. Computers are noisy: caches, scheduling, frequency changes and other programs all disturb a result. Good evaluation designs the experiment, controls what it can, and reports the spread, not just one number.

Key concepts

  • Metrics: latency, throughput, utilisation and error rate, each chosen for the question asked.
  • Benchmarks: micro-benchmarks isolate one operation; macro-benchmarks run a realistic workload.
  • Statistics: repeat runs, report medians and percentiles, and give a confidence interval or the range.
  • Tail latency: the slowest few per cent of requests often matter more than the average.
  • Queueing and load: response time rises sharply as a system nears saturation.

Vocabulary

  • Percentile: the value below which a given share of observations fall, such as the 99th.
  • Warm-up: the early phase, with cold caches, that is excluded from the measurement.
  • Baseline: the reference result that a change is compared against.
  • Confounder: an uncontrolled factor that explains a difference you wrongly attribute to your change.

What to study, in order

  1. Descriptive statistics: mean, median, variance and percentiles, and why the mean can mislead.
  2. Write a micro-benchmark and see how an optimiser or a cold cache ruins a naive one.
  3. Repeat a measurement thirty times, plot the distribution and report it honestly.
  4. Compare two versions of a program with a significance test and a stated confidence.
  5. Generate load against a service and plot latency against request rate.

Related tags

#performance #probability #simulation #observability #testing

Inicia sesión o únete para responder.