Saltar al contenido ForoHackers

Measurement and evaluation: start here

Medición y evaluación
#3393 · · Citar · Denunciar

Performance evaluation is the science of measuring how fast and how well a system works, and of trusting the numbers. Computers are noisy: caches, scheduling, frequency changes and other programs all disturb a result. Good evaluation designs the experiment, controls what it can, and reports the spread, not just one number.

Key concepts

  • Metrics: latency, throughput, utilisation and error rate, each chosen for the question asked.
  • Benchmarks: micro-benchmarks isolate one operation; macro-benchmarks run a realistic workload.
  • Statistics: repeat runs, report medians and percentiles, and give a confidence interval or the range.
  • Tail latency: the slowest few per cent of requests often matter more than the average.
  • Queueing and load: response time rises sharply as a system nears saturation.

Vocabulary

  • Percentile: the value below which a given share of observations fall, such as the 99th.
  • Warm-up: the early phase, with cold caches, that is excluded from the measurement.
  • Baseline: the reference result that a change is compared against.
  • Confounder: an uncontrolled factor that explains a difference you wrongly attribute to your change.

What to study, in order

  1. Descriptive statistics: mean, median, variance and percentiles, and why the mean can mislead.
  2. Write a micro-benchmark and see how an optimiser or a cold cache ruins a naive one.
  3. Repeat a measurement thirty times, plot the distribution and report it honestly.
  4. Compare two versions of a program with a significance test and a stated confidence.
  5. Generate load against a service and plot latency against request rate.

Related tags

#performance #probability #simulation #observability #testing

No has iniciado sesión, así que esto se publica como anónimo: no se guarda con ello ninguna cuenta, nombre ni dirección, y no puede llevar enlaces ni imágenes. Inicia sesión o únete para publicar con tu nombre. Los registros de nuestro servidor web guardan la dirección y la hora de cada petición hasta 14 días; para que la tuya no quede en ellos, usa Tor. Para escribir al equipo de forma confidencial, usa el buzón. Firmar como fuente habitual (una clave guardada solo en este navegador)

Enviar tarda unos segundos aquí: tu navegador resuelve antes una prueba de trabajo contra el spam.

Responder