Skip to the content ForoHackers
Log inJoin

Measurement and evaluation: start here

Measurement and evaluation
#3393 · · Quote · Report

Performance evaluation is the science of measuring how fast and how well a system works, and of trusting the numbers. Computers are noisy: caches, scheduling, frequency changes and other programs all disturb a result. Good evaluation designs the experiment, controls what it can, and reports the spread, not just one number.

Key concepts

  • Metrics: latency, throughput, utilisation and error rate, each chosen for the question asked.
  • Benchmarks: micro-benchmarks isolate one operation; macro-benchmarks run a realistic workload.
  • Statistics: repeat runs, report medians and percentiles, and give a confidence interval or the range.
  • Tail latency: the slowest few per cent of requests often matter more than the average.
  • Queueing and load: response time rises sharply as a system nears saturation.

Vocabulary

  • Percentile: the value below which a given share of observations fall, such as the 99th.
  • Warm-up: the early phase, with cold caches, that is excluded from the measurement.
  • Baseline: the reference result that a change is compared against.
  • Confounder: an uncontrolled factor that explains a difference you wrongly attribute to your change.

What to study, in order

  1. Descriptive statistics: mean, median, variance and percentiles, and why the mean can mislead.
  2. Write a micro-benchmark and see how an optimiser or a cold cache ruins a naive one.
  3. Repeat a measurement thirty times, plot the distribution and report it honestly.
  4. Compare two versions of a program with a significance test and a stated confidence.
  5. Generate load against a service and plot latency against request rate.

Related tags

#performance #probability #simulation #observability #testing

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply