Performance evaluation is the science of measuring how fast and how well a system works, and of trusting the numbers. Computers are noisy: caches, scheduling, frequency changes and other programs all disturb a result. Good evaluation designs the experiment, controls what it can, and reports the spread, not just one number.
Key concepts
- Metrics: latency, throughput, utilisation and error rate, each chosen for the question asked.
- Benchmarks: micro-benchmarks isolate one operation; macro-benchmarks run a realistic workload.
- Statistics: repeat runs, report medians and percentiles, and give a confidence interval or the range.
- Tail latency: the slowest few per cent of requests often matter more than the average.
- Queueing and load: response time rises sharply as a system nears saturation.
Vocabulary
- Percentile: the value below which a given share of observations fall, such as the 99th.
- Warm-up: the early phase, with cold caches, that is excluded from the measurement.
- Baseline: the reference result that a change is compared against.
- Confounder: an uncontrolled factor that explains a difference you wrongly attribute to your change.
What to study, in order
- Descriptive statistics: mean, median, variance and percentiles, and why the mean can mislead.
- Write a micro-benchmark and see how an optimiser or a cold cache ruins a naive one.
- Repeat a measurement thirty times, plot the distribution and report it honestly.
- Compare two versions of a program with a significance test and a stated confidence.
- Generate load against a service and plot latency against request rate.
Related tags
#performance #probability #simulation #observability #testing