Observability is how well you can understand what a running system is doing from the outside, using the data it emits. Monitoring tells you that something is wrong; observability lets you ask why, even about a failure nobody predicted. It rests on metrics, logs and traces, plus the habit of using them.
Key concepts
- Metrics: numbers over time, cheap to store and good for dashboards and alerts.
- Logs: timestamped records of events; structured logs with fields are far easier to search.
- Traces: the path of one request across services, with the time spent in each step.
- Instrumentation: adding code that emits telemetry, ideally through a common standard.
- Alerting: notify a person only for symptoms users feel and that need action now.
Vocabulary
- Telemetry: the metrics, logs and traces a system sends out.
- Cardinality: the number of distinct label values in a metric; too many makes it expensive.
- Latency percentile: the time under which a given share of requests finishes, such as p99.
- Golden signals: latency, traffic, errors and saturation, the four things to watch first.
What to study, in order
- Write a small web service and log each request as structured lines.
- Expose a request counter and a latency histogram, and chart them.
- Learn what an average hides by comparing it with the 50th and 99th percentiles.
- Add a trace that follows one request through two services.
- Write an alert on a user-visible symptom and test that it fires.
- Inject a slow dependency and find it using only the telemetry.