Skip to the content ForoHackers
Log inJoin

Observability: start here

Observability
#697 · Ophuscado · · Report

Observability is how well you can understand what a running system is doing from the outside, using the data it emits. Monitoring tells you that something is wrong; observability lets you ask why, even about a failure nobody predicted. It rests on metrics, logs and traces, plus the habit of using them.

Key concepts

  • Metrics: numbers over time, cheap to store and good for dashboards and alerts.
  • Logs: timestamped records of events; structured logs with fields are far easier to search.
  • Traces: the path of one request across services, with the time spent in each step.
  • Instrumentation: adding code that emits telemetry, ideally through a common standard.
  • Alerting: notify a person only for symptoms users feel and that need action now.

Vocabulary

  • Telemetry: the metrics, logs and traces a system sends out.
  • Cardinality: the number of distinct label values in a metric; too many makes it expensive.
  • Latency percentile: the time under which a given share of requests finishes, such as p99.
  • Golden signals: latency, traffic, errors and saturation, the four things to watch first.

What to study, in order

  1. Write a small web service and log each request as structured lines.
  2. Expose a request counter and a latency histogram, and chart them.
  3. Learn what an average hides by comparing it with the 50th and 99th percentiles.
  4. Add a trace that follows one request through two services.
  5. Write an alert on a user-visible symptom and test that it fires.
  6. Inject a slow dependency and find it using only the telemetry.

Related tags

#sre #performance #distributed-systems #debugging

Log in or join to reply.