Skip to the content ForoHackers
Log inJoin

Observability: start here

Observability
#3203 · · Quote · Report

Observability is how well you can understand what a running system is doing from the outside, using the data it emits. Monitoring tells you that something is wrong; observability lets you ask why, even about a failure nobody predicted. It rests on metrics, logs and traces, plus the habit of using them.

Key concepts

  • Metrics: numbers over time, cheap to store and good for dashboards and alerts.
  • Logs: timestamped records of events; structured logs with fields are far easier to search.
  • Traces: the path of one request across services, with the time spent in each step.
  • Instrumentation: adding code that emits telemetry, ideally through a common standard.
  • Alerting: notify a person only for symptoms users feel and that need action now.

Vocabulary

  • Telemetry: the metrics, logs and traces a system sends out.
  • Cardinality: the number of distinct label values in a metric; too many makes it expensive.
  • Latency percentile: the time under which a given share of requests finishes, such as p99.
  • Golden signals: latency, traffic, errors and saturation, the four things to watch first.

What to study, in order

  1. Write a small web service and log each request as structured lines.
  2. Expose a request counter and a latency histogram, and chart them.
  3. Learn what an average hides by comparing it with the 50th and 99th percentiles.
  4. Add a trace that follows one request through two services.
  5. Write an alert on a user-visible symptom and test that it fires.
  6. Inject a slow dependency and find it using only the telemetry.

Related tags

#sre #performance #distributed-systems #debugging

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply