Skip to main content

The problem observability solves

Summary: without instrumentation, a team learns that an outage exists from its own customers, then spends hours guessing the cause instead of observing it. This lesson details the five structural problems of an unobserved production, puts a number on the real cost of downtime, and shows why the two metrics that matter are time to detect and time to diagnose.

1. The scenario every team has lived through​

The most revealing detail in this scenario: out of two hours of incident, more than one hour was spent on diagnosis, not on the fix. The fix itself took a few minutes. This is the case in the vast majority of incidents — understanding costs far more than repairing.

2. The five structural problems​

Problem 5 is the one that blocks improvement. Without measurement, every technical discussion becomes a debate of opinions: "I feel it has been slower since the last version" versus "I don't see any difference". An instrumented team settles it in thirty seconds with a graph.

3. The two metrics that really matter​

What to take away from this breakdown: when management asks "how long to fix it?", the real question is "how long until we know there is a problem, and until we understand which one?". That is where observability delivers its return on investment, and it is an effective argument for getting a budget.

4. The real cost of downtime​

Cost 5 is the one that kills teams in the long run. A team spending half its time putting out fires no longer delivers value. And since it no longer delivers, it has no time to improve reliability — which increases the number of fires. It is a vicious circle, and observability is the first lever to break out of it.

5. The signals that announced the outage​

Almost all serious incidents are preceded by weak signals that nobody was watching.

Signal 4 deserves particular attention and will be developed in lesson 4. A stable average latency can perfectly mask a severe degradation for a subset of users. If 95% of requests respond in 50 ms and 5% in 8 seconds, the average remains flattering — but one user in twenty experiences an unacceptable service.

6. What observability concretely changes​

Point 6 connects to the rest of the DevOps approach. As seen in the CI/CD course, a canary release consists of exposing 5% of users to a new version and comparing the metrics. Without observability, that comparison is impossible — you expose users without being able to detect the degradation. Observability is a precondition for modern deployment practices.

7. The classic beginner's mistake​

This reversal of perspective is the central message of the course. Monitor first the symptoms experienced by the user (errors, slowness), and only then the technical causes (CPU, memory, disk). Lesson 4 formalizes this principle with the four golden signals.

Remember in 30 seconds​

  • Without observability, you learn about outages from your customers, which is already a failure.
  • Diagnosis often represents 60 to 80% of the duration of an incident; the fix is the shortest item.
  • Five structural problems: late detection, guesswork diagnosis, no history, no prevention, no proof.
  • Three metrics: MTTD (detect), MTTI (identify), MTTR (repair). Observability acts on the first two.
  • The most destructive cost is the opportunity cost: a team putting out fires makes no progress.
  • Almost all serious incidents are preceded by weak signals that nobody was watching.
  • The average hides everything: 5% of requests at 8 seconds remain invisible in a flattering average.
  • Classic mistake: monitoring CPU load instead of user experience. A lightly loaded server can return 100% errors.
  • Observability is the prerequisite of canary release and continuous deployment.

Next: Monitoring or observability? The distinction finally made clear →