What exactly should you measure?
Summary: the question is not "how to measure?" but "what to measure?". Google formalized the answer with the four golden signals: latency, traffic, errors, saturation. This lesson details them, presents the complementary USE and RED methods, explains why the average is misleading, and why monitoring CPU load is a bad reflex.
1. The four golden signals
They are defined in chapter 6 of Google's book "Site Reliability Engineering" (2016), which remains the reference of the field.
Signal 4 is the only predictive one, and that is why it is precious. Latency, traffic and errors describe what is happening now. Saturation announces what is going to happen: a disk at 92% growing by 1% per day gives you eight days to act.
The trap of signal 1, very frequent: if you measure the latency of all requests combined, an outage that returns instant errors will improve your average latency. Your graph improves while your service is broken. Always measure separately the latency of successes and that of failures.
2. Why the average is misleading
This is the most costly statistical mistake in observability.
Understanding percentiles
| Percentile | Meaning | What it reveals |
|---|---|---|
| p50 (median) | Half of the requests are faster | The typical experience |
| p90 | 90% of requests are faster | The beginning of the slow zone |
| p95 | 95% are faster | The experience of disadvantaged users |
| p99 | 99% are faster | The real problem cases |
| p99.9 | 99.9% are faster | The extreme cases, often revealing bugs |
Amazon's often-cited argument: at scale, the p99 is not a marginal case. On one million requests per day, the p99 represents 10,000 requests — hence potentially thousands of unhappy customers. And paradoxically, it is often your best customers who sit in the p99, because they are the ones with the most data to load.
Practical recommendation: monitor the p50 and the p99 at a minimum. The p50 tells you whether the typical experience is good, the p99 tells you whether you are mistreating a minority.
3. The USE method — for resources
Complementary to the golden signals, the USE method by Brendan Gregg applies to resources rather than services.
The most useful point of the USE method: saturation is almost always more revealing than utilization. A CPU at 100% utilization without a queue works perfectly — it is simply fully employed, which is even desirable. A CPU at 60% with ten processes waiting indicates a real problem.
A very frequent concrete application: the database connection pool. Utilization may seem modest, but if requests are waiting for a free connection, your application slows down without anything appearing saturated. Monitor the pool's queue, not just its occupancy rate.
4. The RED method — for services
RED's practical asset: its standardization. With RED, you build a single dashboard template and duplicate it for each of your forty services. Every engineer immediately knows how to read any dashboard, with no learning curve. That is a considerable operational gain in a microservices architecture.
5. How to combine the three approaches
6. Symptom or cause — the fundamental alerting rule
This is the most important principle of this lesson. Most teams suffering from alert fatigue made exactly this mistake: they alert on causes (CPU, memory) instead of symptoms (errors, slowness). As a result, they receive dozens of alerts that correspond to no user impact, and end up ignoring them all.
The convincing counter-example: your database is unreachable. Your web servers barely work anymore, so their CPU load is low. All your machine alerts are green, and your service is completely broken. Only an alert on the error rate would have warned you.
7. What to measure, by component type
| Component | What to measure first |
|---|---|
| Web API | Error rate (by status code), latency p50/p95/p99, requests per second |
| Database | Query latency, active and waiting connections, slow queries, replication lag |
| Message queue | Queue length, consumer lag (the most important), throughput, failed messages |
| Cache | Hit ratio, latency, evictions, memory used |
| Scheduled job | Last successful run, duration, failure rate |
| Web frontend | Core Web Vitals (LCP, INP, CLS), JavaScript errors, real network latency |
| Kubernetes | Pod restarts, unschedulable pods, limits reached, failing probes |
| Business | Orders per hour, sign-ups, successful payments — the most useful of all |
The last row is the one systematically forgotten, and that is a shame. A business metric is often the best outage detector there is. If the number of orders per hour collapses while all your technical indicators are green, there is a problem — even if no error is being reported. It is the kind of silent outage that nothing else detects: a broken client-side form, a button made invisible after a style update, a payment failing silently.
8. The most frequent measurement mistakes
Remember in 30 seconds
- Four golden signals (Google): latency, traffic, errors, saturation. If you only measure four things, these are the ones.
- Saturation is the only predictive signal — it announces outages before they happen.
- Never measure the average latency alone. Use the p50 and the p99. At scale, the p99 represents thousands of real customers.
- Separate the latency of successes and failures — otherwise an outage will embellish your graphs.
- USE method (resources): Utilization, Saturation, Errors. Saturation is more revealing than utilization.
- RED method (services): Rate, Errors, Duration. Enables a standardized dashboard per service.
- Fundamental alerting rule: alert on symptoms (user impact), display the causes on dashboards.
- Don't forget the business metrics — orders, sign-ups, payments. They are the best detectors of silent outages.
- 20 well-understood metrics are worth more than 2,000 ignored ones.