The tools and the art of alerting
Summary: the observability ecosystem is organized around Prometheus for metrics, Grafana for visualization, and OpenTelemetry as the instrumentation standard. This lesson compares the open source and commercial tools, then addresses the most neglected topic in the field: designing alerts that people do not end up ignoring.
1. The tooling landscape
The two names to absolutely remember: Prometheus (metrics, the de facto standard) and OpenTelemetry (instrumentation, an emerging standard that has become essential). These two words appear in nearly every job posting mentioning observability.
2. Prometheus — the metrics standard
Why Prometheus won in the Kubernetes world: its automatic discovery. In a cluster where pods appear and disappear constantly, a system that requires manually declaring each target is unmanageable. Prometheus queries the Kubernetes API and discovers on its own what it needs to monitor. That is what made the difference.
3. The Grafana stack — the unified approach
Loki's design idea deserves to be understood: unlike Elasticsearch which indexes the entire content of every log, Loki indexes only the labels (service, environment, level) and stores the text compressed without an index. Consequence: a significantly lower storage cost, at the price of a slower full-text search. For the majority of uses — "show me the logs of the payment service, error level, between 2 pm and 3 pm" — it is perfectly suited.
4. OpenTelemetry — the standard to know
This is the most durably useful piece of advice in this lesson. Instrumentation is the most expensive part to redo in all of your observability — it is spread throughout your code. By choosing OpenTelemetry, you make that work reusable whatever tool you choose in three years.
5. Open source or commercial solution
A useful warning about commercial solutions: their billing is often based on the volume of ingested data. An application that is a bit too chatty in its logs can triple your bill from one month to the next, without anything warning you. Configure volume limits and budget alerts, exactly as for the cloud.
6. Alert fatigue — the real problem
The sentence to remember: "too many alerts equals no alerts". A team receiving forty notifications a day is blind, exactly like a team receiving none — with exhaustion on top.
7. The rules of a useful alert
Rule 5 is the one that most transforms the on-call experience. Being woken up at 3 am by an alert accompanied by a link to "Procedure: connection pool saturation" detailing five concrete checks is radically different from receiving a terse "error orders service" message. It is often the difference between ten minutes and two hours of resolution.
8. What a good alert looks like
Here are two wordings for the same incident.
What the good alert brings, point by point: it names the business impact (180 failed payments, not an abstract percentage), gives the context (the budget consumed), indicates the urgency (burn rate), provides the tools (dashboard, procedure), and suggests a lead (a recent deployment). The person woken up can act immediately.
The line about the recent deployment is particularly precious: in a majority of incidents, the cause is a recent change. Automatically correlating alerts with deployments saves considerable time.
9. Dashboards — a few principles
Principle 3 is the most often neglected. A graph displaying "latency: 340 ms" tells you nothing without a reference. Is that normal? Is it twice as slow as yesterday? Always display either the SLO threshold, or the same period of the previous week. Without a reference, a graph is decorative.
Remember in 30 seconds
- Prometheus = the metrics standard, a pull-based collection model and automatic discovery (hence its success on Kubernetes).
- Grafana = the undisputed visualization standard. Loki (logs, economical), Tempo (traces), Pyroscope (profiles).
- Loki indexes only the labels, not the content — hence its low cost compared to Elasticsearch.
- OpenTelemetry = the instrumentation standard. Instrument once, change tools freely.
- Open source or commercial: small team without an expert → commercial. Kubernetes with skills → Prometheus. Large scale → self-hosted, more economical.
- Beware of commercial bills: they follow the log volume. Set limits.
- Alert fatigue: "too many alerts equals no alerts".
- Rule number one: every alert must require immediate human action. Otherwise it is a dashboard.
- Every alert must point to a written procedure (runbook). It is the most powerful lever on resolution time.
- Mark the deployments on your graphs — the cause of an incident is most often a recent change.