Skip to main content

Real-world use cases + complete FAQ

Summary: observability is not a theory — it is what allows Google to operate millions of machines with small teams, and what distinguishes a team that learns from its incidents from a team that repeats them. This final lesson presents five documented cases, the blameless post-mortem culture, and answers the 14 frequent questions.

1. Use case 1 — Google and the birth of SRE​

The 50% rule is a remarkably well-designed incentive mechanism. It does not merely protect SREs from burnout: it creates structural pressure on the development team to improve reliability. If your service generates too much manual work, you get that work back. It is organizational engineering, not just technical engineering.

Good to know: Google's "Site Reliability Engineering" book is available free online. It is probably the best free technical read in the field.

2. Use case 2 — The anatomy of a well-handled incident​

Compare with the scenario of lesson 1: there, two hours of incident including one hour twenty of blind diagnosis, discovered by a customer. Here, ten minutes in total, detected automatically, diagnosed in four minutes thanks to the metric → trace → log chain.

The difference is not the team's talent. It is the instrumentation.

3. Use case 3 — The blameless post-mortem culture​

Reframing the questions is the heart of the practice. "Who made the mistake?" produces defensiveness and no learning. "How did our system allow this mistake?" produces a guardrail that will protect all future people. Human error is never the root cause — it is the symptom of a system that made it possible.

An important clarification: "blameless" does not mean "without responsibility". The corrective actions have named owners and deadlines. What is ruled out is the search for a culprit of the incident, not the responsibility for improvement.

4. Use case 4 — An SMB starting from zero​

This is the most useful case to remember for the majority of readers. No sophisticated tool, no dedicated SRE team, under 100 euros per month — and a complete transformation of the ability to operate the service.

Note the order of the steps: external monitoring of the home page in week 1. It is the action with the best benefit-to-effort ratio in all of observability, and it takes fifteen minutes to set up.

5. Use case 5 — When observability costs too much​

A principle to keep in mind: the goal is not to collect everything, it is to collect what answers useful questions. A DEBUG-level log that nobody will ever read costs money every month and degrades the readability of what matters.

6. The 14 most frequent questions​

1. Where do I start if I have nothing at all?

In this precise order, from most to least profitable:

  1. External monitoring of your home page or a health endpoint, every 60 seconds, with an alert. Fifteen minutes of setup, a few euros per month. You will no longer discover outages from your customers.
  2. The error rate of your main entry points, with an alert.
  3. The p99 latency of those same entry points.
  4. Centralized structured logs, with a short retention.
  5. Distributed tracing, only if you have more than three services.

Do not start by installing a complete stack. Start with point 1, today.

2. How much does observability cost?

Monthly orders of magnitude:

  • Personal project: free (Grafana Cloud free tiers, self-hosted Prometheus).
  • Small team, a few machines: between 0 and 100 euros.
  • Medium team, 20 to 50 machines: between 300 and 2,000 euros depending on the solution.
  • Large organization: from a few thousand to several tens of thousands of euros — it is a budget item in its own right.

A useful, often-cited benchmark: beyond 10 to 15% of your infrastructure cost, your observability deserves an audit. Most often, the cause is the log volume.

3. How long should the data be kept?

Recommendations by data type:

  • Metrics: 15 months for aggregated data (useful for year-over-year comparison), 15 days at full resolution. They are cheap.
  • Logs: 7 to 30 days is enough in the vast majority of cases. It is the most expensive item, hence the most important to limit.
  • Traces: 3 to 7 days. They are almost always consulted immediately after an incident.
  • Exception: audit and compliance logs may require several years — store them separately, in cheap archive storage.
4. Should you instrument manually or automatically?

Start with automatic instrumentation. OpenTelemetry provides agents that automatically instrument the common libraries (HTTP servers, database clients, HTTP clients) without modifying your code. You immediately get the basic traces and metrics.

Add manual instrumentation for what has business meaning: "duration of the discount calculation", "number of items in the cart". That is where the real value lies, because no automatic agent knows your business.

5. How do you manage on-call without exhausting the team?

Five principles that make a real difference:

  1. Few alerts, but all relevant. An on-call should be woken up rarely. If it is every night, the problem is the alert design.
  2. A written procedure for each alert. See lesson 6.
  3. A fair rotation, known in advance, with compensation.
  4. The right not to fully resolve at 3 am: restoring the service is enough, understanding can wait until the next day.
  5. Every nighttime wake-up generates a corrective action so it does not happen again. Otherwise you implicitly accept that it is normal.
6. SRE, DevOps, platform engineer: what is the difference?
  • DevOps — a culture and a set of practices: shared responsibility, automation, fast feedback loops. It was not originally a job, even though the title exists everywhere.
  • SRE — the concrete implementation of DevOps as defined by Google, with its own tools: SLOs, error budget, the 50% rule, toil reduction.
  • Platform engineer — builds an internal platform that product teams consume in self-service. Focuses on developer experience.

A useful formula: "DevOps is the goal, SRE is a method to reach it, the platform is a means to industrialize it".

7. How do you measure the real experience of users?

The server can respond in 40 ms and the user wait 4 seconds — because of the network, the browser, the JavaScript. Two complementary approaches:

RUM (Real User Monitoring) — you measure from the browser of real users. Google's Core Web Vitals are the standard: LCP (time to display the main content), INP (responsiveness to interactions), CLS (visual stability of the layout).

Synthetic monitoring — a robot simulates a user journey every X minutes, from several regions of the world. Useful for detecting outages even without real traffic, at night for example.

Both are useful: RUM tells you what your users experience, synthetic detects outages proactively.

8. What do you do when all indicators are green but it doesn't work?

This is the classic sign that you are measuring causes instead of symptoms. Three checks:

  1. Are you measuring the error rate from the user's point of view or only the state of the machines?
  2. Do you have business metrics? If the number of orders collapses while everything is green, there is a problem — often on the client side (broken JavaScript, invisible button).
  3. Do your probes check something real? A probe that systematically returns "OK" without checking the database proves nothing.

The textbook case: your database is unreachable, so your web servers barely work anymore, so their load is low and all your machine lights are green.

9. Do you need a different tool per pillar?

No, and it is rather discouraged. The critical point is correlation: being able to go from a metric to a trace and then to the corresponding logs in one click.

Three viable approaches:

  • A coherent stack: Prometheus + Loki + Tempo + Grafana. Native correlation, all open source.
  • A unified commercial solution: Datadog, New Relic, Grafana Cloud.
  • Separate tools connected via OpenTelemetry — possible, but correlation requires work.

To avoid: three independent tools without a common trace identifier. Your engineers will spend their time comparing timestamps by hand.

10. How do I convince my management to invest in observability?

Speak in incident duration and in euros:

  • "Our last incident lasted 2 hours, including 1h20 just searching for the cause."
  • "With observability, that diagnosis would take 10 minutes."
  • Put a number on the cost of one hour of downtime for your business.
  • Multiply by the number of incidents last year.
  • Compare with the annual cost of the tooling — the gap is generally considerable.

A complementary, often decisive argument: "we discover our outages from our customers". No management likes hearing that twice.

11. Is observability useful in development, or only in production?

Mainly in production, but it also delivers in development:

  • Local tracing immediately reveals cascading-request problems (the "N+1" problem) even before going to production.
  • Profiling identifies costly functions during development.
  • Instrumenting early avoids having to do it in a rush during an incident.

The real benefit of starting early: instrumentation becomes a code-writing habit, instead of a project bolted on afterwards.

12. How do you observe a Kubernetes system?

Kubernetes adds a layer to monitor, on top of your applications:

  • Cluster metrics: nodes, pending pods, unschedulable pods, resources requested versus actually used.
  • Pod restarts — a pod restarting in a loop (CrashLoopBackOff) is the number one symptom.
  • Failing liveness and readiness probes.
  • Limits reached: a container killed for exceeding memory (OOMKilled) is very frequent and often silent.
  • Kubernetes events — a very rich and under-exploited source of information.

The standard tooling: kube-prometheus-stack, a distribution that installs Prometheus, Grafana, Alertmanager and preconfigured Kubernetes dashboards. See our Kubernetes course.

13. What are the salaries in the field?

Orders of magnitude in France, highly variable by region and sector:

  • Junior DevOps with observability basics: between 35,000 and 45,000 euros gross per year.
  • Experienced SRE or DevOps (3 to 5 years): between 50,000 and 70,000 euros.
  • Senior SRE: between 70,000 and 90,000 euros.
  • Senior platform engineer or observability specialist: beyond that, especially in the Paris region, in consulting or in product companies.

The most valued skills: Prometheus and PromQL, OpenTelemetry, SLO definition, Kubernetes, and the ability to concretely reduce incident durations.

14. Where do I concretely start tomorrow morning?

A five-day plan, applicable to a real project:

  1. Day 1 — Set up external monitoring of your home page, every 60 seconds, with an SMS alert. Fifteen minutes. Immediate benefit.
  2. Day 2 — Instrument the error rate of your three main entry points. Create a single alert on it.
  3. Day 3 — Add latency in percentiles (p50, p99) on those same points. Look at the p99 — it will probably surprise you.
  4. Day 4 — Switch your logs to a structured format with a request identifier. Centralize them.
  5. Day 5 — Write a procedure for your day 2 alert: what to do, what to check, who to contact.

The following week: define a single SLO on your most critical journey, and validate it with the business.

The principle: each day brings a usable benefit. Do not plan a six-month project.

7. What you have learned in this course​

8. Your next step​

Track 1 · Round out your technical culture​

Track 2 · Move on to practice​

Track 3 · Read the references of the field​

  • "Site Reliability Engineering" (Google, 2016) — available free online, the absolute reference.
  • "The SRE Workbook" (Google, 2018) — the sequel, focused on putting SLOs into practice.
  • "Observability Engineering" (Charity Majors et al., 2022) — the modern vision of observability.

9. One last piece of advice​

Do the simplest thing, today.

Do not wait to have the budget for Datadog, nor the time to install a complete Prometheus stack. Set up, this week, external monitoring of your home page with an SMS alert. Fifteen minutes of work.

From that moment on, you will no longer discover your outages from your customers. And that is already half the benefit of observability, for almost nothing.

The rest — the metrics, the traces, the SLOs — will come afterwards, one brick at a time.

Remember in 30 seconds​

  • Google invented SRE in 2003 by entrusting operations to developers, so that they would automate it.
  • The 50% rule limits repetitive work and creates healthy pressure on reliability.
  • The book "Site Reliability Engineering" is free online — the best read in the field.
  • The same incident takes 2 hours without observability and 10 minutes with it. The difference is the instrumentation, not the talent.
  • Blameless post-mortem: replace "who made the mistake?" with "how did our system allow it?".
  • An SMB can transform everything in 8 person-days and under 100 euros per month.
  • Observability gets expensive if it is not managed: DEBUG level in production, traces at 100%, excessive retention.
  • The most profitable action: external monitoring of your home page. Fifteen minutes.
  • Benchmark: beyond 10 to 15% of your infrastructure cost, audit your observability.
  • Start small, today. Do not plan a six-month project.

Thank you for taking this observability discovery course! 🎉

Want to keep going? Back to the table of contents or move on to the Premium Monitoring and Logs course.

Last step: take the end-of-course quiz — five corrected and explained questions, three minutes, to check that the essentials have been acquired.