Introduction to Observability

---

1 - What is Observability?

Observability = The ability to understand the internal state of a system from its external outputs.

---

2 - The 3 pillars

2.1 Metrics

Numeric data aggregated over time.

Type	Description	Example
Counter	Increasing value	Number of requests
Gauge	Variable value	CPU usage
Histogram	Distribution	Latency
Summary	Percentiles	P99 latency

# Metric examples
http_requests_total{method="GET", status="200"}
node_cpu_seconds_total
http_request_duration_seconds_bucket

2.2 Logs

Textual records of events.

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "ERROR",
  "service": "api-gateway",
  "message": "Connection timeout",
  "trace_id": "abc123",
  "user_id": "user-456",
  "latency_ms": 5000
}

Type	Advantage	Disadvantage
Structured (JSON)	Queryable	More verbose
Unstructured	Simple	Hard to parse

2.3 Traces

Tracking a request across services.

---

3 - Monitoring vs Observability

Aspect	Monitoring	Observability
Focus	Known problems	Unknown problems
Approach	Predefined dashboards	Ad-hoc exploration
Questions	"Is the system up?"	"Why is it slow?"
Data	Predefined metrics	Rich contextual data

---

4 - SRE and SLOs

4.1 Concepts

Term	Definition
SLI	Service Level Indicator - Measured metric
SLO	Service Level Objective - Internal target
SLA	Service Level Agreement - Customer commitment
Error Budget	Allowed margin of error

4.2 Example

service: api-gateway
slos:
  availability:
    description: "API available"
    sli: sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
    target: 99.9%
    
  latency:
    description: "P99 latency < 200ms"
    sli: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
    target: 0.2  # 200ms
    
error_budget:
  monthly_budget: 43.2 minutes  # 0.1% of 30 days
  current_remaining: 30 minutes

4.3 Error Budget

---

5 - Observability Architecture

5.1 Modern stack

5.2 Stack options

Stack	Components	Usage
LGTM	Loki, Grafana, Tempo, Mimir	Cloud-native OSS
ELK	Elasticsearch, Logstash, Kibana	Advanced logs
Datadog	SaaS all-in-one	Enterprise
New Relic	SaaS all-in-one	APM focus

---

6 - Instrumentation

6.1 Types of instrumentation

Type	Description	Effort
Auto	Automatic agent/sidecar	Minimal
Library	SDK in the code	Medium
Manual	Custom code	High

6.2 Node.js example

const { MeterProvider } = require('@opentelemetry/sdk-metrics');
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

// Setup Prometheus exporter
const exporter = new PrometheusExporter({ port: 9464 });
const meterProvider = new MeterProvider();
meterProvider.addMetricReader(exporter);

const meter = meterProvider.getMeter('my-service');

// Counter
const requestCounter = meter.createCounter('http_requests_total', {
  description: 'Total HTTP requests',
});

// Histogram
const latencyHistogram = meter.createHistogram('http_request_duration_seconds', {
  description: 'HTTP request latency',
});

// Usage in the code
app.use((req, res, next) => {
  const start = Date.now();
  
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    requestCounter.add(1, { method: req.method, status: res.statusCode });
    latencyHistogram.record(duration, { method: req.method });
  });
  
  next();
});

---

7 - Golden Signals

7.1 Google's 4 signals

Signal	Description	Metric
Latency	Response time	http_request_duration_seconds
Traffic	System load	http_requests_total
Errors	Error rate	http_requests_total{status=~"5.."}
Saturation	Capacity used	node_cpu_seconds_total

7.2 RED Method (Services)

Metric	Focus
Rate	Requests per second
Errors	Number of errors
Duration	Response time

7.3 USE Method (Infrastructure)

Metric	Focus
Utilization	% of resource used
Saturation	Queue
Errors	System errors

---

Summary

In this chapter, we discovered:

• The 3 pillars of observability
• The difference between Monitoring vs Observability
• The SRE concepts (SLI, SLO, SLA)
• The architecture of a modern stack
• Application instrumentation
• The Golden Signals

---

Next step

In the next chapter, we will look at Prometheus in detail.

→ Next chapter: Prometheus

---

← Back to table of contents