Skip to main content

Best practices

1 - Naming Conventions​

1.1 Metrics​

# Recommended format
<namespace>_<subsystem>_<name>_<unit>

# Examples
http_requests_total
http_request_duration_seconds
node_memory_MemAvailable_bytes
process_cpu_seconds_total
RuleGoodBad
Snake casehttp_requests_totalhttpRequestsTotal
Unit as suffixrequest_duration_secondsrequest_duration_ms
Base unitbytes not kilobytesmemory_mb
_total for counterserrors_totalerror_count

1.2 Labels​

# Recommended labels
http_requests_total{
  service="api-gateway",
  method="GET",
  status="200",
  endpoint="/api/users"
}

# Labels to avoid
http_requests_total{
  request_id="abc123",        # Infinite cardinality
  timestamp="2024-01-15",     # Never as a label
  user_email="[email protected]"  # PII
}

1.3 Logs​

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "INFO",
  "service": "api-gateway",
  "trace_id": "abc123",
  "span_id": "def456",
  "message": "Request processed",
  "context": {
    "user_id": "user-123",
    "endpoint": "/api/orders",
    "method": "POST",
    "duration_ms": 150
  }
}

2 - Cardinality Management​

2.1 The problem​

2.2 Solutions​

StrategyDescription
Limit labelsMax 5-7 labels per metric
Avoid IDsNo request_id, user_id
BucketizeConvert to ranges
Recording rulesPre-aggregate
# Bad - high cardinality
http_requests_total{user_id="..."}

# Good - grouping
http_requests_total{user_tier="premium"}

2.3 Verification​

# Count series per metric
count by (__name__) ({__name__=~".+"})

# Top metrics by cardinality
topk(10, count by (__name__) ({__name__=~".+"}))

3 - Retention and Storage​

3.1 Strategy by type​

DataHotWarmColdDelete
Metrics7d30d90d1 year
Logs3d14d30d90d
Traces3d7d14d30d

3.2 Prometheus configuration​

# prometheus.yml
global:
  scrape_interval: 15s

storage:
  tsdb:
    retention.time: 15d
    retention.size: 50GB

3.3 Downsampling (Thanos)​

# thanos-compact.yml
downsample:
  resolution: 5m  # After 2 weeks
  resolution: 1h  # After 1 month

4 - Structured Logging​

4.1 Format​

// ❌ Bad
logger.info(`User ${userId} logged in from ${ip}`);

// ✅ Good
logger.info('User logged in', {
  user_id: userId,
  ip_address: ip,
  event: 'user.login'
});

4.2 Log Levels​

LevelUsageExample
ERRORError requiring attentionDB connection failed
WARNNon-blocking anomalyRate limit approaching
INFOBusiness eventOrder created
DEBUGTechnical detailsQuery executed

4.3 Correlation IDs​

const { v4: uuidv4 } = require('uuid');

// Express middleware
app.use((req, res, next) => {
  req.correlationId = req.headers['x-correlation-id'] || uuidv4();
  res.setHeader('x-correlation-id', req.correlationId);
  
  // Add to all logs
  req.log = logger.child({ correlationId: req.correlationId });
  next();
});

5 - Dashboard Design​

5.1 Typical layout​

┌─────────────────────────────────────────────────────────────┐
│                      FILTERS (Variables)                    │
├─────────────────────────────────────────────────────────────┤
│  Requests │  Errors  │  Latency  │  Saturation │  Uptime   │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│                     MAIN GRAPH                             │
│                    (Request Rate + Errors)                  │
│                                                             │
├──────────────────────────┬──────────────────────────────────┤
│                          │                                  │
│    Latency Distribution  │       Top Endpoints              │
│                          │                                  │
├──────────────────────────┴──────────────────────────────────┤
│                                                             │
│                    LOGS / EVENTS                           │
│                                                             │
└─────────────────────────────────────────────────────────────┘

5.2 Principles​

PrincipleDescription
Progressive disclosureGeneral → Detailed
Drill-downClick for details
ContextVariables to filter
Visible alertsRed = problem

6 - Alerting Best Practices​

6.1 Symptoms vs Causes​

# ❌ Alert on cause
- alert: HighCPU
  expr: node_cpu_usage > 80%

# ✅ Alert on symptom
- alert: HighLatency
  expr: http_request_duration_seconds_p99 > 1
  
# ✅ With runbook
annotations:
  runbook: "https://wiki/runbooks/high-latency"
  possible_causes: "CPU, Memory, Database, Dependencies"

6.2 Multi-window Alerts​

# Alert with multiple windows
- alert: HighErrorRate
  expr: |
    (
      # Rate over 5 minutes
      sum(rate(http_errors_total[5m])) / sum(rate(http_requests_total[5m])) > 0.05
    ) and (
      # Rate over 1 hour also high
      sum(rate(http_errors_total[1h])) / sum(rate(http_requests_total[1h])) > 0.02
    )
  for: 5m

6.3 Checklist​

For each alert:
- [ ] Symptom, not cause?
- [ ] Actionable?
- [ ] Runbook linked?
- [ ] Thresholds based on data?
- [ ] Appropriate for-duration?
- [ ] Tested?

7 - Observability Maturity​

7.1 Levels​

LevelCharacteristics
0 - NoneNo monitoring
1 - BasicInfra metrics, basic alerts
2 - MetricsApp metrics, dashboards
3 - LogsCentralized logs, correlation
4 - TracesDistributed tracing
5 - AdvancedSLOs, Error budgets, Auto-remediation

7.2 Checklist by level​

level_1:
  - [ ] CPU/Memory/Disk metrics
  - [ ] Availability alerts
  - [ ] Basic dashboard

level_2:
  - [ ] Application metrics
  - [ ] Golden signals
  - [ ] Service dashboards

level_3:
  - [ ] Structured logs
  - [ ] Centralized search
  - [ ] Correlation IDs

level_4:
  - [ ] OpenTelemetry
  - [ ] Distributed tracing
  - [ ] Service maps

level_5:
  - [ ] SLIs/SLOs defined
  - [ ] Error budgets
  - [ ] Auto-scaling/remediation

8 - Cost Optimization​

8.1 Strategies​

StrategySavings
Trace sampling50-90%
Prod log level30-50%
Short retentionVariable
Recording rules20-40%
Compression30-50%

8.2 Smart sampling​

# otel-collector-config.yaml
processors:
  tail_sampling:
    policies:
      # Keep 100% of errors
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      # Keep 100% of slow traces
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000
      # 10% of the rest
      - name: default
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

Summary​

In this chapter, we covered:

  • Naming conventions
  • Cardinality management
  • Retention strategies
  • Structured logging
  • Dashboard design
  • Alerting best practices
  • Observability maturity
  • Cost optimization

Next step​

In the next chapter, we will put things into practice with Exercises and Projects.

→ Next chapter: Exercises and Projects

← Back to table of contents