Skip to main content

Distributed Tracing

1 - Concepts​

1.1 Why tracing?​

Problem: A slow request... but where exactly?

1.2 Anatomy of a Trace​

1.3 Terminology​

TermDescription
TraceThe complete journey of a request
SpanA unit of work within a trace
ContextPropagated metadata
BaggageCustom propagated data
Sampling% of traces captured

2 - OpenTelemetry​

2.1 Architecture​

2.2 Installation (Node.js)​

// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'my-service',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
// index.js
require('./tracing'); // Before any import

const express = require('express');
const app = express();
// ...

2.3 Manual Instrumentation​

const { trace } = require('@opentelemetry/api');

const tracer = trace.getTracer('my-service');

async function processOrder(orderId) {
  // Create a span
  return tracer.startActiveSpan('process-order', async (span) => {
    try {
      span.setAttribute('order.id', orderId);
      
      // Child span
      const result = await tracer.startActiveSpan('validate-order', async (childSpan) => {
        // Validation logic
        childSpan.setAttribute('validation.passed', true);
        childSpan.end();
        return validated;
      });
      
      // Event
      span.addEvent('order-validated', { orderId });
      
      return result;
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

3 - Jaeger​

3.1 Installation​

# docker-compose.yml
version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.52
    ports:
      - "16686:16686"  # UI
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
      - "14268:14268"  # Jaeger HTTP
    environment:
      - COLLECTOR_OTLP_ENABLED=true

3.2 Kubernetes​

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install jaeger jaegertracing/jaeger \
    --namespace monitoring \
    --set provisionDataStore.cassandra=false \
    --set storage.type=memory

3.3 Collector configuration​

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

4 - Grafana Tempo​

4.1 Architecture​

4.2 Installation​

# docker-compose.yml
tempo:
  image: grafana/tempo:2.3.0
  ports:
    - "3200:3200"   # Tempo API
    - "4317:4317"   # OTLP gRPC
    - "4318:4318"   # OTLP HTTP
  volumes:
    - ./tempo-config.yaml:/etc/tempo/tempo.yaml
  command: ["-config.file=/etc/tempo/tempo.yaml"]
# tempo-config.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks
    wal:
      path: /tmp/tempo/wal

compactor:
  compaction:
    block_retention: 48h

metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

4.3 Grafana Integration​

# datasources.yaml
datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanEndTimeShift: '1h'
        tags: [{ key: 'service.name', value: 'service' }]
      tracesToMetrics:
        datasourceUid: prometheus
        spanEndTimeShift: '1h'
        tags: [{ key: 'service.name', value: 'service' }]
      serviceMap:
        datasourceUid: prometheus

5 - Context Propagation​

5.1 W3C Trace Context​

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=t61rcWkgMzE

Format: version-trace_id-parent_id-flags

5.2 Configuration​

const { W3CTraceContextPropagator } = require('@opentelemetry/core');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');

// Default propagator
const propagator = new W3CTraceContextPropagator();

// Automatic HTTP headers
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation({
      // Headers propagated automatically
    }),
  ],
});

5.3 Cross-service Example​

// Service A
app.get('/api/orders', async (req, res) => {
  const span = trace.getActiveSpan();
  span.setAttribute('request.path', '/api/orders');
  
  // Call to Service B - context propagated automatically
  const response = await axios.get('http://service-b/validate');
  
  res.json(response.data);
});

// Service B - receives the context
app.get('/validate', async (req, res) => {
  // This span is automatically a child of Service A's span
  const span = trace.getActiveSpan();
  span.setAttribute('validation.type', 'order');
  
  res.json({ valid: true });
});

6 - Sampling​

6.1 Strategies​

StrategyDescriptionUsage
Always On100% of tracesDev/Debug
Always Off0% of tracesN/A
ProbabilisticX% of tracesProduction
Rate LimitingN traces/secHigh traffic
Parent-basedFollows the parent decisionDistributed

6.2 Configuration​

const { TraceIdRatioBasedSampler, ParentBasedSampler } = require('@opentelemetry/sdk-trace-base');

// 10% of traces
const sampler = new ParentBasedSampler({
  root: new TraceIdRatioBasedSampler(0.1),
});

const sdk = new NodeSDK({
  sampler: sampler,
  // ...
});

6.3 Tail-based Sampling (Collector)​

# otel-collector-config.yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1000
      - name: probabilistic-sample
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

7 - Service Map​

Grafana Tempo automatically generates a Service Map from traces.

Summary​

In this chapter, we learned:

  • The concepts of distributed tracing
  • OpenTelemetry SDK and instrumentation
  • Jaeger for trace storage
  • Cloud-native Grafana Tempo
  • Context Propagation
  • Sampling strategies
  • Generating Service Maps

Next step​

In the next chapter, we will look at Application Performance Monitoring (APM).

→ Next chapter: APM

← Back to table of contents