Grafana Loki

1 - Overview

Loki = Prometheus for logs. A horizontally scalable log system, inspired by Prometheus.

2 - Differences with ELK

Aspect	Loki	Elasticsearch
Indexing	Labels only	Full-text
Cost	Very low	High
Complexity	Simple	Complex
Queries	LogQL	Query DSL
Scalability	Native	Configuration required

Loki philosophy: Don't index the log content, only the labels.

3 - Installation

3.1 Docker Compose

# docker-compose.yml
version: '3.8'
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:2.9.0
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
    command: -config.file=/etc/promtail/config.yml

volumes:
  loki_data:

3.2 Loki configuration

# loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_streams_per_user: 10000
  max_entries_limit_per_query: 5000

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  retention_enabled: true
  retention_delete_delay: 2h

ruler:
  alertmanager_url: http://alertmanager:9093

3.3 Kubernetes (Helm)

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
    --namespace monitoring \
    --set promtail.enabled=true \
    --set grafana.enabled=true

4 - Promtail

4.1 Configuration

# promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # File logs
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
          
  # Docker logs
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'

  # Kubernetes logs
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

4.2 Pipeline stages

scrape_configs:
  - job_name: app-logs
    static_configs:
      - targets: [localhost]
        labels:
          job: app
          __path__: /var/log/app/*.log
    pipeline_stages:
      # Parse JSON
      - json:
          expressions:
            level: level
            message: message
            timestamp: ts
            
      # Extract labels
      - labels:
          level:
          
      # Parse the timestamp
      - timestamp:
          source: timestamp
          format: RFC3339Nano
          
      # Filter
      - match:
          selector: '{level="debug"}'
          action: drop
          
      # Regex
      - regex:
          expression: 'user=(?P<user>\w+)'
          
      # Final output
      - output:
          source: message

5 - LogQL

5.1 Selectors

// Simple selector
{job="app"}

// With multiple labels
{job="app", level="error"}

// Regex
{job=~"app.*", level!="debug"}

// Combinations
{namespace="production", app=~"api|web"}

5.2 Line filters

// Contains
{job="app"} |= "error"

// Does not contain
{job="app"} != "debug"

// Regex
{job="app"} |~ "user=\\d+"

// Chaining
{job="app"} |= "error" != "expected" |~ "connection"

5.3 Parser

// JSON
{job="app"} | json

// Logfmt
{job="app"} | logfmt

// Regex
{job="app"} | regexp `user=(?P<user>\w+)`

// Pattern
{job="app"} | pattern `<ip> - - <_> "<method> <path> <_>" <status>`

// Access parsed fields
{job="app"} | json | level="error"

5.4 Aggregation functions

// Count logs
count_over_time({job="app"}[5m])

// Rate (logs per second)
rate({job="app"}[1m])

// Bytes rate
bytes_rate({job="app"}[1m])

// Sum by label
sum(rate({job="app"}[5m])) by (level)

// Top K
topk(5, sum(rate({job="app"}[5m])) by (path))

// Quantile (estimation)
quantile_over_time(0.99, {job="app"} | json | unwrap latency [5m])

5.5 Practical examples

// Error rate
sum(rate({job="app", level="error"}[5m])) 
/ 
sum(rate({job="app"}[5m])) * 100

// P99 latency
quantile_over_time(0.99,
  {job="app"} | json | unwrap response_time [5m]
) by (endpoint)

// Top endpoints with errors
topk(10,
  sum(rate({job="app", level="error"} | json [1h])) by (endpoint)
)

// Logs with latency > 1s
{job="app"} 
| json 
| latency > 1000 
| line_format "{{.endpoint}} - {{.latency}}ms"

6 - Alerting

6.1 Ruler Configuration

# rules.yml
groups:
  - name: app-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate({job="app", level="error"}[5m])) > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} errors/s"
          
      - alert: NoLogs
        expr: |
          absent(rate({job="app"}[5m]))
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "No logs from app"

7 - Grafana Integration

7.1 Data Source

# provisioning/datasources/loki.yml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      maxLines: 1000
      derivedFields:
        - name: TraceID
          matcherRegex: "trace_id=(\\w+)"
          url: "$${__value.raw}"
          datasourceUid: tempo

7.2 Explore & Dashboard

// Panel with logs
{
  "type": "logs",
  "targets": [
    {
      "expr": "{namespace=\"production\", app=\"api\"} |= \"error\"",
      "refId": "A"
    }
  ],
  "options": {
    "showTime": true,
    "showLabels": true,
    "wrapLogMessage": true
  }
}

8 - Performance Tips

Tip	Description
Label cardinality	Limit the number of unique values
Avoid dynamic labels	No timestamp/UUID as a label
Use filters	Filter before parsing
Chunk size	Adjust based on volume
Retention	Configure based on needs

# Good use of labels
labels:
  environment: production    # Few values
  service: api-gateway       # Few values
  # ❌ Avoid: request_id, user_id, timestamp

# Bad
labels:
  request_id: abc123  # Too many unique values

Summary

In this chapter, we learned:

The architecture of Loki
Configuring Promtail
The LogQL language
Alerting with Ruler
Grafana integration
Performance best practices

Next step

In the next chapter, we will look at Alerting.

→ Next chapter: Alerting

← Back to table of contents

1 - Overview​

2 - Differences with ELK​

3 - Installation​

3.1 Docker Compose​

3.2 Loki configuration​

3.3 Kubernetes (Helm)​

4 - Promtail​

4.1 Configuration​

4.2 Pipeline stages​

5 - LogQL​

5.1 Selectors​

5.2 Line filters​

5.3 Parser​

5.4 Aggregation functions​

5.5 Practical examples​

6 - Alerting​

6.1 Ruler Configuration​

7 - Grafana Integration​

7.1 Data Source​

7.2 Explore & Dashboard​

8 - Performance Tips​

Summary​

Next step​