High availability

---

Chapter objectives

• Understand high availability in Swarm
• Configure fault tolerance of managers
• Ensure service resilience
• Plan for disaster recovery

---

1 - High availability concepts

Definitions

Term	Description
HA	High Availability
Failover	Automatic switchover to a backup
Quorum	Minimum number of nodes required to operate
Split-brain	Situation where the cluster splits

HA architecture

┌─────────────────────────────────────────────────────────────┐
│                  Cluster Haute Disponibilité                 │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   Zone A              Zone B              Zone C            │
│   ┌──────────┐       ┌──────────┐       ┌──────────┐       │
│   │ Manager  │       │ Manager  │       │ Manager  │       │
│   │ (Leader) │◄─────▶│(Reachable)◄─────▶│(Reachable)│       │
│   └────┬─────┘       └────┬─────┘       └────┬─────┘       │
│        │                  │                  │              │
│   ┌────▼─────┐       ┌────▼─────┐       ┌────▼─────┐       │
│   │ Worker 1 │       │ Worker 3 │       │ Worker 5 │       │
│   │ Worker 2 │       │ Worker 4 │       │ Worker 6 │       │
│   └──────────┘       └──────────┘       └──────────┘       │
│                                                             │
│   Si Zone A tombe → Zone B ou C prend le lead              │
│                                                             │
└─────────────────────────────────────────────────────────────┘

---

2 - High availability of managers

Recommended number of managers

Managers	Quorum	Fault tolerance	Recommendation
1	1	0	Dev/Test only
2	2	0	Not recommended
3	2	1	Minimal production
5	3	2	Standard production
7	4	3	Large production

Important

Always use an odd number of managers to avoid split-brain.

Configure 3 managers

# Sur le premier manager
docker swarm init --advertise-addr 192.168.1.10

# Obtenir le token manager
docker swarm join-token manager

# Sur les autres managers
docker swarm join --token SWMTKN-xxx 192.168.1.10:2377

# Vérifier
docker node ls

Geographic distribution

# Ajouter des labels de zone
docker node update --label-add zone=eu-west-1a manager1
docker node update --label-add zone=eu-west-1b manager2
docker node update --label-add zone=eu-west-1c manager3

# Répartir les services par zone
docker service create --name critical-app \
  --replicas 6 \
  --placement-pref 'spread=node.labels.zone' \
  myapp

---

3 - Service fault tolerance

Replicas and availability

# Plus de réplicas = plus de résilience
docker service create --name web \
  --replicas 5 \
  nginx

# Si un nœud tombe, les tâches sont redéployées

Restart policy

services:
  api:
    deploy:
      restart_policy:
        condition: on-failure    # none, on-failure, any
        delay: 5s               # Délai avant redémarrage
        max_attempts: 3         # Nombre max de tentatives
        window: 120s           # Fenêtre de comptage des échecs

Healthchecks

services:
  api:
    image: myapi
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    deploy:
      replicas: 3

Behavior:

1. Swarm monitors the health of containers
2. If unhealthy, the container is stopped
3. A new task is created
4. The old task is removed

---

4 - Handling node failures

Automatic behavior

Panne d'un worker :

Avant:              Node 1    Node 2    Node 3
                    [web.1]   [web.2]   [web.3]
                    [web.4]   [web.5]

                        ↓ Node 2 tombe ↓

Après (auto):       Node 1    Node 2    Node 3
                    [web.1]     ❌      [web.3]
                    [web.4]             [web.5]
                    [web.2*]            [NEW]
                    
* web.2 redéployé sur Node 1 ou 3

Behavior when a manager fails

3 Managers :
┌──────────┐  ┌──────────┐  ┌──────────┐
│ Manager1 │  │ Manager2 │  │ Manager3 │
│ (Leader) │  │(Reachable)│  │(Reachable)│
└──────────┘  └──────────┘  └──────────┘
    │              │              │
    ▼ Tombe        ▼              ▼
    ❌         ┌──────────┐  ┌──────────┐
               │ Manager2 │  │ Manager3 │
               │ (Leader) │  │(Reachable)│
               └──────────┘  └──────────┘

Quorum maintenu (2/3) → Cluster fonctionnel
Nouveau leader élu automatiquement

Loss of quorum

# Si vous perdez le quorum (ex: 2 managers sur 3 tombent)
# Le cluster devient non fonctionnel

# Récupération forcée (sur le manager restant)
docker swarm init --force-new-cluster --advertise-addr <IP>

# Reconstruire ensuite en ajoutant de nouveaux managers

---

5 - Drain mode and maintenance

Put a node in maintenance

# Drainer le nœud (les tâches sont déplacées)
docker node update --availability drain worker1

# État du nœud
docker node ls
# AVAILABILITY: Drain

# Les tâches sont reschedulées sur les autres nœuds
docker service ps web

# Réactiver après maintenance
docker node update --availability active worker1

Planned maintenance

#!/bin/bash
# maintenance.sh

NODE=$1

echo "Draining node $NODE..."
docker node update --availability drain $NODE

echo "Waiting for tasks to migrate..."
sleep 30

echo "Performing maintenance..."
# Vos commandes de maintenance ici
ssh $NODE "apt update && apt upgrade -y && reboot"

echo "Waiting for node to come back..."
sleep 120

echo "Reactivating node..."
docker node update --availability active $NODE

echo "Done!"

---

6 - Backup and restore

Back up the Swarm state

# Sur un manager (de préférence le leader)
# Arrêter Docker
sudo systemctl stop docker

# Sauvegarder le répertoire swarm
sudo tar -cvzf swarm-backup-$(date +%Y%m%d).tar.gz \
  /var/lib/docker/swarm

# Redémarrer Docker
sudo systemctl start docker

Data to back up

Data	Location	Critical
Swarm state	/var/lib/docker/swarm	Yes
Secrets	Included in swarm	Yes
Configs	Included in swarm	Yes
Volumes	/var/lib/docker/volumes	Yes
Images	External registry	No

Complete backup script

#!/bin/bash
# backup-swarm.sh

BACKUP_DIR="/backups/swarm"
DATE=$(date +%Y%m%d_%H%M%S)

# Créer le répertoire
mkdir -p $BACKUP_DIR

# Sauvegarder l'état Swarm (manager uniquement)
if docker node ls > /dev/null 2>&1; then
  echo "Backing up Swarm state..."
  sudo systemctl stop docker
  sudo tar -czf $BACKUP_DIR/swarm-state-$DATE.tar.gz /var/lib/docker/swarm
  sudo systemctl start docker
fi

# Sauvegarder les volumes
echo "Backing up volumes..."
for vol in $(docker volume ls -q); do
  docker run --rm \
    -v $vol:/source:ro \
    -v $BACKUP_DIR:/backup \
    alpine tar -czf /backup/volume-$vol-$DATE.tar.gz -C /source .
done

# Exporter les secrets (métadonnées seulement)
echo "Exporting secrets list..."
docker secret ls > $BACKUP_DIR/secrets-list-$DATE.txt

# Exporter les configs
echo "Exporting configs..."
docker config ls > $BACKUP_DIR/configs-list-$DATE.txt

# Nettoyer les vieux backups
find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR"

Restore

# Restaurer l'état Swarm
sudo systemctl stop docker
sudo rm -rf /var/lib/docker/swarm
sudo tar -xzf swarm-backup.tar.gz -C /
sudo systemctl start docker

# Récupérer le cluster
docker swarm init --force-new-cluster

---

7 - Availability monitoring

Key metrics

# État des nœuds
docker node ls

# Services unhealthy
docker service ls --filter "replicas=0/N"

# Tâches en échec
docker service ps --filter "desired-state=running" --filter "actual-state=failed" myservice

Alerting with Prometheus

# alert-rules.yml
groups:
  - name: swarm-ha
    rules:
      - alert: SwarmNodeDown
        expr: swarm_node_manager_reachable == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Swarm node {{ $labels.node_id }} is down"

      - alert: SwarmQuorumAtRisk
        expr: count(swarm_node_manager_leader) < 2
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Swarm quorum at risk"

      - alert: ServiceReplicasMismatch
        expr: swarm_service_running_replicas != swarm_service_desired_replicas
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Service {{ $labels.service_name }} has replica mismatch"

---

8 - HA best practices

Architecture

# ✅ 3 ou 5 managers (jamais pair)
# ✅ Managers dans différentes zones/racks
# ✅ Au moins 3 réplicas pour les services critiques
# ✅ Healthchecks sur tous les services
# ✅ Limites de ressources définies

# ❌ Éviter
# - 2 managers (pas de quorum en cas de panne)
# - Tous les managers dans la même zone
# - Services critiques avec 1 seul réplica

HA checklist

## Checklist Haute Disponibilité

### Infrastructure
- [ ] 3+ managers (nombre impair)
- [ ] Managers dans différentes zones
- [ ] Workers répartis géographiquement
- [ ] Réseau redondant

### Services
- [ ] Réplicas >= 3 pour services critiques
- [ ] Healthchecks configurés
- [ ] Restart policy définie
- [ ] Rolling update configuré

### Données
- [ ] Volumes sur stockage persistant
- [ ] Backups automatisés
- [ ] Test de restauration régulier

### Monitoring
- [ ] Alertes sur état des nœuds
- [ ] Alertes sur quorum
- [ ] Alertes sur réplicas manquants

---

Summary

Concept	Description
Quorum	Majority of managers required
Failover	Automatic election of a new leader
Drain	Move tasks before maintenance
Backup	Back up /var/lib/docker/swarm

Key points

• 3 or 5 managers for production
• Distribute managers geographically
• Healthchecks are mandatory for resilience
• Regularly test restoring backups

---

Practical exercises

1. Configure a cluster with 3 managers
2. Simulate a manager failure and observe the failover
3. Put a worker in drain mode and observe the migration
4. Create a backup script and test the restore

---

← Scaling and Load Balancing | Best practices →