Troubleshooting and diagnostics

---

Table of contents

1. Diagnostic methodology
2. Boot problems
3. Performance problems
4. Network problems
5. Application problems
6. Practical exercises

---

1 - Diagnostic methodology

Structured approach

Questions to ask

1. What - What is the exact symptom?
2. When - Since when? After which change?
3. Who - All users? Just one?
4. Where - One server? All?
5. How - How to reproduce it?

Essential tools

Domain	Tools
System	dmesg, journalctl, top, htop
Processes	ps, strace, lsof
Network	ss, tcpdump, ping, traceroute
Disk	df, du, iostat, iotop
Memory	free, vmstat, /proc/meminfo

First-level checklist

# État général
uptime
free -h
df -h
dmesg | tail -50
journalctl -p err --since "1 hour ago"

# Processus
top -bn1 | head -20
ps aux --sort=-%cpu | head -10

# Réseau
ss -tunapl
ping -c 3 google.com

# Services critiques
systemctl --failed

🔝 Back to table of contents

---

2 - Boot problems

Common symptoms

Symptom	Possible cause
Kernel panic	Corrupted module, hardware
Stuck at GRUB	Invalid GRUB config
Emergency mode	Incorrect fstab, corrupted fs
Black screen	Graphics driver

Recovery mode

# Au boot GRUB, éditer l'entrée (e)
# Ajouter à la ligne linux:
systemd.unit=rescue.target
# ou
init=/bin/bash

# Puis Ctrl+X pour booter

Repair GRUB

# Depuis un live USB
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot/efi  # Si UEFI
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys

chroot /mnt
grub-install /dev/sda
update-grub
exit

umount -R /mnt
reboot

Repair fstab

# En mode recovery, remonter en RW
mount -o remount,rw /

# Éditer fstab
nano /etc/fstab

# Commenter la ligne problématique
# /dev/sdb1 /data ext4 defaults 0 0

# Ou utiliser nofail
/dev/sdb1 /data ext4 defaults,nofail 0 0

Check the filesystem

# Depuis recovery (FS démonté)
fsck -y /dev/sda2

# Forcer au prochain boot
touch /forcefsck

🔝 Back to table of contents

---

3 - Performance problems

CPU diagnostics

# Identifier les processus CPU
top -b -n1 -o %CPU | head -15

# Processus en état D (I/O wait)
ps aux | awk '$8 ~ /D/'

# Interruptions
cat /proc/interrupts

# Profiling avec perf
perf top
perf record -g -p PID
perf report

Memory diagnostics

# Vue mémoire
free -h
cat /proc/meminfo

# Processus consommateurs
ps aux --sort=-%mem | head

# OOM killer
dmesg | grep -i "out of memory"
journalctl | grep -i oom

# Cache pressure
vmstat 1
# Si si/so élevés = swapping

I/O diagnostics

# I/O par processus
iotop -oP

# Statistiques disque
iostat -xz 1

# Colonnes importantes:
# await : latence moyenne
# %util : saturation

# Queue I/O
cat /sys/block/sda/queue/nr_requests

# Fichiers ouverts
lsof +D /var/log

Network diagnostics

# Connexions
ss -tunapl | wc -l
ss -s

# Trafic par interface
sar -n DEV 1

# Paquets droppés
netstat -i
cat /proc/net/dev

# Congestion
ss -ti

🔝 Back to table of contents

---

4 - Network problems

No connectivity

# 1. Interface UP ?
ip link show
ip addr show

# 2. IP configurée ?
ip addr show eth0

# 3. Route par défaut ?
ip route show
ip route get 8.8.8.8

# 4. DNS ?
cat /etc/resolv.conf
nslookup google.com
dig google.com

# 5. Firewall ?
iptables -L -n
ufw status

Port not accessible

# Service écoute ?
ss -tlnp | grep :80
netstat -tlnp | grep :80

# Firewall local ?
iptables -L INPUT -n --line-numbers

# Firewall distant ?
nmap -p 80 target

# Test connexion
telnet target 80
nc -zv target 80

Traffic capture

# Tout le trafic sur une interface
tcpdump -i eth0

# Port spécifique
tcpdump -i eth0 port 80

# Avec détails
tcpdump -i eth0 -nn -A port 80

# Sauvegarder
tcpdump -i eth0 -w capture.pcap

# Analyser
tcpdump -r capture.pcap

Debug DNS

# Test résolution
dig example.com
nslookup example.com

# DNS spécifique
dig @8.8.8.8 example.com

# Trace
dig +trace example.com

# Cache DNS local
systemd-resolve --status
systemd-resolve --flush-caches

🔝 Back to table of contents

---

5 - Application problems

Analyze a process

# État du processus
ps aux | grep myapp
cat /proc/PID/status

# Fichiers ouverts
lsof -p PID

# Descripteurs de fichiers
ls -la /proc/PID/fd/

# Limites
cat /proc/PID/limits

# Mémoire mappée
pmap -x PID
cat /proc/PID/maps

Trace system calls

# Tracer un processus
strace -p PID

# Tracer une commande
strace -f -e trace=network ./myapp

# Avec timing
strace -T -p PID

# Résumé
strace -c -p PID
# Ctrl+C pour voir les stats

Analyze application logs

# Suivre en temps réel
tail -f /var/log/myapp/app.log

# Chercher les erreurs
grep -i error /var/log/myapp/app.log | tail -50

# Compter par type
grep -oE "(ERROR|WARN|INFO)" app.log | sort | uniq -c

# Timeline des erreurs
grep ERROR app.log | awk '{print $1, $2}' | uniq -c

Debug with GDB

# Attacher à un processus
gdb -p PID

# Commandes GDB:
# bt - backtrace
# info threads - lister les threads
# thread N - sélectionner un thread
# continue - reprendre l'exécution

# Core dump
ulimit -c unlimited
gdb ./myapp core

Generating a core dump

# Activer
ulimit -c unlimited

# Configuration système
echo "/var/crash/core.%e.%p" > /proc/sys/kernel/core_pattern

# Forcer un core dump
kill -ABRT PID
gcore PID  # Sans tuer

🔝 Back to table of contents

---

6 - Practical exercises

Exercise 1: Diagnosing slowness

A web server is slow. Find the cause:

Approach

# 1. Vue d'ensemble
top
vmstat 1 5

# 2. Si CPU élevé
ps aux --sort=-%cpu | head

# 3. Si I/O wait élevé
iotop
iostat -xz 1

# 4. Si mémoire saturée
free -h
ps aux --sort=-%mem | head

# 5. Vérifier le service
systemctl status nginx
tail -f /var/log/nginx/error.log

Exercise 2: Port not accessible

Port 8080 is not accessible from the outside:

Approach

# 1. Service écoute ?
ss -tlnp | grep 8080

# 2. Écoute sur 0.0.0.0 ou 127.0.0.1 ?
ss -tlnp | grep 8080
# Si 127.0.0.1 -> problème de binding

# 3. Firewall ?
iptables -L INPUT -n | grep 8080
ufw status

# 4. Connexion locale OK ?
curl localhost:8080

# 5. SELinux ?
getenforce
ausearch -m avc -ts recent

Quiz

Q1. Which command shows processes in I/O wait?

Answer

ps aux | awk '$8 ~ /D/' or top (look at %wa)

Q2. How do you trace a process's system calls?

Answer

strace -p PID

🔝 Back to table of contents

---

Key takeaways

• Methodology: symptom → collection → hypotheses → test
• dmesg and journalctl for system logs
• top, vmstat, iostat for performance
• ss, tcpdump for the network
• strace to trace system calls
• lsof for open files
• Always document the resolution

🔝 Back to table of contents

---

← Previous chapter | Next chapter: Final project →