Toil Elimination
1 - Definition of Toil
1.1 What is Toil?
Toil = Manual, repetitive, automatable, tactical work with no lasting value.
1.2 Toil vs Engineering
| Toil | Engineering |
|---|---|
| Manual | Automated |
| Repetitive | New |
| Reactive | Proactive |
| No lasting value | Permanent improvement |
| Scales O(n) | Scales O(1) |
1.3 Impact of Toil
toil_impact:
individual:
- Burnout
- Frustration
- Skill stagnation
- Less time for innovation
team:
- Reduced velocity
- Difficulty recruiting
- High turnover
organization:
- Limited scalability
- Growing costs
- Reduced innovation
2 - Identifying Toil
2.1 Checklist
toil_identification:
questions:
- "Is it manual?"
- "Is it repetitive?"
- "Can it be automated?"
- "Is it tactical (not strategic)?"
- "Does it grow with the service?"
- "Does it have lasting value?"
scoring:
4-6_yes: "Definitely toil"
2-3_yes: "Probably toil"
0-1_yes: "Probably not toil"
2.2 Common examples
| Category | Toil | Non-Toil |
|---|---|---|
| Deployments | Manual deployment | CI/CD pipeline |
| Incidents | Manual restart | Auto-healing |
| Scaling | Manually adding servers | Auto-scaling |
| Config | Manual change | GitOps |
| Monitoring | Manual check | Automatic alerts |
| Tickets | Repetitive manual resolution | Self-service |
2.3 Toil Tracking
# toil-tracking.yaml
tracking:
method: "Weekly survey + Ticket analysis"
categories:
- deployments
- incidents
- configuration
- access_requests
- capacity_management
- on_call
- monitoring
metrics:
- hours_per_week
- percentage_of_time
- trend_over_months
3 - Measuring Toil
3.1 Metrics
toil_metrics:
time_spent:
description: "Hours spent on toil per week"
target: "< 50% of time"
alert_threshold: "> 60%"
toil_tickets:
description: "Number of toil tickets"
tracking: "'toil' label in JIRA"
toil_ratio:
description: "Toil / Total work"
formula: "toil_hours / total_work_hours"
3.2 Dashboard
dashboard:
panels:
- title: "Toil Time This Week"
type: stat
query: sum(toil_hours_weekly)
- title: "Toil by Category"
type: piechart
query: toil_hours_by_category
- title: "Toil Trend"
type: timeseries
query: toil_percentage_weekly
- title: "Top Toil Sources"
type: table
query: topk(10, toil_hours_by_task)
4 - Elimination strategies
4.1 Prioritization
4.2 Decision matrix
| Frequency | Time | Action |
|---|---|---|
| High | Long | Automate immediately |
| High | Short | Automate if possible |
| Low | Long | Document, then automate |
| Low | Short | Accept or document |
4.3 Elimination techniques
elimination_techniques:
automation:
- Scripts
- CI/CD pipelines
- Infrastructure as Code
- Self-healing systems
elimination:
- Remove the need
- Change the architecture
- Self-service for users
reduction:
- Simplify the process
- Better tools
- Improved documentation
transfer:
- Shared responsibility
- Dedicated team (temporary)
5 - Automation
5.1 Automation levels
automation_levels:
level_0_manual:
description: "Entirely manual"
example: "SSH and restart"
level_1_documented:
description: "Documented but manual"
example: "Detailed runbook"
level_2_scripted:
description: "Script available"
example: "Restart script to run"
level_3_triggered:
description: "Script triggered automatically"
example: "Webhook launches the script"
level_4_autonomous:
description: "Fully automatic"
example: "Auto-healing with no intervention"
5.2 Example: Automatic restart
# Level 0 → Level 4
# Level 0: Manual
manual_process: |
1. SSH to server
2. Check process status
3. Kill process
4. Start process
5. Verify health
# Level 2: Script
script: |
#!/bin/bash
systemctl restart myapp
sleep 5
curl -f http://localhost:8080/health
# Level 4: Auto-healing (Kubernetes)
kubernetes:
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 3
5.3 Self-Service
# Self-service portal
self_service_examples:
database_access:
before: "Ticket → SRE review → Manual grant"
after: "User request → Auto-approval (policy) → Auto-grant"
environment_creation:
before: "Ticket → SRE creates → Days of waiting"
after: "User clicks button → Terraform → Ready in minutes"
certificate_renewal:
before: "Alert → SRE renews manually"
after: "cert-manager auto-renewal"
6 - Toil Budget
6.1 The 50% rule
sre_time_allocation:
target:
engineering: 50% # Minimum
toil_operations: 50% # Maximum
actions_if_exceeded:
- Prioritize toil elimination
- Get more SRE headcount
- Push back on requests
- Escalate to management
6.2 Quarterly Tracking
quarterly_review:
metrics:
- Total toil hours
- Toil percentage
- Top toil sources
- Toil eliminated this quarter
goals:
- Reduce toil by 10% quarter over quarter
- Eliminate top 3 toil sources
actions:
- Update automation backlog
- Plan next quarter's automation work
7 - Practical examples
7.1 Before/After
example_deployments:
before:
process: |
1. Build locally
2. SSH to server
3. Stop service
4. Copy files
5. Start service
6. Test manually
time: 30 minutes
frequency: 10x/week
toil_hours: 5 hours/week
after:
process: "git push → auto deploy"
time: 0 minutes (automated)
frequency: 10x/week
toil_hours: 0
automation_cost: "20 hours one-time"
roi: "4 weeks"
7.2 ROI Calculation
toil_automation_roi:
current_toil:
hours_per_occurrence: 0.5
occurrences_per_week: 20
hours_per_week: 10
automation_investment:
development_hours: 40
maintenance_hours_per_month: 2
savings:
hours_saved_per_week: 10
hours_saved_per_year: 520
break_even: "4 weeks"
recommendation: "High priority automation"
Summary
In this chapter, we learned:
- The definition of Toil
- How to identify toil
- The tracking metrics
- The elimination strategies
- The automation levels
- The Toil Budget and ROI
Next step
In the next chapter, we will look at Incident Management.
→ Next chapter: Incident Management