Site Reliability Engineering

This course teaches how to keep a fleet of services running: to predict how a design fails before it does, to bound the damage when it does, and to argue with numbers about what reliability is worth. It assumes the System Design course, not experience of running a fleet, and aims past what a senior staff engineer is expected to know. System Design teaches the performance laws, the serving path in normal operation and the basics of service-level objectives; this course starts where those end — what happens when parts fail, slow down or are overloaded, how failures spread and keep themselves going, how to contain them, and how changes, the most common trigger, are made safe. It also builds on Computer Networking and Mathematics.

Reliability as a number makes availability, service-level indicators and error budgets quantities you can compute and spend. How distributed systems fail covers correlated, gray and cascading failure and the health checks meant to catch them. Timeouts, deadlines and retries shows how well-meant retries multiply load, and overload shows how a system can stay broken after its trigger has gone, and the load shedding and throttling that prevent it. Containing the blast radius builds bulkheads, cells and shuffle sharding, and dependencies and degraded modes decides which failures a service must survive.

Safe change covers progressive delivery, canary analysis, rollback and feature flags. Incidents, learning and experiments covers incident response, postmortems and chaos engineering, and capacity, cost and the reliability review sizes headroom, prices each extra nine of availability, and closes by walking one service through the whole course.

48 topics across 9 modules, in dependency order.