Testing & Verifying Distributed Systems

Many engineers can explain Raft and name the isolation anomalies; far fewer can say what it would take to show that a system really is linearizable, why a test that passed for a week proves nothing about whether the system always eventually makes progress, or which of a specification, a fault-injection run, a deterministic simulator or a property test would have caught last quarter's outage. This course closes that gap. It answers one question from every side — how do you know a distributed system does what it claims? — for any engineer who will decide where a team spends its testing and verification effort, taught from nothing to a depth beyond what a senior staff engineer is expected to know. It builds on System Design, DBMS, Operating Systems, Data Structures, Algorithms and Mathematics.

What "correct" means states a claim precisely, as properties of a system's possible behaviours: safety (nothing bad ever happens) and liveness (something good eventually does). Specifying systems as state machines writes a design down so a tool can check it, refinement says when an implementation matches its specification, and model checking explores every reachable state of a small model. Checking histories asks the question after the fact: was this recorded execution linearizable, or serializable?

Then the implementation itself. Black-box fault testing drives a real cluster into failure, the way Jepsen — Kyle Kingsbury's testing project — does; deterministic simulation testing runs a whole cluster inside one reproducible process; and property-based testing and fuzzing generates inputs that find bugs no hand-written test reaches. Binding models to code connects the specification to what actually runs, and closes by choosing among the techniques by risk and cost.

45 topics across 9 modules, in dependency order.