Where to start if you want to harden a distributed system
Summary
To harden distributed systems, skip straight to studying accident analysis rather than tools—Nancy Leveson's "Engineering a Safer World" applies lessons from aerospace disasters like Therac 25 and Columbia to help engineers understand why complex systems fail catastrophically.
Key Takeaways
- Study accident investigation frameworks from high-stakes industries (aerospace, medical devices) before implementing distributed system tools—understanding failure modes prevents costly production incidents.
- Nancy Leveson's systems thinking approach, developed through investigating real disasters (Therac 25 radiation case, Columbia shuttle disaster), provides actionable mental models for anticipating distributed system failures.
- Apply safety engineering principles from aeronautics to software architecture—these disciplines have decades of proven methodologies for preventing failures in complex, interconnected systems.
- Prioritize understanding "how systems break" over tool selection—foundational knowledge of failure patterns enables better architectural decisions and risk mitigation strategies.
Related topics
Transcript Excerpt
If I'm a software engineer and I work on complicated distributed [music] systems, what tools, what techniques would you recommend that I look into to harden these systems? >> I'm going to just completely go 90° here and I'm recommend this book by Nancy Leveson called Engineering a Safer World. She was an aeronautics engineer who investigated things like the Therac 25 radiation case and the [music] Columbia disaster. And she was really fascinated in how accidents in systems happen, why accidents [music] could happen in complicated systems. And I found her writing on this to be incredibly insightful and incredibly valuable in understanding how these systems can break. So, that's the thing I first recommend is checking out that book.…