How S3 is built
Summary
AWS S3 manages over 500 trillion objects across 38 regions, processing a quadrillion requests annually using tens of millions of hard drives. What started as an eventual consistency model in 2006 has evolved into a massively scalable, strongly consistent cloud storage platform.
Key Takeaways
- S3 manages 500 trillion objects across 120 availability zones and 38 global regions
- Original S3 design optimized for durability and availability with eventual consistency
- Entire S3 hard drive infrastructure could stack from Earth to International Space Station
- Individual enterprise customers now manage exabytes of data in 'data oceans'
Related topics
Transcript Excerpt
AWS S3 is the world's largest cloud storage service, but just how big is it and how is it engineered to be as reliable as it is at such a massive scale? Milon is the VP of data and analytics at AWS and has been running S3 for 13 years. Today we discuss the sheer scale of S3 in the data stored and the number of servers it runs on. How seemingly overnight AWS went from an eventually consistent data store to a strongly consistent one and the massive injury and complexity behind this move. What is correlated failure, crash consistency, and failure allowances, and why engineers on S3 live and breathe these concepts, the importance of formal methods to ensure correctness at S3 scale, and many more. A lot of these topics are ones that AWS engineing rarely talks about in public. I hope you enjoy t…