Loading...
Loading...
Highly scalable object storage architecture and bucket indexing
Highly durable cloud object store. Data is managed as flat key-value objects inside buckets and accessed via RESTful HTTP API (PUT, GET, DELETE).
You already know from the storage-contracts module how block devices sell rewritable sectors for databases while file systems sell shared hierarchies, and why neither holds billions of immutable blobs cheaply. Ten billion photos. No renames, no appends, no directory tree, just store a blob under a key (a long name like photos/2026/beach.jpg that looks like a path but is really one flat label) and fetch it back, forever, from anywhere. Strip a filesystem down to that and you get object storage, a flat HTTP store for whole blobs with no in-place edits, infinite, simple, and nearly free per gigabyte.
Think of a giant warehouse of labeled bins, the one comparison we will use here: you hand over a sealed box with a label and later reclaim it by label, but you never open the box on the shelf to change one sock. In the lab above, store a photo, fetch it from another region, then try to append one byte and watch the store refuse, because whole-box replacement is the only write it knows.
The naive move keeps every blob on one attached disk and outgrows it. We rejected scaling that disk upward because no single volume reaches petabytes and every photo would still depend on one machine. The real choice is which contract fits the access pattern: whole blobs over HTTP, raw disk sectors for databases, or shared folders for programs that expect files. Work the scale math: 10 billion 2MB photos equal 20 petabytes, which no single disk holds but a flat key space absorbs without rebalancing.
One flat namespace from key to bytes. PUT writes a whole object, GET reads it, DELETE drops it. Ideal for photos, videos, logs, and backups that are written once and read many times.
In the lab, list a million keys by prefix and watch it page instead of stalling, because slashes are just characters the listing groups, not real folders to traverse.
Block storage (a virtual hard drive attached to one server, measured in IOPS, meaning input-output operations per second) offers sector reads at sub-millisecond speed for databases and filesystems that rewrite in place. Fast, bounded in size, and gone with the machine unless snapshotted elsewhere.
File storage (a shared folder tree served over network protocols like NFS, speaking POSIX, the classic file-system dialect of open, seek, and lock) lets many machines open the same paths. Familiar and slower across distance, right for home directories and legacy tools that demand real files.
Eleven nines (99.999999999% yearly survival per object) means a 0.000000001 chance of losing one object in a year. Store 10 billion photos and the expected loss is 10 billion times 1e-11, about 0.1 photo per year, meaning one photo per decade across the whole fleet. The mechanism is mundane: every object is copied across at least three isolated buildings (availability zones) in the region, so one flood, fire, or bad deploy cannot take all copies. In the lab, lose one zone and watch reads continue, because two copies still answer.
Strong read-after-write consistency (once a write is acknowledged, every later read sees it) holds for creates, replacements, and deletes in all regions. Overwrite beach.jpg and the next GET returns the new bytes, never a mix. The edge case this does not solve is your own caching: a delivery network or browser holding the old copy still serves stale bytes until its expiry passes, so version the key (beach.v2.jpg) when freshness matters.
Multipart upload splits a 100GB video into, say, 1,000 100MB parts sent in parallel, then joins them server-side. Ten parallel parts at 50MB/s each finish in about 200 seconds instead of 2,000, and a failed part retries alone instead of restarting the world.
A million 1KB files cost a million key entries plus request fees that dwarf the byte bill. At 0.4 cents per thousand writes, one million writes cost about $4 before storing a single byte. Zip small events into larger blobs when you can.
Asking for all keys under photos/ returns pages of about 1,000 with a token for the next page. Code the loop or miss most of the warehouse. Prefixes help humans browse, but every key still lives flat.
Two features carry most real designs. Presigned URLs (time-boxed links the store mints so a browser uploads or downloads directly without proxying bytes through you) move heavy lifting to the edge: mint in milliseconds, let gigabytes flow around you. Lifecycle rules (policies like after 30 days move to cheap cold storage, after a year delete) match price to age, because last month's logs are rarely worth hot-disk prices. Set both in the lab and watch uploads bypass your server while old objects cool off on schedule.
No folders, no edits, no ceiling, just keys over HTTP that almost never lose data. The trade is total: you gain infinite cheap blobs and give up everything filesystems do. If your blobs must be searchable by content rather than by exact key, what second structure would you build beside the warehouse to find them?
Try this in the playground
Open a template and build it yourself — then take a quiz.