Loading...
Loading...
Architecture of Amazon S3, buckets, objects, and consistency models
OAuth left you with short-lived scoped tokens that die on their own schedule instead of one shared secret, and storage needs the same delegation trick. You store 100M images, each PUT once and GET many times, and uploads arrive straight from browsers holding presigned POSTs, meaning server-signed policies authorizing direct browser-to-storage uploads that expire in minutes. A POSIX tree, meaning the standard directory hierarchy with per-file metadata hops, chokes on listing and metadata at that count, and you never need in-place edits. Object storage, meaning flat string keys over HTTP with immutable values, fits instead: PUT a key, GET a key, at exabyte scale, with durability from erasure coding, meaning splitting each object into data plus parity shards scattered so concurrent losses still reconstruct.
PUT /bucket/user/42/avatar.jpg Body=... Content-Type=image/jpeg → shards + erasure coded (e.g., 9+3) across AZs → <200ms ack after quorum GET /bucket/user/42/avatar.jpg → replica selection, range get supported Bucket key is flat string "user/42/avatar.jpg". Slashes are illusion; LIST is prefix scan
AZs, meaning availability zones or separate data-center buildings in one region, hold different shards so one building fire cannot take all copies. Quorum acknowledgment, meaning enough shards to guarantee reconstructability, lets the PUT return before every shard lands.
The naive mental model still says S3-style storage is eventually consistent everywhere, which was true for listings before December 2020 and is now wrong same-region. Same-region PUT, DELETE, and LIST are strongly read-after-write, meaning a reread immediately reflects the write, while cross-region replication, meaning async copying to a second region, still lags. Durability near eleven nines comes from erasure coding plus cross-building spread, billed down a ladder from Standard to Infrequent Access to Glacier tiers with slower retrievals.
Strongly read-after-write for PUT and DELETE including overwrite and list same-region since late 2020. Still eventual across replicated regions. Large files use multipart, meaning split into 5MB to 5GB parts uploaded separately then stitched by one Complete call, enabling 100GB objects.
Eleven nines is a design target from coding plus cross-building spread, not a promise per file. Versioning, meaning keeping every overwrite as a separate version, plus lifecycle rules, meaning automatic transitions and expirations, manage history. Never use objects as a block device, since mutation means full replace.
Per-prefix request rates auto-partition toward effectively unlimited, but LIST stays O(prefix) pagination, so avoid per-key HEAD loops. Use range GET, meaning fetching only byte ranges such as Parquet footers, and signed URLs, meaning time-boxed direct-upload links, for browser uploads without proxying.
// Presigned URL for direct upload
const url = await s3.getSignedUrl("putObject", {
Bucket, Key, Expires: 3600, ContentType: "image/jpeg"
});Database write-ahead logs need random writes with fsync latency, meaning durable acknowledgment in sub-milliseconds, so they belong on block. Shared homes need POSIX locks and directories, so they belong on file. Media and lakes need scale, HTTP access, and the cheapest durable byte, so they belong on object. The need names the interface before pricing does.
| Need | Choose | Why |
|---|---|---|
| DB WAL | Block | Random write, fsync latency |
| Shared home | File | POSIX locks, directories |
| Media/data lake | Object | Scale, durability, HTTP, cheapest |
Rejected alternative: keep the 100M thumbnails as bytea rows in Postgres so metadata and bytes commit together. At 200KB average that is 20TB inside the primary, every replica and nightly backup carries all 20TB, and one thumbnail range scan evicts the buffer pool that live queries need. The pointer wins: a narrow row holds the key while the bytes rest where 1.5x erasure overhead beats 3x replication.
Your nightly dump is 100GB and a single PUT dies at 94% after forty minutes. Retrying from scratch is not a strategy. Multipart upload, meaning splitting one object into independently retryable parts stitched server-side by one Complete call, exists for exactly this: twenty parallel streams each retry their own parts while the final stitch makes one atomic object appear.
Rules that matter:
part size: 5MB – 5GB each (last part may be smaller), max 10,000 parts
100GB file → 1,000 parts × 100MB → 20 parallel streams × 50MB/s ≈ 35 min
single PUT cap: 5GB, anything bigger MUST be multipart
Flow:
CreateMultipartUpload → {uploadId}
PUT part 1..1000 (each with ETag, retryable independently)
CompleteMultipartUpload(uploadId, [ETags]) → one atomic object appears
AbortMultipartUpload on cancel, or abandoned parts bill you silentlyParquet readers, meaning columnar-file readers that consult a footer index, fetch only the last kilobytes to learn offsets, then range-GET those ranges. A 2GB file answers a filtered query near 5MB transferred. Full GETs on columnar data in object storage is how teams discover the bill.
Eleven nines, meaning 99.999999999% or one lost object per ten million per year, sounds like marketing until erasure coding does the division. Each object becomes data plus parity shards, for example 6 data plus 3 parity, scattered across buildings, so any three concurrent losses still reconstruct from the surviving six. Overhead is 9/6 equal to 1.5x versus 3x for triple replication. Durability is not availability: eleven nines of not-losing sits beside four nines of reachable-when-asked.
Erasure sketch (illustrative 6+3): object → 6 data shards + 3 parity shards, one per AZ/rack lose any ≤3 shards → reconstruct from surviving 6 overhead: 9/6 = 1.5x vs 3x for triple replication durability 11 nines (design target) ≠ availability 99.99% (four nines) Cost ladder per GB-month (directional): Standard < Infrequent Access < Glacier Instant < Glacier Flexible < Deep Archive retrieval: Standard ms → Glacier minutes–hours → Deep Archive ~12–48h
Your bucket holds 800M Parquet files, meaning columnar analytics files, and the nightly job opens with LIST. At 1,000 keys per page that is 800,000 requests before one data byte moves: hours, throttling, and per-thousand fees. The fix pairs layout with avoidance. Hash-prefixes spread write load, Hive-style date prefixes prune reads by path, Inventory reports replace scans with a daily manifest, and S3 Select filters CSV or JSON server-side.
Bad: s3://lake/events/2025/09/05/part-*.parquet (one hot prefix, LIST pages forever)
Good: s3://lake/events/dt=2025-09-05/hr=14/a3f9-part-0001.parquet
hash-prefix (a3f9) spreads partitions; Hive-style dt=/hr= prunes by path
800M keys ÷ 1000/page = 800k LIST calls ≈ 4+ hours + throttle backoff
Instead: S3 Inventory (daily CSV/Parquet manifest of all keys) → query with Athena
S3 Select: filter CSV/JSON server-side, return matching rows onlyHive-style prefixes such as year, month, day let Spark and Trino, both distributed query engines, skip subtrees before listing. A one-day query lists thousands of keys, not hundreds of millions. Random hash prefixes spread write QPS, meaning queries per second; date prefixes prune read scans. Production layouts nest both.
Same-region PUT, DELETE, and LIST are strongly consistent, but cross-region replication lags seconds to minutes, so a western reader after an eastern write can briefly 404. Design failover reads with retry plus backoff, not assumptions.
Objects have no locks and no directories, yet teams coordinate daily with conventions over conditional writes. A zero-byte _SUCCESS marker means a partition is complete, an external lease row with TTL, meaning time-boxed ownership, means exactly one writer owns a prefix, and versioning means no overwrite destroys evidence. Like a restaurant pass where the bell means the tray is complete, readers trust the marker, not the aroma of partial files.
| Need | Convention over objects | Watch out |
|---|---|---|
| Job completion | Write data parts, then _SUCCESS last; readers ignore partitions without it | Crash between parts and marker leaves orphans; cleaner sweeps unmarked prefixes |
| Single writer | Conditional PUT (If-None-Match) or external lease row with TTL | Lease expiry during long writes forks writers; fence with version IDs |
| Direct browser upload | Presigned POST with content-length and content-type conditions | Never proxy gigabytes through your API tier; validate on download or via trigger |
A deploy script overwrites the production config key with an empty file and services restart broken. Without versioning, meaning per-PUT version retention, that is data loss. With versioning every PUT mints a version ID and DELETE writes a delete marker, meaning a tombstone hiding versions without erasing them, so recovery deletes the marker or restores the prior version. The discipline is the bill: each version is full-size until lifecycle rules expire current and noncurrent generations, and MFA Delete, meaning requiring a hardware code to permanently erase, guards buckets holding Terraform state and billing exports.
PUT config.json (v1: abc) → PUT config.json (v2: empty, bad deploy) → outage recovery: GET ?versions → restore v1 as v3, or DELETE v2's marker semantics DELETE without versionId → delete marker (object "vanishes", versions remain) DELETE with versionId=v2 → surgically removes the bad version Lifecycle that ships: current → IA after 30d → Glacier after 90d → expire after 1y noncurrent versions: expire after 30–90d (the rule everyone forgets to add) MFA Delete on the bucket holding Terraform state and billing exports.
Ten thousand users uploading 50MB videos through your API tier means proxying 500GB the tier never needed to touch. Presigned POSTs, meaning server-signed policies authorizing direct browser-to-storage uploads, move bytes straight to storage: your server signs bucket, key prefix, size range, content-type, and expiry, the browser uploads directly, and arrival notifications trigger thumbnails and moderation. The API tier handles bytes of policy, never bytes of video.
policy: { bucket, key starts-with "uploads/{userId}/", 1B–100MB, image/*, expires 15min }
browser POSTs file + policy + signature straight to S3 → 200, no proxy hop
abuse guard: randomize key suffix (no overwrite games), scan on ObjectCreated trigger
multipart from browser for >100MB: sign each part URL, Complete from your API (one call)
Cost: 10k × 50MB proxied = 500GB egress/compute through your tier vs ~0 with direct.Part 731 of 1000 fails its checksum after 99GB already landed. The naive retry restarts CreateMultipartUpload from scratch and re-sends everything, paying the full 35 minutes twice. The working resume lists already-received parts by upload ID, re-sends only missing or corrupt parts with fresh ETags, meaning per-part receipts the Complete call verifies, then completes with the full ETag manifest. Abandoned uploads without lifecycle-abort linger as billable orphans, so every starter sets a 7-day abort rule at creation. The Complete call itself is the atomic commit: no object exists under the key until it succeeds, so readers never see halves.
Upload 100GB, 1000 × 100MB: parts 1–730 ✓, 731 checksum ✗, 732–1000 pending
naive: abort + restart → resend 100GB (~35min) + double request fees
resume: ListParts(uploadId) → 730 ETags present → PUT 731 (retry) + 732–1000
→ Complete(uploadId, [1000 ETags]) → one atomic object appears
orphan guard: lifecycle abort IncompleteMultipartUpload after 7d at bucket creation.
rule: parts are expendable and Complete is the commit, so resume, never restart.Flat keys survive three concurrent building losses at 1.5x overhead, and the Complete call commits a 100GB object atomically. Yet no PUT overwrites a single byte in place, so where does a Postgres WAL that fsyncs every commit in under a millisecond actually live, and where do shared home directories with locks go?