Block vs File vs Object Storage

Three ways to persist bytes when one interface cannot serve all three

Object storage just committed a 100GB backup atomically through one Complete call at 1.5x erasure overhead, yet it never overwrites a single byte in place. Your app must also store a Postgres write-ahead log, meaning the append-only journal the database fsyncs before acknowledging commits, plus a user home directory, plus 10M thumbnails. One needs low-latency random overwrites, one needs shared directories with locks, one needs cheap durable blobs over HTTP. Block storage, meaning raw numbered sectors you overwrite in place, file storage, meaning a POSIX hierarchy, which is the standard path plus directory plus lock API, with shared paths, and object storage, meaning immutable keys you replace whole via HTTP, each optimize one axis. Picking by habit instead of mutation pattern means fighting the system for years.

What each interface feels like at 2am under load

Picture one restaurant kitchen with three pantries: a spice rack you grab in milliseconds but only one cook touches, a shared walk-in every cook enters with a sign-out sheet, and a distant warehouse that ships any box cheaply in minutes. Block is the spice rack, file the walk-in, object the warehouse. The naive move forces one pantry for everything; the working move matches access to pantry.

What one Postgres commit feels on block, file, and object

Picture Postgres committing a transaction. On block it issues an 8KB random write, gets fsync, meaning durable-write acknowledgment, back in under a millisecond, and moves on. On a file share the same write crosses NFS, meaning the standard network file protocol, waits on attribute caches and lock checks, and lands in 2–8ms. On object there is no overwrite at all: it must PUT a whole new multi-MB object and wait 50–150ms. Same bytes, three performance universes.

Worked sense-check: 1M thumbnails at 200KB equals 200GB. On block that is 200GB of expensive provisioned volume plus snapshots; on object, meaning S3-style key storage, it is 200GB at lake prices with per-key HTTP access and lifecycle to cheaper tiers. Immutable read-heavy key access picks object before pricing pages open.

Where each abstraction quietly breaks in production

Running one Postgres across all three interfaces at once

The naive layout puts everything on one volume for simplicity, then pays block prices for cold backups and loses history with the availability zone. The working layout uses every interface for its strength: live data files on block for fsync-grade random writes, WAL archives and base backups streaming to object for eleven-nines retention, and shared config plus staging dumps on file where operators and jobs share one tree.

Live:  pgdata on EBS gp3, 16KB pages, checkpoint every 5min
         WAL segment 16MB, fsync per commit → 0.5ms p99 on block
Backup: WAL-G / pgBackRest streams each 16MB segment to S3 on archive
         base backup weekly (snapshot to object), WAL replay for point-in-time
         30-day retention: 30 × daily delta ≈ cheap object GBs, not block GBs
Shared: /mnt/ops (EFS) holds configs, migration scripts, CSV staging
          every app host + cron box mounts the same tree with no rsync chains

Budgeting one division of latency and one of dollars for thumbnails

A thumbnail service at 10k GETs per second picks storage with per-op latency times parallelism and per-GB price times retention. Block wins a latency contest nobody asked for here; object, meaning S3-style keys plus a CDN, meaning edge caches serving repeats, wins the bill by an order of magnitude while absorbing parallelism natively.

Striping volumes when one disk ceiling triples your p99

Your analytics replica needs 12,000 IOPS and a single gp3 volume, meaning baseline cloud SSD, tops near 16,000. Technically enough until a Monday report doubles load and latency triples at the ceiling, because queues grow nonlinearly past saturation. Striping four volumes multiplies throughput but also failure surface: lose one stripe and the filesystem is gone, so stripes always pair with snapshots. Rejected alternative: stay on the single gp3 and trust bursting past its 16k ceiling because the replica needs only 12k today. Monday doubles load to 24k, queues grow nonlinearly past saturation, and p99 triples while the burst credits drain in minutes. The rule names all three: stripe for throughput, replicate for survival, snapshot to object for history.

Need: 25k IOPS sustained, 3TB, crash-safe Postgres replica
  option A: 1× io2 3TB @ 25k IOPS. Simple, single AZ fate, highest per-volume price
  option B: 4× gp3 750GB @ 6.5k IOPS each, striped (mdadm RAID-0)
            → ~26k aggregate IOPS, 4x the snapshot chores, lose-one-lose-all
  option C: local NVMe (i3) + streaming replica + object backups
            → 100k+ IOPS, ephemeral (AZ failure = rebuild from peer), cheapest fast GB

Rule: stripe for throughput, replicate for survival, snapshot to object for history.
  One mechanism never covers all three, so the design always names all three.

Feeding eight GPUs from one shared tree without copying 8TB

Training on 8 GPUs across 2 hosts reads the same 4TB image dataset randomly each epoch. Copies per host waste 8TB and drift; object adds 50ms per random read and starves GPUs. A shared POSIX mount such as EFS, a managed NFS service, Filestore, or Lustre-class FSx gives every worker one path with read-ahead and caching. It is the workload where file semantics genuinely beat both alternatives.

Choosing io2 over gp3 by ceiling, latency, and lost-write cost

A finance database needs 20k random-write IOPS at sub-millisecond latency with durability auditors accept. gp3, meaning baseline cloud SSD, caps near 16k IOPS per volume with generous but vague bursting. io2, meaning provisioned-IOPS cloud SSD with published durability, provisions past 64k with per-IOPS pricing that stings. Compare ceiling, latency, and what a lost write costs: boot and replicas default to gp3, latency-sensitive primaries past 16k move to io2, and hot primaries with streaming replica plus object backups can ride local NVMe, meaning server-attached flash, for 100k-plus.

Migrating a live database from gp3 to io2 without a maintenance window

Your primary rides gp3, meaning baseline cloud SSD, at 15k IOPS against a 16k ceiling with p99 climbing weekly. The naive migration schedules downtime, snapshots, and restores, which trades a latency problem for an availability incident. The working path replicates first: attach a streaming replica on io2, meaning provisioned-IOPS cloud SSD, let it catch up, promote it during a seconds-long failover, then repoint readers. The replica absorbs the copy load while the primary serves, and rollback means promoting back rather than restoring. Watch replication lag during the copy, since a primary writing 5k IOPS while the new volume initializes can keep the replica minutes behind until the backlog drains.

Live migration: primary (gp3, 15k/16k IOPS, p99 2ms climbing)
  1. provision replica on io2 @ 25k IOPS → streaming replication catches up (~hours)
  2. lag < 1s → promote replica (failover ~5–30s) → readers repoint
  3. old primary becomes standby or retires; no snapshot-restore window
Lag math: 5k writes/s × 3h copy = 54M writes of backlog → replay at 8k/s ≈ 2h catch-up.
  start Friday night, promote Monday; never promote with lag above your RPO.

The one-line picker

Three interfaces cover one Postgres cleanly, and the gp3 to io2 failover takes seconds. But none of them sorts 10TB of logs with 80,000 parallel readers beside the bytes. What filesystem splits files into 128MB replicated blocks so the compute moves to the data instead?