Loading...
Loading...
Real-time analytics with Flink and Spark Streaming
Eighty thousand mappers sort 1TB across the wire in minutes, yet the answer lands when the whole job finishes. Your fraud detector must block a transaction in 300ms, not after tonight's batch. Stream processing, meaning treating data as continuous events where each event updates state and may emit immediately, fits: events flow through ingest, windows, state, and sinks without hourly boundaries. Flink, a true event-time streaming engine, handles each event with 10 to 50ms latency, while Spark Structured Streaming, a micro-batch engine reusing the batch optimizer, batches every 100ms to 1s for simpler operations at coarser latency.
Batch: collect 1TB → run hourly → output Stream: event → [ingest] → window → state → sink (continuous, low latency) Engines: Flink (true streaming, event time) vs Spark Structured Streaming (micro-batch ~100ms)
The naive stream counts by arrival order, which misattributes every delayed phone upload. Rejected alternative: count by arrival and accept the skew. With 5% of phone uploads arriving 30s late, 5% of fraud windows close on incomplete data and the 300ms alert fires on lies. Event time, meaning when the event happened, must separate from processing time, meaning when the system saw it. Watermarks, meaning signals declaring no earlier events expected, close windows deterministically despite out-of-order arrival, like a restaurant kitchen firing a table's order only after the ticket window closes. State survives crashes through checkpoint barriers snapshotted to durable storage plus log offsets, with transactional sinks for end-to-end exactly-once.
Event time is when the event happened; processing time is when the system saw it. Watermarks advance the event clock and declare no earlier events expected, closing windows deterministically despite out-of-order arrival.
Tumbling windows never overlap such as 5m chunks, sliding windows overlap such as 5m every 1m, session windows split on 30m gaps. Flink keys state per window; late data takes allowed lateness, meaning a grace period that refires windows, or side outputs, meaning overflow streams for stragglers.
Checkpoint barriers, meaning coordinated snapshot markers flowing with events, snapshot operators to S3 every 30s or so. On failure replay from checkpoint plus Kafka offsets, meaning per-partition log positions. Sinks need two-phase commit for true exactly-once.
Flink keeps per-event state in RocksDB, an embedded sorted store on local SSD, with Chandy-Lamport barriers, meaning consistent snapshots without stopping the flow, for rich window joins and CEP, meaning complex-event pattern matching. Spark Structured Streaming reuses the Catalyst batch optimizer with write-ahead-log checkpoints, which simplifies operations for Spark shops at second-scale latency. The trade is latency and expressiveness against operational familiarity.
Per-event state in RocksDB, checkpoint barriers every 30s, exactly-once via Chandy-Lamport. Latency 10–50ms, rich window joins and CEP. Cost: operate RocksDB, tune checkpoint timeout and backpressure.
Batch every 100ms–1s, reuse batch optimizer (Catalyst), easy SQL. Latency higher, watermark also needed, but checkpoint via WAL and simpler ops if you already run Spark. Good for 1s SLA, not 50ms.
| Concern | Flink | Spark |
|---|---|---|
| State | RocksDB per key, async snap | In-memory map, checkpoint to S3 |
| Late data | Watermark + side output | Watermark + drop or update mode |
Keying the fraud stream by merchant hash-partitions evenly until one marketplace seller produces 40% of events. That partition's RocksDB compactions lag, its checkpoints stretch to minutes while siblings finish in seconds, and backpressure, meaning a slow subtask throttling shared sources, slows every partition. Averages hide it; per-subtask watermark lag exposes it.
32 partitions keyed by merchantId, 100k events/s: average: ~3.1k/s per subtask; comfortable hot seller: 40k/s on partition 7 → RocksDB compaction lags, checkpoint grows backpressure: partition 7 → source slows → ALL partitions slow (shared input) Fixes that ship: key splitting: merchantId + salt(0..9) → 10 sub-keys, two-stage aggregate isolate hot keys: dedicated operator chain with higher parallelism monitor per-subtask watermark lag, not just job average; the max tells the story
Flink draws barriers every N seconds, snapshots RocksDB to S3, and commits Kafka offsets alongside. Shorter intervals replay less after crashes but snapshot constantly; longer intervals run cheaply and recover terrifyingly. At 500k events/s a 3min interval means about 90M events to replay, near 90s catch-up at 1M/s plus snapshot load, versus 20s for a 30s interval at 6x snapshot cost. Start near 1min with 10min timeouts and 30s minimum pauses so snapshots never overlap, then move with measured checkpoint duration.
Checkpoints alone give at-least-once sinks. True exactly-once needs a transactional sink: Flink pre-commits on barrier, commits on checkpoint-complete (two-phase against Kafka transactions or upsert-by-key into a store). Without that second phase, replays duplicate every alert your fraud detector ever sent.
Recovery math: 500k events/s × 3min interval ≈ 90M events to replay at 1M events/s replay → ~90s catch-up + snapshot load + warmup → 3min interval ≈ 2min worst-case unavailability; 30s interval ≈ 20s, 6x snapshot cost
Windows never close and the global watermark, meaning the job-wide event-time clock, stalls. The usual cause is one idle Kafka partition, meaning a log shard with no recent events, holding the minimum back, since the global watermark is the minimum of per-partition watermarks. Flink's per-partition watermarks plus idleness timeouts near 1min of silence let quiet shards stop freezing the job. Monitor the maximum per-subtask lag rather than the job average to spot it.
Flink buys 10 to 50ms latency with RocksDB state you operate; Spark buys simpler SQL at second latency with coarser windows. Choose by the alert deadline: sub-second blocking needs per-event streaming, second-scale dashboards tolerate micro-batch.
Ride receipts must total correctly even when a phone uploads Tuesday's trips Friday, while fraud alerts must fire in 300ms despite out-of-order events. One pipeline serves both with allowed lateness plus side outputs: fast approximate results now, corrected restatements when stragglers land. Lambda architecture, meaning parallel batch-correct plus streaming-approximate pipelines that reconcile, doubles pipelines for defensible totals; Kappa, meaning one streaming engine replayed for both, replays the log instead. Prefer Kappa unless batch restatements are cheaper separately.
Fraud window (5-min tumbling, watermark lag 2s): on-time event → counted, alert in ~300ms 30s-late event → within allowed lateness (5min): window refires, alert amended 2h-late event → past lateness: routed to side output, reviewed offline, never blocks Billing window (1-hour tumbling, allowed lateness 24h): Tuesday trip uploaded Friday → side-output replay job restates Tuesday's total dashboard shows v1 fast, v2 corrected; version the aggregates, don't overwrite blindly
Bounded out-of-orderness of 2–30s covers normal jitter; idle-source timeouts near 1min stop one silent Kafka partition from freezing every window; per-partition watermarks in Flink prevent the slowest shard from dictating global progress.
Lambda keeps batch (correct, slow) plus streaming (fast, approximate) and reconciles; double the pipelines, defensible totals. Kappa replays the log through one streaming engine for both. Choose Kappa unless the business needs batch-correct restatements the stream cannot afford to recompute.
Throughput graphs stay flat until collapse because streaming fails as state growth and checkpoint stretch, not error spikes. Watch per-subtask checkpoint duration p99, meaning the near-worst snapshot time, state size per subtask bounded by key TTLs, and Kafka consumer lag, meaning unread backlog. Flat lines are healthy; slopes lasting days are the page arriving early.
| Signal | Healthy | Action when it drifts |
|---|---|---|
| Checkpoint duration p99 | Seconds, stable across days | Move to incremental snapshots, raise interval, or split hot keys |
| State size per subtask | Bounded by key TTLs (hours–days) | Add state TTL, compact old windows, pre-aggregate upstream |
| Consumer lag (Kafka) | Near zero with brief spikes | Scale subtasks (savepoint, rescale, resume); lag that grows linearly never self-heals |
The fraud model shipped inverted for three days and every score since Tuesday is garbage. Kafka, a durable log bus, still holds every event, so replay: savepoint the running job, meaning snapshot state plus offsets, deploy fixed code starting from Tuesday's offsets into fresh state, sink restatements to v2 tables without overwriting v1, and promote after unaffected windows converge. Immutable logs make wrong code recoverable; mutable databases would not.
Replay recipe (Flink + Kafka): 1. savepoint (consistent snapshot of state + offsets) → keep the bad job running 2. new job version: same topology, fixed logic, start offsets = Tuesday 00:00 3. sink to restated tables (v2), never overwrite v1 mid-replay 4. compare v1 vs v2 on unaffected windows → converge → promote v2, retire v1 Retention math: 500k events/s × 3 days ≈ 130B events ≈ 65TB at 500B each → Kafka retention (7d, tiered to S3) must exceed your worst "code was wrong" window.
Which clicks converted within an hour needs both streams held in state, which grows unbounded unless the join carries expiry. Interval joins, meaning matches constrained to purchase minus click within 0 to 1h, keep each click for one event-time hour with TTL eviction. For 1M users times 20 clicks times 200B near 4GB across subtasks in RocksDB, bounded and checkpointed, plus side outputs for 2h-late purchases the attribution job decides. Regular joins keep both sides forever like batch; interval joins expire by business rule.
clicks.keyBy(user) ⨝ purchases.keyBy(user) WHERE purchase.time - click.time ∈ [0, 1h] state per user: clicks of the last hour (≈ events/user/hour × size), bounded by TTL 1M users × 20 clicks × 200B ≈ 4GB across subtasks; RocksDB, checkpointed, fine. late purchase (2h after click): outside interval → side output, attribution job decides. Flink interval join vs regular join: regular keeps BOTH sides forever (batch-shaped); interval join is the streaming-native answer; always ask "what expires this state?"
A Flink job with 64 subtasks reading a 16-partition Kafka topic, meaning a log split into 16 ordered shards, leaves 48 subtasks idle, since one partition feeds at most one subtask. Parallelism beyond partition count buys nothing; below it subtasks multiplex and the hottest partition still caps one subtask. Size job parallelism as a multiple of source partitions, repartition by business key only where demanded, and treat idle subtasks as the cheapest scaling signal.
The job restarts into a 6-hour backlog at 500k events per second, nearly 11B events waiting. The naive resume processes at normal parallelism and never catches up, since arrival rate matches service rate and the backlog persists indefinitely. The working drain scales temporarily: raise parallelism against the 16-plus source partitions, meaning log shards bounding per-subtask assignment, switch checkpoints to longer intervals to spend I/O on catch-up rather than snapshots, and shed side outputs such as late-data branches until lag nears zero. Backpressure, meaning slow subtasks throttling shared sources, inverts during drain: sources must run unthrottled while state backends absorb the write storm. Watch RocksDB compaction debt, since 11B replays compacting at once can stall the job just as lag clears.
Backlog: 500k/s × 6h ≈ 10.8B events; normal service 520k/s → net drain 20k/s ≈ 6 days
scaled: 2x parallelism (32 → 64 subtasks) + 3min checkpoints → 900k/s → net 400k/s
→ drains in ~7.5h while arrivals continue; then scale back, restore intervals.
RocksDB edge: 10.8B writes compact at once → SSTable count spikes → throttle drain
to compaction throughput, not source throughput, for the final stretch.Streams emit v1 in 300ms and restate v2 when Tuesday's trips land Friday, with intervals expiring state by business rule. Yet v1 and v2 both need a home analysts can scan in seconds without touching the transactional primary. What stores columns apart from rows so last year's revenue stops taking five minutes?