Loading...
Loading...
Ethernet, IP, ARP, routing tables and how packets traverse the internet
Last topic priced failover at 10 to 15 seconds to notice, RTO in minutes for standby, and fencing through leases and epochs so the old leader stays buried. All of that assumes pings, votes, and writes actually travel between machines. Click a link and your request crosses a home router, an ISP, meaning your internet provider that carries you to the wider net, undersea cables, and a load balancer, meaning a box that spreads requests across backends. The rejected mental model is one smart box understanding the whole trip, which fails because no single box can track every wire, address, and app at planetary scale: a core router holding per-application state for billions of flows would drown in memory and control traffic. Instead each box only reads its own layer, makes a local decision, and forwards. That division of labor is the entire idea behind network layers.
Application HTTP, gRPC, WebSocket → "what does the user want?" Transport TCP / UDP / QUIC → "reliable stream or datagram?" Network IP (v4/v6) → "which host?" Data Link Ethernet, Wi-Fi (802.11) → "which device on this wire/air?" Physical electrons / photons / radio → "bits on a medium"
Reliability lives in one specific place here: not the cable and not IP, but TCP's retransmission, sequencing, and flow control at the transport layer. When someone says “TCP is reliable,” that bookkeeping is what they mean.
Before anything can cross the internet, it has to leave your local network. L2, the data-link layer, moves frames, meaning packets scoped to one local network, between devices on the same wire or Wi-Fi, like your laptop to your router, using burned-in MAC addresses, meaning hardware identifiers on each network card. It only knows the building it is in.
MAC identifies a network card. ARP, meaning Address Resolution Protocol, maps IP → MAC on a local network via broadcast: “who has 10.0.0.5?” The owner replies. Entries are cached for minutes and expire.
MTU, meaning the maximum transmission unit or largest frame a link allows, is 1500 bytes on Ethernet. A 3000-byte IP packet is split into frames here, called fragmentation. But because Wi-Fi retries per frame, large frames hurt loss recovery. Path MTU discovery, meaning probing to learn the smallest MTU along the route, avoids fragmentation.
Switches learn MAC-to-port tables and forward by MAC only. They do not understand IP. That is why your VPC, meaning a virtual private cloud or isolated virtual network, needs both subnets for local delivery and route tables for cross-network hops.
Client ──[Ethernet frame: src MAC aa:.. dst MAC gateway]──> Home router
payload = IP packet (10.0.0.12 → 93.184.216.34)
payload = TCP segment
payload = HTTP GETEach hop strips outer headers, makes a forwarding decision, and re-encapsulates, meaning it wraps the payload in a fresh outer header.
Once your packet leaves the street, it needs a global address and someone willing to forward it hop by hop. That is IP, meaning Internet Protocol, which offers no promises about speed, order, or even arrival. It provides best-effort delivery plus routing, meaning each router picking the next hop toward the destination, which is the reason the packet moves toward its target at all.
IPv4 is 32-bit (~4B addresses, exhausted), IPv6 is 128-bit. CIDR, meaning classless addressing notation like 10.0.0.0/16 where the number after the slash counts network bits, expresses a block: 10.0.0.0/16 means first 16 bits are network, rest host, which supports 65k hosts.
Each router has a table: destination → next hop interface. BGP, meaning the protocol carriers use to exchange routes between networks, handles the global hops. OSPF and IS-IS, meaning interior protocols used inside one organization, handle intra-datacenter hops. Packets hop router by router. TTL, meaning a hop-count that decrements each hop, prevents loops.
ip route show 10.0.0.0/16 via 10.0.0.1 dev eth0 0.0.0.0/0 via 192.0.2.1 dev eth0 # default1. DNS: client asks "A record for api.sysiq.com?" → 93.184.216.34
2. SYN: client → server (TCP handshake, see TCP topic)
3. Encapsulation:
HTTP GET → TCP seq=1001 len=512 → IP src 10.0.0.12 dst 93.184.216.34 → Ethernet dst gateway MAC
4. Home router: strips Ethernet, looks up IP route 0.0.0.0/0 → ISP, re-encapsulates on WAN link
5. ISP routers: hop-by-hop IP forward via BGP/OSPF
6. Server NIC: delivers IP packet to kernel; TCP reassembles stream; user-space reads HTTP
7. Reverse path symmetric for responseEvery layer adds its own wrapper, and the wrappers add up. A typical HTTP response byte travels with roughly 20 bytes of IP header plus 20 bytes of TCP header, before TLS, meaning the encryption layer, and HTTP add theirs. On a 1500-byte Ethernet frame that leaves about 1460 bytes for your data, called the MSS. Send 10k RPS of 1 KB API responses and the math is 10 MB/s of payload but closer to 11 MB/s on the wire once 40 bytes of headers ride each 1,000-byte payload, or about 88 Mbps once headers are counted and bytes are multiplied by 8. The rejected shortcut is ignoring headers in capacity math: the team that prices 10k times 1 KB as exactly 80 Mbps discovers the missing 8 Mbps as drops at the first saturated link.
| Wrapper | Typical size | What to remember |
|---|---|---|
| Ethernet frame | 14 bytes + 1500 MTU | Going bigger needs jumbo frames end to end |
| IPv4 / IPv6 header | 20 / 40 bytes | v6 costs double per packet. Matters at high PPS |
| TCP header | 20–60 bytes | Options like timestamps and SACK eat into MSS |
| Small-packet penalty | A 100-byte RPC carries ~40% overhead | Chatty microservices drown in headers, so batch or multiplex |
Sizing habit worth building: when asked to size a link, add 5–10% header overhead to the payload math, then double for peak plus retries. The candidate who says “10k times 1 KB is 80 Mbps, call it 100 with overhead” has priced the wrappers instead of ignoring them.
Layering means failures surface one layer away from their cause. The classics all look like “the app is slow” from above and “the network is fine” from below, until someone checks the layer in between.
Small requests fly, large ones hang: a middlebox with a smaller MTU drops fragments while ICMP “too big” messages, meaning the feedback packets Path MTU Discovery needs, are filtered, so discovery never learns. Users see uploads and big API responses stall. On-call sees clean pings, because pings are small. The fix is clamping TCP MSS to 1460, or 1360 behind VPNs, meaning encrypted tunnels with extra headers, and setting Don't Fragment so failures surface fast instead of hanging.
NAT, meaning network address translation that lets many private IPs share one public IP, tracks every outbound connection in a table. A pool misconfigured to open thousands of short connections exhausts the table, and new connections fail while old ones look healthy. It presents as random timeouts scaling with traffic. The fix is connection reuse plus raising conntrack limits, meaning the kernel's connection-tracking capacity, and graphing table usage before it hits 100%.
None of this is classroom-only. Cloudflare, a CDN and edge-security provider, absorbs multi-terabit floods at L3 and L4 before they ever reach customer origins, because raw pipes cannot. Netflix keeps its Open Connect caches, meaning Netflix's own video boxes placed inside ISP networks, so evening video traverses one short hop instead of a continent. And every cloud provider begs you to keep chatty internal calls inside one availability zone, where round trips stay under a millisecond instead of crossing regions at a hundred times the cost.
The working version: distance sets the latency floor, headers set the throughput tax. Put bytes near users, keep services near their dependencies, and the lower layers stop being the reason pages feel slow.
Layers deliver to 93.184.216.34 by reading one wrapper at a time, with 1460 bytes of room per 1500-byte frame and route flaps showing up as p99 spikes. But where did 93.184.216.34 come from? Nobody types it. Something turned a name into that number before the first SYN ever left.