No router knows the whole internet. Every packet you send is handed between networks that don’t trust each other — and at the far end, between two stacks on one machine joined by nothing but a pointer. This guide works downward through both.
Why does networking look the way it does?
Almost nothing here was designed. Each piece was added by someone with a specific problem, on top of whatever was already deployed — which is why the result layers the way it does.
routed in 4.2BSD and was on every machine before anyone wrote it down. RFC 1058 documented it five years later rather than specifying it.Two administrations, no shared trust, and a boundary that gets negotiated rather than measured — plus the label-switched core that carries the result.
Why isn’t there just one routing protocol?
Inside a network, routing is an engineering question: which way is fastest? Between networks, it is a business question: whose traffic am I willing to carry?
What are all these protocols, and how do they differ?
Every routing protocol differs on two axes: who you talk to, and what you tell them.
How do two networks that don’t trust each other agree on a route?
Carrying the full AS path solves the loop problem outright — and doubles as the raw material for policy. Here is what actually moves between two speakers.
AS_PATH, e.g. 64500 3356 15169.LOCAL_PREF (what I want)AS_PATHORIGINMED (what they asked for)Between two autonomous systems. This is the boundary BGP was designed around, and the only place its caution makes sense.
AS_PATH — which is simultaneously the loop check, the distance metric, and the evidence policy is written against.NEXT_HOP to itself, so the far side is given an address it can actually reach.ebgp-multihop, stated deliberately.The same protocol between two routers inside one AS. Almost everything that looks strange about BGP network design comes from this half.
AS_PATH is unchanged inside the AS — which is exactly why the next card exists.NEXT_HOP. It stays the far eBGP neighbour’s address, so your IGP has to be able to reach it — or you configure next-hop-self and take the job on yourself.AS_PATH does not grow internally, there is no loop detection available. The prohibition has to be structural instead.ORIGINATOR_ID and CLUSTER_LIST as the loop check; confederations fake sub-ASes instead.Why forward on a label instead of an address?
MPLS puts a short fixed label in front of the packet. Every router after the first one matches that label exactly, and never looks at the IP address again.
That is the whole MPLS header: 32 bits, shimmed between the layer-2 frame and the IP packet — which is why MPLS is called “layer 2.5”. The S bit marks the bottom of the stack, because labels stack: an outer one for the tunnel across the core, an inner one identifying the customer at the far end.
10.0.0.0/8 across one core that never learns either.How can two customers both use 10.0.0.0/8 on one network?
Two customers both use 10.0.0.0/8 and both cross the same core. Nothing collides, because in the core the packet is never routed by its address — the label carries the tenant.
10.0.0.0/8 and red’s are rows in two different tables that never meet — no more magic than two network namespaces on one host — and Linux has the VRF abstraction itself, in section 17.65000:1:10.0.0.0/8. Purely a uniquifier — it carries no policy.10.1.1.1 and red’s still cannot talk to each other — joining them needs NAT or a shared-services VRF with non-overlapping addressing. Overlap is only free while the two tenants stay strangers. The same shape turns up in VXLAN/EVPN, where the VNI plays the VPN label’s part and the RD/RT model is borrowed wholesale: the tenant identifier has to travel with the packet, because the address stopped being unique.What has to come up, and in what order, before anything works?
Everything in this part runs at once, on the same routers, over the same links. What makes it work is the order — and what makes it fail is that the order is never enforced.
A pedantic but load-bearing distinction: OSPF and IS-IS form adjacencies, not sessions. There is no TCP — OSPF rides IP protocol 89 with its own acknowledgement scheme, IS-IS sits straight on layer 2 — and they discover each other with multicast hellos. LDP and BGP have real TCP sessions, on ports 646 and 179, terminated on loopbacks. That difference is why an IGP adjacency survives things a session does not, and why a session can be down while the adjacency beside it looks perfect.
Interface up. BFD starts a bare heartbeat — typically 3 × 50 ms — that every protocol above subscribes to instead of running its own slow timers. It carries no routing information at all; its entire job is to say down in 150 ms rather than 40 seconds.
Hellos, database sync, SPF. The important output is not the transit routes — it is that every loopback in the network is now reachable, because every session above is going to be built between loopbacks.
Discovery is per link — hellos to 224.0.0.2 on UDP 646. The session itself is TCP 646 between the two loopbacks, opened by the higher LSR-ID. Both ends then advertise a label for every prefix in the IGP, unsolicited, whether or not they are the next hop for it.
PE to PE, or PE to route reflector, TCP 179 loopback to loopback — a session that exists only because step 02 made those loopbacks reachable. VPN routes arrive carrying their VPN labels, and each one’s BGP next hop resolves to an LSP that step 03 already built.
mpls ldp router-id Loopback0 force is not a formality: an unreachable loopback breaks LDP while every interface and adjacency still looks perfectly healthy.One owner, full trust, and a free hand to optimise. This is the only place a metric means what it says.
How does a router learn about a network it cannot see?
This single choice decides the memory a router needs, how fast the network heals, and how badly it fails.
| Distance vector | Link state | Path vector | |
|---|---|---|---|
| What you store | one row per destination | the whole topology graph | every path you were offered |
| Who you tell | direct neighbours only | every router in the area | configured peers only |
| What you tell | your whole table | your own links | prefix + full AS path + attributes |
| When | periodic, e.g. 30 s | on change, plus refresh | on change, incremental |
| Loop defence | hop ceiling and hacks | impossible — everyone shares one map | see your own AS number, discard |
| Picks by | metric | metric | policy, then metric |
| Scales to | tens of routers | hundreds per area | the entire internet |
Where do routes live before they reach the routing table?
A protocol’s database is not the routing table. Each one maintains a private view, argues with its peers inside that view, and submits a single candidate per prefix to the router it happens to be running on.
224.0.0.9 is believed.224.0.0.10 / IP proto 88.Four protocols offer the same prefix. Which one wins?
Protocols only propose. A single router may hear about the same destination from four of them at once.
Decided before any protocol matters. 10.1.2.0/24 beats 10.0.0.0/8 for one packet, even if the /8 came from a “better” protocol. More specific always wins.
Same prefix from two protocols? The one you trust more wins. Cisco’s defaults: connected 0, static 1, eBGP 20, EIGRP 90, OSPF 110, IS-IS 115, RIP 120, iBGP 200. It is a local preference, not part of any protocol.
Same prefix, same protocol, two paths? Now the protocol’s own arithmetic decides — OSPF cost, IS-IS metric, EIGRP’s composite, RIP hops. Metrics are never comparable across protocols, which is exactly why step 02 exists.
The winner is installed in the routing table (RIB) — the control plane’s conclusion. It is then compiled into the forwarding table (FIB) in hardware, which is what actually moves packets, millions per second, without consulting anything.
Which protocol should I expect to find where?
| Protocol | Family | Metric | Runs on | Converges | Where you meet it |
|---|---|---|---|---|---|
| RIPv2 | IGP · distance vector | hop count, max 15 | UDP 520 · 224.0.0.9 | ~30 s per hop | small flat networks, textbooks |
| EIGRP | IGP · distance vector | composite: bandwidth + delay | IP proto 88 | sub-second | Cisco enterprise estates |
| OSPFv2/v3 | IGP · link state | cost = ref-bandwidth ÷ link | IP proto 89 | tens of ms | enterprise, data centres |
| IS-IS | IGP · link state | wide metric, 24-bit | directly on layer 2 | tens of ms | carrier cores, large fabrics |
| BGP-4 | EGP · path vector | policy first, AS_PATH second | TCP 179 | seconds to minutes | every network boundary on earth |
Two of these have outlived their category. IS-IS won the carrier core because it doesn’t ride on IP and can carry new address families without a new protocol version. BGP escaped the border entirely — it now distributes VPN routes, EVPN MAC addresses and data-centre fabrics inside networks, doing a job its name says it shouldn’t.
One broadcast domain — the set of machines that reach each other with no router in between. Everything above this was about getting between such places; this is what happens inside one.
How does a machine find the MAC address for an IP it has never talked to?
A subnet is not an addressing convention, it is a scope — and inside it you can ask a question of everyone at once. That single ability is what makes address resolution work, what lets a switch learn without ever being configured, and what eventually stops the whole thing scaling.
10.0.1.5 lives, so it sends the frame everywhere except back the way it came.ip neigh — with states you can watch: REACHABLE, STALE, DELAY, PROBE, FAILED./24 is not a property of the wire; it is the number a host consults to answer exactly one question: is this destination somewhere I can shout at, or do I hand the frame to my gateway? Get the mask wrong on one host and it ARPs for addresses nobody will answer, or routes to things that were sitting beside it all along.Section 05 solved this for routes: give every tenant its own table and carry an identifier with the packet. Two layers down, the same answer gets rebuilt twice — once in twelve bits, once in twenty-four.
How does one cable carry several separate networks?
One physical Ethernet, many broadcast domains. A four-byte tag, inserted on the way in and removed on the way out, so the hosts on either end never learn they were sharing.
Four bytes, wedged in after the source MAC — so a tagged frame is 1522 bytes on a 1500-byte network, which is why switches quietly grew a “baby giant” MTU. PCP is the class-of-service field every QoS design starts from. VID is the whole point, and its twelve bits are the reason the rest of this part exists: 0 and 4095 are reserved, leaving 4094 segments per L2 domain.
ip link add link eth0 name eth0.100 type vlan id 100 makes a subinterface — its own netdev, MAC, MTU and sysctls, tagging everything it sends.bridge vlan add dev eth0 vid 100 pvid untagged gives you real access and trunk ports.tcpdump on the host often shows no tag at all on a frame that definitely had one.What do you do when 4,094 VLANs isn’t enough?
Wrap the whole frame in an ordinary IP packet and the L2 domain stops being a cabling problem. Twenty-four bits of tenant id instead of twelve, and a routed fabric underneath that can use every link it has.
The encapsulation costs 50 bytes — 14 outer Ethernet, 20 outer IP, 8 UDP, 8 VXLAN — so a guest on a 1500-byte underlay must be told to use 1450, or the fabric must run jumbo frames. Nothing announces this. It presents as small packets working and large ones vanishing, which is the single most common VXLAN outage there is.
# a VTEP, with no flooding and no multicast $ ip link add vxlan0 type vxlan id 5000 dstport 4789 local 10.0.0.11 nolearning $ bridge fdb show dev vxlan0 00:1a:4b:7c:2e:91 dst 10.0.0.12 self permanent # inner MAC → remote VTEP 00:00:00:00:00:00 dst 10.0.0.13 self permanent # all-zero = flood to this VTEP too
Note dstport 4789. Linux’s VXLAN driver predates the IANA assignment and still defaults to 8472 — two vendors, two ports, no packets, and nothing in the logs to say so.
tx-udp_tnl-segmentation) moves it into the NIC; without it, throughput falls off a cliff.tcpdump on the underlay shows UDP, not the traffic you care about.What tells every switch where a tenant’s addresses are?
VXLAN moves the packet. Something still has to decide where to send it — and that job is split across two routing protocols that must never be handed each other’s work.
| Route type | Carries | What it replaces |
|---|---|---|
| Type 2 | a MAC, optionally with its IP and its VNI | flood-and-learn, and ARP flooding with it |
| Type 3 | inclusive multicast — “I hold VNI 5000, send me its broadcast” | hand-built flood lists and underlay multicast groups |
| Type 5 | an IP prefix, in a VRF | routing between subnets rather than bridging within one |
| Type 1 · 4 | Ethernet auto-discovery and Ethernet segment routes | the guesswork in dual-homing a server to two leaves |
The last three metres, where a cable is a pointer, a stack is a struct, a routing table is a device and a tenant is a number in a map. Not all of it is host-scoped — one of these is a router concept and one is a cluster concept — but this is where you meet them as code.
How does a container’s network reach the outside world?
Below the NIC, the same problem repeats at a thousandth of the scale: two network stacks on one machine that need a link between them. The kernel’s answer is a pair of devices that are each other’s transmit path.
Create one and you get two devices at once — ip link add v0 type veth peer name v1 — because they are one kernel object with two faces. Put one end in a container’s namespace, leave the other in the host’s, and you have the standard pod interface. Delete either end and both disappear.
# in the pod — which host device is the other end? $ cat /sys/class/net/eth0/iflink 12 # on the host — that ifindex, and the namespace it reaches back into $ ip -d link show | grep -A1 '^12:' 12: veth9a3f21@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni0 link/ether 5a:c2:1f:0b:77:e4 link-netnsid 0
@if2 is the peer’s ifindex in the other namespace — the pod’s eth0. That cross-reference is the only reliable way to map a container interface to its host-side device when you are staring at tcpdump output on a busy node.
tcpdump reports “incorrect” checksums on a perfectly healthy pod.IFF_NO_QUEUE, so the default qdisc is noqueue. Shaping has to be added deliberately, on the host-side device, usually tbf plus an ifb for the other direction.rp_filter, accept_ra, forwarding are all properties of the new netns, not the device.br_netfilter loaded, bridged frames traverse iptables as well — surprising, required by kube-proxy, and a real cost per packet.bpf_redirect_peer() drops a pod-to-pod packet straight into the target namespace, and bpf_redirect_neigh() sends it to the wire — skipping the host’s routing and netfilter entirely. Same veth, half the work.What exactly does a container get its own copy of?
The kernel’s answer to tenancy is the most thorough one in this guide and the only one that puts nothing on the wire at all.
eth0 carrying 10.0.0.1, both can bind :80, and a conntrack entry in one is invisible in the other. The kernel object is struct net; every device, socket, route and netfilter hook holds a pointer to exactly one of them, and that pointer is the isolation.# a namespace is a reference, not a file — `ip netns add` bind-mounts one so it survives $ ls -l /proc/4821/ns/net lrwxrwxrwx 1 root root 0 → 'net:[4026533871]' # enter a running pod's namespace without its filesystem, its cgroups or its binaries $ nsenter -t 4821 -n ip route default via 10.244.3.1 dev eth0 10.244.3.0/24 dev eth0 proto kernel scope link src 10.244.3.47
That inode number is the namespace’s real identity. A namespace lives while something still references it — a running process, or a bind mount under /var/run/netns — and is destroyed when the last reference goes. On destruction its virtual devices are deleted and any physical device it held is returned to the initial namespace rather than disappearing with it.
localhost and a single port space.ss, tcpdump and conntrack -L all report the namespace you happen to be in, which is usually the wrong one.How does one Linux box hold several routing tables?
You have already met VRF, in section 05, as what a provider edge router keeps per tenant. It is not a host concept and has no scale of its own — a PE, a switch and a laptop all mean the same thing by it. What is specific here is the implementation, and what it buys: separate routing tables, but one firewall, one conntrack and one agent that can see all of them.
$ ip link add vrf-blue type vrf table 10 $ ip link set vrf-blue up $ ip link set eth1 master vrf-blue # eth1's traffic now resolves in table 10 $ ip rule show 0: from all lookup local 1000: from all lookup [l3mdev-table] # one rule covers every VRF 32766: from all lookup main 32767: from all lookup default
l3mdev device. Enslaving an interface to it makes the kernel resolve that interface’s traffic in the master’s table.[l3mdev-table] is a single rule standing in for all of them — add a hundred VRFs and it stays one rule.ip rule can select a table by source, input interface, fwmark or uid. VRF is the tidy special case that won.SO_BINDTODEVICE to the VRF device, or ip vrf exec vrf-blue <cmd>, which attaches a small BPF program to bind the process’s sockets for you.net.ipv4.tcp_l3mdev_accept=1 and the UDP equivalent — after which you must read the incoming device to know which tenant you are talking to.How do you write a firewall rule when addresses are recycled every minute?
Every mechanism so far assumes an address identifies a tenant. In a cluster it identifies nothing: pod addresses are handed out from a node’s range and reused within seconds.
10.244.3.47 protects whatever lands there in ninety seconds.host, remote-node, world for everything outside the cluster, and unmanaged.NetworkPolicy is opt-in. Isolation exists only while the policy is right.Everything above, applied once to a real building: 64 GPU nodes, 8 storage nodes, twelve racks, and tenants who must not be able to reach each other.
What does all of this look like in one actual building?
A concrete build, with the arithmetic shown. Numbers are worked from stated assumptions rather than quoted — change an assumption and the shape changes with it.
Twelve racks: 8 compute, 1 storage, 3 network and edge. Power sets that count — an 8-GPU node draws roughly 15 kW, so eight of them is a 120 kW liquid-cooled rack, near the practical ceiling today. Everything downstream, including how many rails the backend needs, follows from that kilowatt figure.
Half the switching never reaches those network racks. Thirty-one switches live in the racks they serve — two frontend leaves and one out-of-band switch per rack, plus the storage pair. What has to be central is everything that spans racks, and that is what fills the other three.
| In the network racks | Count | U each | U |
|---|---|---|---|
| Backend rail leaves | 8 | 2 | 16 |
| Backend spines | 8 | 2 | 16 |
| Frontend spines | 2 | 2 | 4 |
| OOB aggregation | 2 | 1 | 2 |
| Border routers | 2 | 2 | 4 |
| Firewalls | 2 | 2 | 4 |
| SoftGates | 4 | 2 | 8 |
| Patch panels | ~15 | 1–2 | ~25 |
| Cable managers, PDUs, blanking | — | — | ~10 |
| Total | 28 devices | — | ≈ 89U |
Two 42U racks give 84U, so this needs three — and the binding constraint is not the switches but the patching. 512 backend fibres converging on eight rail switches is the hard part of rail optimisation, and it is why those switches must be central: each one takes GPU i from every node in the cluster, so none of them can live in a compute rack. Power is comfortable either way — roughly 43 kW total, about 15 kW a rack across three.
| Fabric | Per node | Switching | Oversubscription | Tenant isolation |
|---|---|---|---|---|
| Backend | 8 × 400G | 8 rail leaves + 8 spines | 1:1 | whole nodes per tenant — or PKeys on InfiniBand |
| Frontend | 2 × 200G | 18 leaves + 2 spines | 3:1 | VPC → VRF, V-Net → L2VNI |
| Storage | 2 × 100G | 2 leaves | 2:1 | its own VRF, exported per tenant |
| Out-of-band | 1 × 1G | 9 ToR + 2 agg | n/a | operator only — never reachable by a tenant |
10.0.0.0/8 between tenants is free on the frontend and meaningless on the backend.The guide has now asked the same question at five scales and been given five different answers. Read together, they disagree about something fundamental.
Is there a single idea underneath all of it?
| VLAN | MPLS L3VPN | VXLAN | netns · VRF | identity | |
|---|---|---|---|---|---|
| identifier | VID | VPN label | VNI | struct net · table id | security identity |
| width | 12 bits | 20 bits | 24 bits | 32 bits | unbounded |
| assigned by | the operator | the egress PE | the orchestrator | the kernel | the control plane |
| on the wire? | a tag in the frame | a shim in the packet | a UDP header | nothing | sometimes |
| shared core sees | a tagged frame | a label | a UDP flow | the same kernel | an address |
| boundary is | a broadcast domain | a VRF table | a bridge domain | a namespace | a policy engine |
| crosses a machine? | yes, one L2 domain | yes, globally | yes, over any IP | no | yes, with the map |
| fails as | 4094, and spanning tree | a wrong route target | an MTU black hole | the wrong table bound | a stale identity |
Every term the guide leans on more than once. Each also appears as a dotted underline at its first use in any section — hover it there instead of coming here.