LabHub

Blog

BGP Internet Routing Deep Dive 2025: AS, Peering, Path Selection, RPKI, Anycast — How the Internet Really Works

2026-04-15 · 14 min read · #bgp#internet-routing#as#peering#rpki#anycast#networking#iana#isp#hijacking

한국어English日本語

Intro: The Internet Is Not Actually One Network

What you call "the internet" is not a single network. It is a loose federation of 75,000+ independent networks. Each one is called an AS (Autonomous System).

When you reach Google from home Wi-Fi, packets cross dozens of ASes. Each AS tells the next, "that prefix goes this way." This runs on BGP (Border Gateway Protocol).

Why BGP Matters

BGP is as essential as TCP/IP but far less known.

1. Autonomous System: Internet Building Block

An AS is a set of IP networks under a single administrative entity:

Examples: ISPs (Comcast, KT, NTT), hyperscalers (Google, Amazon), universities, CDNs (Cloudflare, Akamai).

ASN Allocation

IANA manages ASNs and delegates to regional RIRs: ARIN (NA), RIPE NCC (EU/ME), APNIC (APAC), LACNIC (SA), AFRINIC (Africa).

Prefix Allocation

Google prefix examples:
- 8.8.8.0/24  (DNS)
- 74.125.0.0/16
- 142.250.0.0/15

/24 = 256 addresses, /16 = 65536 addresses. Each AS advertises its prefixes via BGP.

Current Scale (2025)

2. BGP Basics: Path Vector Protocol

BGP is unusual among routing protocols:

BGP Messages

Path Attributes

BGP UPDATE Example

UPDATE from 10.0.0.1:
  Withdrawn: []
  Attributes:
    ORIGIN = IGP
    AS_PATH = [65001, 7018, 15169]
    NEXT_HOP = 10.0.0.1
    LOCAL_PREF = 100
    MED = 50
  NLRI: [8.8.8.0/24, 8.8.4.0/24]

Session States

Idle → Connect → OpenSent → OpenConfirm → Established (exchange routes).

3. Peering and Transit

Two ways ASes connect:

Transit — customer-provider:

Peering — equal exchange:

Tier 1, 2, 3

Tier 1 (~12 networks): peering only, buys no transit. Level3, Cogent, AT&T, NTT, Telia, Deutsche Telekom, Tata.

Tier 2: buys some transit, peers a lot. Most ISPs.

Tier 3: mostly transit, limited peering.

Modern hyperscalers (Google, Microsoft, Facebook) built their own global backbones and sit effectively at Tier 1 level.

Peering Flavors

4. BGP Path Selection

BGP picks by policy, not shortest path. Why: money, politics, performance, trust.

Decision process (standard order):

  1. Highest LOCAL_PREF
  2. Shortest AS_PATH
  3. Lowest ORIGIN (IGP < EGP < Incomplete)
  4. Lowest MED (from same AS)
  5. eBGP over iBGP
  6. Lowest IGP cost to NEXT_HOP
  7. Oldest route
  8. Lowest Router ID

Most decisions end at steps 1-3.

LOCAL_PREF Economics

Customer route: LOCAL_PREF = 200 (revenue)
Peer route:     LOCAL_PREF = 100 (free)
Transit route:  LOCAL_PREF = 50  (cost)

AS_PATH Prepending

Repeat your own ASN to make a path look longer and less attractive:

Normal:    AS_PATH = [65001, 7018, 15169]
Prepended: AS_PATH = [65001, 65001, 65001, 7018, 15169]

Used for traffic engineering — primary/backup links, load balancing.

5. iBGP vs eBGP

Full Mesh Problem

N iBGP routers need N(N-1)/2 sessions. 100 routers → 4950 sessions.

Solutions:

iBGP Next-Hop Trap

iBGP does not modify NEXT_HOP. Internal routers must be able to reach the external peer's IP, or the route is invalid. Fix: next-hop-self on the edge router.

6. Convergence and Stability

7. BGP Hijacking and RPKI

BGP is trust-based — anyone can claim "I own this prefix."

Famous Incidents

RPKI: Resource Public Key Infrastructure

  1. RIRs issue certificates to ASes.
  2. AS creates a ROA (Route Origin Authorization) for each prefix.
  3. ROA = "AS 15169 may originate 8.8.8.0/24."
  4. Routers validate BGP announcements against ROAs.
  5. Unauthorized announcements are dropped.

Adoption (2025)

Limits

8. Anycast: One IP, Many Locations

Advertise the same IP from many places; BGP routes each user to the nearest.

1.1.1.1 (Cloudflare DNS)
Advertised from: US East, US West, London, Tokyo, ... 300+ POPs.

Benefits

Use Cases

Limits

HTTPS + anycast was historically awkward (TCP) — solved via sticky routing, and natively by QUIC / HTTP/3 with connection migration.

9. Historical BGP Outages

Common lessons: BGP is fragile, automation amplifies mistakes, peer validation is essential, RPKI must spread, keep out-of-band management.

10. BGP Operations in Practice

Full Table vs Default Route

Multi-homing

        Your AS
       /       \
   ISP A      ISP B

Redundancy, load balancing, negotiating leverage.

Route Filtering (Cisco example)

ip prefix-list FILTER-IN deny 0.0.0.0/0 le 7
ip prefix-list FILTER-IN deny 0.0.0.0/0 ge 25
ip prefix-list FILTER-IN permit 0.0.0.0/0 le 32

Block bogons (RFC 1918, unallocated) and martians.

RPKI Validation

rtr client rpki-server 192.0.2.10
route-map RPKI-CHECK permit 10
  match rpki valid
  set local-preference 200
route-map RPKI-CHECK permit 20
  match rpki not-found
  set local-preference 100
route-map RPKI-CHECK permit 30
  match rpki invalid
  drop

Looking Glass

Public BGP viewers: bgp.he.net, looking-glass.nl. Check how others see your announcements.

Debug Commands

Quiz

Q1. Why does BGP use "policy-based" routing instead of "shortest path"?

A. IGPs (OSPF, IS-IS) operate within a single administrative domain where shortest path is adequate. BGP connects tens of thousands of independently run ASes with conflicting interests.

Economics first: transit costs money; peering is free. An operator prefers the cheaper path even if longer. LOCAL_PREF encodes this — customer (200) > peer (100) > transit (50).

Performance vs hop count: 3 congested hops can be worse than 5 uncongested ones. Policy lets you override naive metrics.

Trust and politics: operators avoid suspect ASes, bypass hostile jurisdictions, comply with regulation.

Traffic engineering: AS_PATH prepending, MED, communities balance load and build primary/backup paths.

Valley-free routing: peering contracts say "only advertise my customer routes to you" — pure shortest path cannot express this.

BGP is "the internet is politics." It finds an economically and politically acceptable path, not a technically optimal one. That is a feature: 75,000 independent organizations only cooperate because each is allowed to encode its own interests.

Q2. What separates peering from transit, and why do Tier 1 ISPs only peer?

A. Transit = customer pays provider for full internet reachability. Peering = two ASes exchange traffic between their own customers, usually free.

Transit (rough): $500–2000/mo per Gbps in metro, much more in secondary markets. Peering (IXP): $500–5000/mo for the port; private peering adds a cross-connect fee (~$200/mo); traffic is free.

Tier 1 = defined as "AS that reaches the whole internet without buying transit." About 12–20 networks: AT&T, Verizon, Lumen (Level 3), Cogent, NTT, Telia, Deutsche Telekom, Tata.

They peer only because:

  1. Buying transit would disqualify them by definition.
  2. Settlement-free peering enforces mutual equality; paying would imply subordination.
  3. They already reach everyone via their own backbones and customers.
  4. Their business is selling transit to others — buying transit would cut margin.

Peering requires roughly symmetric traffic, geography, and scale. Asymmetric peering requests (Netflix pushing vastly more than it pulls) historically get rejected or converted to paid peering — like Netflix / Comcast 2014, which triggered the net neutrality debate.

Tier 1 relationships are brittle. Cogent vs Level 3 (2005): Level 3 depeered Cogent; for three weeks parts of the internet were partitioned. "One internet" depends on political agreement.

Hyperscalers (Google, Facebook, Netflix, Amazon) now run their own global networks and effectively act as Tier 1 peers. The power structure has shifted from classic ISPs to content networks.

Q3. How does BGP hijacking work, and how does RPKI defend against it?

A. BGP was designed in the 1980s on implicit trust. It does not verify that an advertised prefix really belongs to the originator.

Prefix hijacking: attacker announces someone else's prefix. Because BGP prefers more specific prefixes, announcing 8.8.8.0/25 beats Google's 8.8.8.0/24. Traffic now flows to the attacker.

AS path manipulation: attacker forges an AS_PATH to impersonate a relationship with the victim AS.

Route leak: accidentally relaying a peer's routes to a transit provider (2019 Verizon).

Incidents: 2008 Pakistan/YouTube, 2018 Amazon Route 53 crypto-wallet theft, 2021 Twitter.

RPKI (2011):

  1. RIRs issue certificates binding prefixes to ASNs.
  2. Owners publish ROAs: signed statements "AS X may originate prefix Y up to max-length N."
  3. Validators fetch the RPKI repository and feed routers.
  4. Routers check each BGP UPDATE against ROAs: Valid / Invalid / NotFound.
  5. Invalid routes are dropped.

Limits:

  • Origin-only — middle-path tampering still possible.
  • ~50% of prefixes covered in 2025.
  • Misconfigured ROAs can reject legitimate routes (Cloudflare 2019).

BGPsec would sign the full path but has near-zero deployment because of CPU and memory cost and the need for universal upgrade.

Complementary defenses: IRR-based filtering, BGP monitoring (BGPStream, RIPE RIS, Cloudflare Radar), MANRS (800+ participating ASes committing to best practices).

Lesson: BGP will remain partially insecure for a long time. Defense in depth — encryption, authentication, active monitoring — is mandatory for anything important.

Q4. How does Anycast deliver "one IP, many locations"?

A. Advertise the same prefix from multiple sites over BGP. Each client's ISP picks one route via normal BGP path selection — usually to the nearest site by AS hops.

Cloudflare: 1.1.1.0/24 advertised from 300+ POPs via the same AS 13335.
Korean user → Tokyo POP
UK user     → London POP
US user     → Virginia POP

BGP "closeness" is AS-hop based, not physical distance. Peering choices of the local ISP can put a user on a surprising POP.

Great for: DNS (1.1.1.1, 8.8.8.8), CDNs, DDoS absorption, NTP, QUIC/HTTP/3.

TCP historically struggled: path shifts mid-connection → RST. Solutions: sticky routing (short-lived flows tolerate it), session sync (rare, expensive), L4 load balancers behind the anycast VIP, or moving to QUIC with connection IDs that survive IP changes.

Cloudflare's design: every customer behind the same anycast IP range, every POP serves every customer, adding a POP auto-balances. Effect: sub-50ms global latency, automatic failover, DDoS distributed across the edge.

Tradeoffs: non-deterministic routing (hard to debug), needs BGP expertise, requires replicated data at every POP, international routing can look weird.

Anycast is a small idea with disproportionate consequences: "advertise from many places" lets a 40-year-old protocol power modern global services. Users see 1.1.1.1; BGP quietly chooses the best POP for them.

Q5. What really caused Facebook's 6-hour outage in October 2021?

A. 2021-10-04 15:40 UTC: a Facebook engineer ran a backbone capacity-assessment command. A bug in the config validation let a dangerous variant through, and the command removed all BGP routes reaching Facebook's DNS servers.

The DNS servers had a safety feature: if they detect they are isolated from the backbone, they withdraw their own BGP announcements so stale responses don't leak. Normally good; here every DNS server withdrew simultaneously. Facebook, Instagram, WhatsApp, Messenger all became unresolvable.

Cascading failures:

  1. Internal tools (VPN, SSO, mail, Workplace) all lived on facebook.com — engineers couldn't log in remotely.
  2. Data-center badge readers depended on Facebook's auth API. Staff couldn't physically enter either.
  3. Engineers coordinated on Twitter and Discord.

Recovery required on-site engineers, manual router reconfiguration, and staged BGP re-advertisement to avoid a traffic storm. Full restore at 21:05 UTC — about 6 hours.

Facebook's post-mortem improvements: stricter config validation, out-of-band management independent of BGP, gradual DNS withdraw, badge systems decoupled from the main service auth, internal tools independent of the public domain.

Industry lessons:

  1. Blast radius — one command should not destroy an entire organization.
  2. Dependency cycles — if your fix tools depend on your broken service, you can't fix it.
  3. BGP == existence — with BGP down, physically healthy servers are invisible to the internet.
  4. Change is the cause — most major outages trace to a recent change.
  5. Post-mortem culture — Facebook publishing the root cause publicly is what lets the industry learn.

Similar patterns: AWS S3 2017 (typo), Slack 2022 (network config), Google Cloud 2020 (auth). The pattern repeats: automation amplifies small mistakes into global incidents.

BGP is simultaneously the most essential and one of the most fragile layers of the internet.

Closing

Summary

  1. AS: internet building blocks — 75,000+.
  2. BGP: policy-based path vector protocol.
  3. Peering vs Transit: the economic structure of the internet.
  4. Path Selection: many-stage decision process.
  5. RPKI: cryptographic origin validation.
  6. Anycast: BGP-powered global distribution.
  7. Hijacking: consequence of trust-based BGP.
  8. Outages: BGP mistakes have outsized impact.

What BGP Teaches

BGP is an extreme case of distributed systems:

That it works at all after 40+ years is remarkable. Understanding BGP means understanding that "the internet" is tens of thousands of contracts, dozens of Tier 1 networks, hundreds of IXPs, millions of prefixes, thousands of BGP updates per second — held together by agreement, not by design.

Next time a packet lands on your screen, remember each hop involved money, politics, and policy. BGP is what makes that miracle routine — and also what makes it fragile.

References

Comments

No comments yet.

Sign in to leave a comment