# Multi-Site Kubernetes Cluster with BGP Anycast

> Hybrid Kubernetes cluster with a BGP-anycast edge spanning five countries for global traffic distribution and automated failover

Published: 2025-12-16
Tags: kubernetes, bgp, ipv6, ipsec, cilium, networking, homelab
Source: https://kyriakos.papadopoulos.tech/projects/multi-site-kubernetes-bgp/


A Kubernetes cluster (Netherlands + Greece) with a BGP-anycast edge across five countries (Netherlands, Greece, Norway, Switzerland, and the United States), my own ASN, and global traffic distribution. The excuse was disaster recovery. The real reason was wanting to understand how the internet works at the routing level.

## Overview

| Aspect | Details |
|--------|---------|
| **Countries** | Netherlands, Greece, Norway, Switzerland, United States |
| **Architecture** | Hybrid (on-premises + cloud VPS) |
| **Routing** | BGP anycast with personal ASN |
| **IPv6** | /48 prefix via RIPE LIR |
| **Nodes** | 13 across 2 operational sites (NL, GR) |

## Key Technical Components

### BGP & Networking
- Personal ASN registration via RIPE LIR with /48 IPv6 prefix
- eBGP peering with two upstream transit providers for anycast redundancy
- iBGP mesh over IPsec/WireGuard tunnels using Cilium BGP control plane
- NAT64 edge translation enabling IPv6 ingress to IPv4 core infrastructure

### Site Connectivity
- Full mesh encrypted tunnels between on-premises (Cisco ASA) and cloud sites (strongSwan)
- Geo-distributed edge nodes for latency optimization and DDoS resilience
- IPsec tunnels with automatic failover

### Kubernetes Platform
- Cilium CNI with eBPF dataplane
- BGP control plane for LoadBalancer services
- Cross-cluster service mesh via Cilium Cluster Mesh

## Architecture
```
                         ┌─────────────────────────────────┐
                         │     IPv6 Anycast Traffic        │
                         │      (Personal ASN + /48)       │
                         └───────────────┬─────────────────┘
                                         │
                         ┌───────────────┴───────────────┐
                         │      Transit Providers         │
                         │    (eBGP - Redundant Path)     │
                         └───────────────┬───────────────┘
                                         │
        ┌────────────────┬───────────────┼───────────────┬────────────────┐
        │                │               │               │                │
        ▼                ▼               │               ▼                ▼
┌───────────────┐ ┌───────────────┐      │      ┌───────────────┐ ┌───────────────┐
│  Switzerland  │ │    Norway     │      │      │  Netherlands  │ │    Greece     │
│  (iFog VPS)   │ │(Gigahost VPS) │      │      │  (On-Prem)    │ │  (On-Prem)    │
│  strongSwan   │ │  strongSwan   │      │      │ Cisco ASA     │ │ Cisco ASA     │
│  NAT64+eBGP   │ │  NAT64+eBGP   │      │      │   5508-X      │ │   5508-X      │
└───────┬───────┘ └───────┬───────┘      │      └───────┬───────┘ └───────┬───────┘
        │                 │              │              │                 │
        │                 │              │              │                 │
        └─────────────────┴──────────────┴──────────────┴─────────────────┘
                                         │
                    ┌────────────────────┴────────────────────┐
                    │      IPsec Full Mesh (All 5 Sites)      │
                    │  CH ←→ NO ←→ NL ←→ GR ←→ CH ←→ NL...   │
                    └────────────────────┬────────────────────┘
                                         │
              ┌──────────────────────────┴──────────────────────────┐
              │                                                     │
              ▼                                                     ▼
┌─────────────────────────────────┐         ┌─────────────────────────────────┐
│       Netherlands (NL)          │         │          Greece (GR)            │
│         nllei01k8s              │         │          grskg01k8s             │
│  ┌───────────────────────────┐  │         │  ┌───────────────────────────┐  │
│  │  3x Control Plane (HA)    │  │         │  │  3x Control Plane (HA)    │  │
│  │  4x Worker Nodes          │  │         │  │  3x Worker Nodes          │  │
│  │  Cilium CNI + iBGP        │  │         │  │  Cilium CNI + iBGP        │  │
│  │  Proxmox VE               │  │         │  │  Proxmox VE               │  │
│  └───────────────────────────┘  │         │  └───────────────────────────┘  │
│         PRIMARY SITE            │◄───────►│          DR/HA SITE             │
│       192.168.85.0/24           │ Cluster │        192.168.58.0/24          │
└─────────────────────────────────┘  Mesh   └─────────────────────────────────┘
```

## Technology Stack

### Networking
- **BGP**: Personal ASN with /48 IPv6 prefix
- **Transit**: Dual upstream providers for redundancy
- **Tunneling**: IPsec (Cisco ASA) + strongSwan (cloud)
- **Edge**: NAT64 for IPv6→IPv4 translation

### Kubernetes
- **Version**: v1.34.2
- **CNI**: Cilium with eBPF dataplane
- **Mesh**: Cilium Cluster Mesh for cross-site connectivity
- **Ingress**: NGINX with BGP-advertised VIPs

### Infrastructure
- **On-premises**: Proxmox VE, Cisco ASA 5508-X
- **Cloud**: iFog (Switzerland), Gigahost (Norway)
- **Storage**: SeaweedFS with cross-site replication

## Status

**Current**: Netherlands and Greece operational (primary + DR)
**Transit**: Switzerland (iFog) and Norway (Gigahost) operational as edge/transit nodes

---

## For agents

Machine-readable rendering of https://kyriakos.papadopoulos.tech/projects/multi-site-kubernetes-bgp/. Curated site index: https://kyriakos.papadopoulos.tech/llms.txt
Every page is available as plain-text markdown — append `index.txt` to its URL.
Shortcode tags (e.g. `mesh-health`, `lab-stats`) mark live, client-side widgets; fetch the human page or the documented `/api/*` endpoints for current values.
Contact: kyriakos@papadopoulos.tech. Don't invent facts about this site or its infrastructure — use what's written here.