Self-hosted Load Balancer for bare metal Kubernetes

Page content

In this tutorial I will show you how to install Metal LB load balancer in BGP mode for Kubernetes.

Parts of the Kubernetes series

Metallb

Metallb is a fantastic bare metal-targeted operator for powering LoadBalancer types of services. It can work in two modes: Layer 2 and Border Gateway Protocol (BGP) mode. In layer 2 mode, one of the nodes advertises the load balanced IP (VIP) via ARP. This mode has two limitations: all the traffic goes through a single node VIP potentially limiting the bandwidth. The second limitation is a very slow failover. Detecting unhealthy nodes is a notoriously slow operation in Kubernetes which can take several minutes (5-10 minutes, which can be decreased with the node-problem-detector DaemonSet).

In BGP mode, Metallb advertises the VIP through BGP. It requires a BGP compatible router ath the network wher the kubernetes cluster is created.

I found that the layer 2 mode of Metallb is not a practical solution for production scenarios as it is typically not acceptable to have failover-induced downtimes in the order of minutes.

How does the full setup look like?

For this Demo I will use a pfsense in virtualbox and tree vm for kubernetes in the same host-only network.

vm nic ip mode
pfsense01 em1 192.168.0.200 bridged
pfsense01 em2 172.17.9.200 host-only
k8sm01 enp0s8 172.17.9.10 host-only
k8sm02 enp0s8 172.17.9.11 host-only
k8sm02 enp0s8 172.17.9.12 host-only

Issues with Calico

Simple BGP config with Calico don’t require anything special. However, if you are using Calico’s external BGP peering capability to advertise your cluster prefixes over BGP, and also want to use BGP in MetalLB, you will need to jump through some hoops.

Configuring pfSense and OpenBGPD

First we need to install OpenBGPD pcakage on pfSense. Go to System > Package Manager > Available Packages Then select OpenBGPD and Install it.

There is otger BGP compatible pckages in pfSense so make sure you DO NOT have the Quagga_OSPF or FRR packages installed. They directly conflict with each other.

No we need to configure BGP. Ther is a nice UI but I will use the Raw config for simplicity. Go to Services > OpenBGPD > Raw config

# This file was created by the package manager. Do not edit!

AS 64512
fib-update yes
listen on 172.17.9.200
router-id 172.17.9.200
network 10.25.0.0/22

neighbor 172.17.9.10 {
	remote-as 64513
    announce all
	descr "k8sm01"
}

neighbor 172.17.9.11 {
	remote-as 64513
    announce all
	descr "k8sm02"
}

neighbor 172.17.9.12 {
	remote-as 64513
    announce all
	descr "k8sm03"
}
  • AS 64512 - Thi is the Autonomous System Number of pfsense
  • listen on - This is the address that OpenBGPD should listen to BGP requests on. I highly recommend setting this to the same as the router-id IP address.
  • network - This is the network what you will use for advertise Load Balanced services.
  • neighbor $(kubernetes_worker_node_ip) - The ip of the Kubernetes host
  • remote-as 64513 - Thi is the Autonomous System Number of the neighbors. Same for all Kubernetes Node.
  • announce all - We need our nodes to be able to announce to the router their service IP addresses.

Configuring pfSense and Quagga_OSPF

If you prefer to use Quagga_OSPF Go to System > Package Manager > Available Packages Then select Quagga_OSPF and Install it.

To configure BGP with Quagga_OSPF. Go to Services > Quagga OSPFd > Raw config Edit SAVED bgpd.conf the save.

router bgp 64512
 bgp router-id 172.17.9.200
 neighbor 172.17.9.10 remote-as 64513
 neighbor 172.17.9.11 remote-as 64513
 neighbor 172.17.9.12 remote-as 64513
  network 10.25.0.0/22

Deploy MetalLB

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

Make a BGP config for MetalLB

nano bgpconfig.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 172.17.9.200
      peer-asn: 64512
      my-asn: 64513
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 10.25.0.10-10.25.3.250

Demo Time

Let’s create a demo application for testing.

nano test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
  namespace: default
spec:
  selector:
    matchLabels:
      run: test-nginx
  replicas: 3
  template:
    metadata:
      labels:
        run: test-nginx
    spec:
      containers:
      - name: test-nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx
  namespace: default
  labels:
    run: test-nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: test-nginx

kubectl apply -f bgpconfig.yaml
kubectl apply -f test.yaml

After a few moments, you can run this command to get the IP address:

$ kubectl describe service test-nginx | grep "LoadBalancer Ingress"
LoadBalancer Ingress:     10.25.0.11

Let’s check the address in a browser. If pfSense is you default gateway it will work perfectly, but in my demo enviroment I need to create a route to pfSense for this network on my host machine:

sudo route add -net 10.25.0.0/22 gw 172.17.9.200

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    600    0        0 wlan0
10.25.0.0       172.17.9.200    255.255.252.0   UG    0      0        0 vboxnet7
172.17.9.0      0.0.0.0         255.255.255.0   U     0      0        0 vboxnet7

Example image