Use Cilium BGP integration with OPNsense
In this tutorial I will show you how to install Cilium with BGP integration for Kubernetes.
In the upcoming release, version 1.13, Cilium will include a functionality which allow to advertise routes to Service IPs via BGP, so we didn’t need to install MetalLB.
Cilium version 1.13 is still in pre-release phase.
How does the full setup look like?
For this Demo I will use a pfsense in virtualbox and tree vm for kubernetes in the same host-only network.
| vm | nic | ip | mode |
|---|---|---|---|
| opnsense01 | em1 | 192.168.0.200 | bridged |
| opnsense01 | em2 | 172.17.9.200 | host-only |
| k8sm01 | enp0s8 | 172.17.9.10 | host-only |
| k8sm02 | enp0s8 | 172.17.9.11 | host-only |
| k8sm02 | enp0s8 | 172.17.9.12 | host-only |
Install BGP to OPNsense
Go to System > Firmware > Plugins and install os-frr
Configure BGP on OPNsense
Go tp Routing > General and enable enable the plugin. Next go to Routing > BGP and enble, then add AS Number.
Configure Neighbor
Go to Routing > BGP the switch to the Neighbor tab and add the following three neighbors.
Cilium configuration
BGP support is enabled by providing the BGP configuration via a ConfigMap and by setting a few Helm values. Otherwise, BGP is disabled by default.
helm install cilium cilium/cilium \
--set bgp.enabled=true
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
name: opensense
spec:
virtualRouters:
- localASN: 64513
neighbors:
- peerASN: 64512
peerAddress: 172.17.9.200
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: default
spec:
cidrs:
- cidr: 10.25.0.10-10.25.3.250
disabled: false
Demo Time
Let’s create a demo application for testing.
nano test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-nginx
namespace: default
spec:
selector:
matchLabels:
run: test-nginx
replicas: 3
template:
metadata:
labels:
run: test-nginx
spec:
containers:
- name: test-nginx
image: nginx
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: test-nginx
namespace: default
labels:
run: test-nginx
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
selector:
run: test-nginx
kubectl apply -f bgpconfig.yaml
kubectl apply -f test.yaml
After a few moments, you can run this command to get the IP address:
$ kubectl describe service test-nginx | grep "LoadBalancer Ingress"
LoadBalancer Ingress: 10.25.0.11
Let’s check the address in a browser. If pfSense is you default gateway it will work perfectly, but in my demo enviroment I need to create a route to pfSense for this network on my host machine:
sudo route add -net 10.25.0.0/22 gw 172.17.9.200
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 600 0 0 wlan0
10.25.0.0 172.17.9.200 255.255.252.0 UG 0 0 0 vboxnet7
172.17.9.0 0.0.0.0 255.255.255.0 U 0 0 0 vboxnet7
