Use Cilium BGP integration with OPNsense

Kubernetes
Page content

In this tutorial I will show you how to install Cilium with BGP integration for Kubernetes.

In the upcoming release, version 1.13, Cilium will include a functionality which allow to advertise routes to Service IPs via BGP, so we didn’t need to install MetalLB.

Cilium version 1.13 is still in pre-release phase.

How does the full setup look like?

For this Demo I will use a pfsense in virtualbox and tree vm for kubernetes in the same host-only network.

vm nic ip mode
opnsense01 em1 192.168.0.200 bridged
opnsense01 em2 172.17.9.200 host-only
k8sm01 enp0s8 172.17.9.10 host-only
k8sm02 enp0s8 172.17.9.11 host-only
k8sm02 enp0s8 172.17.9.12 host-only

Install BGP to OPNsense

Go to System > Firmware > Plugins and install os-frr

Configure BGP on OPNsense

Go tp Routing > General and enable enable the plugin. Next go to Routing > BGP and enble, then add AS Number.

Enable BGP

Configure Neighbor

Go to Routing > BGP the switch to the Neighbor tab and add the following three neighbors.

Neighbor

Neighbor

Neighbor

Cilium configuration

BGP support is enabled by providing the BGP configuration via a ConfigMap and by setting a few Helm values. Otherwise, BGP is disabled by default.

helm install cilium cilium/cilium \
--set bgp.enabled=true

apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: opensense
spec:
  virtualRouters:
    - localASN: 64513
      neighbors:
        - peerASN: 64512
          peerAddress: 172.17.9.200
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: default
spec:
  cidrs:
    - cidr: 10.25.0.10-10.25.3.250
  disabled: false

Demo Time

Let’s create a demo application for testing.

nano test.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
  namespace: default
spec:
  selector:
    matchLabels:
      run: test-nginx
  replicas: 3
  template:
    metadata:
      labels:
        run: test-nginx
    spec:
      containers:
      - name: test-nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-nginx
  namespace: default
  labels:
    run: test-nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: TCP
  selector:
    run: test-nginx

kubectl apply -f bgpconfig.yaml
kubectl apply -f test.yaml

After a few moments, you can run this command to get the IP address:

$ kubectl describe service test-nginx | grep "LoadBalancer Ingress"
LoadBalancer Ingress:     10.25.0.11

Let’s check the address in a browser. If pfSense is you default gateway it will work perfectly, but in my demo enviroment I need to create a route to pfSense for this network on my host machine:

sudo route add -net 10.25.0.0/22 gw 172.17.9.200

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    600    0        0 wlan0
10.25.0.0       172.17.9.200    255.255.252.0   UG    0      0        0 vboxnet7
172.17.9.0      0.0.0.0         255.255.255.0   U     0      0        0 vboxnet7

Demo