Kubernetes with external Ingress Controller with Haproxy and BGP

March 03, 2024

Page content

In this post I will show you how to nstall HAProxy Igress Controller on a separate VM instad of running it in the Kubernetes cluster as a pod. For this I will use cilium BGP pod CIDR export option.

Parts of the K8S Security Lab series

At the company I working wor we head to create a kubernetes cluster with the DMZ network integration, but I didn’t want to place the kubernetes cluster to the DMZ network. So I started to find a way to place only the Ingress Controller to a separate node and place only that to the DMZ. I found thet ther is an option with the HAProxy Igress Controller called external mode wher you can run the Ingress Controller not in a pod in the cluster but on a separate node.

Ciliium config

Enable BGP in cilium:

nano cilium-helm-values.yaml
...
bgpControlPlane:
  enabled: true
...

helm upgrade cilium cilium/cilium -f cilium-helm-values.yaml -n kube-system

# test ciliumBGP CRD
k api-resources | grep -i ciliumBGP

# If not exists delete the operator pod
k delete pod -n kube-system cilium-operator-768959858c-zjjnc

# now exists
k api-resources | grep -i ciliumBGP
ciliumbgppeeringpolicies    bgpp    cilium.io/v2alpha1                     false        CiliumBGPPeeringPolicy

I will use 3179 port as BGP port besause cilium ha no privilage for standerd 179

Annotate all the nodes to use port 3179

kubectl annotate node m102 cilium.io/bgp-virtual-router.65002="local-port=3179"
kubectl annotate node m102 cilium.io/bgp-virtual-router.65002="local-port=3179"
kubectl annotate node m102 cilium.io/bgp-virtual-router.65002="local-port=3179"

Ceate the Cilium BGP pearing policy:

nano cilium-bgp-policy.yaml
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumBGPPeeringPolicy
metadata:
 name: bgp-peering-policy
spec:
 virtualRouters:
 - localASN: 65001
   exportPodCIDR: true
   neighbors:
    - peerAddress: '192.168.56.13/32'
      peerASN: 65001

When localASN and peerASN are the same, iBGP peering is used. When localASN and peerASN are different, eBGP peering is used.

‘‘‘External Border Gateway Protocol (EBGP)''': EBGP is used between autonomous systems. It is used and implemented at the edge or border router that provides inter-connectivity for two or more autonomous system. It functions as the protocol responsible for interconnection of networks from different organizations or the Internet.

‘‘‘Internal Border Gateway Protocol (IBGP)''': IBGP is used inside the autonomous systems. It is used to provide information to your internal routers. It requires all the devices in same autonomous systems to form full mesh topology or either of Route reflectors and Confederation for prefix learning.

kubectl apply -f cilium-bgp-policy.yaml

Install BIRD

On the External Ingress COntroller’s VM we vill install the BRD BGP client to estabish connection betwean the VM and the K8S INternal network:

sudo apt-get install software-properties-common
sudo add-apt-repository -y ppa:cz.nic-labs/bird
sudo apt update
sudo apt install bird

nano /etc/bird/bird.conf

router id 192.168.56.15;
log syslog all;debug protocols all;

# cluster nodes (add all nodes)
protocol bgp m101 {
  local 192.168.56.15 as 65001;
  neighbor 192.168.56.141 port 3179 as 65001;
  import all;
  export all;
  multihop;
  graceful restart;
}
protocol bgp m102 {
  local 192.168.56.15 as 65001;
  neighbor 192.168.56.142 port 3179 as 65001;
  import all;
  export none;
  multihop;
  graceful restart;
}
protocol bgp m103 {
  local 192.168.56.15 as 65001;
  neighbor 192.168.56.143 port 3179 as 65001;
  import all;
  export none;
  multihop;
  graceful restart;
}

# Inserts routes into the kernel routing table
protocol kernel {
   scan time 20;
   import all;
   export all;
   persist;
}

# Gets information about network interfaces from the kernel
protocol device {
   scan time 56;
}

sudo systemctl enable bird
sudo systemctl restart bird

Test the BGP resoults

On cilium:

cilium bgp peers
Node        Local AS   Peer AS   Peer Address     Session State   Uptime   Family         Received   Advertised
m101        65001      65001     192.168.56.15    established     6s       ipv4/unicast   0          2
                                                                           ipv6/unicast   0          0
m102        65001      65001     192.168.56.15    established     2s       ipv4/unicast   0          2
                                                                           ipv6/unicast   0          0
m103        65001      65001     192.168.56.15    established     3s       ipv4/unicast   0          2
                                                                           ipv6/unicast   0          0

On VM:

sudo birdc show protocols
BIRD 1.6.8 ready.
name     proto    table    state  since       info
m101     BGP      master   up     11:37:24    Established
m102     BGP      master   up     11:37:28    Established
m103     BGP      master   up     11:37:27    Established
kernel1  Kernel   master   up     11:37:23
device1  Device   master   up     11:37:23

birdc show route
BIRD 1.6.8 ready.
10.244.3.0/24      via 192.168.56.141 on enp0s8 [m102 11:37:29] * (100/-) [i]
10.244.4.0/24      via 192.168.56.142 on enp0s8 [m101 11:37:25] * (100/-) [i]
10.244.5.0/24      via 192.168.56.143 on enp0s8 [m103 11:37:28] * (100/-) [i]

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    100    0        0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 enp0s3
_gateway        0.0.0.0         255.255.255.255 UH    100    0        0 enp0s3
10.244.3.0      192.168.56.141  255.255.255.0   UG    0      0        0 enp0s8
10.244.4.0      192.168.56.142  255.255.255.0   UG    0      0        0 enp0s8
10.244.5.0      192.168.56.143  255.255.255.0   UG    0      0        0 enp0s8
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 enp0s8

Install the ingress controller outside of your cluster

Copy the /etc/kubernetes/admin.conf kubeconfig file from the control plane server to this server and store it in the root user’s home directory. The ingress controller will use this to connect to the Kubernetes API.

sudo mkdir -p /root/.kube
sudo cp admin.conf /root/.kube/config
sudo chown -R root:root /root/.kube

kubectl create configmap -n default haproxy-kubernetes-ingress

HAProxy Kubernetes Ingress Controller is compatible with a specific version of HAProxy. Install the HAProxy package on your Linux distribution based on the table below. For Ubuntu and Debian, follow the install steps at haproxy.debian.net

Ingress controller version	Compatible HAProxy version
1.11	2.8
1.10	2.7
1.9	2.6
1.8	2.5
1.7	2.4

In my case I use Ingress controller version 1.11 and HAProxy version 2.8:

curl https://haproxy.debian.net/bernat.debian.org.gpg \
      | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
echo deb "[signed-by=/usr/share/keyrings/haproxy.debian.net.gpg]" \
      http://haproxy.debian.net bookworm-backports-2.8 main \
      > /etc/apt/sources.list.d/haproxy.list

apt-get update
apt-get install haproxy=2.8.\*

Stop and disable the HAProxy service.

sudo systemctl stop haproxy
sudo systemctl disable haproxy

Call the setcap command to allow HAProxy to bind to ports 80 and 443:

sudo setcap cap_net_bind_service=+ep /usr/sbin/haproxy

Download the ingress controller from the project’s GitHub Releases page Extract it and then copy it to the /usr/local/bin directory.

wget https://github.com/haproxytech/kubernetes-ingress/releases/download/v1.11.3/haproxy-ingress-controller_1.11.3_Linux_x86_64.tar.gz

tar -xzvf haproxy-ingress-controller_1.11.3_Linux_x86_64.tar.gz
sudo cp ./haproxy-ingress-controller /usr/local/bin/

Create the file /lib/systemd/system/haproxy-ingress.service and add the following to it:

cat << EOF > /lib/systemd/system/haproxy-ingress.service
[Unit]
Description="HAProxy Kubernetes Ingress Controller"
Documentation=https://www.haproxy.com/
Requires=network-online.target
After=network-online.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/haproxy-ingress-controller --external --default-ssl-certificate=ingress-system/default-cert --configmap=default/haproxy-kubernetes-ingress --program=/usr/sbin/haproxy --disable-ipv6 --ipv4-bind-address=0.0.0.0 --http-bind-port=80 --https-bind-port=443  --ingress.class=ingress-public
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

Enable and start the service.

sudo systemctl enable haproxy-ingress
sudo systemctl start haproxy-ingress

Demo Aapp

nano demo-app.yaml
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  annotations:
    ingressclass.kubernetes.io/is-default-class: "false"
  labels:
    app.kubernetes.io/instance: ingress-public
  name: ingress-public
spec:
  controller: haproxy.org/ingress-controller/ingress-public
---
apiVersion: apps/v1
kind: Deployment
metadata:
 labels:
   run: app
 name: app
spec:
 replicas: 5
 selector:
   matchLabels:
     run: app
 template:
   metadata:
     labels:
       run: app
   spec:
     containers:
     - name: app
       image: jmalloc/echo-server
       ports:
       - containerPort: 8080

---
apiVersion: v1
kind: Service
metadata:
 labels:
   run: app
 name: app
spec:
 selector:
   run: app
 ports:
 - name: port-1
   port: 80
   protocol: TCP
   targetPort: 8080

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: test-ingress
 namespace: default
spec:
  ingressClassName: ingress-public
  rules:
  - host: "example.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80

Parts of the K8S Security Lab series

Container Runetime Security

Advanced Kernel Security

Network Security

Secure Kubernetes Install

User Security

Image Security

Pod Security

Secret Security

Monitoring and Observability

Backup

Ciliium config

Install BIRD

Test the BGP resoults

Install the ingress controller outside of your cluster

Demo Aapp

Your support is our everlasting motivation, that cup of coffee is what keeps us going!

Your support is our everlasting motivation,
that cup of coffee is what keeps us going!