Kubernetes Multi-Tenancy With vCluster

K3S
Page content

In this post I will use vCluster to run virtual Kubernetes clusters inside a Kubernetes cluster.

Parts of the K3S series

What is vCluster

We all know about k3d is not, it is a lightweight wrapper to run k3s n docker. In that scenario a Kubernetes node is a container running on docker. Probably k3sd was the inspiration for vCluster but they take the idea to the next level. With vCluster you can run a k3s cluster in a single namespace This solution is similar the kcp but they used the tools that already exists.

vcluster - Architecture

vCluster runs a component called syncert that is synchronize the Low-Level component from the k3s cluster like pods, services, ingress. So in reality the parent cluster will run these objects.

Getting Started

vCluster give you a cli to ease the task of installation:

curl -s -L "https://github.com/loft-sh/vcluster/releases/latest" | sed -nE 's!.*"([^"]*vcluster-linux-amd64)".*!https://github.com\1!p' | xargs -n 1 curl -L -o vcluster && chmod +x vcluster;
sudo mv vcluster /usr/local/bin;

In the background vCluster CLI use helm and kubectl to do the magic. So you can easily modify the configuration in the values.yaml file if you want.

kubectl create ns vcluster-1

vcluster create vcluster-1 -n vcluster-1

Exposing vcluster

By default, vcluster is not reachable. To directly access vcluste you need to use one of the following methods:

  • port-forwarding
  • LoadBalancer service
  • NodePort service
  • Ingress
# Use --expose to create a vcluster with an LoadBalancer Service
vcluster create vcluster-1 -n vcluster-1 --expose

I have a preinstalled MetalLB so the LoadBalancer Service will work perfectly for me, but if you want to use ingress you can do just fine.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: vcluster-1
  namespace: vcluster-1
spec:
  rules:
  - host: vcluster-1.example.com
    http:
      paths:
      - backend:
          serviceName: vcluster-1
          servicePort: 443
        path: /

When you manually add an ingress or service you mast add the ip or the hostname to the certificate of the k3s cluster

syncer:
  extraArgs:
  - --tls-san=my-vcluster.example.com
# or
syncer:
  extraArgs:
  - --tls-san=10.10.10.5

vcluster create vcluster-1 -n vcluster-1 -f values.yaml

External Datastorage

In the default scenario k3s in the namespace will use SQLite as the Datastorage but vCluster can run k3s with all the other supported Datastorage:

  • Embedded SQLite (default)
  • PostgreSQL
  • MySQL
  • MariaDB
  • etcd
vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: https://etcd-host-1:2379,https://etcd-host-2:2379,https://etcd-host-3:2379

vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: postgres://username:password@hostname:5432/k3s

vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: 'mysql://username:password@tcp(hostname:3306)/k3s'
  - name: K3S_DATASTORE_CERTFILE
    value: '/path/to/client.crt' 
  - name: K3S_DATASTORE_KEYFILE
    value: '/path/to/client.key' 
  volumeMounts:
    - mountPath: /data
      name: data
    - mountPath: /path/to
      name: datastore-tls
volumes:
- name: datastore-tls
  secret:
    secretName: my-datastore-secret
    items:
    - key: tls.key
      path: client.key
    - key: tls.crt
      path: client.crt

For this demo I will use the default Datastorage.

Access the vcluster

vcluster connect vcluster-1 -n vcluster-1

export KUBECONFIG=./kubeconfig.yaml

kubectl get ns