---
title: Kubernetes Multi-Tenancy With vCluster
url: https://devopstales.github.io/kubernetes/vcluster/
date: 2021-08-03
keywords: kubernetes deployment, K3S, vCluster, Multi-Tenancy
---


In this post I will use vCluster to run virtual Kubernetes clusters inside a Kubernetes cluster.

<!--more-->

{{< content "/filedir/k3s.html" >}}


### What is vCluster

We all know about `k3d` is not, it is a lightweight wrapper to run `k3s` n docker. In that scenario a Kubernetes node is a container running on docker. Probably `k3sd` was the inspiration for `vCluster` but they take the idea to the next level. With `vCluster` you can run a `k3s` cluster in a single namespace This solution is similar the [kcp](https://github.com/kcp-dev/kcp) but they used the tools that already exists.  

![vcluster - Architecture](/img/include/vcluster-architecture.svg)

`vCluster` runs a component called `syncert` that is synchronize the Low-Level component from the `k3s` cluster like pods, services, ingress. So in reality the parent cluster will run these objects. 

### Getting Started

`vCluster` give you a cli to ease the task of installation:

```bash
curl -s -L "https://github.com/loft-sh/vcluster/releases/latest" | sed -nE 's!.*"([^"]*vcluster-linux-amd64)".*!https://github.com\1!p' | xargs -n 1 curl -L -o vcluster && chmod +x vcluster;
sudo mv vcluster /usr/local/bin;
```

In the background `vCluster CLI` use `helm` and `kubectl` to do the magic. So you can easily modify the configuration in the `values.yaml` file if you want.

```bash
kubectl create ns vcluster-1

vcluster create vcluster-1 -n vcluster-1
```

### Exposing vcluster

By default, vcluster is not reachable. To directly access vcluste you need to use one of the following methods:

* port-forwarding
* LoadBalancer service
* NodePort service
* Ingress

```bash
# Use --expose to create a vcluster with an LoadBalancer Service
vcluster create vcluster-1 -n vcluster-1 --expose 
```

I have a preinstalled `MetalLB` so the `LoadBalancer Service` will work perfectly for me, but if you want to use ingress you can do just fine.

```yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: vcluster-1
  namespace: vcluster-1
spec:
  rules:
  - host: vcluster-1.example.com
    http:
      paths:
      - backend:
          serviceName: vcluster-1
          servicePort: 443
        path: /
```

When you manually add an ingress or service you mast add the ip or the hostname to the certificate of the `k3s` cluster

```yaml
syncer:
  extraArgs:
  - --tls-san=my-vcluster.example.com
# or
syncer:
  extraArgs:
  - --tls-san=10.10.10.5
```

```bash
vcluster create vcluster-1 -n vcluster-1 -f values.yaml
```

### External Datastorage

In the default scenario `k3s` in the namespace will use SQLite as the Datastorage but `vCluster` can run `k3s` with all the other supported Datastorage:

* Embedded SQLite (default)
* PostgreSQL
* MySQL
* MariaDB
* etcd

```yaml
vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: https://etcd-host-1:2379,https://etcd-host-2:2379,https://etcd-host-3:2379
```

```yaml
vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: postgres://username:password@hostname:5432/k3s
```

```yaml
vcluster:
  env:
  - name: K3S_DATASTORE_ENDPOINT
    value: 'mysql://username:password@tcp(hostname:3306)/k3s'
  - name: K3S_DATASTORE_CERTFILE
    value: '/path/to/client.crt' 
  - name: K3S_DATASTORE_KEYFILE
    value: '/path/to/client.key' 
  volumeMounts:
    - mountPath: /data
      name: data
    - mountPath: /path/to
      name: datastore-tls
volumes:
- name: datastore-tls
  secret:
    secretName: my-datastore-secret
    items:
    - key: tls.key
      path: client.key
    - key: tls.crt
      path: client.crt
```

For this demo I will use the default Datastorage.

### Access the vcluster

```bash
vcluster connect vcluster-1 -n vcluster-1

export KUBECONFIG=./kubeconfig.yaml

kubectl get ns
```