Kubernetes Ceph RBD volume with CSI driver
In this post I will show you how can you use CEPH RBD with CSI driver for persistent storagi on Kubernetes.
Parst of the Kubernetes series
Parst of the Kubernetes series
- Part1a: Install K8S with ansible
- Part1b: Install K8S with kubeadm
- Part1c: Install K8S with kubeadm in HA mode
- Part2: Intall metal-lb with K8S
- Part2: Intall metal-lb with BGP
- Part3: Install Nginx ingress to K8S
- Part4: Install cert-manager to K8S
- Part5a: Use local persisten volume with K8S
- Part5b: Use ceph persisten volume with K8S
- Part5c: Use ceph CSI persisten volume with K8S
- Part5d: Use Project Longhorn as persisten volume with K8S
- Part5e: Use OpenEBS as persisten volume with K8S
- Part5f: vSphere persistent storage for K8S
- Part6a: Install k8s with IPVS mode
- Part6b: Install k8s with IPVS mode
- Part7: Use Helm with K8S
- Part8: Tillerless helm2 install
- Part9: Kubernetes Dashboard SSO
- Part10: Kuberos for K8S
- Part11: Gangway for K8S
- Part12: Velero Backup for K8S
The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage storage systems to Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing storage systems in Kubernetes. Bbefore we begin lets ensure that we have the following requirements:
- Kubernetes cluster v1.14+
- allow-privileged flag enabled for both kubelet and API server
Running Ceph cluster
git clone https://github.com/ceph/ceph-csi.git cd ceph-csi/deploy/rbd/kubernetes/v1.14+/ kubectl create -f csi-nodeplugin-rbac.yaml kubectl create -f csi-provisioner-rbac.yaml
nano csi-config-map.yaml --- apiVersion: v1 kind: ConfigMap data: config.json: |- [ { "clusterID": "k8s-ceph", "monitors": [ "192.168.1.31:6789", "192.168.1.32:6789", "192.168.1.33:6789" ] } ] metadata: name: ceph-csi-config kubectl create -fcsi-config-map.yaml
kubectl create -f csi-rbdplugin-provisioner.yaml kubectl create -f csi-rbdplugin.yaml
ceph auth get-key client.admin|base64 QVFDTDliVmNEb21I32SHoPxXNGhmRkczTFNtcXM0ZW5VaXlTZEE977== nano csi-rbd-secret.yaml --- apiVersion: v1 kind: Secret metadata: name: csi-rbd-secret namespace: default data: userID: admin userKey: QVFDTDliVmNEb21I32SHoPxXNGhmRkczTFNtcXM0ZW5VaXlTZEE977== nano rbd-csi-sc.yaml --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rbd provisioner: rbd.csi.ceph.com parameters: monitors: 192.168.1.31:6790,192.168.1.32:6790,192.168.1.33:6790 clusterID: k8s-ceph pool: rbd imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: default csi.storage.k8s.io/node-publish-secret-name: csi-rbd-secret csi.storage.k8s.io/node-publish-secret-namespace: default adminid: admin csi.storage.k8s.io/fstype: ext4 reclaimPolicy: Delete mountOptions: - discard kubectl create -f csi-rbd-secret.yaml kubectl create -f rbd-csi-sc.yaml kubectl get storageclass NAME PROVISIONER AGE csi-rbd rbd.csi.ceph.com 15s
nano raw-block-pvc.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: raw-block-pvc spec: accessModes: - ReadWriteMany volumeMode: Block resources: requests: storage: 1Gi storageClassName: csi-rbd kubectl create -f raw-block-pvc.yaml kubectl get pvc NAME STATUS VOLUME raw-block-pvc Bound pvc-fd66b4d6-757d-22e9-8f9e-4f86e2356a59