使用s3(minio)為kubernetes提供pv存儲
我們可以通過csi使用s3為kubernetes提供pv存儲，當我們申請pvc時，會自動在s3上創(chuàng)建bucket。這里我們使用minio作為s3使用。
注: 不建議生產(chǎn)環(huán)境使用
要求
kubernetes集群需要滿足以下幾個條件：
Kubernetes 1.16+（CSI v1.0.0兼容性） Kubernetes必須允許特權(quán)容器 Docker守護程序必須允許共享掛載（systemd標志 MountFlags=shared）
安裝
首先創(chuàng)建一個secrets用來提供minio的憑據(jù)
apiVersion:?v1
kind:?Secret
metadata:
??name:?csi-driver-s3-secret
??namespace:?kube-system
stringData:
??accessKeyID:?"xxxxx"
??secretAccessKey:?"xxxxxx"
??#s3地址，如果是aws需要設(shè)置為https://s3..amazonaws.com
??endpoint:?https://minio.lishuai.fun
??＃如果不在S3上，請將其設(shè)置為“”
??region:?""
接下來安裝csi驅(qū)動
provisioner.yaml
apiVersion:?v1
kind:?ServiceAccount
metadata:
??name:?csi-provisioner-sa
??namespace:?kube-system
---
kind:?ClusterRole
apiVersion:?rbac.authorization.k8s.io/v1
metadata:
??name:?external-provisioner-runner
rules:
??-?apiGroups:?[""]
????resources:?["secrets"]
????verbs:?["get",?"list"]
??-?apiGroups:?[""]
????resources:?["persistentvolumes"]
????verbs:?["get",?"list",?"watch",?"create",?"delete"]
??-?apiGroups:?[""]
????resources:?["persistentvolumeclaims"]
????verbs:?["get",?"list",?"watch",?"update"]
??-?apiGroups:?["storage.k8s.io"]
????resources:?["storageclasses"]
????verbs:?["get",?"list",?"watch"]
??-?apiGroups:?[""]
????resources:?["events"]
????verbs:?["list",?"watch",?"create",?"update",?"patch"]
---
kind:?ClusterRoleBinding
apiVersion:?rbac.authorization.k8s.io/v1
metadata:
??name:?csi-provisioner-role
subjects:
??-?kind:?ServiceAccount
????name:?csi-provisioner-sa
????namespace:?kube-system
roleRef:
??kind:?ClusterRole
??name:?external-provisioner-runner
??apiGroup:?rbac.authorization.k8s.io
---
kind:?Service
apiVersion:?v1
metadata:
??name:?csi-provisioner-s3
??namespace:?kube-system
??labels:
????app:?csi-provisioner-s3
spec:
??selector:
????app:?csi-provisioner-s3
??ports:
????-?name:?dummy
??????port:?12345
---
kind:?StatefulSet
apiVersion:?apps/v1
metadata:
??name:?csi-provisioner-s3
??namespace:?kube-system
spec:
??serviceName:?"csi-provisioner-s3"
??replicas:?1
??selector:
????matchLabels:
??????app:?csi-provisioner-s3
??template:
????metadata:
??????labels:
????????app:?csi-provisioner-s3
????spec:
??????serviceAccount:?csi-provisioner-sa
??????containers:
????????-?name:?csi-provisioner
??????????image:?quay.io/k8scsi/csi-provisioner:v1.6.0
??????????args:
????????????-?"--provisioner=s3.csi.metal-stack.io"
????????????-?"--csi-address=$(ADDRESS)"
????????????-?"--v=4"
??????????env:
????????????-?name:?ADDRESS
??????????????value:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io/csi.sock
??????????imagePullPolicy:?"IfNotPresent"
??????????volumeMounts:
????????????-?name:?socket-dir
??????????????mountPath:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io
????????-?name:?csi-driver-s3
??????????image:?majst01/csi-driver-s3:v0.2.0
??????????args:
????????????-?"--endpoint=$(CSI_ENDPOINT)"
????????????-?"--nodeid=$(NODE_ID)"
????????????-?"--v=4"
??????????env:
????????????-?name:?CSI_ENDPOINT
??????????????value:?unix:///var/lib/kubelet/plugins/s3.csi.metal-stack.io/csi.sock
????????????-?name:?NODE_ID
??????????????valueFrom:
????????????????fieldRef:
??????????????????fieldPath:?spec.nodeName
??????????imagePullPolicy:?"IfNotPresent"
??????????volumeMounts:
????????????-?name:?socket-dir
??????????????mountPath:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io
??????volumes:
????????-?name:?socket-dir
??????????emptyDir:?{}
attacher.yaml
apiVersion:?v1
kind:?ServiceAccount
metadata:
??name:?csi-attacher-sa
??namespace:?kube-system
---
kind:?ClusterRole
apiVersion:?rbac.authorization.k8s.io/v1
metadata:
??name:?external-attacher-runner
rules:
??-?apiGroups:?[""]
????resources:?["secrets"]
????verbs:?["get",?"list"]
??-?apiGroups:?[""]
????resources:?["events"]
????verbs:?["get",?"list",?"watch",?"update"]
??-?apiGroups:?[""]
????resources:?["persistentvolumes"]
????verbs:?["get",?"list",?"watch",?"update"]
??-?apiGroups:?[""]
????resources:?["nodes"]
????verbs:?["get",?"list",?"watch"]
??-?apiGroups:?["storage.k8s.io"]
????resources:?["volumeattachments"]
????verbs:?["get",?"list",?"watch",?"update",?"patch"]
---
kind:?ClusterRoleBinding
apiVersion:?rbac.authorization.k8s.io/v1
metadata:
??name:?csi-attacher-role
subjects:
??-?kind:?ServiceAccount
????name:?csi-attacher-sa
????namespace:?kube-system
roleRef:
??kind:?ClusterRole
??name:?external-attacher-runner
??apiGroup:?rbac.authorization.k8s.io
---
#?needed?for?StatefulSet
kind:?Service
apiVersion:?v1
metadata:
??name:?csi-attacher-s3
??namespace:?kube-system
??labels:
????app:?csi-attacher-s3
spec:
??selector:
????app:?csi-attacher-s3
??ports:
????-?name:?dummy
??????port:?12345
---
kind:?StatefulSet
apiVersion:?apps/v1
metadata:
??name:?csi-attacher-s3
??namespace:?kube-system
spec:
??serviceName:?"csi-attacher-s3"
??replicas:?1
??selector:
????matchLabels:
??????app:?csi-attacher-s3
??template:
????metadata:
??????labels:
????????app:?csi-attacher-s3
????spec:
??????serviceAccount:?csi-attacher-sa
??????containers:
????????-?name:?csi-attacher
??????????image:?quay.io/k8scsi/csi-attacher:v2.2.0
??????????args:
????????????-?"--v=4"
????????????-?"--csi-address=$(ADDRESS)"
??????????env:
????????????-?name:?ADDRESS
??????????????value:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io/csi.sock
??????????imagePullPolicy:?"IfNotPresent"
??????????volumeMounts:
????????????-?name:?socket-dir
??????????????mountPath:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io
??????volumes:
????????-?name:?socket-dir
??????????hostPath:
????????????path:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io
????????????type:?DirectoryOrCreate
csi-s3.yaml
apiVersion:?v1
kind:?ServiceAccount
metadata:
??name:?csi-driver-s3
??namespace:?kube-system
---
kind:?ClusterRole
apiVersion:?rbac.authorization.k8s.io/v1
metadata:
??name:?csi-driver-s3
rules:
??-?apiGroups:?[""]
????resources:?["secrets"]
????verbs:?["get",?"list"]
??-?apiGroups:?[""]
????resources:?["nodes"]
????verbs:?["get",?"list",?"update"]
??-?apiGroups:?[""]
????resources:?["namespaces"]
????verbs:?["get",?"list"]
??-?apiGroups:?[""]
????resources:?["persistentvolumes"]
????verbs:?["get",?"list",?"watch",?"update"]
??-?apiGroups:?["storage.k8s.io"]
????resources:?["volumeattachments"]
????verbs:?["get",?"list",?"watch",?"update"]
---
kind:?ClusterRoleBinding
apiVersion:?rbac.authorization.k8s.io/v1
metadata:
??name:?csi-driver-s3
subjects:
??-?kind:?ServiceAccount
????name:?csi-driver-s3
????namespace:?kube-system
roleRef:
??kind:?ClusterRole
??name:?csi-driver-s3
??apiGroup:?rbac.authorization.k8s.io
---
kind:?DaemonSet
apiVersion:?apps/v1
metadata:
??name:?csi-driver-s3
??namespace:?kube-system
spec:
??selector:
????matchLabels:
??????app:?csi-driver-s3
??template:
????metadata:
??????labels:
????????app:?csi-driver-s3
????spec:
??????serviceAccount:?csi-driver-s3
??????hostNetwork:?true
??????containers:
????????-?name:?driver-registrar
??????????image:?quay.io/k8scsi/csi-node-driver-registrar:v1.3.0
??????????args:
????????????-?"--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
????????????-?"--v=4"
????????????-?"--csi-address=$(ADDRESS)"
??????????env:
????????????-?name:?ADDRESS
??????????????value:?/csi/csi.sock
????????????-?name:?DRIVER_REG_SOCK_PATH
??????????????value:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io/csi.sock
????????????-?name:?KUBE_NODE_NAME
??????????????valueFrom:
????????????????fieldRef:
??????????????????fieldPath:?spec.nodeName
??????????volumeMounts:
????????????-?name:?plugin-dir
??????????????mountPath:?/csi
????????????-?name:?registration-dir
??????????????mountPath:?/registration/
????????-?name:?csi-driver-s3
??????????securityContext:
????????????privileged:?true
????????????capabilities:
??????????????add:?["SYS_ADMIN"]
????????????allowPrivilegeEscalation:?true
??????????image:?majst01/csi-driver-s3:v0.2.0
??????????args:
????????????-?"--endpoint=$(CSI_ENDPOINT)"
????????????-?"--nodeid=$(NODE_ID)"
????????????-?"--v=4"
??????????env:
????????????-?name:?CSI_ENDPOINT
??????????????value:?unix:///csi/csi.sock
????????????-?name:?NODE_ID
??????????????valueFrom:
????????????????fieldRef:
??????????????????fieldPath:?spec.nodeName
??????????imagePullPolicy:?"IfNotPresent"
??????????volumeMounts:
????????????-?name:?plugin-dir
??????????????mountPath:?/csi
????????????-?name:?pods-mount-dir
??????????????mountPath:?/var/lib/kubelet/pods
??????????????mountPropagation:?"Bidirectional"
????????????-?name:?fuse-device
??????????????mountPath:?/dev/fuse
??????volumes:
????????-?name:?registration-dir
??????????hostPath:
????????????path:?/var/lib/kubelet/plugins_registry/
????????????type:?DirectoryOrCreate
????????-?name:?plugin-dir
??????????hostPath:
????????????path:?/var/lib/kubelet/plugins/s3.csi.metal-stack.io
????????????type:?DirectoryOrCreate
????????-?name:?pods-mount-dir
??????????hostPath:
????????????path:?/var/lib/kubelet/pods
????????????type:?Directory
????????-?name:?fuse-device
??????????hostPath:
????????????path:?/dev/fuse
psp.yaml
apiVersion:?policy/v1beta1
kind:?PodSecurityPolicy
metadata:
??name:?psp-s3
spec:
??allowPrivilegeEscalation:?true
??fsGroup:
????rule:?RunAsAny
??privileged:?true
??runAsUser:
????rule:?RunAsAny
??seLinux:
????rule:?RunAsAny
??supplementalGroups:
????rule:?RunAsAny
??hostNetwork:?true
??allowedHostPaths:?[]
??allowedCapabilities:
????-?"SYS_ADMIN"
??volumes:
??-?'*'
---
apiVersion:?rbac.authorization.k8s.io/v1
kind:?Role
metadata:
??name:?psp-s3-role
??namespace:?kube-system
rules:
-?apiGroups:
??-?policy
??resources:
??-?podsecuritypolicies
??resourceNames:
??-?psp-s3
??verbs:
??-?use
---
apiVersion:?rbac.authorization.k8s.io/v1
kind:?RoleBinding
metadata:
??name:?psp-rolebinding-s3
??namespace:?kube-system
roleRef:
??apiGroup:?rbac.authorization.k8s.io
??kind:?Role
??name:?psp-s3-role
subjects:
-?apiGroup:?""
??kind:?ServiceAccount
??name:?csi-attacher-sa
-?apiGroup:?""
??kind:?ServiceAccount
??name:?csi-driver-s3
-?apiGroup:?""
??kind:?ServiceAccount
??name:?csi-provisioner-sa
-?apiGroup:?""
??kind:?ServiceAccount
??name:?csi-resizer-sa
接下來創(chuàng)建storageclass
storageclass.yaml
---
kind:?StorageClass
apiVersion:?storage.k8s.io/v1
metadata:
??name:?csi-driver-s3
provisioner:?s3.csi.metal-stack.io
parameters:
??#?specify?which?mounter?to?use
??#?currently?only?s3fs?is?supported
??mounter:?s3fs
??csi.storage.k8s.io/provisioner-secret-name:?csi-driver-s3-secret
??csi.storage.k8s.io/provisioner-secret-namespace:?kube-system
??csi.storage.k8s.io/controller-publish-secret-name:?csi-driver-s3-secret
??csi.storage.k8s.io/controller-publish-secret-namespace:?kube-system
??csi.storage.k8s.io/node-stage-secret-name:?csi-driver-s3-secret
??csi.storage.k8s.io/node-stage-secret-namespace:?kube-system
??csi.storage.k8s.io/node-publish-secret-name:?csi-driver-s3-secret
??csi.storage.k8s.io/node-publish-secret-namespace:?kube-system
mounter有以下幾種配置參數(shù)：
S3不是真正的文件系統(tǒng)，因此這里有一些限制要考慮。根據(jù)所使用的安裝程序，您將具有不同級別的POSIX兼容性。另外，取決于您使用的是哪個S3存儲后端，并不總是可以保證一致性。
可以將驅(qū)動程序配置為使用以下安裝程序之一來安裝存儲桶：
rclone s3fs goofys s3backer
可以將安裝程序設(shè)置為存儲類中的參數(shù)。如果愿意，還可以為每個安裝程序創(chuàng)建多個存儲類。
根據(jù)您的使用情況，所有安裝程序都有不同的優(yōu)點和缺點。以下是一些可以幫助您選擇貼片機的特征：
rclone
幾乎完全兼容POSIX（取決于緩存模式） 可以使用任何S3客戶端正常查看文件
s3fs
POSIX的較大子集 可以使用任何S3客戶端正常查看文件 不支持追加或隨機寫入
goofys
POSIX兼容性弱 表現(xiàn)第一 可以使用任何S3客戶端正常查看文件 不支持追加或隨機寫入
s3backer（實驗性）
表示存儲在S3上的塊設(shè)備 允許使用真實的文件系統(tǒng) 其他S3客戶端無法讀取文件 支持附件 支持上傳前壓縮（此驅(qū)動程序尚未實現(xiàn)） 支持上傳前加密（此驅(qū)動程序尚未實現(xiàn)）
s3backer目前處于試驗階段，因為在Kubernetes節(jié)點或CSI Pod意外關(guān)閉的情況下，卷損壞可能很快發(fā)生。s3backer二進制文件未與普通docker映像捆綁在一起，以使其盡可能小。使用
使用
我們創(chuàng)建一個pvc并創(chuàng)建一個pod綁定這個pvc
apiVersion:?v1
kind:?PersistentVolumeClaim
metadata:
??name:?csi-s3-pvc
??namespace:?default
spec:
??accessModes:
??-?ReadWriteOnce
??resources:
????requests:
??????storage:?1Gi
??storageClassName:?csi-s3
---
apiVersion:?v1
kind:?Pod
metadata:
??name:?csi-s3-test-nginx
??namespace:?default
spec:
??containers:
???-?name:?csi-s3-test-nginx
?????image:?nginx
?????volumeMounts:
???????-?mountPath:?/var/lib/www/html
?????????name:?webroot
??volumes:
???-?name:?webroot
?????persistentVolumeClaim:
???????claimName:?csi-s3-pvc
???????readOnly:?false
我們創(chuàng)建后查看pvc已經(jīng)是bound狀態(tài)了
[root@master-01?sample]#?kubectl?get?pvc?
NAME????????????????STATUS???VOLUME?????????????????????????????????????CAPACITY???ACCESS?MODES???STORAGECLASS????AGE
csi-driver-s3-pvc???Bound????pvc-db7ab3e1-e3f4-4b4f-9877-c82051e60063???5Gi????????RWO????????????csi-driver-s3???6s
并且我們在minio上也可以看到這個bucket
我們進入pod內(nèi)的/var/lib/www/html目錄下創(chuàng)建一個文件
[root@master-01?sample]#?kubectl?exec?-it?csi-driver-s3-test-nginx??--?bash?
root@csi-driver-s3-test-nginx:/#?cd?/var/lib/www/html/
root@csi-driver-s3-test-nginx:/var/lib/www/html#?touch?`date?+"%Y-%m-%d"`.txt
root@csi-driver-s3-test-nginx:/var/lib/www/html#?ls
2021-01-07.txt
此時我們也可以在minio的相應(yīng)的bucket下看到該文件
CKA 認證培訓(xùn)
?點擊屏末?|?閱讀原文?|?即刻學(xué)習(xí)評論
圖片
表情