基于k8s手動部署rabbitmq集群
目錄
1、RabbitMQ介紹
2、RabbitMQ的持久化和鏡像隊列
3、RabbitMQ集群在k8s中的部署
3.1 環(huán)境介紹
3.2 創(chuàng)建configmap
3.3 創(chuàng)建service
3.4 創(chuàng)建rbac授權(quán)
3.5 創(chuàng)建statefulset
3.6 部署檢查
1、RabbitMQ介紹
RabbitMQ是實現(xiàn)了高級消息隊列協(xié)議AMQP的開源消息代理軟件（亦稱面向消息的中間件）。RabbitMQ服務(wù)器是用Erlang語言編寫的，而集群和故障轉(zhuǎn)移是構(gòu)建在開放電信平臺框架上的。AMQP：Advanced Message Queue，高級消息隊列協(xié)議。它是應(yīng)用層協(xié)議的一個開放標(biāo)準(zhǔn)，為面向消息的中間件設(shè)計，基于此協(xié)議的客戶端與消息中間件可傳遞消息，并不受產(chǎn)品、開發(fā)語言燈條件的限制
AMQP具有如下的特性：
可靠性 Reliablity：使用了一些機制來保證可靠性，比如持久化、傳輸確認(rèn)、發(fā)布確認(rèn)靈活的路由 Flexible Routing：在消息進(jìn)入隊列之前，通過Exchange來路由消息。對于典型的路由功能，Rabbit已經(jīng)提供了一些內(nèi)置的Exchange來實現(xiàn)。針對更復(fù)雜的路由功能，可以將多個Exchange綁定在一起，也通過插件機制實現(xiàn)自己的Exchange消息集群 Clustering：多個RabbitMQ服務(wù)器可以組成一個集群，形成一個邏輯Broker高可用 Highly Avaliable Queues：隊列可以在集群中的機器上進(jìn)行鏡像，使得在部分節(jié)點出問題的情況下隊列仍然可用多種協(xié)議 Multi-protocol：支持多種消息隊列協(xié)議，如STOMP、MQTT等多種語言客戶端 Many Clients：幾乎支持所有常用語言，比如Java、.NET、Ruby等管理界面 Management UI：提供了易用的用戶界面，使得用戶可以監(jiān)控和管理消息Broker的許多方面跟蹤機制 Tracing：如果消息異常，RabbitMQ提供了消息的跟蹤機制，使用者可以找出發(fā)生了什么插件機制 Plugin System：提供了許多插件，來從多方面進(jìn)行擴展，也可以編輯自己的插件
2、RabbitMQ的持久化和鏡像隊列
RabbitMQ持久化分為Exchange、Queue、Message
Exchange和Queue持久化：指持久化Exchange、Queue元數(shù)據(jù)，持久化的是自身，服務(wù)宕機Exchange和Queue自身就沒有了Message持久化：顧名思義就是把每一條消息體持久化，服務(wù)宕機，消息不丟失
RabbitMQ的隊列Queue鏡像，指master node在接受到請求后，會同步到其他節(jié)點上，以此來保證高可用。在confirm模式下，具體過程如下
clientpublisher發(fā)送消息 –> master node接到消息 –> master node將消息持久化到磁盤 –> 將消息異步發(fā)送給其他節(jié)點 –> master將ack返回給client publisher
3、RabbitMQ集群在k8s中的部署
將RabbitMQ以集群的方式部署在k8s中，前提是RabbitMQ的每個節(jié)點都能像傳統(tǒng)方式一樣進(jìn)行相互的服務(wù)發(fā)現(xiàn)。因此RabbitMQ在k8s集群中通過rabbitmq_peer_discovery_k8s plugin與k8s apiserver進(jìn)行交互，獲取各個服務(wù)的URL，且RabbitMQ在k8s集群中必須用statefulset和headless service進(jìn)行匹配
需要注意的是，rabbitmq_peer_discovery_k8s是RabbitMQ官方基于第三方開源項目rabbitmq-autocluster開發(fā)，對3.7.X及以上版本提供的Kubernetes下的對等發(fā)現(xiàn)插件，可實現(xiàn)rabbitmq集群在k8s中的自動化部署，因此低于3.7.X版本請使用rabbitmq-autocluster
3.1 環(huán)境介紹
本文部署的版本是3.8.3
默認(rèn)部署在default命名空間下，
持久化存儲為storageclass動態(tài)存儲，底層為nfs提供
鏡像地址rabbitmq:3.8.3-management
以下yaml參考自官方示例https://github.com/rabbitmq/diy-kubernetes-examples
3.2 創(chuàng)建configmap
01-rabbitmq-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: rabbitmq-cluster-config
namespace: default
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
enabled_plugins: |
[rabbitmq_management,rabbitmq_peer_discovery_k8s].
rabbitmq.conf: |
default_user = admin
default_pass = 123!@#
## Cluster formation. See https://www.rabbitmq.com/cluster-formation.html to learn more.
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
## Should RabbitMQ node name be computed from the pod's hostname or IP address?
## IP addresses are not stable, so using [stable] hostnames is recommended when possible.
## Set to "hostname" to use pod hostnames.
## When this value is changed, so should the variable used to set the RABBITMQ_NODENAME
## environment variable.
cluster_formation.k8s.address_type = hostname
## How often should node cleanup checks run?
cluster_formation.node_cleanup.interval = 30
## Set to false if automatic removal of unknown/absent nodes
## is desired. This can be dangerous, see
## * https://www.rabbitmq.com/cluster-formation.html#node-health-checks-and-cleanup
## * https://groups.google.com/forum/#!msg/rabbitmq-users/wuOfzEywHXo/k8z_HWIkBgAJ
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
## See https://www.rabbitmq.com/ha.html#master-migration-data-locality
queue_master_locator=min-masters
## See https://www.rabbitmq.com/access-control.html#loopback-users
loopback_users.guest = false
cluster_formation.randomized_startup_delay_range.min = 0
cluster_formation.randomized_startup_delay_range.max = 2
# default is rabbitmq-cluster's namespace
# hostname_suffix
cluster_formation.k8s.hostname_suffix = .rabbitmq-cluster.default.svc.cluster.local
# memory
vm_memory_high_watermark.absolute = 1GB
# disk
disk_free_limit.absolute = 2GB
部分參數(shù)說明：
enabled_plugins：聲明開啟的插件名 default_pass/default_pass：聲明用戶名和密碼（雖然有部分文章記錄可以通過環(huán)境變量的方式聲明，但是經(jīng)測試，針對此版本如果指定了 configmap即rabbitmq的配置文件，聲明的環(huán)境變量是沒有用的，都需要在配置文件中指定）cluster_formation.k8s.address_type：從 k8s返回的Pod容器列表中計算對等節(jié)點列表，這里只能使用主機名，官方示例中是ip，但是默認(rèn)情況下在k8s中pod的ip都是不固定的，因此可能導(dǎo)致節(jié)點的配置和數(shù)據(jù)丟失，后面的yaml中會通過引用元數(shù)據(jù)的方式固定pod的主機名。
更多參數(shù)請參考官方文檔，這里就不贅述了
3.3 創(chuàng)建service
02-rabbitmq-service.yaml
kind: Service
apiVersion: v1
metadata:
labels:
app: rabbitmq-cluster
name: rabbitmq-cluster
namespace: default
spec:
clusterIP: None
ports:
- name: rmqport
port: 5672
targetPort: 5672
selector:
app: rabbitmq-cluster
---
kind: Service
apiVersion: v1
metadata:
labels:
app: rabbitmq-cluster
name: rabbitmq-cluster-manage
namespace: default
spec:
ports:
- name: http
port: 15672
protocol: TCP
targetPort: 15672
selector:
app: rabbitmq-cluster
type: NodePort
上面定義了兩個Service，一個是rabbitmq的服務(wù)端口，一個是管理界面的端口，用戶外部訪問，這里通過NodePort方式進(jìn)行暴露
3.4 創(chuàng)建rbac授權(quán)
前面的介紹中提到了RabbitMQ通過插件與k8s apiserver交互獲得集群中節(jié)點相關(guān)信息，因此需要對其進(jìn)行RBAC授權(quán)
03-rabbitmq-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: rabbitmq-cluster
namespace: default
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rabbitmq-cluster
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rabbitmq-cluster
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rabbitmq-cluster
subjects:
- kind: ServiceAccount
name: rabbitmq-cluster
namespace: default
3.5 創(chuàng)建statefulset
RabbitMQ在k8s中作為一個有狀態(tài)應(yīng)用進(jìn)行部署，因此控制器類型為StatefulSet，yaml中還定義了pvc相關(guān)內(nèi)容
04-rabbitmq-cluster-sts.yaml
kind: StatefulSet
apiVersion: apps/v1
metadata:
labels:
app: rabbitmq-cluster
name: rabbitmq-cluster
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: rabbitmq-cluster
serviceName: rabbitmq-cluster
template:
metadata:
labels:
app: rabbitmq-cluster
spec:
containers:
- args:
- -c
- cp -v /etc/rabbitmq/rabbitmq.conf ${RABBITMQ_CONFIG_FILE}; exec docker-entrypoint.sh
rabbitmq-server
command:
- sh
env:
- name: TZ
value: 'Asia/Shanghai'
- name: RABBITMQ_ERLANG_COOKIE
value: 'SWvCP0Hrqv43NG7GybHC95ntCJKoW8UyNFWnBEWG8TY='
- name: K8S_SERVICE_NAME
value: rabbitmq-cluster
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: RABBITMQ_USE_LONGNAME
value: "true"
- name: RABBITMQ_NODENAME
value: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local
- name: RABBITMQ_CONFIG_FILE
value: /var/lib/rabbitmq/rabbitmq.conf
image: rabbitmq:3.8.3-management
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- rabbitmq-diagnostics
- status
# See https://www.rabbitmq.com/monitoring.html for monitoring frequency recommendations.
initialDelaySeconds: 60
periodSeconds: 60
timeoutSeconds: 15
name: rabbitmq
ports:
- containerPort: 15672
name: http
protocol: TCP
- containerPort: 5672
name: amqp
protocol: TCP
readinessProbe:
exec:
command:
- rabbitmq-diagnostics
- status
initialDelaySeconds: 20
periodSeconds: 60
timeoutSeconds: 10
volumeMounts:
- mountPath: /etc/rabbitmq
name: config-volume
readOnly: false
- mountPath: /var/lib/rabbitmq
name: rabbitmq-storage
readOnly: false
- name: timezone
mountPath: /etc/localtime
readOnly: true
serviceAccountName: rabbitmq-cluster
terminationGracePeriodSeconds: 30
volumes:
- name: config-volume
configMap:
items:
- key: rabbitmq.conf
path: rabbitmq.conf
- key: enabled_plugins
path: enabled_plugins
name: rabbitmq-cluster-config
- name: timezone
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
volumeClaimTemplates:
- metadata:
name: rabbitmq-storage
spec:
accessModes:
- ReadWriteMany
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 2Gi
3.6 部署檢查
? rabbitmq-cluster pwd
/Users/ssgeek/Documents/k8s-manifests/rabbitmq-cluster
? rabbitmq-cluster ls
01-rabbitmq-configmap.yaml 02-rabbitmq-service.yaml 03-rabbitmq-rbac.yaml 04-rabbitmq-cluster-sts.yaml
? rabbitmq-cluster kubectl apply -f .
configmap/rabbitmq-cluster-config created
service/rabbitmq-cluster created
service/rabbitmq-cluster-manage created
serviceaccount/rabbitmq-cluster created
role.rbac.authorization.k8s.io/rabbitmq-cluster created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster created
statefulset.apps/rabbitmq-cluster created
等待一段時間，查看創(chuàng)建的相關(guān)資源
? rabbitmq-cluster kubectl get po,sts -l app=rabbitmq-cluster
NAME READY STATUS RESTARTS AGE
pod/rabbitmq-cluster-0 1/1 Running 0 3m22s
pod/rabbitmq-cluster-1 1/1 Running 0 2m18s
pod/rabbitmq-cluster-2 1/1 Running 0 2m23s
NAME READY AGE
statefulset.apps/rabbitmq-cluster 3/3 3m25s
查看日志，從日志的最后部分觀察集群建立的狀態(tài)
? rabbitmq-cluster kubectl logs -f rabbitmq-cluster-0
'/etc/rabbitmq/rabbitmq.conf' -> '/var/lib/rabbitmq/rabbitmq.conf'
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: list of feature flags found:
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] drop_unroutable_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] empty_basic_get_metric
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] implicit_default_bindings
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] quorum_queue
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: [ ] virtual_host_metadata
2021-02-17 03:30:39.445 [info] <0.9.0> Feature flags: feature flag states written to disk: yes
2021-02-17 03:30:39.544 [info] <0.269.0> ra: meta data store initialised. 0 record(s) recovered
2021-02-17 03:30:39.547 [info] <0.274.0> WAL: recovering []
2021-02-17 03:31:10.676 [info] <0.313.0>
Starting RabbitMQ 3.8.3 on Erlang 22.3.4.1
Copyright (c) 2007-2020 Pivotal Software, Inc.
Licensed under the MPL 1.1. Website: https://rabbitmq.com
## ## RabbitMQ 3.8.3
## ##
########## Copyright (c) 2007-2020 Pivotal Software, Inc.
###### ##
########## Licensed under the MPL 1.1. Website: https://rabbitmq.com
Doc guides: https://rabbitmq.com/documentation.html
Support: https://rabbitmq.com/contact.html
Tutorials: https://rabbitmq.com/getstarted.html
Monitoring: https://rabbitmq.com/monitoring.html
Logs: <stdout>
Config file(s): /var/lib/rabbitmq/rabbitmq.conf
Starting broker...2021-02-17 03:31:10.678 [info] <0.313.0>
node : rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
home dir : /var/lib/rabbitmq
config file(s) : /var/lib/rabbitmq/rabbitmq.conf
cookie hash : H+IQL2spD4MDV4jPi7mMAg==
log(s) : <stdout>
database dir : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
2021-02-17 03:31:10.695 [info] <0.313.0> Running boot step pre_boot defined by app rabbit
...省略中間內(nèi)容
2021-02-17 03:31:13.273 [info] <0.824.0> Statistics database started.
2021-02-17 03:31:13.273 [info] <0.823.0> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-02-17 03:31:13.467 [info] <0.9.0> Server startup complete; 5 plugins started.
* rabbitmq_peer_discovery_k8s
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_management_agent
* rabbitmq_peer_discovery_common
completed with 5 plugins.
2021-02-17 03:32:30.060 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:32:31.264 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:33:31.280 [info] <0.566.0> node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up
2021-02-17 03:33:32.627 [info] <0.566.0> rabbit on node 'rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local' up
進(jìn)入到pod中通過客戶端查看集群狀態(tài)
? rabbitmq-cluster kubectl exec -it rabbitmq-cluster-0 bash
root@rabbitmq-cluster-0:/# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local ...
Basics
Cluster name: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
Disk Nodes
rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local
Running Nodes
rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local
Versions
rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local: RabbitMQ 3.8.3 on Erlang 22.3.4.1
Alarms
(none)
Network Partitions
(none)
Listeners
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-0.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-1.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rabbitmq-cluster-2.rabbitmq-cluster.default.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Feature flags
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: quorum_queue, state: enabled
Flag: virtual_host_metadata, state: enabled
通過NodePort訪問管理界面
kubectl get svc -l app=rabbitmq-cluster
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rabbitmq-cluster ClusterIP None <none> 5672/TCP 8m47s
rabbitmq-cluster-manage NodePort 10.1.239.191 <none> 15672:30888/TCP 8m47s
到這里，在k8s中手動部署一個RabbitMQ集群就完成啦~
參考：https://www.rabbitmq.com/cluster-formation.html https://github.com/rabbitmq/diy-kubernetes-examples
K8S 進(jìn)階訓(xùn)練營
點擊屏末 | 閱讀原文 | 即刻學(xué)習(xí)評論
圖片
表情