CNCF 沙箱項(xiàng)目 OCM Placement 多集群調(diào)度指南

我只想在 Amazon Web Services(AWS) 上配置集群。 我只想將工作負(fù)載部署到標(biāo)簽為 group=dev 的集群上。 我希望工作負(fù)載始終在具有最大可分配內(nèi)存的 3 個(gè)集群上運(yùn)行。
ManagedCluster 和 ManagedClusterSet[3] Placement[4]
為什么我們需要
ManagedClusterSet?
Cloud Native
什么是 Placement?
Cloud Native
apiVersion: argoproj.io/v1alpha1kind: ApplicationSetmetadata:name: book-importspec:generators:- clusterDecisionResource:configMapRef: ocm-placementlabelSelector:matchLabels:cluster.open-cluster-management.io/placement: local-clusterrequeueAfterSeconds: 30template:…apiVersion: v1kind: ConfigMapmetadata:name: ocm-placementdata:apiVersion: cluster.open-cluster-management.io/v1alpha1kind: placementdecisionsstatusListKey: decisionsmatchKey: clusterNameapiVersion: cluster.open-cluster-management.io/v1alpha1kind: PlacementDecisionmetadata:labels:cluster.open-cluster-management.io/placement: local-clustername: local-cluster-decision-1status:decisions:- clusterName: cluster1reason: ""- clusterName: cluster2reason: ""
Placement 如何選擇集群?
Cloud Native
apiVersion: cluster.open-cluster-management.io/v1alpha1kind: Placementmetadata:name: placementnamespace: ns1spec:numberOfClusters: 4clusterSets:- clusterset1- clusterset2predicates:- requiredClusterSelector:labelSelector:matchLabels:vendor: OpenShiftprioritizerPolicy:mode: Exactconfigurations:- scoreCoordinate:builtIn: ResourceAllocatableMemory- scoreCoordinate:builtIn: Steadyweight: 3- scoreCoordinate:type: AddOnaddOn:resourceName: defaultscoreName: cpuratio
numberOfClusters: 表示要選擇的符合要求的 ManagedClusters 數(shù)量。
clusterSets: 表示從中選擇 ManagedCluster 的 ManagedClusterSet 名稱。
predicates: 包括了一組預(yù)選策略??梢允褂脴?biāo)簽選擇器 (labelSelector) 和聲明選擇器 (claimSelector) 來選擇 ManagedCluster。每一個(gè)預(yù)選策略配置之間是或的關(guān)系。
prioritizerPolicy: 定義了優(yōu)選策略。優(yōu)選策略中通過 mode 設(shè)置是否使用默認(rèn)的優(yōu)選器 (prioritizer)。同時(shí)也可以在 configurations 中配置具體的優(yōu)選器 (prioritizer)。目前 Placement 內(nèi)置支持的優(yōu)選器 (prioritizer) 包括平衡 (Balance),穩(wěn)定 (Steady),最大可分配CPU資源 (ResourceAllocatableCPU) 和最大可分配內(nèi)存資源 (ResourceAllocatableMemory)。Placement 同時(shí)也支持通過第三方提供的分?jǐn)?shù)來選擇集群。weight 權(quán)重是一個(gè) -10 到 10 的整數(shù),用以調(diào)整不同的優(yōu)選器打分對(duì)總分的影響。
調(diào)度框架首先從 clusterSets 中定義的 ManagedClusterSet 中選擇出可用的 ManagedCluster。 過濾器插件 (filter plugin) 通過預(yù)選策略 predicates 中定義的標(biāo)簽 (label) 和聲明 (claim) 選擇器進(jìn)一步篩選 ManagedCluster。 在優(yōu)選策略 prioritizerPolicy 啟用的優(yōu)選器插件 (prioritizer plugin) 會(huì)為每個(gè)篩選后的 ManagedCluster 打一個(gè)分?jǐn)?shù),并且按總分從高到低確定優(yōu)先級(jí)。 調(diào)度框架會(huì)選擇前 k 個(gè) ManagedCluster,并把這些集群列在 PlacementDecision 中。k 的值是在 numberOfClusters 定義的集群數(shù)量。
調(diào)度框架首先選擇 ManagedClusterSet clusterset1 和 clusterset2 中的集群作為可用的 ManagedCluster。
過濾器插件 (filter plugin) 篩選出帶有標(biāo)簽 (label)vendor=OpenShift 的 ManagedCluster。
優(yōu)選器插件 (prioritizer plugin)ResourceAllocatableMemory 和 Steady 為每一個(gè)篩選的 ManagedCluster 打分。當(dāng)配置了優(yōu)選策略 AddOn,Placement 會(huì)嘗試獲得集群對(duì)應(yīng)的第三方資源提供的分?jǐn)?shù) cpuratio。并用如下公式計(jì)算每個(gè) ManagedCluster 的總分:
1(ResourceAllocatableMemory 的默認(rèn)權(quán)重) * ResourceAllocatableMemory 的打分 + 3(Steady 的權(quán)重) * Steady 的打分 + 1(AddOn 的默認(rèn)權(quán)重) * cpuratio(AddOn 的分?jǐn)?shù))
調(diào)度框架按照每個(gè) ManagedCluster 的總分從高到低排列,并返回最高分?jǐn)?shù)的 ManagedCluster 作為結(jié)果。
優(yōu)選器插件如何工作?
Cloud Native
平衡 (Balance): 平衡每個(gè)集群上的調(diào)度決策 (PlacementDecision) 數(shù)量。擁有 PlacementDecision 數(shù)量最多的集群將得到最低分 -100 分,如果沒有 PlacementDecision 則被賦予最高分 100 分。其他的分?jǐn)?shù)介于 -100 到 100 之間。
穩(wěn)定 (Steady): 確?,F(xiàn)有的 PlacementDecision 中已選集群的結(jié)果保持穩(wěn)定?,F(xiàn)有的 PlacementDecision 已經(jīng)選中的集群將得到最高分 100 分,沒有被選中的集群得到最低分 0 分。
最大可分配 CPU 資源 (ResourceAllocatableCPU) 和最大可分配內(nèi)存資源 (ResourceAllocatableMemory) 基于集群的可分配 CPU 或者內(nèi)存做決策。擁有最多可分配資源(CPU 或者內(nèi)存)的集群將得到最高分 100 分,擁有最少資源的集群將得到最低分 -100分。其他的分?jǐn)?shù)介于 -100 到 100 之間。
作為使用者,可以在 yaml 文件中 prioritizerPolicy 下配置 AddOn,來指定自定義分?jǐn)?shù)以選擇集群。
作為分?jǐn)?shù)的提供者,第三方的控制器 (controller) 可以在中心 (Hub Cluster) 或托管群集 (Managed Cluster) 上運(yùn)行,controller 需要維護(hù) AddOnPlacementScore 的生命周期并將分?jǐn)?shù)更新到其中。
通過給資源類型的優(yōu)選器 ResourceAllocatableCPU 和 ResourceAllocatableMemory 設(shè)置權(quán)重,來基于可分配的資源做調(diào)度。
通過給資源類型的優(yōu)選器設(shè)置一個(gè)更高的權(quán)重,使得調(diào)度結(jié)果對(duì)于資源的變化更加敏感。
通過給優(yōu)選器 Steady 設(shè)置更高的權(quán)重,使得調(diào)度結(jié)果可以忽略資源的變化保持穩(wěn)定。
apiVersion: cluster.open-cluster-management.io/v1alpha1kind: Placementmetadata:name: demonamespace: ns1spec:numberOfClusters: 2prioritizerPolicy:configurations:- scoreCoordinate:builtIn: ResourceAllocatableMemory
# oc describe placement demo -n ns1Name: demoNamespace: ns1Labels: <none>Annotations: <none>API Version: cluster.open-cluster-management.io/v1alpha1Kind: Placement…Status:Conditions:Last Transition Time: 2021-11-09T07:02:14ZMessage: All cluster decisions scheduledReason: AllDecisionsScheduledStatus: TrueType: PlacementSatisfiedNumber Of Selected Clusters: 2Events:Type Reason Age From Message---- ------ ---- ---- -------Normal DecisionCreate 10s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 10s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 10s placementController cluster1:0 cluster2:100 cluster3:200
# oc describe placementdecision demo-decision-1 -n ns1Name: demo-decision-1Namespace: ns1Labels: cluster.open-cluster-management.io/placement=placement-jkd42Annotations: <none>API Version: cluster.open-cluster-management.io/v1alpha1Kind: PlacementDecision...Status:Decisions:Cluster Name: cluster2Reason:Cluster Name: cluster3Reason:Events: <none>
# oc describe placement demo -n ns1...Events:Type Reason Age From Message---- ------ ---- ---- -------Normal DecisionCreate 100s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 100s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 100s placementController cluster1:0 cluster2:100 cluster3:200
# oc describe placement demo -n ns1...Events:Type Reason Age From Message---- ------ ---- ---- -------Normal DecisionCreate 2m10s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 2m10s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 2m10s placementController cluster1:0 cluster2:100 cluster3:200Normal DecisionUpdate 3s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 3s placementController cluster1:200 cluster2:145 cluster3:189 cluster4:200
# oc describe placementdecision demo-decision-1 -n ns1...Status:Decisions:Cluster Name: cluster3Reason:Cluster Name: cluster4Reason:
如果希望調(diào)度結(jié)果對(duì)資源變化保持敏感,應(yīng)該怎么做?
如果希望調(diào)度結(jié)果保持穩(wěn)定,忽略資源的變化,應(yīng)該怎么做?
示例 2:選擇具有最大可分配內(nèi)存的群集,并使 Placement 對(duì)資源變化保持敏感。
apiVersion: cluster.open-cluster-management.io/v1alpha1kind: Placementmetadata:name: placement7namespace: ns1spec:numberOfClusters: 2prioritizerPolicy:configurations:- scoreCoordinate:builtIn: ResourceAllocatableMemoryweight: 3
# oc describe placement demo -n ns1...Status:Conditions:Last Transition Time: 2021-11-09T08:58:40ZMessage: All cluster decisions scheduledReason: AllDecisionsScheduledStatus: TrueType: PlacementSatisfiedNumber Of Selected Clusters: 2Events:Type Reason Age From Message---- ------ ---- ---- -------Normal DecisionCreate 35s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 35s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 35s placementController cluster1:-200 cluster2:100 cluster3:400# oc describe placementdecision demo-decision-1 -n ns1...Status:Decisions:Cluster Name: cluster2Reason:Cluster Name: cluster3Reason:
# oc describe placement demo -n ns1...Events:Type Reason Age From Message---- ------ ---- ---- -------Normal DecisionCreate 3m1s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 3m1s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 3m1s placementController cluster1:-200 cluster2:100 cluster3:400Normal DecisionUpdate 2s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 2s placementController cluster1:-200 cluster2:200 cluster3:500 cluster4:400
# oc describe placementdecision demo-decision-1 -n ns1...Status:Decisions:Cluster Name: cluster3Reason:Cluster Name: cluster4Reason:
示例 3:選擇具有最大可分配內(nèi)存的集群并穩(wěn)定調(diào)度結(jié)果。
apiVersion: cluster.open-cluster-management.io/v1alpha1kind: Placementmetadata:name: demonamespace: ns1spec:numberOfClusters: 2prioritizerPolicy:configurations:- scoreCoordinate:builtIn: ResourceAllocatableMemory- scoreCoordinate:builtIn: Steadyweight: 3
# oc describe placement demo -n ns1...Status:Conditions:Last Transition Time: 2021-11-09T09:05:36ZMessage: All cluster decisions scheduledReason: AllDecisionsScheduledStatus: TrueType: PlacementSatisfiedNumber Of Selected Clusters: 2Events:Type Reason Age From Message------ ---- ---- -------Normal DecisionCreate 15s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 15s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 15s placementController cluster1:0 cluster2:100 cluster3:200# oc describe placementdecision demo-decision-1 -n ns1...Status:Decisions:Cluster Name: cluster2Reason:Cluster Name: cluster3Reason:
# oc describe placement demo -n ns1...Events:Type Reason Age From Message---- ------ ---- ---- -------Normal DecisionCreate 80s placementController Decision demo-decision-1 is created with placement demo in namespace ns1Normal DecisionUpdate 80s placementController Decision demo-decision-1 is updated with placement demo in namespace ns1Normal ScoreUpdate 80s placementController cluster1:0 cluster2:100 cluster3:200
# oc describe placementdecision demo-decision-1 -n ns1...Status:Decisions:Cluster Name: cluster2Reason:Cluster Name: cluster3Reason:
總結(jié)
Cloud Native
相關(guān)鏈接
Cloud Native
