Kubernetes Operator Finalizers 使用

最近在寫 k8s Operator,在看示例的時(shí)候看到 controller 都會(huì)設(shè)置 Finalizers,今天來聊一聊 Finalizers 和相關(guān)實(shí)現(xiàn)。
Finalizers 允許 Operator 控制器實(shí)現(xiàn)異步的 pre-delete hook。比如你給 API 類型中的每個(gè)對(duì)象都創(chuàng)建了對(duì)應(yīng)的外部資源,你希望在 k8s 刪除對(duì)應(yīng)資源時(shí)同時(shí)刪除關(guān)聯(lián)的外部資源,那么可以通過 Finalizers 來實(shí)現(xiàn)。
Finalizers 是由字符串組成的列表,當(dāng) Finalizers 字段存在時(shí),相關(guān)資源不允許被強(qiáng)制刪除。存在 Finalizers 字段的的資源對(duì)象接收的第一個(gè)刪除請(qǐng)求設(shè)置 metadata.deletionTimestamp 字段的值, 但不刪除具體資源,在該字段設(shè)置后, finalizer 列表中的對(duì)象只能被刪除,不能做其他操作。
當(dāng) metadata.deletionTimestamp 字段非空時(shí),controller watch 對(duì)象并執(zhí)行對(duì)應(yīng) finalizers 的動(dòng)作,當(dāng)所有動(dòng)作執(zhí)行完后,需要清空 finalizers ,之后 k8s 會(huì)刪除真正想要?jiǎng)h除的資源。
Operator finalizers 使用
介紹了 Finalizers 概念,那么我們來看看在 Operator 中如何使用,在 Operator Controller 中,最重要的邏輯就是 Reconcile 方法,finalizers 也是在 Reconcile 中實(shí)現(xiàn)的。要注意的是,設(shè)置了 Finalizers 會(huì)導(dǎo)致 k8s 的 delete 動(dòng)作轉(zhuǎn)為設(shè)置 metadata.deletionTimestamp 字段,如果你通過 kubectl get 命令看到資源存在這個(gè)字段,則表示資源正在刪除(deleting)。
有以下幾點(diǎn)需要理解:
如果資源對(duì)象未被刪除且未設(shè)置 finalizers,則添加 finalizer并更新 k8s 資源對(duì)象; 如果正在刪除資源對(duì)象并且 finalizers 仍然存在于 finalizers 列表中,則執(zhí)行 pre-delete hook并刪除 finalizers ,更新資源對(duì)象; 由于以上兩點(diǎn),需要確保 pre-delete hook是冪等的。
kuberbuilder 示例
我們來看一個(gè) kubebuilder 官方示例:
func?(r?*CronJobReconciler)?Reconcile(req?ctrl.Request)?(ctrl.Result,?error)?{
????ctx?:=?context.Background()
????log?:=?r.Log.WithValues("cronjob",?req.NamespacedName)
????var?cronJob?batch.CronJob
????if?err?:=?r.Get(ctx,?req.NamespacedName,?&cronJob);?err?!=?nil?{
????????log.Error(err,?"unable?to?fetch?CronJob")
????????return?ctrl.Result{},?ignoreNotFound(err)
????}
????//?聲明?finalizer?字段,類型為字符串
????myFinalizerName?:=?"storage.finalizers.tutorial.kubebuilder.io"
????//?通過檢查?DeletionTimestamp?字段是否為0?判斷資源是否被刪除
????if?cronJob.ObjectMeta.DeletionTimestamp.IsZero()?{
????????//?如果為0?,則資源未被刪除,我們需要檢測(cè)是否存在?finalizer,如果不存在,則添加,并更新到資源對(duì)象中
????????if?!containsString(cronJob.ObjectMeta.Finalizers,?myFinalizerName)?{
????????????cronJob.ObjectMeta.Finalizers?=?append(cronJob.ObjectMeta.Finalizers,?myFinalizerName)
????????????if?err?:=?r.Update(context.Background(),?cronJob);?err?!=?nil?{
????????????????return?ctrl.Result{},?err
????????????}
????????}
????}?else?{
????????//?如果不為?0?,則對(duì)象處于刪除中
????????if?containsString(cronJob.ObjectMeta.Finalizers,?myFinalizerName)?{
????????????//?如果存在?finalizer?且與上述聲明的?finalizer?匹配,那么執(zhí)行對(duì)應(yīng)?hook?邏輯
????????????if?err?:=?r.deleteExternalResources(cronJob);?err?!=?nil?{
????????????????//?如果刪除失敗,則直接返回對(duì)應(yīng)?err,controller?會(huì)自動(dòng)執(zhí)行重試邏輯
????????????????return?ctrl.Result{},?err
????????????}
????????????//?如果對(duì)應(yīng)?hook?執(zhí)行成功,那么清空?finalizers,?k8s?刪除對(duì)應(yīng)資源
????????????cronJob.ObjectMeta.Finalizers?=?removeString(cronJob.ObjectMeta.Finalizers,?myFinalizerName)
????????????if?err?:=?r.Update(context.Background(),?cronJob);?err?!=?nil?{
????????????????return?ctrl.Result{},?err
????????????}
????????}
????????return?ctrl.Result{},?err
????}
}
func?(r?*Reconciler)?deleteExternalResources(cronJob?*batch.CronJob)?error?{
????//
????//?刪除?crobJob關(guān)聯(lián)的外部資源邏輯
????//
????//?需要確保實(shí)現(xiàn)是冪等的
}
func?containsString(slice?[]string,?s?string)?bool?{
????for?_,?item?:=?range?slice?{
????????if?item?==?s?{
????????????return?true
????????}
????}
????return?false
}
func?removeString(slice?[]string,?s?string)?(result?[]string)?{
????for?_,?item?:=?range?slice?{
????????if?item?==?s?{
????????????continue
????????}
????????result?=?append(result,?item)
????}
????return
}
cluster-api-provider-vsphere 實(shí)現(xiàn)
看完了示例,我們來找一個(gè)具體項(xiàng)目看看,cluster-api-provider-vsphere 是 cluster-api 相關(guān)項(xiàng)目,用于提供 vsphere 相關(guān)資源創(chuàng)建的 Operator,采用 kubebuilder 來實(shí)現(xiàn)的。
vspheremachine_controller.go 中實(shí)現(xiàn)了 Reconcile 方法:
//?Reconcile?ensures?the?back-end?state?reflects?the?Kubernetes?resource?state?intent.
func?(r?*VSphereMachineReconciler)?Reconcile(req?ctrl.Request)?(_?ctrl.Result,?reterr?error)?{
?...
?//?Always?close?the?context?when?exiting?this?function?so?we?can?persist?any?VSphereMachine?changes.
?defer?func()?{
??if?err?:=?machineContext.Patch();?err?!=?nil?&&?reterr?==?nil?{
???reterr?=?err
??}
?}()
?//?Handle?deleted?machines
?if?!vsphereMachine.ObjectMeta.DeletionTimestamp.IsZero()?{
??return?r.reconcileDelete(machineContext)
?}
?//?Handle?non-deleted?machines
?return?r.reconcileNormal(machineContext)
}
在 Reconcile 中檢測(cè)了 DeletionTimestamp 是否為0 ,如果不為0 ,則表示資源處于正在刪除中,那么來看下 reconcileDelete 實(shí)現(xiàn):
func?(r?*VSphereMachineReconciler)?reconcileDelete(ctx?*context.MachineContext)?(reconcile.Result,?error)?{
?ctx.Logger.Info("Handling?deleted?VSphereMachine")
?var?vmService?services.VirtualMachineService?=?&govmomi.VMService{}
????//?執(zhí)行刪除虛擬機(jī)邏輯
?vm,?err?:=?vmService.DestroyVM(ctx)
?if?err?!=?nil?{
????????//?如果刪除失敗,則直接返回錯(cuò)誤,controller?會(huì)自動(dòng)重試
??return?reconcile.Result{},?errors.Wrapf(err,?"failed?to?destroy?VM")
?}
?//?重新調(diào)度刪除虛擬機(jī)邏輯,直到虛擬機(jī)狀態(tài)處于?notfound?狀態(tài)
?if?vm.State?!=?infrav1.VirtualMachineStateNotFound?{
??ctx.Logger.V(6).Info("requeuing?operation?until?vm?state?is?reconciled",?"expected-vm-state",?infrav1.VirtualMachineStateNotFound,?"actual-vm-state",?vm.State)
??return?reconcile.Result{RequeueAfter:?config.DefaultRequeue},?nil
?}
?//?pre-delete?hook執(zhí)行成功,也就是上面的刪除虛擬機(jī)邏輯執(zhí)行成功,則清空?Finalizers
?ctx.VSphereMachine.Finalizers?=?clusterutilv1.Filter(ctx.VSphereMachine.Finalizers,?infrav1.MachineFinalizer)
?return?reconcile.Result{},?nil
}
可以看到整體邏輯與示例的使用是一致的,主要通過這種方式來達(dá)到 pre-delete hook 的效果。
k8s-initializer-finalizer-practice
在搜索相關(guān)資料的時(shí)候,看到有人在 SO 上問了如何使用的問題,其中有個(gè)回答中附上了一個(gè)練習(xí)項(xiàng)目,項(xiàng)目很小,很適合了解 Finalizers 概念。
相關(guān)邏輯如下:
}else{
?customdeployment:=obj.(*crdv1alpha1.CustomDeployment).DeepCopy()
?fmt.Println("Event..............................")
?if?customdeployment.DeletionTimestamp?!=?nil{
??//?check?if?it?has?finalizer
??if?customdeployment.GetFinalizers()!=nil{
???finalizers:=customdeployment.GetFinalizers()
???//?check?if?first?finalizer?match?with?deletepod.crd.emruz.com
???if?finalizers[0]=="deletepods.crd.emruz.com"{
????//
????_,err:=myutil.PatchCustomDeployment(c.clientset,customdeployment,?func(deployment?*crdv1alpha1.CustomDeployment)?*crdv1alpha1.CustomDeployment?{
?????//?delete?pods?under?this?deployment
?????err:=myutil.DeletePods(c.kubeclient,c.podLabel)
?????if?err!=nil{
??????fmt.Println("Failed?to?remove?all?pods.?Reason:?",err)
??????return?nil
?????}
?????//?pods?sucessfully?removed.?remove?the?finalizer
?????customdeployment.ObjectMeta=myutil.RemoveFinalizer(customdeployment.ObjectMeta)
?????return?customdeployment
????})
????if?err!=nil{
?????return?err
????}
???}
???????????}
總結(jié)
在開發(fā) Operator 時(shí),pre-delete hook 是一個(gè)很常見的需求,目前只發(fā)現(xiàn)了 Finalizers 適合實(shí)現(xiàn)這個(gè)功能,需要好好掌握。
參考鏈接
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#finalizers https://book.kubebuilder.io/reference/using-finalizers.html https://stackoverflow.com/questions/53057185/kubernetes-crd-finalizer
“原文鏈接:https://zdyxry.github.io/2019/09/13/Kubernetes-%E5%AE%9E%E6%88%98-Operator-Finalizers/
”
?點(diǎn)擊屏末?|?閱讀原文?|?即刻學(xué)習(xí)