Cron HPA動(dòng)態(tài)擴(kuò)縮容實(shí)踐

Kubernetes 默認(rèn) HPA 功能可實(shí)現(xiàn)基于Metric CPU 與 MEM 的使用率來進(jìn)行動(dòng)態(tài)擴(kuò)縮容，在實(shí)現(xiàn)工作中針對(duì)如果已知的突峰流量的場(chǎng)景（如指定時(shí)間段的業(yè)務(wù)推廣等）可能存在擴(kuò)容置后不及時(shí)的問題，為了實(shí)現(xiàn)此類場(chǎng)景可以通過類似 Crontab 定時(shí)機(jī)制來動(dòng)態(tài)擴(kuò)縮容。本次實(shí)踐是使用了一個(gè)開源的 FinOps 項(xiàng)目Crane ，項(xiàng)目在 k8s 默認(rèn)的 HPA基礎(chǔ)上封裝了更智能的 ehpa、 Analytics、tsp 等功能，如果有興趣可以進(jìn)一步了解。今天主要實(shí)踐了基于 cron 方式的ehpa 功能。

一、 安裝

k8s 集群內(nèi)安裝 Crane (推薦采用自定義安裝方式)

  1. clone 最新代碼并切到最新分支內(nèi)容
git clone https://github.com/gocrane/crane.git
cd crane

CRANE_LATEST_VERSION=$(curl -s https://api.github.com/repos/gocrane/crane/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")')
git checkout $CRANE_LATEST_VERSION
  1. 應(yīng)用 CRD 資源對(duì)象的創(chuàng)建
kubectl apply -f deploy/manifests
  1. 修改 craned 連接的 prometheus server 的地址配置
  vi deploy/craned/deployment.yaml 
  command:
    - /craned
    - --prometheus-address=http://<ipaddr>:<port>
  1. 應(yīng)用核心服務(wù)部署
kubectl apply -f deploy/craned 
kubectl apply -f deploy/metric-adapter

二、 配置

定義定時(shí)擴(kuò)縮調(diào)度策略配置
#配置實(shí)例，創(chuàng)建 cron-scale.yaml 
apiVersion: autoscaling.crane.io/v1alpha1
kind: EffectiveHorizontalPodAutoscaler
metadata:
  name: ehpa-cron-test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment               
    name: nginx                       #指定scale deployment對(duì)象名稱
  minReplicas: 1                      #最大復(fù)本數(shù)
  maxReplicas: 5                      #最小復(fù)本數(shù)
  scaleStrategy: Auto   
  crons:
    - name: "biz-push-cron-1"
      timezone: "Asia/Shanghai"       #時(shí)區(qū)定義
      description: "scale up"         #擴(kuò)容至3個(gè)POD復(fù)本
      start: "36 17 ? * *"            ##17:36~17:37 執(zhí)行scale up復(fù)本數(shù)為3
      end: "37 17 ? * *"
      targetReplicas: 3
    - name: "biz-push-cron-2"         #縮容至1個(gè)POD復(fù)本 
      timezone: "Asia/Shanghai"
      description: "scale down"
      start: "37 17 ? * *"
      end: "45 17 ? * *"              ##scale down復(fù)本數(shù)為1，注意start-end須>5m
      targetReplicas: 1      
      
# 應(yīng)用策略
kubectl  apply -f cron-scale.yaml

三、測(cè)試驗(yàn)證

  • 查看應(yīng)用策略狀態(tài)
[crane]# kubectl get hpa
NAME                  REFERENCE          TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
ehpa-ehpa-cron-test   Deployment/nginx   1/1 (avg)   1         5         1          3m
[root@itg-k8s-master01 crane]# kubectl get ehpa
NAME             STRATEGY   MINPODS   MAXPODS   SPECIFICPODS   REPLICAS   AGE
ehpa-cron-test   Auto       1         5                        1          3m3s
  • 時(shí)間到達(dá)觸發(fā)后狀態(tài)查看，擴(kuò)容到達(dá)三個(gè)
[crane]# kubectl get hpa
NAME                  REFERENCE          TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
ehpa-ehpa-cron-test   Deployment/nginx   1/1 (avg)   1         5         3          5m29s

[crane]# kubectl get ehpa
NAME             STRATEGY   MINPODS   MAXPODS   SPECIFICPODS   REPLICAS   AGE
ehpa-cron-test   Auto       1         5                        3          5m58s
  • 查看 pod craned 日志
I0624  1 effective_hpa_controller.go:40] Got ehpa default/ehpa-cron-test
I0624  1 effective_hpa_controller.go:139] Update EffectiveHorizontalPodAutoscaler status successful, ehpa default/ehpa-cron-test
  • 時(shí)間到達(dá)觸發(fā)后查看 ehpa 實(shí)例的狀態(tài)信息
# 第一次cron調(diào)度(biz-push-cron-1)
# kubectl get ehpa ehpa-cron-test -oyaml
Status:
  Conditions:
    Last Transition Time:  2022-06-29T09:36:07Z
    Message:               Effective HPA is ready
    Reason:                EffectiveHorizontalPodAutoscalerReady
    Status:                True
    Type:                  Ready
    Last Transition Time:  2022-06-29T09:36:07Z
    Message:               the HPA controller was able to update the target scale to 3
    Reason:                SucceededRescale
    Status:                True
    Type:                  AbleToScale
    Last Transition Time:  2022-06-29T09:36:07Z
    Message:               the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
    Reason:                ValidMetricFound
    Status:                True
    Type:                  ScalingActive
    Last Transition Time:  2022-06-29T09:36:07Z
    Message:               the desired count is within the acceptable range
    Reason:                DesiredWithinRange
    Status:                False
    Type:                  ScalingLimited
  Current Replicas:        1
  Expect Replicas:         3
---------------  
Status:
  Conditions:
    Last Transition Time:  2022-06-29T09:37:07Z
    Message:               Effective HPA is ready
    Reason:                EffectiveHorizontalPodAutoscalerReady
    Status:                True
    Type:                  Ready
    Last Transition Time:  2022-06-29T09:37:07Z
    Message:               recent recommendations were higher than current one, applying the highest recent recommendation
    Reason:                ScaleDownStabilized
    Status:                True
    Type:                  AbleToScale
    Last Transition Time:  2022-06-29T09:37:07Z
    Message:               the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
    Reason:                ValidMetricFound
    Status:                True
    Type:                  ScalingActive
    Last Transition Time:  2022-06-29T09:37:07Z
    Message:               the desired count is within the acceptable range
    Reason:                DesiredWithinRange
    Status:                False
    Type:                  ScalingLimited
  Current Replicas:        3
  Expect Replicas:         3
  

# (上次結(jié)束時(shí)間+5分鐘)第二次 cron 調(diào)度 (biz-push-cron-2)
Status:
  Conditions:
    Last Transition Time:  2022-06-29T09:41:53Z
    Message:               Effective HPA is ready
    Reason:                EffectiveHorizontalPodAutoscalerReady
    Status:                True
    Type:                  Ready
    Last Transition Time:  2022-06-29T09:41:53Z
    Message:               the HPA controller was able to update the target scale to 1
    Reason:                SucceededRescale
    Status:                True
    Type:                  AbleToScale
    Last Transition Time:  2022-06-29T09:41:53Z
    Message:               the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
    Reason:                ValidMetricFound
    Status:                True
    Type:                  ScalingActive
    Last Transition Time:  2022-06-29T09:41:53Z
    Message:               the desired count is within the acceptable range
    Reason:                DesiredWithinRange
    Status:                False
    Type:                  ScalingLimited
  Current Replicas:        3                         
  Expect Replicas:         1                             #觸發(fā)期望復(fù)本數(shù)

---------------  
Status:
  Conditions:
    Last Transition Time:  2022-06-29T09:42:08Z
    Message:               Effective HPA is ready
    Reason:                EffectiveHorizontalPodAutoscalerReady
    Status:                True
    Type:                  Ready
    Last Transition Time:  2022-06-29T09:42:08Z
    Message:               recommended size matches current size
    Reason:                ReadyForNewScale
    Status:                True
    Type:                  AbleToScale
    Last Transition Time:  2022-06-29T09:42:08Z
    Message:               the HPA was able to successfully calculate a replica count from external metric ehpa-cron-test(&LabelSelector{MatchLabels:map[string]string{autoscaling.crane.io/effective-hpa-uid: f2525179-a3bf-44b7-babe-11a71c73a252,},MatchExpressions:[]LabelSelectorRequirement{},})
    Reason:                ValidMetricFound
    Status:                True
    Type:                  ScalingActive
    Last Transition Time:  2022-06-29T09:42:08Z
    Message:               the desired count is within the acceptable range
    Reason:                DesiredWithinRange
    Status:                False
    Type:                  ScalingLimited
  Current Replicas:        1                                   # HPA 達(dá)成所期望復(fù)本數(shù)
  Expect Replicas:         1

～～～ Finish ～～～

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容