生產(chǎn)環(huán)境中，遇到coredns Pending問題，如下
# kubectl get pod -n kube-system |grep coredns
coredns-5479d79657-6gvvs 1/1 Running 3 28d
coredns-5479d79657-7l7tn 1/1 Running 3 28d
coredns-5479d79657-98qz8 1/1 Running 3 28d
coredns-5479d79657-bsx7h 1/1 Running 3 28d
coredns-5479d79657-btbl8 1/1 Running 3 28d
coredns-5479d79657-f6pwq 1/1 Running 3 28d
coredns-5479d79657-fbht5 0/1 Pending 0 28d
coredns-5479d79657-g7xhz 1/1 Running 3 28d
coredns-5479d79657-gw27m 1/1 Running 5 28d
coredns-5479d79657-h7g29 1/1 Running 3 28d
coredns-5479d79657-jqhj9 1/1 Running 2 28d
coredns-5479d79657-k94lh 1/1 Running 0 28d
coredns-5479d79657-kg5hv 1/1 Running 3 28d
coredns-5479d79657-khjdk 1/1 Running 3 28d
coredns-5479d79657-khp2l 0/1 Pending 0 2d22h
coredns-5479d79657-lwjb7 0/1 Pending 0 28d
coredns-5479d79657-p7ks6 1/1 Running 6 28d
coredns-5479d79657-p8c4v 1/1 Running 3 28d
coredns-5479d79657-tqdhz 0/1 Pending 0 7h15m
coredns-5479d79657-v6qfb 1/1 Running 3 28d
coredns-5479d79657-wcq7t 1/1 Running 3 28d
coredns-5479d79657-zbbck 0/1 Pending 0 28d
當(dāng)前業(yè)務(wù)并無異常，只是pod狀態(tài)不正常。刪除pending狀態(tài)的coredns，會(huì)立即啟動(dòng)一個(gè)，但依然是pending狀態(tài)。查看一個(gè)pending狀態(tài)的pod詳細(xì)描述，可以看到報(bào)錯(cuò)原因
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m18s (x2460351 over 21d) default-scheduler 0/17 nodes are available: 17 node(s) didn't match pod affinity/anti-affinity, 17 node(s) didn't satisfy existing pods anti-affinity rules.
大概意思是現(xiàn)有的17個(gè)節(jié)點(diǎn)不滿足節(jié)點(diǎn)親和性，所以pod無法運(yùn)行。
# kubectl get pod -n kube-system |grep coredns |wc -l
22
# kubectl get pod -n kube-system |grep coredns |grep Pending |wc -l
5
當(dāng)前k8s集群里有17個(gè)node，coredns啟動(dòng)了22個(gè)，有5個(gè)為Pending狀態(tài)。結(jié)合當(dāng)前業(yè)務(wù)正常的情況猜測節(jié)點(diǎn)親和性設(shè)置為每個(gè)節(jié)點(diǎn)只能運(yùn)行一個(gè)coredns，于是有5個(gè)pod在其節(jié)點(diǎn)上由于已經(jīng)有運(yùn)行的coredns pod，無法運(yùn)行，只能為Pending狀態(tài)。查看節(jié)點(diǎn)親和性。
# kubectl get ep -n kube-system
NAME ENDPOINTS AGE
coredns 10.233.64.116:53,10.233.65.138:53,10.233.66.25:53 + 48 more... 33d
kube-controller-manager <none> 33d
kube-scheduler <none> 33d
kubernetes-dashboard 10.233.67.14:8443 33d
prometheus-operator-coredns 10.233.64.116:9153,10.233.65.138:9153,10.233.66.25:9153 + 14 more... 33d
prometheus-operator-kube-controller-manager <none> 33d
prometheus-operator-kube-etcd <none> 33d
prometheus-operator-kube-scheduler <none> 33d
prometheus-operator-kubelet 172.29.11.10:10255,172.29.11.12:10255,172.29.11.14:10255 + 48 more... 33d
tiller-deploy 10.233.66.24:44134 33d
# kubectl edit deployment coredns -n kube-system
//只關(guān)注親和性/反親和性設(shè)置這一段
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: In
values:
- ""
weight: 100
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
k8s-app: coredns
topologyKey: kubernetes.io/hostname
集群中node節(jié)點(diǎn)是17個(gè)，為什么coredns設(shè)置為22個(gè)？只好先看下副本管理器中coredns設(shè)置的副本數(shù)。查看舊版本的k8s副本管理器用kubectl get rc -n kube-system，而比較新的版本用rs代替rc。
# kubectl get rs -n kube-system
NAME DESIRED CURRENT READY AGE
coredns-5479d79657 22 22 17 28d
dns-autoscaler-55944959bd 1 1 1 28d
kubernetes-dashboard-86b759667 1 1 1 28d
tiller-deploy-597b9b5f7c 1 1 1 28d
看到副本管理器中確實(shí)設(shè)定了副本數(shù)位22，先將副本數(shù)改為17，觀察Pending狀態(tài)的pod是否會(huì)被刪除。
# kubectl edit rs coredns-5479d79657 -n kube-system
//這個(gè)命令可以修改coredns-5479d79657這個(gè)rs的配置，按照文檔只修改spec.replicas的值
spec:
replicas: 17 //改為跟node數(shù)一致
selector:
matchLabels:
k8s-app: coredns
pod-template-hash: 5479d7965
//修改完畢保存退出
replicaset.extensions/coredns-5479d79657 edited
但是保存此配置后，刪除Pending狀態(tài)的pod，還是會(huì)自動(dòng)啟動(dòng)一個(gè)，總數(shù)并沒有改變。使用命令修改副本數(shù)：
# kubectl scale rs coredns-5479d79657 -n kube-system --replicas=17
replicaset.extensions/coredns-5479d79657 scaled
# kubectl get pod -n kube-system |grep coredns |wc -l
22
提示修改成功，但coredns pod數(shù)量還是沒有改變。
嘗試修改deployments
kubectl edit deployments coredns -n kube-system
//只修改spec.replicas的值
spec:
progressDeadlineSeconds: 2147483647
replicas: 17 //修改為與node數(shù)量一致
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: coredns
保存退出后，發(fā)現(xiàn)pod數(shù)量依然是22個(gè)。
使用patch修改deployment副本數(shù)，結(jié)果pod數(shù)量還是不變。
kubectl patch deployment coredns -p '{"spec":{"replicas":17}}' -n kube-system
由此可以推測coredns數(shù)量由某個(gè)進(jìn)程或配置管理，不受rs、deployments管理。
這時(shí)注意到dns-autoscaler這個(gè)deployment，然后聯(lián)想到cluster-autoscaler。CA（cluster-autoscaler）是用來彈性伸縮kubernetes集群的，dns-autoscaler應(yīng)該是彈性伸縮coredns這個(gè)pod集群的。
# kubectl get deployment -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
coredns 22 22 22 22 33d
dns-autoscaler 1 1 1 1 33d
kubernetes-dashboard 1 1 1 1 33d
tiller-deploy 1 1 1 1 33d
為了驗(yàn)證猜想，先停掉dns-autoscaler，再將pod數(shù)量調(diào)整為17個(gè)。
# kubectl scale deployment --replicas=0 dns-autoscaler -n kube-system
deployment.extensions/dns-autoscaler scaled
# kubectl patch deployment coredns -p '{"spec":{"replicas":17}}' -n kube-system
deployment.extensions/coredns patched
# kubectl get pod -n kube-system |grep coredns |wc -l
17
再查看coredns pod數(shù)量，已經(jīng)變?yōu)?7個(gè)， 而且全都是running狀態(tài)，問題解決。后續(xù)再研究下dns-autoscaler為什么會(huì)把coredns pod目標(biāo)數(shù)量設(shè)定為22個(gè)，怎么修改這個(gè)預(yù)設(shè)數(shù)量。
