kubernetes:v1.12.2
docker:18.06.1-ce
helm:v2.12.0

通過helm刪除服務(wù)后，pods無法刪除:

~]#  helm del  --purge simp-ser           
release "simp-ser" deleted  //刪除成功
~]#  helm install simp-ser --name=simp-ser
Error: release simp-ser failed: object is being deleted: persistentvolumes "simp-ser" already exists //創(chuàng)建時報錯，pvc已經(jīng)存在

突如其來的報錯讓我有點懵，因為定義的這個pvc是跟隨helm服務(wù)一起刪除，我趕緊去看了下pvc服務(wù):

~]#   kubectl get pvc
simp-ser                   Terminating   simp-ser         300Mi      RWO,RWX                       3d21h//正在刪除

pvc正在刪除中，但是一直沒有刪除成功，因為關(guān)聯(lián)了pod，我又查看了相關(guān)pod：

~]# kubectl get pods -o wide
simp-ser-6b8f54bb64-tdxlw     1/1     Terminating   0          3d21h   10.244.1.71    k8s.node1   <none>//也是正在刪除
~]#  kubectl describe pods simp-ser-6b8f54bb64-tdxlw
...
 Container ID:  docker://70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1
...
 Warning  FailedKillPod  13m (x324 over 21h)  kubelet, k8s.node1  error killing pod: [failed to "KillContainer" for "simp-ser" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
, failed to "KillPodSandbox" for "0e9a9fa9-6328-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
...
] //pod內(nèi)的容器是70494faefcf...

看的出來是問題出在了node節(jié)點上，繼續(xù)查看node的狀態(tài)：

// 查看node是否正常
~]# kubectl get node 
k8s.node1    Ready      <none>   146d   v1.12.2
//切換到node節(jié)點查看kubelet 狀態(tài)
~]#systemctl status kubelet
... 
Active: active (running) since 五 2019-03-29 10:07:30 CST; 3 weeks 5 days ago.//kubelet狀態(tài)也是正常的
...
//查看kubelet日志
~]# journalctl -u  kubelet -f 
···
4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.155788    2147 kuberuntime_manager.go:810] Failed to stop sandbox {"docker" "ffc1eccb5b0ba196fe86f9987632194ee3c2e63372adb2d283a05ac0c2e68f4d"}
4月 24 11:08:48 k8s.node1 kubelet[2147]: E0424 11:08:48.155934    2147 kubelet.go:1551] error killing pod: [failed to "KillContainer" for "simp-layui" with KillContainerError: "rpc error: code = Unknown desc = operation timeout: context deadline exceeded"
4月 24 11:08:48 k8s.node1 kubelet[2147]: , failed to "KillPodSandbox" for "c5cae9f9-6321-11e9-bd84-000c2975aae4" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
4月 24 11:08:48 k8s.node1 kubelet[2147]: ]
··· //

既然kubelet沒有問題，那再去看看pod中的容器：

~]# docker ps 
70494faefcf3        f4b4fccc65bb           "sh -c 'java -jar /o…"   3 days ago          Up 3 days                               k8s_simp-ser_simp-ser-6b8f54bb64-tdxlw_jx_0e9a9fa9-6328-11e9-bd84-000c2975aae4_0 // 容器還在運(yùn)行
//查看下docker日志
~]#journalctl -u docker -f 
···
msg="Container 70494faefcf334b906a112a7e7f9a9821a0aaff314f9e21f4ae949d527daf7e1 failed to exit within 30 seconds of signal 15 - using the force"
···//docker 無法 kill 掉容器

初步得出的判斷是因為容器化后，由于單容器單進(jìn)程，已經(jīng)沒有傳統(tǒng)意義上的 init 進(jìn)程了。應(yīng)用進(jìn)程直接占用
了 pid 1 的進(jìn)程號，Linux 內(nèi)核中會對 pid 1 進(jìn)程發(fā)送特殊的信號量。具體可參考Docker init 進(jìn)程。

最終解決辦法：

~]# docker restart // 重啟nodes節(jié)點