二進(jìn)制安裝-k8s高可用集群13-部署kubelet組件

kublet 運(yùn)行在每個(gè) worker 節(jié)點(diǎn)上，接收 kube-apiserver 發(fā)送的請(qǐng)求，管理 Pod 容器，執(zhí)行交互式命令，如 exec、run、logs 等。

kublet 啟動(dòng)時(shí)自動(dòng)向 kube-apiserver 注冊(cè)節(jié)點(diǎn)信息，內(nèi)置的 cadvisor 統(tǒng)計(jì)和監(jiān)控節(jié)點(diǎn)的資源使用情況。

為確保安全，本文檔只開啟接收 https 請(qǐng)求的安全端口，對(duì)請(qǐng)求進(jìn)行認(rèn)證和授權(quán)，拒絕未授權(quán)的訪問 (如 apiserver、heapster)。

下載分發(fā)，在部署 master 節(jié)點(diǎn)時(shí)已經(jīng)做過，依賴包也在 work 節(jié)點(diǎn)當(dāng)中安裝過，現(xiàn)在直接進(jìn)入配置

1、創(chuàng)建 kubelet bootstrap kubeconfig 文件

cat > magic57_create_kubeconfig_file.sh << "EOF"
#!/bin/bash
# 創(chuàng)建 kubelet bootstrap kubeconfig 文件
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do
    echo ">>> ${node_name}"
    # 創(chuàng)建 token
    export BOOTSTRAP_TOKEN=$(kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:${node_name} \
      --kubeconfig ~/.kube/config)
    # 設(shè)置集群參數(shù)
    kubectl config set-cluster kubernetes \
      --certificate-authority=/etc/kubernetes/cert/ca.pem \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 設(shè)置客戶端認(rèn)證參數(shù)
    kubectl config set-credentials kubelet-bootstrap \
      --token=${BOOTSTRAP_TOKEN} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 設(shè)置上下文參數(shù)
    kubectl config set-context default \
      --cluster=kubernetes \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 設(shè)置默認(rèn)上下文
    kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
done
EOF
  • 證書中寫入 Token 而非證書，證書后續(xù)由 controller-manager 創(chuàng)建

查看 kubeadm 為各節(jié)點(diǎn)創(chuàng)建的 token：

kubeadm token list --kubeconfig ~/.kube/config
TOKEN   TTL    EXPIRES                     USAGES                   DESCRIPTION               EXTRA GROUPS
XX.XX   XXh    20XX-XX-XXTXX:XX:XX+XX:XX   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:kube-node2
XX.XX   XXh    20XX-XX-XXTXX:XX:XX+XX:XX   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:kube-node3
XX.XX   XXh    20XX-XX-XXTXX:XX:XX+XX:XX   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:kube-node1
  • 創(chuàng)建的 token 有效期為 1 天，超期后將不能再被使用，且會(huì)被 kube-controller-manager 的 tokencleaner 清理 (如果啟用該 controller 的話)；
  • kube-apiserver 接收 kubelet 的 bootstrap token 后，將請(qǐng)求的 user 設(shè)置為 system:bootstrap:，group 設(shè)置為 system:bootstrappers；

各 token 關(guān)聯(lián)的 Secret：

kubectl get secrets  -n kube-system
NAME                                     TYPE                                  DATA      AGE
bootstrap-token-3v99js                   bootstrap.kubernetes.io/token         7         6h
bootstrap-token-904crw                   bootstrap.kubernetes.io/token         7         6h
bootstrap-token-v75f7h                   bootstrap.kubernetes.io/token         7         6h
default-token-sqk8s                      kubernetes.io/service-account-token   3         6h

2、分發(fā) bootstrap kubeconfig 文件到所有 worker 節(jié)點(diǎn)

cat > magic58_distribute_kubeconfig_file.sh << "EOF"
#!/bin/bash
# 分發(fā) bootstrap kubeconfig 文件到所有 worker 節(jié)點(diǎn)
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do
    echo ">>> ${node_name}" 
    scp kubelet-bootstrap-${node_name}.kubeconfig k8s@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
done
EOF

3、創(chuàng)建和分發(fā) kubelet 參數(shù)配置文件

創(chuàng)建 kubelet 參數(shù)配置模板文件：

source /opt/k8s/bin/environment.sh

cat > kubelet.config.json.template <<EOF
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/cert/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "##NODE_IP##",
  "port": 10250,
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "${CLUSTER_DNS_DOMAIN}",
  "clusterDNS": ["${CLUSTER_DNS_SVC_IP}"]
}
EOF
  • address：API 監(jiān)聽地址，不能為 127.0.0.1，否則 kube-apiserver、heapster 等不能調(diào)用 kubelet 的 API；
  • readOnlyPort=0：關(guān)閉只讀端口 (默認(rèn) 10255)，等效為未指定；
  • authentication.anonymous.enabled：設(shè)置為 false，不允許匿名訪問 10250 端口；
  • authentication.x509.clientCAFile：指定簽名客戶端證書的 CA 證書，開啟 HTTP 證書認(rèn)證；
  • authentication.webhook.enabled=true：開啟 HTTPs bearer token 認(rèn)證；
  • 對(duì)于未通過 x509 證書和 webhook 認(rèn)證的請(qǐng)求 (kube-apiserver 或其他客戶端)，將被拒絕，提示 Unauthorized；
  • authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查詢 kube-apiserver 某 user、group 是否具有操作資源的權(quán)限 (RBAC)；
  • featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自動(dòng) rotate 證書，證書的有效期取決于 kube-controller-manager 的 –experimental-cluster-signing-duration 參數(shù)；
  • 需要 root 賬戶運(yùn)行；

為各節(jié)點(diǎn)創(chuàng)建和分發(fā) kubelet 配置文件：

cat > magic59_createNode_distribute_file.sh << "EOF"
#!/bin/bash
# 為各節(jié)點(diǎn)創(chuàng)建和分發(fā) kubelet 配置文件
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    sed -e "s/##NODE_IP##/${node_ip}/" kubelet.config.json.template > kubelet.config-${node_ip}.json
    scp kubelet.config-${node_ip}.json root@${node_ip}:/etc/kubernetes/kubelet.config.json
done
EOF

4、創(chuàng)建和分發(fā) kubelet systemd unit 文件

創(chuàng)建 kubelet systemd unit 文件模板：

cat > kubelet.service.template <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet \\
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/etc/kubernetes/cert \\
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  --config=/etc/kubernetes/kubelet.config.json \\
  --hostname-override=##NODE_NAME## \\
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\
  --allow-privileged=true \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
  • 如果設(shè)置了 --hostname-override 選項(xiàng)，則 kube-proxy 也需要設(shè)置該選項(xiàng)，否則會(huì)出現(xiàn)找不到 Node 的情況；
  • --bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用該文件中的用戶名和 token 向 kube-apiserver 發(fā)送 TLS Bootstrapping 請(qǐng)求；
  • K8S approve kubelet 的 csr 請(qǐng)求后，在 --cert-dir 目錄創(chuàng)建證書和私鑰文件，然后寫入 --kubeconfig 文件

為各節(jié)點(diǎn)創(chuàng)建和分發(fā) kubelet systemd unit 文件：

cat > magic60_create_distribute_kubelet_file.sh << "EOF"
#!/bin/bash
# 為各節(jié)點(diǎn)創(chuàng)建和分發(fā) kubelet systemd unit 文件
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do 
    echo ">>> ${node_name}"
    sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template > kubelet-${node_name}.service
    scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.service
    scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
done
EOF

5、Bootstrap Token Auth 和授予權(quán)限

kublet 啟動(dòng)時(shí)查找配置的 –kubeletconfig 文件是否存在，如果不存在則使用 --bootstrap-kubeconfig 向 kube-apiserver 發(fā)送證書簽名請(qǐng)求 (CSR)。

kube-apiserver 收到 CSR 請(qǐng)求后，對(duì)其中的 Token 進(jìn)行認(rèn)證（事先使用 kubeadm 創(chuàng)建的 token），認(rèn)證通過后將請(qǐng)求的 user 設(shè)置為 system:bootstrap:，group 設(shè)置為 system:bootstrappers，這一過程稱為 Bootstrap Token Auth。

默認(rèn)情況下，這個(gè) user 和 group 沒有創(chuàng)建 CSR 的權(quán)限，kubelet 啟動(dòng)失敗，錯(cuò)誤日志如下：

sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests'
XX XX XX:XX:XX kube-node1 kubelet[26986]: F0506 XX:XX:XX.314378   26986 server.go:233] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:lemy40" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope
XX XX XX:XX:XX kube-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
XX XX XX:XX:XX kube-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

解決：創(chuàng)建一個(gè) clusterrolebinding 將 group system:bootstrappers 和 clusterrole system:node-bootstrapper 綁定

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --group=system:bootstrappers

6、啟動(dòng) kubelet 服務(wù)

cat > magic61_start_kubelet_server.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh root@${node_ip} "mkdir -p /var/lib/kubelet"
    ssh root@${node_ip} "/usr/sbin/swapoff -a"
    ssh root@${node_ip} "mkdir -p /var/log/kubernetes && chown -R k8s /var/log/kubernetes"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
done
EOF
  • 關(guān)閉 swap 分區(qū)，否則 kubelet 會(huì)啟動(dòng)失??；
  • 必須先創(chuàng)建工作和日志目錄；
journalctl -u kubelet |tail
XX XX XX:XX:XX kube-node1 kubelet[17331]: I1124 XX:XX:XX.255251   17331 container_manager_linux.go:427] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
XX XX XX:XX:XX kube-node1 kubelet[17331]: I1124 XX:XX:XX.255711   17331 container_manager_linux.go:427] [ContainerManager]: Discovered runtime cgroups name: 
...........

kubelet 啟動(dòng)后使用 --bootstrap-kubeconfig 向 kube-apiserver 發(fā)送 CSR 請(qǐng)求，當(dāng)這個(gè) CSR 被 approve 后，kube-controller-manager 為 kubelet 創(chuàng)建 TLS 客戶端證書、私鑰和 --kubeletconfig 文件。
注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 參數(shù)，才會(huì)為 TLS Bootstrap 創(chuàng)建證書和私鑰

kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk   43s       system:bootstrap:zkiem5   Pending
node-csr-oVbPmU-ikVknpynwu0Ckz_MvkAO_F1j0hmbcDa__sGA   27s       system:bootstrap:mkus5s   Pending
node-csr-u0E1-ugxgotO_9FiGXo8DkD6a7-ew8sX2qPE6KPS2IY   13m       system:bootstrap:k0s2bj   Pending

kubectl get nodes
  No resources found.
  • 三個(gè) work 節(jié)點(diǎn)的 csr 均處于 pending 狀態(tài)

1）approve kubelet CSR 請(qǐng)求
可以手動(dòng)或自動(dòng) approve CSR 請(qǐng)求。推薦使用自動(dòng)的方式，因?yàn)閺?v1.8 版本開始，可以自動(dòng)輪轉(zhuǎn) approve csr 后生成的證書

2）手動(dòng) approve CSR 請(qǐng)求
查看 CSR 列表：

kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk   43s       system:bootstrap:zkiem5   Pending
node-csr-oVbPmU-ikVknpynwu0Ckz_MvkAO_F1j0hmbcDa__sGA   27s       system:bootstrap:mkus5s   Pending
node-csr-u0E1-ugxgotO_9FiGXo8DkD6a7-ew8sX2qPE6KPS2IY   13m       system:bootstrap:k0s2bj   Pending

approve CSR：

kubectl certificate approve node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk
certificatesigningrequest.certificates.k8s.io "node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk" approved

查看 Approve 結(jié)果：

kubectl describe  csr node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk
Name:               node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Wed, XX XX 20XX XX:XX:XX +0800
Requesting User:    system:bootstrap:zkiem5
Status:             Approved
Subject:
         Common Name:    system:node:kube-node2
         Serial Number:
         Organization:   system:nodes
Events:  <none>
  • Requesting User：請(qǐng)求 CSR 的用戶，kube-apiserver 對(duì)它進(jìn)行認(rèn)證和授權(quán)；
  • Subject：請(qǐng)求簽名的證書信息；
  • 證書的 CN 是 system:node:kube-node2， Organization 是 system:nodes，kube-apiserver 的 Node 授權(quán)模式會(huì)授予該證書的相關(guān)權(quán)限；

3）自動(dòng) approve CSR 請(qǐng)求
創(chuàng)建三個(gè) ClusterRoleBinding，分別用于自動(dòng) approve client、renew client、renew server 證書：

cat > csr-crb.yaml <<EOF
 # Approve all CSRs for the group "system:bootstrappers"
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: auto-approve-csrs-for-group
 subjects:
 - kind: Group
   name: system:bootstrappers
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
   apiGroup: rbac.authorization.k8s.io
---
 # To let a node of the group "system:nodes" renew its own credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-client-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
   apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
 # To let a node of the group "system:nodes" renew its own server credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-server-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: approve-node-server-renewal-csr
   apiGroup: rbac.authorization.k8s.io
EOF
  • auto-approve-csrs-for-group：自動(dòng) approve node 的第一次 CSR； 注意第一次 CSR 時(shí)，請(qǐng)求的 Group 為 system:bootstrappers；
  • node-client-cert-renewal：自動(dòng) approve node 后續(xù)過期的 client 證書，自動(dòng)生成的證書 Group 為 system:nodes;
  • node-server-cert-renewal：自動(dòng) approve node 后續(xù)過期的 server 證書，自動(dòng)生成的證書 Group 為 system:nodes;

生效配置：

kubectl apply -f csr-crb.yaml

7、查看 kublet 的情況

等待一段時(shí)間 (1-10 分鐘)，三個(gè)節(jié)點(diǎn)的 CSR 都被自動(dòng) approve：

kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
csr-98h25                                              6m        system:node:kube-node2    Approved,Issued
csr-lb5c9                                              7m        system:node:kube-node3    Approved,Issued
csr-m2hn4                                              14m       system:node:kube-node1    Approved,Issued
node-csr-7q7i0q4MF_K2TSEJj16At4CJFLlJkHIqei6nMIAaJCU   28m       system:bootstrap:k0s2bj   Approved,Issued
node-csr-ND77wk2P8k2lHBtgBaObiyYw0uz1Um7g2pRvveMF-c4   35m       system:bootstrap:mkus5s   Approved,Issued
node-csr-Nysmrw55nnM48NKwEJuiuCGmZoxouK4N8jiEHBtLQso   6m        system:bootstrap:zkiem5   Approved,Issued
node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk   1h        system:bootstrap:zkiem5   Approved,Issued
node-csr-oVbPmU-ikVknpynwu0Ckz_MvkAO_F1j0hmbcDa__sGA   1h        system:bootstrap:mkus5s   Approved,Issued
node-csr-u0E1-ugxgotO_9FiGXo8DkD6a7-ew8sX2qPE6KPS2IY   1h        system:bootstrap:k0s2bj   Approved,Issued

所有節(jié)點(diǎn)均 ready：

kubectl get node
NAME         STATUS    ROLES     AGE       VERSION
kube-node1   Ready     <none>    6h        v1.10.4
kube-node2   Ready     <none>    6h        v1.10.4
kube-node3   Ready     <none>    6h        v1.10.4

kube-controller-manager 為各 node 生成了 kubeconfig 文件和公私鑰：

ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2296 XX XX XX:XX /etc/kubernetes/kubelet.kubeconfig

ls -l /etc/kubernetes/cert/|grep kubelet
-rw-r--r-- 1 root root 1046 XX XX XX:XX kubelet-client.crt
-rw------- 1 root root  227 XX XX XX:XX kubelet-client.key
-rw------- 1 root root 1334 XX XX XX:XX kubelet-server-20XX-XX-XX-XX-XX-XX.pem
lrwxrwxrwx 1 root root   59 XX XX XX:XX kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-20XX-XX-XX-XX-XX-XX.pem
  • kubelet-server 證書會(huì)周期輪轉(zhuǎn)；

8、kubelet 提供的 API 接口

kublet 啟動(dòng)后監(jiān)聽多個(gè)端口，用于接收 kube-apiserver 或其它組件發(fā)送的請(qǐng)求：

sudo netstat -lnpt|grep kubelet
tcp        0      0 172.68.96.101:10250     0.0.0.0:*               LISTEN      17331/kubelet
tcp        0      0 172.68.96.101:4194      0.0.0.0:*               LISTEN      17331/kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      17331/kubelet
  • 4194: cadvisor http 服務(wù)；
  • 10248: healthz http 服務(wù)；
  • 10250: https API 服務(wù)；注意：未開啟只讀端口 10255；

例如執(zhí)行 kubectl exec -it nginx-ds-5rmws -- sh 命令時(shí)，kube-apiserver 會(huì)向 kubelet 發(fā)送如下請(qǐng)求：

POST /exec/default/nginx-ds-5rmws/my-nginx?command=sh&input=1&output=1&tty=1

kubelet 接收 10250 端口的 https 請(qǐng)求：

  • /pods、/runningpods
  • /metrics、/metrics/cadvisor、/metrics/probes
  • /spec
  • /stats、/stats/container
  • /logs
  • /run/、”/exec/”, “/attach/”, “/portForward/”, “/containerLogs/” 等管理

詳情參考：https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go#L434:3

由于關(guān)閉了匿名認(rèn)證，同時(shí)開啟了 webhook 授權(quán)，所有訪問 10250 端口 https API 的請(qǐng)求都需要被認(rèn)證和授權(quán)。

預(yù)定義的 ClusterRole system:kubelet-api-admin 授予訪問 kubelet 所有 API 的權(quán)限：

kubectl describe clusterrole system:kubelet-api-admin
Name:         system:kubelet-api-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources      Non-Resource URLs  Resource Names  Verbs
  ---------      -----------------  --------------  -----
  nodes          []                 []              [get list watch proxy]
  nodes/log      []                 []              [*]
  nodes/metrics  []                 []              [*]
  nodes/proxy    []                 []              [*]
  nodes/spec     []                 []              [*]
  nodes/stats    []                 []              [*]

9、kublet api 認(rèn)證和授權(quán)

kublet 配置了如下認(rèn)證參數(shù)：

  • authentication.anonymous.enabled：設(shè)置為 false，不允許匿名訪問 10250 端口
  • authentication.x509.clientCAFile：指定簽名客戶端證書的 CA 證書，開啟 HTTPs 證書認(rèn)證
  • authentication.webhook.enabled=true：開啟 HTTPs bearer token 認(rèn)證；

同時(shí)配置了如下授權(quán)參數(shù)：

  • authroization.mode=Webhook：開啟 RBAC 授權(quán)；

kubelet 收到請(qǐng)求后，使用 clientCAFile 對(duì)證書簽名進(jìn)行認(rèn)證，或者查詢 bearer token 是否有效。如果兩者都沒通過，則拒絕請(qǐng)求，提示 Unauthorized

curl -s --cacert /etc/kubernetes/cert/ca.pem https://172.68.96.103:10250/metrics
Unauthorized

curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer 123456" https://172.68.96.103:10250/metrics
Unauthorized

通過認(rèn)證后，kubelet 使用 SubjectAccessReview API 向 kube-apiserver 發(fā)送請(qǐng)求，查詢證書或 token 對(duì)應(yīng)的 user、group 是否有操作資源的權(quán)限 (RBAC)；

證書認(rèn)證和授權(quán)：

# 權(quán)限不足的證書；
curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://172.68.96.103:10250/metrics
Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)

# 使用部署 kubectl 命令行工具時(shí)創(chuàng)建的、具有最高權(quán)限的 admin 證書；
curl -s --cacert /etc/kubernetes/cert/ca.pem --cert ./admin.pem --key ./admin-key.pem https://172.68.96.103:10250/metrics|head
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0
  • --cacert、--cert、--key 的參數(shù)值必須是文件路徑，如上面的 ./admin.pem 不能省略 ./，否則返回 401 Unauthorized；

bear token 認(rèn)證和授權(quán)：

創(chuàng)建一個(gè) ServiceAccount，將它和 ClusterRole system:kubelet-api-admin 綁定，從而具有調(diào)用 kubelet API 的權(quán)限：

kubectl create sa kubelet-api-test
kubectl create clusterrolebinding kubelet-api-test --$clusterrole=system:kubelet-api-admin --

serviceaccount=default:kubelet-api-test
SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
echo ${TOKEN}

curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://172.27.129.111:10250/metrics|head
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0

10，cadvisor 和 metrics

cadvisor 統(tǒng)計(jì)所在節(jié)點(diǎn)各容器的資源 (CPU、內(nèi)存、磁盤、網(wǎng)卡) 使用情況，分別在自己的 http web 頁(yè)面 (4194 端口) 和 10250 以 promehteus metrics 的形式輸出

瀏覽器訪問 http://172.68.96.103:4194/containers/ 可以查看到 cadvisor 的監(jiān)控頁(yè)面：

image.png

11、獲取 kublet 的配置

從 kube-apiserver 獲取各 node 的配置：

source /opt/k8s/bin/environment.sh

# 使用部署 kubectl 命令行工具時(shí)創(chuàng)建的、具有最高權(quán)限的 admin 證書；
curl -sSL --cacert /etc/kubernetes/cert/ca.pem --cert ./admin.pem --key ./admin-key.pem ${KUBE_APISERVER}/api/v1/nodes/kube-node1/proxy/configz | jq \
  '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'
{
  "syncFrequency": "1m0s",
  "fileCheckFrequency": "20s",
  "httpCheckFrequency": "20s",
  "address": "172.27.129.80",
  "port": 10250,
  "readOnlyPort": 10255,
  "authentication": {
    "x509": {},
    "webhook": {
      "enabled": false,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": true
    }
  },
  "authorization": {
    "mode": "AlwaysAllow",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "registryPullQPS": 5,
  "registryBurst": 10,
  "eventRecordQPS": 5,
  "eventBurst": 10,
  "enableDebuggingHandlers": true,
  "healthzPort": 10248,
  "healthzBindAddress": "127.0.0.1",
  "oomScoreAdj": -999,
  "clusterDomain": "cluster.local.",
  "clusterDNS": [
    "10.254.0.2"
  ],
  "streamingConnectionIdleTimeout": "4h0m0s",
  "nodeStatusUpdateFrequency": "10s",
  "imageMinimumGCAge": "2m0s",
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "volumeStatsAggPeriod": "1m0s",
  "cgroupsPerQOS": true,
  "cgroupDriver": "cgroupfs",
  "cpuManagerPolicy": "none",
  "cpuManagerReconcilePeriod": "10s",
  "runtimeRequestTimeout": "2m0s",
  "hairpinMode": "promiscuous-bridge",
  "maxPods": 110,
  "podPidsLimit": -1,
  "resolvConf": "/etc/resolv.conf",
  "cpuCFSQuota": true,
  "maxOpenFiles": 1000000,
  "contentType": "application/vnd.kubernetes.protobuf",
  "kubeAPIQPS": 5,
  "kubeAPIBurst": 10,
  "serializeImagePulls": false,
  "evictionHard": {
    "imagefs.available": "15%",
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionPressureTransitionPeriod": "5m0s",
  "enableControllerAttachDetach": true,
  "makeIPTablesUtilChains": true,
  "iptablesMasqueradeBit": 14,
  "iptablesDropBit": 15,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "failSwapOn": true,
  "containerLogMaxSize": "10Mi",
  "containerLogMaxFiles": 5,
  "enforceNodeAllocatable": [
    "pods"
  ],
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1"
}

或者參考代碼中的注釋：https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go

12、參考

kubelet 認(rèn)證和授權(quán)：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容