限制容器在node節(jié)點上的資源占用。
1. 節(jié)點信息總覽
1.1 master 信息輸出如下
"Capacity"和"Allocatable" 處可見，資源全部被允許被分配，即沒有預(yù)留：
[root@devops-master ~]# kubectl describe nodes devops-master
Name: devops-master
Roles: master
#以下是給角色打的標(biāo)簽，架構(gòu)和操作系統(tǒng)，等都會在里邊。
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=devops-master
kubernetes.io/os=linux
node-role.kubernetes.io/master=
# flannel網(wǎng)卡的虛擬MAC地址，也可以在ip a 中看到
Annotations: flannel.alpha.coreos.com/backend-data: {"VtepMAC":"9e:d1:1a:e6:83:2e"}
# vxlan指 可擴展的虛擬網(wǎng)絡(luò)
flannel.alpha.coreos.com/backend-type: vxlan
flannel.alpha.coreos.com/kube-subnet-manager: true
flannel.alpha.coreos.com/public-ip: 10.252.97.56
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0ke
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 29 Apr 2020 16:33:15 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:33:11 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:33:11 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:33:11 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 12 Aug 2020 18:50:58 +0800 Wed, 29 Apr 2020 16:45:45 +0800 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.252.97.56
Hostname: devops-master
#所有硬件資源
Capacity:
cpu: 8
ephemeral-storage: 25792732Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32765896Ki
pods: 110
#以下是可分配資源
Allocatable:
cpu: 8
ephemeral-storage: 23770581772
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32663496Ki
pods: 110
System Info:
Machine ID: dff543df0a0c44e2962f1438f92b6868
System UUID: 42277530-DD16-E8F5-B3AB-C6831B9F49FA
Boot ID: 45ec85f2-0237-4d25-b684-6ec886f0c824
Kernel Version: 3.10.0-514.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.6.1
Kubelet Version: v1.15.2
Kube-Proxy Version: v1.15.2
PodCIDR: 10.244.0.0/24
Non-terminated Pods: (8 in total)
#以下列出所有pod的信息
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-bccdc95cf-vrxck 100m (1%) 0 (0%) 70Mi (0%) 170Mi (0%) 105d
kube-system etcd-devops-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-apiserver-devops-master 250m (3%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-controller-manager-devops-master 200m (2%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-flannel-ds-amd64-bh5gv 100m (1%) 100m (1%) 50Mi (0%) 50Mi (0%) 105d
kube-system kube-proxy-6r9sg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 105d
kube-system kube-scheduler-devops-master 100m (1%) 0 (0%) 0 (0%) 0 (0%) 105d
monitoring prometheus-operator-prometheus-node-exporter-qb48r 0 (0%) 0 (0%) 0 (0%) 0 (0%) 96d
# 以下是已分配資源
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 750m (9%) 100m (1%)
memory 120Mi (0%) 220Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
1.2 node信息如下
同樣查看node節(jié)點信息，可見資源同樣都被分配了。
Capacity:
cpu: 4
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16247820Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16247820Ki
pods: 110
說明：下文會修改這個節(jié)點的cgroup資源限制
2. 配置docker的 cgroup驅(qū)動
- 確認(rèn)docker驅(qū)動
# docker info | grep "Cgroup Driver"
Cgroup Driver: cgroupfs
- 如果不是 cgroupfs，則可以通過以下方法配置
# vim /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
.............
}
3. 配置kubelete的cgroup驅(qū)動
3.1 配置文件
/var/lib/kubelet/kubeadm-flags.env
作用：
用來為Kube組件和System進(jìn)程預(yù)留資源，從而保證當(dāng)節(jié)點出現(xiàn)滿負(fù)荷時也能保證Kube和System進(jìn)程有足夠的資源。
3.2 默認(rèn)配置
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1"
參數(shù)說明
- Node Capacity： 是Node的所有硬件資源
- kube-reserved： 是給kube組件預(yù)留的資源
- system-reserved： 是給System進(jìn)程預(yù)留的資源
- eviction-threshold： 驅(qū)逐閾值
- allocatable： 可配置值
節(jié)點上可配置值 = 總量 - kube組件預(yù)留值 - 系統(tǒng)預(yù)留值 - 驅(qū)逐閾值
3.3 修改如下
KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs \
--network-plugin=cni \
--pod-infra-container-image=nexus.10010sh.cn/pause:3.1 \
--enforce-node-allocatable=pods,kube-reserved,system-reserved \
--kube-reserved-cgroup=/system.slice/kubelet.service \
--system-reserved-cgroup=/system.slice \
--kube-reserved=cpu=1,memory=1Gi \
--system-reserved=cpu=1,memory=1Gi \
--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \
--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \
--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \
--eviction-max-pod-grace-period=30 \
--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"
注解：
--cgroup-driver=cgroupfs \
--network-plugin=cni \
--pod-infra-container-image=nexus.10010sh.cn/pause:3.1 \
#開啟為kube組件和系統(tǒng)守護進(jìn)程預(yù)留資源的功能
--enforce-node-allocatable=pods,kube-reserved,system-reserved \
#設(shè)置k8s組件的cgroup
--kube-reserved-cgroup=/system.slice/kubelet.service \
#設(shè)置系統(tǒng)守護進(jìn)程的cgroup
--system-reserved-cgroup=/system.slice \
# kubernetes預(yù)留
--kube-reserved=cpu=1,memory=1Gi \
# 系統(tǒng)預(yù)留
--system-reserved=cpu=1,memory=1Gi \
#驅(qū)逐pod的硬限制
--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \
#驅(qū)逐pod的軟限制
--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \
#達(dá)到驅(qū)逐閾值后多久開始驅(qū)逐
--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \
#驅(qū)逐前最大等待時間
--eviction-max-pod-grace-period=30 \
#至少回收多少資源才停止驅(qū)逐
--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"
3.4 修改kubelet 啟動文件
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
[Service]
ExecStart=/usr/bin/kubelet
#添加如下兩行
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
4. 重啟服務(wù)查看結(jié)果
重啟服務(wù)
如果修改了docker則需重啟docker
重啟kubelet查看修改結(jié)果
Capacity:
cpu: 4
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 16247820Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 43400496Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 13658395636
pods: 110
