在容器實(shí)踐線路圖中介紹了容器技術(shù)選型,關(guān)于容器運(yùn)行時(shí),提到了CRI規(guī)范與CRI-O實(shí)現(xiàn),使用CRI-O可以在運(yùn)行時(shí)完全替代docker。CRI-O提供了crictl工具,類似docker client,可以pull鏡像、ps容器進(jìn)程、attach到容器進(jìn)程內(nèi)等等,除了build與tag/push鏡像沒提供之外,其他都有了。至于為何不提供鏡像build/tag/push操作,官方解釋是crictl不是替代docker,呵呵。
本文介紹如何使用CRI-O運(yùn)行時(shí)替換docker運(yùn)行時(shí),基于CRI-O對接Kubernetes編排。本文所有環(huán)境都基于 CentOS7.6 操作系統(tǒng),內(nèi)核版本為 3.10.0-957.21.3.el7.x86_64 。
1. 安裝CRI-O
- 啟用內(nèi)核模塊
modprobe overlay
modprobe br_netfilter
- 設(shè)置內(nèi)核參數(shù)
# Setup required sysctl params, these persist across reboots.
cat > /etc/sysctl.d/99-kubernetes-cri.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system
- 安裝CRI-O
# Install prerequisites
yum install yum-utils
yum-config-manager --add-repo=https://cbs.centos.org/repos/paas7-crio-311-candidate/x86_64/os/
# Install CRI-O
yum install --nogpgcheck cri-o
- 配置 CRI-O,設(shè)置 crio pause 鏡像下載地址為阿里云鏡像倉庫
默認(rèn)配置在 /etc/crio/crio.conf
sed -i "s/pause_image = .*/pause_image = \"registry.cn-hangzhou.aliyuncs.com\/google_containers\/pause:3.1\"/g" /etc/crio/crio.conf
- 啟動 CRI-O
systemctl start crio
- 驗(yàn)證 CRI-O 部署
curl -v --unix-socket /var/run/crio/crio.sock http://localhost/info
{"storage_driver":"overlay","storage_root":"/var/lib/containers/storage","cgroup_driver":"systemd","default_id_mappings":{"uids":[{"container_id":0,"host_id":0,"size":4294967295}],"gids":[{"container_id":0,"host_id":0,"size":4294967295}]}}s
2. 安裝kubeadm/kubelet/kubectl
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
# 將 SELinux 設(shè)置為 permissive 模式(將其禁用)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet
不過谷歌的包倉庫大陸無法訪問,使用阿里云鏡像站替換:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 將 SELinux 設(shè)置為 permissive 模式(將其禁用)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
CRI-O 運(yùn)行時(shí)使用的 cgroup driver 為 systemd ,因此需要設(shè)置 kubelet 參數(shù)保持一致:
echo "KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" >> /etc/sysconfig/kubelet
# !!在 kubelet 啟動文件 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 增加 KUBELET_CGROUP_ARGS 參數(shù) !!
# ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CGROUP_ARGS
systemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet
確保 kubelet 啟動參數(shù)中有 KUBELET_CGROUP_ARGS 設(shè)置的值:
# systemctl status -l kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since ...
Docs: https://kubernetes.io/docs/
Main PID: 15971 (kubelet)
CGroup: /system.slice/kubelet.service
└─15971 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --cgroup-driver=systemd
否則后續(xù)會因 cgroup driver 不匹配(crio使用的cgroup driver是systemd, kubelet使用的是cgroupfs)報(bào)錯(cuò):
kuberuntime_sandbox.go:68] CreatePodSandbox for pod "kube-scheduler-k8s-master-01_kube-system(...)" failed: rpc error: code = Unknown desc = cri-o configured with systemd cgroup manager, but did not receive slice as parent: /kubepods/burstable/...
3. 使用 kubeadm 安裝 Kubernetes
確定使用哪種網(wǎng)絡(luò)插件實(shí)現(xiàn),kubeadm 只支持 CNI 網(wǎng)絡(luò)插件;這里使用 Flannel 插件
修改 kubeadm 的默認(rèn)倉庫為阿里云倉庫鏡像
由于 Kubernetes 鏡像倉庫在 k8s.gcr.io 上,大陸無法訪問,需要使用阿里云的鏡像站,因此需要修改 kubeadm 默認(rèn)配置。
# 導(dǎo)出默認(rèn)配置
kubeadm config print init-defaults > kubeadm-init-config.yaml
#修改倉庫鏡像地址與kubernetes版本號
sed -i "s/imageRepository: .*/imageRepository: registry.cn-hangzhou.aliyuncs.com\/google_containers/g" kubeadm-init-config.yaml
sed -i "s/kubernetesVersion: .*/kubernetesVersion: v1.15.1/g" kubeadm-init-config.yaml
sed -i "s/criSocket: .*/criSocket: \/var\/run\/crio\/crio.sock/g" kubeadm-init-config.yaml
address=`ifconfig eth0 | egrep "inet\s" | awk '{print $2}'` && sed -i "s/advertiseAddress: .*/advertiseAddress: ${address}/g" kubeadm-init-config.yaml
# 在 kubeadm-init-config.yaml 增加 podSubnet: 參數(shù),如
...
kind: ClusterConfiguration
networking:
podSubnet: 10.244.0.0/16
...
修改后的 kubeadm-init-config.yaml :
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.128.165
bindPort: 6443
nodeRegistration:
criSocket: /var/run/crio/crio.sock
name: k8s-master-01
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.15.1
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.244.0.0/16
scheduler: {}
- kubeadm init
kubeadm init --config ~/kubeadm-init-config.yaml
如果 kubeadm init 命令出錯(cuò)了,修復(fù)之后重試,需要先執(zhí)行 kubeadm reset 。
- kubectl 連接到 Kubernetes 集群
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
如果使用 root 用戶,則直接使用:
export KUECONFIG=/etc/kubernetes/admin.conf
- 安裝網(wǎng)絡(luò)插件 Flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml
4. 增加 node 節(jié)點(diǎn)到集群
在 node 上執(zhí)行上面的大步驟1/2,安裝 CRI-O 與 kubeadm/kubelet/crio-tools
登錄到node節(jié)點(diǎn)上,執(zhí)行 join 的命令
kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>
如果忘記了token,在Master節(jié)點(diǎn)上通過如下命令查看:
kubeadm token list
如果token過期了,在Master節(jié)點(diǎn)上通過如下命令重新生成:
kubeadm token create
--discovery-token-ca-cert-hash 參數(shù)的值,在管理節(jié)點(diǎn)上通過如下命令獲取:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
5. 部署一個(gè) nginx 應(yīng)用
Reference
使用kubeadm 部署 Kubernetes(國內(nèi)環(huán)境)
too hard to install k8s in china
更多云最佳實(shí)踐 https://best.practices.cloud