在CentOS 7上安裝部署Kubernetes V1.10集群
通過(guò)參考Google和官網(wǎng)等資料
使用kubeadm進(jìn)行前期的安裝,本文無(wú)需科學(xué)上網(wǎng)
kubeadm是一個(gè)可以幫您以一種簡(jiǎn)單、合理安全和可擴(kuò)展的方式引導(dǎo)一個(gè)最佳實(shí)踐的Kubernetes集群的工具。它還支持為您管理Bootstrap Tokens和升級(jí)/降級(jí)集群
Before you begin
1. 準(zhǔn)備三臺(tái)虛擬機(jī):一臺(tái)Master、兩臺(tái)node,安裝操作系統(tǒng)CentOS Linux release 7.2,(由于要連互聯(lián)網(wǎng)拉去相關(guān)鏡像,我們用vbox設(shè)置好網(wǎng)絡(luò),啟用NAT和host-only)。
2. 系統(tǒng)配置(三節(jié)點(diǎn)都需執(zhí)行) |可先創(chuàng)建一臺(tái)虛機(jī)配好系統(tǒng)參數(shù)后再克隆兩臺(tái)。
系統(tǒng)配置(三節(jié)點(diǎn)都要執(zhí)行)
- 關(guān)閉SELinux和防火墻:
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config # 修改配置永久生效,需重啟
setenforce 0
#關(guān)閉防火墻
systemctl stop firewalld && systemctl disable firewalld
- RHEL / CentOS 7上的某些用戶報(bào)告了由于iptables被繞過(guò)而導(dǎo)致流量被錯(cuò)誤路由的問(wèn)題。應(yīng)該確保net.bridge.bridge-nf-call-iptables的sysctl配置中被設(shè)置為1
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
- 關(guān)閉系統(tǒng)Swap:Kubernetes 1.8開(kāi)始要求關(guān)閉系統(tǒng)的Swap,如果不關(guān)閉,默認(rèn)配置下kubelet將無(wú)法啟動(dòng)。方法一,通過(guò)kubelet的啟動(dòng)參數(shù)–fail-swap-on=false更改這個(gè)限制。方法二,關(guān)閉系統(tǒng)的Swap。
swapoff -a
修改/etc/fstab文件,注釋掉SWAP的自動(dòng)掛載,使用free -m確認(rèn)swap已經(jīng)關(guān)閉。
Docker安裝與配置(三節(jié)點(diǎn))
- 安裝Docker
mkdir ~/k8s
cd k8s
wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
wget https://download.docker.com/linux/centos/7/x86_64/stable/Packages/docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm
yum install ./docker-ce-selinux-17.03.2.ce-1.el7.centos.noarch.rpm
yum install ./docker-ce-17.03.2.ce-1.el7.centos.x86_64.rpm
systemctl enable docker
systemctl start docker
- 配置Docker
開(kāi)啟iptables filter表的FORWARD鏈
編輯/lib/systemd/system/docker.service,在ExecStart=..上面加入如下內(nèi)容:
ExecStartPost=/usr/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
ExecStart=/usr/bin/dockerd
.....
- 配置國(guó)內(nèi)鏡像加速器。(如果可以科學(xué)上網(wǎng)此步可忽略)因?yàn)閗ubeadm默認(rèn)要從google的鏡像倉(cāng)庫(kù)下載鏡像,但目前國(guó)內(nèi)無(wú)法訪問(wèn)google鏡像倉(cāng)庫(kù),所以需要我們配置成國(guó)內(nèi)的鏡像倉(cāng)庫(kù),并在kubeadm init前pull下所需的鏡像。
使用阿里云鏡像加速器:阿里云容器hub https://dev.aliyun.com/search.html;登錄之后,進(jìn)入管理中心-->鏡像加速器-->操作文檔,根據(jù)提示進(jìn)行設(shè)置即可。
坑點(diǎn):此處網(wǎng)上大部分提示:將鏡像加速器網(wǎng)站配置到 /etc/docker/daemon.json,但是后期我pull鏡像發(fā)現(xiàn),依然找的是國(guó)外的鏡像,花了一番功夫發(fā)現(xiàn),需要在 /etc/systemd/system/multi-user.target.wants/docker.service里修改。
`找到 ExecStart= 這一行,在這行最后添加加速器地址 --registry-mirror=<加速器地址> `
systemctl daemon-reload && systemctl restart docker && systemctl status docker
驗(yàn)證:
docker info
或
ps -ef | grep dockerd
如果從結(jié)果中看到了配置的 --registry-mirror 參數(shù)說(shuō)明配置成功。
拉鏡像:
需要提前pull相關(guān)鏡像并tag,否則 kubeadm init時(shí)會(huì)報(bào)錯(cuò),科學(xué)上網(wǎng)的可忽略。
master節(jié)點(diǎn):
docker pull anjia0532/kube-apiserver-amd64:v1.10.0
docker pull anjia0532/kube-scheduler-amd64:v1.10.0
docker pull anjia0532/kube-controller-manager-amd64:v1.10.0
docker pull anjia0532/kube-proxy-amd64:v1.10.0
docker pull anjia0532/k8s-dns-kube-dns-amd64:1.14.8
docker pull anjia0532/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker pull anjia0532/k8s-dns-sidecar-amd64:1.14.8
docker pull anjia0532/etcd-amd64:3.1.12
docker pull anjia0532/flannel:v0.10.0-amd64
docker pull anjia0532/pause-amd64:3.1
docker tag anjia0532/kube-apiserver-amd64:v1.10.0 k8s.gcr.io/kube-apiserver-amd64:v1.10.0
docker tag anjia0532/kube-scheduler-amd64:v1.10.0 k8s.gcr.io/kube-scheduler-amd64:v1.10.0
docker tag anjia0532/kube-controller-manager-amd64:v1.10.0 k8s.gcr.io/kube-controller-manager-amd64:v1.10.0
docker tag anjia0532/kube-proxy-amd64:v1.10.0 k8s.gcr.io/kube-proxy-amd64:v1.10.0
docker tag anjia0532/k8s-dns-kube-dns-amd64:1.14.8 k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
docker tag anjia0532/k8s-dns-dnsmasq-nanny-amd64:1.14.8 k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag anjia0532/k8s-dns-sidecar-amd64:1.14.8 k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
docker tag anjia0532/etcd-amd64:3.1.12 k8s.gcr.io/etcd-amd64:3.1.12
docker tag anjia0532/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
docker tag anjia0532/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
或執(zhí)行以下腳本:
#!/bin/bash
images=(kube-proxy-amd64:v1.10.0 kube-scheduler-amd64:v1.10.0 kube-controller-manager-amd64:v1.10.0 kube-apiserver-amd64:v1.10.0
etcd-amd64:3.1.12 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.8 k8s-dns-kube-dns-amd64:1.14.8
k8s-dns-dnsmasq-nanny-amd64:1.14.8)
for imageName in ${images[@]} ; do
docker pull anjia0532/$imageName
docker tag anjia0532/$imageName k8s.gcr.io/$imageName
docker rmi anjia0532/$imageName
done
docker images 可查看相關(guān)腳本;
Node節(jié)點(diǎn),在join節(jié)點(diǎn)之前也需要先下載到節(jié)點(diǎn)上面:
docker pull anjia0532/kube-proxy-amd64:v1.10.0
docker pull anjia0532/flannel:v0.10.0-amd64
docker pull anjia0532/pause-amd64:3.1
docker pull anjia0532/kubernetes-dashboard-amd64:v1.8.3
docker pull anjia0532/heapster-influxdb-amd64:v1.3.3
docker pull anjia0532/heapster-grafana-amd64:v4.4.3
docker pull anjia0532/heapster-amd64:v1.4.2
docker tag anjia0532/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
docker tag anjia0532/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker tag anjia0532/kube-proxy-amd64:v1.10.0 k8s.gcr.io/kube-proxy-amd64:v1.10.0
docker tag anjia0532/kubernetes-dashboard-amd64:v1.8.3 k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
docker tag anjia0532/heapster-influxdb-amd64:v1.3.3 k8s.gcr.io/heapster-influxdb-amd64:v1.3.3
docker tag anjia0532/heapster-grafana-amd64:v4.4.3 k8s.gcr.io/heapster-grafana-amd64:v4.4.3
docker tag anjia0532/heapster-amd64:v1.4.2 k8s.gcr.io/heapster-amd64:v1.4.2
或執(zhí)行以下腳本:
#!/bin/bash
images=(kube-proxy-amd64:v1.10.0 flannel:v0.10.0-amd64 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3
heapster-influxdb-amd64:v1.3.3 heapster-grafana-amd64:v4.4.3 heapster-amd64:v1.4.2)
for imageName in ${images[@]} ; do
docker pull anjia0532/$imageName
docker tag anjia0532/$imageName k8s.gcr.io/$imageName
docker rmi anjia0532/$imageName
done
安裝配置kubernetes
- 配置yum源并安裝
注:此小結(jié)三節(jié)點(diǎn)都要執(zhí)行
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
我們是安裝最新版本的,所以直接yum install -y kubeadm即可,它會(huì)安裝相應(yīng)依賴包。
如果要指定版本,可以先看看有那些版本
yum list kubeadm --showduplicates
- 配置kubelet
docker info 查看 --cgroup-driver
確保kubelet的配置文件/etc/systemd/system/kubelet.service.d/10-kubeadm.conf里的--cgroup-driver的一致。
我們安裝的都是一致的無(wú)需修改
安裝完成后:
kubelet設(shè)置開(kāi)機(jī)自動(dòng)運(yùn)行
systemctl enable kubelet
啟動(dòng)kubelet:
systemctl start kubelet
此時(shí)系統(tǒng)日志報(bào)錯(cuò)可忽略,因?yàn)檫€沒(méi)有初始化。
使用kubeadm init初始化集群
該小節(jié)僅在Master節(jié)點(diǎn)上執(zhí)行
初始化是關(guān)鍵的一步,也有不少坑,有的網(wǎng)站給的命令不全或不成功,eg:
kubeadm init --skip-preflight-checks --kubernetes-version=v1.10.0 --pod-network-cidr=10.244.0.0/16
加version是為了避免time out ;加pod-networ是我們后續(xù)節(jié)點(diǎn)間是使用flannel網(wǎng)絡(luò)的默認(rèn)配置,這兩項(xiàng)都需要添加,如果不成功我們通過(guò)/var/log/messages查看相關(guān)信息。
坑點(diǎn):通過(guò)鍵入上述命令,一直會(huì)卡住,主要有兩點(diǎn)一個(gè)是我們沒(méi)有提前pull鏡像導(dǎo)致,kubeadm默認(rèn)會(huì)從Google拉去鏡像,這對(duì)于我們沒(méi)有科學(xué)上網(wǎng)的人一開(kāi)始可能會(huì)掉入此坑;再有一點(diǎn),通過(guò)log日志發(fā)現(xiàn)有類似信息:
k8s.io/kubernetes/pkg/kubelet/kubelet.go:460: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster&limit=500&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
我們發(fā)現(xiàn)了一個(gè)問(wèn)題,日志里的ip是我們?cè)L問(wèn)互聯(lián)網(wǎng)的ip地址,這明顯不是正確的配置,通過(guò)官網(wǎng)找到原因:
Unless otherwise specified, kubeadm uses the default gateway’s network interface to advertise the master’s IP. If you want to use a different network interface, specify --apiserver-advertise-address=<ip-address> argument to kubeadm init.
由于本機(jī)配了多個(gè)網(wǎng)口所以需要指定相應(yīng)的ip地址。
如果初始化失敗,可使用清理命令:
kubeadm reset
- 所以完整的命令是:
kubeadm init --apiserver-advertise-address 192.168.56.20 --pod-network-cidr=10.244.0.0/16 --kubernetes-version=v1.10.0
執(zhí)行成功信息:

初始化主要過(guò)程為:
1.kubeadm 執(zhí)行初始化前的檢查
2.生成 token 和證書(shū)
3.生成 KubeConfig 文件,kubelet 需要這個(gè)文件與 Master 通信。
4.安裝 Master 組件,會(huì)從 goolge 的 Registry 下載組件的 Docker 鏡像,這一步可能會(huì)花一些時(shí)間,主要取決于網(wǎng)絡(luò)質(zhì)量,如果本地已有相關(guān)鏡像則會(huì)優(yōu)先使用本地的。
5.安裝附加組件 kube-proxy 和 kube-dns
6.Kubernetes Master 初始化成功
7.提示后續(xù)操作
根據(jù)安裝成功后的信息,進(jìn)行相關(guān)操作,記下join段的信息,由于別的節(jié)點(diǎn)添加至集群使用。
To make kubectl work for your non-root user, you might want to run these commands (which is also a part of the kubeadm init output):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 查看集群狀態(tài)
[root@master yum.repos.d]# kubectl get cs
NAME STATUS MESSAGE ERROR
etcd-0 Healthy {"health": "true"}
controller-manager Healthy ok
scheduler Healthy ok
- 安裝Pod Network--flannel
注:該小節(jié)僅在Master節(jié)點(diǎn)上執(zhí)行
相當(dāng)于部署軟件的操作
$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io "flannel" created
clusterrolebinding.rbac.authorization.k8s.io "flannel" created
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset.extensions "kube-flannel-ds" created
如果你的節(jié)點(diǎn)有多個(gè)網(wǎng)卡的話,需要在kube-flannel.yml中使用--iface參數(shù)指定集群主機(jī)內(nèi)網(wǎng)網(wǎng)卡的名稱,否則可能會(huì)出現(xiàn)dns無(wú)法解析。flanneld啟動(dòng)參數(shù)加上--iface=<iface-name>
args:
- --ip-masq
- --kube-subnet-mgr
- --iface=enp0s8
安裝完成后:
ifconfig驗(yàn)證是否有flannel網(wǎng)絡(luò)
使用kubectl get pods命令可以查看到我們集群中的組件運(yùn)行狀態(tài):
[root@master ~]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default curl-775f9567b5-69tlv 1/1 Running 2 3d
default httpd-app-77c9c8f99f-l4xct 1/1 Running 1 3d
default httpd-app-77c9c8f99f-srr69 1/1 Running 1 3d
default nginx-deployment-6b5c99b6fd-2b9lx 1/1 Running 1 3d
default nginx-deployment-6b5c99b6fd-qtqxl 1/1 Running 2 3d
kube-system etcd-master 1/1 Running 13 4d
kube-system heapster-676cc864c6-ggcgr 1/1 Running 1 3d
kube-system kube-apiserver-master 1/1 Running 1 1m
kube-system kube-controller-manager-master 1/1 Running 7 4d
kube-system kube-dns-86f4d74b45-swwb8 3/3 Running 6 3d
kube-system kube-flannel-ds-9mljn 1/1 Running 27 4d
kube-system kube-flannel-ds-ffqxn 1/1 Running 3 4d
kube-system kube-proxy-9t2pw 1/1 Running 2 4d
kube-system kube-proxy-jsnsd 1/1 Running 9 5d
kube-system kube-scheduler-master 1/1 Running 7 4d
kube-system kubernetes-dashboard-7d5dcdb6d9-psld9 1/1 Running 1 3d
kube-system monitoring-grafana-69df66f668-g6pzw 1/1 Running 1 3d
kube-system monitoring-influxdb-78d4c6f5b6-d5prw 1/1 Running 1 3d
如果都是running狀態(tài)則表示master節(jié)點(diǎn)安裝成功了。(nginx和httpd是我后續(xù)安裝的,可忽略)
- Master節(jié)點(diǎn)參與工作負(fù)載
使用kubeadm初始化的集群,出于安全考慮Pod不會(huì)被調(diào)度到Master Node上,可使用如下命令使Master節(jié)點(diǎn)參與工作負(fù)載。
kubectl taint nodes master node-role.kubernetes.io/master-
- 向Kubernetes集群添加Node
Run the command that was output by kubeadm init. For example:
kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>
A few seconds later, you should notice this node in the output from kubectl get nodes when run on the master.
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 5d v1.10.0
node1 Ready <none> 5d v1.10.0
同理加入另一個(gè)節(jié)點(diǎn)。
部署Dashboard插件
- 下載Dashboard插件配置文件
cd ~/k8s
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
本文前面已經(jīng)pull了dashboard的鏡像,所以只做配置即可:
-
指定端口類型為 NodePort,這樣外界可以通過(guò)地址 nodeIP:nodePort 訪問(wèn) dashboard;
只添加個(gè)type: NodePort
由于 K8S在1.6版本以后kube-apiserver 啟用了 RBAC 授權(quán),而官方源碼目錄的 dashboard-controller.yaml 沒(méi)有定義授權(quán)的 ServiceAccount,所以后續(xù)訪問(wèn) API server 的 API 時(shí)會(huì)被拒絕
- 創(chuàng)建一個(gè)kubernetes-dashboard-admin的ServiceAccount并授予集群admin的權(quán)限,創(chuàng)建kubernetes-dashboard-admin.rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard-admin
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard-admin
namespace: kube-system
- 創(chuàng)建pod
kubectl create -f kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard-admin.rbac.yaml
查看分配的NodePort
kubectl get svc,pod --all-namespaces | grep dashboard

- 部署heapster插件
安裝Heapster為集群添加使用統(tǒng)計(jì)和監(jiān)控功能,為Dashboard添加儀表盤(pán)。
mkdir -p ~/k8s/heapster
cd ~/k8s/heapster
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml
wget https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml
kubectl create -f ./
-
查看kubernete-dashboard-admin的token
- 訪問(wèn)dashboard
幾個(gè)坑點(diǎn):宿主機(jī)外訪問(wèn),需使用火狐瀏覽器;還有些權(quán)限的問(wèn)題看不到頁(yè)面內(nèi)容,需修改kube-apiserver-master的配置文件添加- --anonymous-auth=false。

參考地址:
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
https://www.centos.bz/2017/12/%E4%BD%BF%E7%94%A8kubeadm%E5%9C%A8centos-7%E4%B8%8A%E5%AE%89%E8%A3%85kubernetes-1-8/

