Kubernetes作為容器應(yīng)用的管理中心,通過(guò)對(duì)Pod的數(shù)量進(jìn)行監(jiān)控,并且根據(jù)主機(jī)或容器失效的狀態(tài)將新的Pod調(diào)度到其他Node上,實(shí)現(xiàn)了應(yīng)用層的高可用性。針對(duì)Kubernetes集群,高可用性還應(yīng)包含以下兩個(gè)層面的考慮:etcd數(shù)據(jù)存儲(chǔ)的高可用性和Kubernetes Master組件的高可用性。
*架構(gòu)*
K8S Master 192.168.81.11? ? Etcd Flannel Kube-apiserver Kube-controller-manager Kube-scheduler
K8S Minion1 192.168.81.12Flannel Kubelet Kube-proxy
K8S Minion2 192.168.81.60Flannel Kubelet Kube-proxy
*準(zhǔn)備工作*
端口轉(zhuǎn)發(fā)
vim /etc/sysctl.confnet.ipv4.ip_
forward=1net.ipv4.conf.all.rp_
filter=0net.ipv4.conf.default.rp_
filter=0
關(guān)閉網(wǎng)絡(luò)管理服務(wù)
systemctl stop NetworkManager.service
systemctl disable firewalld.service
firewalld和iptables(這里是測(cè)試環(huán)境所以全部關(guān)閉了,生產(chǎn)環(huán)境可以參考如下)
#停止firewall
systemctl stop firewalld.service
?#禁止firewall開機(jī)啟動(dòng)
systemctl disable firewalld.service
?#安裝 iptables.service
yum -y install iptables-services
#添加策略
vim /etc/sysconfig/iptables
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 10250 -j ACCEPT
[所有節(jié)點(diǎn)]#注釋此行-A FORWARD -j REJECT --reject-with icmp-host-prohibited #添加此行-A FORWARD -j ACCEPT?
#注釋此行
-A INPUT -j REJECT --reject-with icmp-host-prohibited?
#添加此行
-A INPUT -j ACCEPT?
#重啟防火墻使配置生效
systemctl restart iptables.service
#設(shè)置防火墻開機(jī)啟動(dòng)
systemctl enable iptables.servicedocker
#更新
yumyum update
#配置yum源
vim /etc/yum.repos.d/docker.repo
?[dockerrepo]name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
#安裝yum install docker-engine
#下載鏡像
docker pull google/pause
docker tag google/pause gcr.io/google_containers/pause-amd64:3.0
docker pull siriuszg/kubernetes-dashboard-amd64:v1.4.0
docker tag siriuszg/kubernetes-dashboard-amd64:v1.4.0 10.2.3.223:5000/kubernetes-dashboard-amd64:v1.4.0
一、etc集群的部署etcd在整個(gè)Kubernetes集群中處于中心數(shù)據(jù)庫(kù)的地位,為保證Kubernetes集群的高可用性,首先需要保證數(shù)據(jù)庫(kù)不是單故障點(diǎn)。
一方面,etcd需要以集群的方式進(jìn)行部署,以實(shí)現(xiàn)etcd數(shù)據(jù)存儲(chǔ)的冗余、備份與高可用性;
另一方面,etcd存儲(chǔ)的數(shù)據(jù)本身也應(yīng)考慮使用可靠的存儲(chǔ)設(shè)備。
首先,規(guī)劃一個(gè)至少3臺(tái)服務(wù)器(節(jié)點(diǎn))的etcd集群,在每臺(tái)服務(wù)器上安裝好etcd。
etcd1192.168.81.11etcd2192.168.81.12etcd3192.168.81.60
yum -y install etcd
#etcd實(shí)例名稱? ETCD_NAME
#etcd數(shù)據(jù)保存目錄? ETCD_DATA_DIR
#集群內(nèi)部通信使用的URL? ETCD_LISTEN_PEER_URLS
#供外部客戶端使用的URL? ETCD_LISTEN_CLIENT_URLS
#廣播給集群內(nèi)其他成員使用的URL? ETCD_INITIAL_ADVERTISE_PEER_URLS
#初始集群成員列表? ETCD_INITIAL_CLUSTER
#初始集群狀態(tài)? ETCD_INITIAL_CLUSTER_STATE
#集群名稱? ETCD_INITIAL_CLUSTER_TOKEN
#廣播給外部客戶端使用的URL? ETCD_ADVERTISE_CLIENT_URLS修改每臺(tái)服務(wù)器上etcd的配置文件/etc/etcd/etcd.conf[etcd2]
vim /etc/etcd/etcd.conf
ETCD_NAME=etcd2
ETCD_DATA_DIR="/var/lib/etcd/etcd2.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.81.12:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://192.168.81.12:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.81.12:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.81.11:2380,etcd2=http://192.168.81.12:2380,etcd3=http://192.168.81.60:2380"
ETCD_INITIAL_CLUSTER_STATE="exist"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.81.12:2379"systemctl restart etcd[etcd3]
vim /etc/etcd/etcd.conf
ETCD_NAME=etcd3
ETCD_DATA_DIR="/var/lib/etcd/etcd2.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.81.60:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://192.168.81.60:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.81.60:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.81.11:2380,etcd2=http://192.168.81.12:2380,etcd3=http://192.168.81.60:2380"
ETCD_INITIAL_CLUSTER_STATE="exist"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.81.60:2379"systemctl restart etcd[etcd1]
vim /etc/etcd/etcd.conf
ETCD_NAME=etcd1ETCD_DATA_DIR="/var/lib/etcd/etcd1.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.81.11:2380"
ETCD_LISTEN_CLIENT_URLS="http://127.0.0.1:2379,http://192.168.81.11:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.81.11:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://192.168.81.11:2380,etcd2=http://192.168.81.12:2380,etcd3=http://192.168.81.60:2380"
ETCD_INITIAL_CLUSTER_STATE="new"ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.81.11:2379"
systemctl restart etcd
啟動(dòng)完成后,在任意etcd節(jié)點(diǎn)執(zhí)行etcdctl cluster-health命令來(lái)查詢集群的運(yùn)行狀態(tài)
etcdctl cluster-health
member 618d69366dd8cee3 is healthy: got healthy result from http://192.168.81.12:2379
member acd2ba924953b1ec is healthy: got healthy result from http://192.168.81.60:2379
member f56676081999649a is healthy: got healthy result from http://192.168.81.11:2379
cluster is healthy
在任意etcd節(jié)點(diǎn)上執(zhí)行etcdctl member list命令來(lái)查詢集群的成員列表
etcdctl member list
618d69366dd8cee3: name=etcd2 peerURLs=http://192.168.81.12:2380 clientURLs=http://192.168.81.12:2379 isLeader=true
acd2ba924953b1ec: name=etcd3 peerURLs=http://192.168.81.60:2380 clientURLs=http://192.168.81.60:2379 isLeader=false
f56676081999649a: name=etcd1 peerURLs=http://192.168.81.11:2380 clientURLs=http://192.168.81.11:2379 isLeader=false
編輯啟動(dòng)腳本
vim /usr/lib/systemd/system/etcd.service
? ? ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" --initial-advertise-peer-urls=\"${ETCD_INITIAL_ADVERTISE_PEER_URLS}\" --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\""
啟動(dòng)etcd服務(wù)
systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service
創(chuàng)建網(wǎng)絡(luò)信息
etcdctl mkdir /k8s/network
etcdctl set /k8s/network/config '{"Network":"172.100.0.0/16"}'
二、flannel安裝與配置
yum -y install flannel
創(chuàng)建日志目錄
mkdir -p /var/log/k8s/flannel/
配置
vim /etc/sysconfig/flanneld
FLANNEL_ETCD_ENDPOINTS="http://192.168.81.11:2379"
FLANNEL_ETCD_PREFIX="/k8s/network"
FLANNEL_OPTIONS="--logtostderr=false --log_dir=/var/log/k8s/flannel/ --etcd-endpoints=http://192.168.81.11:2379 --iface=eno16777736"
(如果k8s-master是集群,配置不同之處如下:FLANNEL_ETCD="http://k8s_master_ip1:2379,http://k8s_master_ip2:2379,http://k8s_master_ip3:2379")
啟動(dòng)并添加開機(jī)啟動(dòng)項(xiàng)
systemctl start flanneld
systemctl enable flanneld.service
生成環(huán)境變量
/usr/libexec/flannel/mk-docker-opts.sh -i
檢查環(huán)境變量
cat /run/flannel/subnet.env
FLANNEL_NETWORK=172.100.0.0/16
FLANNEL_SUBNET=172.100.9.1/24
FLANNEL_MTU=1472
FLANNEL_IPMASQ=falsecat /run/docker_opts.env
DOCKER_OPT_BIP="--bip=172.100.9.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1472"
EnvironmentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd ${DOCKER_OPT_BIP} ${DOCKER_OPT_IPMASQ} ${DOCKER_OPT_MTU}
注釋掉#ExecStart=/usr/bin/dockerd-current \
生效
systemctl daemon-reload
啟動(dòng)
systemctl stop docker?
systemctl restart flanneld
systemctl start docker
檢查
ip a? | grep flannel
4: flannel0:mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500inet 172.100.9.0/16 scope global flannel0
ip a? | grep docker
5: docker0:mtu 1500 qdisc noqueue state DOWN
inet 172.100.9.1/24 scope global docker0
在192.168.81.11和192.168.81.12分別啟動(dòng)容器
docker run -ti --net=bridge centos:7 /bin/bash
在192.168.81.11啟動(dòng)的容器中,ping 192.168.81.12啟動(dòng)的容器的IP(172.100.64.2)
[root@8e7cf36a1fb2 /]# ping 172.100.64.2
PING 172.100.64.2 (172.100.64.2) 56(84) bytes of data.
64 bytes from 172.100.64.2: icmp_seq=1 ttl=60 time=8.46 ms
64 bytes from 172.100.64.2: icmp_seq=2 ttl=60 time=0.794 ms
64 bytes from 172.100.64.2: icmp_seq=3 ttl=60 time=0.584 ms
三、部署kubernets
在hosts文件中加入master和node節(jié)點(diǎn)(由于宿主機(jī)性能限制,這里和etcd集群部署在一起)
echo "192.168.81.11 centos-master
> 192.168.81.12 centos-minion
> 192.168.81.60 centos-minion2" >> /etc/hosts
編輯/etc/kubernetes/config
vim /etc/kubernetes/config
KUBE_LOGTOSTDERR="--logtostderr=true"
KUBE_LOG_LEVEL="--v=0"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER="--master=http://192.168.81.11:8080"
在master配置
vim /etc/kubernetes/apiserver
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
KUBE_API_PORT="--port=8080"
KUBELET_PORT="--kubelet-port=10250"
KUBE_ETCD_SERVERS="--etcd-servers=http://centos-master:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=172.100.0.0/16"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
KUBE_API_ARGS=""
啟動(dòng)恰當(dāng)?shù)姆?wù)
for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
配置在node節(jié)點(diǎn)上的kubernetes服務(wù)
minion
vim /etc/kubernetes/kubelet
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
KUBELET_HOSTNAME="--hostname-override=centos-minion"
KUBELET_API_SERVER="--api-servers=http://centos-master:8080"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_ARGS=""
minion2
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
KUBELET_HOSTNAME="--hostname-override=centos-minion2"
KUBELET_API_SERVER="--api-servers=http://centos-master:8080"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_ARGS=""
在node節(jié)點(diǎn)啟動(dòng)恰當(dāng)?shù)姆?wù)
for SERVICES in kube-proxy kubelet docker; do
systemctl restart $SERVICES
systemctl enable $SERVICES
systemctl status $SERVICES
done
檢查以確認(rèn)現(xiàn)在集群中master能夠看到node
kubectl get nodes
NAME? ? ? ? ? ? STATUS? ? AGE
192.168.81.12? ? NotReady? 4m
centos-minion? ? Ready? ? ? 14s
centos-minion2? Ready? ? ? 45s
kubernetes-dashboard部署
鏡像下載
docker pull siriuszg/kubernetes-dashboard-amd64:v1.4.0
docker tag siriuszg/kubernetes-dashboard-amd64:v1.4.0 10.2.3.223:5000/kubernetes-dashboard-amd64:v1.4.0
可以下載google提供的kubernetes-dashboard.yaml進(jìn)行修改,也可以自己創(chuàng)建
https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml
vim kubernetes-dashboard.yaml
-------------------------------------------------------------------------------------
metadata:
labels:
app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: kubernetes-dashboard
template:
metadata:
labels:
app: kubernetes-dashboard
annotations:
scheduler.alpha.kubernetes.io/tolerations: |
[
{
"key": "dedicated",
"operator": "Equal",
"value": "master",
"effect": "NoSchedule"
}
]
spec:
containers:
- name: kubernetes-dashboard
image: 10.2.3.223:5000/kubernetes-dashboard-amd64:v1.4.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9090
protocol: TCP
args:
- --apiserver-host=http://192.168.81.11:8080
livenessProbe:
httpGet:
path: /
port: 9090
initialDelaySeconds: 30
timeoutSeconds: 30
---
kind: Service
apiVersion: v1
metadata:
labels:
app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 80
targetPort: 9090
selector:
app: kubernetes-dashboard
--------------------------------------------------------------------------------
啟動(dòng)
kubectl create -f kubernetes-dashboard.yaml
訪問(wèn)
http://192.168.81.11:8080/ui
