kubeadm部署k8s-1.15.6高可用集群+外部etcd集群

2021-4-16

1、初始化服務(wù)器
2、ETCD集群配置
3、安裝配置keepalived
4、安裝docker
5、安裝配置k8s

服務(wù)器規(guī)劃

1、初始化服務(wù)器(所有節(jié)點(diǎn)執(zhí)行,可以粘貼入腳本執(zhí)行)

#!/bin/bash
#centos7 系統(tǒng)初始化
yum install -y rsync lrzsz* vim telnet ntpdate wget net-tools  nfs-utils.x86_64 
chmod +x /etc/rc.d/rc.local
#關(guān)閉防火墻、swap分區(qū)
setenforce 0
sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld.service
systemctl stop firewalld.service
swapoff -a;sed -i 's/.*swap.*/#&/' /etc/fstab
#修改系統(tǒng)參數(shù)、文件描述符
sed -i 's/4096/65535/' /etc/security/limits.d/20-nproc.conf
echo '* soft nofile 65535' >> /etc/security/limits.conf
echo '* hard nofile 65535' >> /etc/security/limits.conf
echo "ulimit revamped: `ulimit -n`"
#將橋接的IPv4流量傳遞到iptables的鏈,允許 iptables 檢查橋接流量
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

#主機(jī)名寫入hosts
cat >> /etc/hosts << EOF
192.168.2.241   2dot241
192.168.2.242   2dot242
192.168.2.243   2dot243
192.168.2.239   2dot239
192.168.2.240   2dot240
EOF

2、配置需認(rèn)證etcd集群,etcd相當(dāng)于整個(gè)k8s的數(shù)據(jù)庫

——1、下載CFSSL工具;CFSSL 包含一個(gè)命令行工具和一個(gè)用于簽名驗(yàn)證并且捆綁TLS證書的 HTTP API 服務(wù)
curl  -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl  -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
curl  -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x /bin/cfssl*
——2、創(chuàng)建CA證書相關(guān)配置文件--證書過期時(shí)間改為10年
#配置證書生成策略,規(guī)定CA可以頒發(fā)那種類型的證書
cat >  ca-config.json <<EOF
{
"signing": {
"default": {
  "expiry": "87600h"
},
"profiles": {
  "kubernetes": {
    "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
    ],
   "expiry": "87600h"
 }
}
}
}
EOF
——?jiǎng)?chuàng)建CA證書簽名請(qǐng)求,crs文件
cat >  ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
——?jiǎng)?chuàng)建etcd相關(guān)證書文件 !??!#etcd集群ip地址根據(jù)自己的來
cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.2.241",
    "192.168.2.242",
    "192.168.2.243"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
——3、生成證書和私鑰,分發(fā)到其他etcd服務(wù)器,免密登錄不在贅述,已經(jīng)提前配置
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
#執(zhí)行完成后會(huì)多2個(gè)pem文件和一個(gè)crs;ca-key.pem、ca.pem、ca.csr
cfssl gencert -ca=ca.pem \
 -ca-key=ca-key.pem \
 -config=ca-config.json \
 -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
[root@2dot241 test]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem
[root@2dot241 test]# mkdir -p /etc/etcd/ssl
[root@2dot241 test]# cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
[root@2dot241 test]# scp -r /etc/etcd 192.168.2.242:/etc/
[root@2dot241 test]# scp -r /etc/etcd 192.168.2.243:/etc/
——4、下載etcd軟件,3臺(tái)服務(wù)器全部執(zhí)行,或者直接拷貝etcd兩個(gè)腳本到另兩臺(tái)服務(wù)器/usr/local/bin/
[root@2dot241 test]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
[root@2dot241 test]# tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
[root@2dot241 test]# cp -a etcd-v3.3.10-linux-amd64/etcd* /usr/local/bin/
——5、生成etcd啟動(dòng)服務(wù)文件,拷貝至另兩臺(tái)服務(wù)器??!注意修改相關(guān)配置(四個(gè)ip一個(gè)name)
[root@2dot241 test]# cat >  /etc/systemd/system/etcd.service <<EOF
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
   --name=etcd-host1 \
   --cert-file=/etc/etcd/ssl/etcd.pem \
   --key-file=/etc/etcd/ssl/etcd-key.pem \
   --peer-cert-file=/etc/etcd/ssl/etcd.pem \
   --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
   --trusted-ca-file=/etc/etcd/ssl/ca.pem \
   --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
   --initial-advertise-peer-urls=https://192.168.2.241:2380 \
   --listen-peer-urls=https://192.168.2.241:2380 \
   --listen-client-urls=https://192.168.2.241:2379,http://127.0.0.1:2379 \
   --advertise-client-urls=https://192.168.2.241:2379 \
   --initial-cluster-token=etcd-cluster-0 \
   --initial-cluster=etcd-host1=https://192.168.2.241:2380,etcd-host2=https://192.168.2.242:2380,etcd-host3=https://192.168.2.243:2380 \
   --initial-cluster-state=new \
   --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

EOF
[root@2dot241 test]# scp  /etc/systemd/system/etcd.service 192.168.2.242:/etc/systemd/system/
[root@2dot241 test]# scp  /etc/systemd/system/etcd.service 192.168.2.243:/etc/systemd/system/
####################拷貝完成后修改相關(guān)配置,主要為以下5項(xiàng)########################
--name                               #對(duì)應(yīng)下面-initial-cluster=的名字
--initial-advertise-peer-urls        #本節(jié)點(diǎn)ip
--listen-peer-urls                   #本節(jié)點(diǎn)ip
--listen-client-urls                 #本節(jié)點(diǎn)ip
--advertise-client-urls              #本節(jié)點(diǎn)ip
——6、啟動(dòng)etcd集群,驗(yàn)證集群狀態(tài),三臺(tái)都啟動(dòng),第一個(gè)啟動(dòng)后會(huì)處于等待狀態(tài),等待其他兩個(gè)服務(wù)器啟動(dòng)
[root@2dot241 ~]# mkdir -p /var/lib/etcd
[root@2dot241 ~]# systemctl daemon-reload
[root@2dot241 ~]# systemctl enable etcd
[root@2dot241 ~]# systemctl start etcd
[root@2dot241 ~]# etcdctl --endpoints=https://192.168.2.241:2379  \
        --ca-file=/etc/etcd/ssl/ca.pem  \
        --cert-file=/etc/etcd/ssl/etcd.pem  \
        --key-file=/etc/etcd/ssl/etcd-key.pem \
        cluster-health
#顯示 member 1500ba7df8eae435 is healthy: got healthy result from https://192.168.2.241:2379
#顯示 member e2dd7103115d4825 is healthy: got healthy result from https://192.168.2.243:2379
#顯示 member ef5a0092d902bbfe is healthy: got healthy result from https://192.168.2.242:2379

3、安裝keepalived,三臺(tái)服務(wù)器執(zhí)行

yum -y install keepalived
#241服務(wù)器
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://192.168.2.5:6443" # vip
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160 # 本地網(wǎng)卡名稱
    virtual_router_id 61
    priority 120 # 權(quán)重,要唯一!值越大權(quán)重越高,其他兩個(gè)節(jié)點(diǎn)修改要小于他
    advert_int 1
    mcast_src_ip 192.168.2.241 # 本地IP
    nopreempt
    authentication {
        auth_type PASS
        auth_pass sqP05dQgMSlzrxHj
    }
    unicast_peer {
        #192.168.2.241  #注釋掉本地ip
        192.168.2.242
        192.168.2.243
    }
    virtual_ipaddress {
        192.168.2.5/24 # VIP
    }
    track_script {
        CheckK8sMaster
    }
}
EOF
#242服務(wù)器
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://192.168.2.5:6443" # vip
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160 # 本地網(wǎng)卡名稱
   virtual_router_id 61
    priority 110 # 權(quán)重,要唯一!值越大權(quán)重越高
    advert_int 1
    mcast_src_ip 192.168.2.242 # 本地IP
    nopreempt
    authentication {
        auth_type PASS
        auth_pass sqP05dQgMSlzrxHj
    }
    unicast_peer {
        192.168.2.241
        #192.168.2.242 #注釋掉本地ip
        192.168.2.243
    }
    virtual_ipaddress {
        192.168.2.5/24 # VIP
    }
    track_script {
        CheckK8sMaster
    }
}
EOF
#243服務(wù)器
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://192.168.2.5:6443" # vip
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160 # 本地網(wǎng)卡名稱
    virtual_router_id 61
    priority 100 # 權(quán)重,要唯一!值越大權(quán)重越高,其他兩個(gè)節(jié)點(diǎn)修改要小于他
    advert_int 1
    mcast_src_ip 192.168.2.243 # 本地IP
    nopreempt
    authentication {
        auth_type PASS
        auth_pass sqP05dQgMSlzrxHj
    }
    unicast_peer {
        192.168.2.241
        192.168.2.242 
        #192.168.2.243  #注釋掉本地ip
    }
    virtual_ipaddress {
        192.168.2.5/24 # VIP
    }
    track_script {
        CheckK8sMaster
    }
}
EOF
#分別啟動(dòng)服務(wù)器
systemctl enable keepalived && systemctl start keepalived
#權(quán)重最高的服務(wù)器上驗(yàn)證服務(wù),是否有虛擬ip
[root@2dot241 test]# ip addr
ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default 
qlen 1000
link/ether 00:50:56:b1:e4:82 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.241/24 brd 192.168.2.255 scope global noprefixroute ens160
   valid_lft forever preferred_lft forever
inet 192.168.2.5/24 scope global secondary ens160
   valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb1:e482/64 scope link 
   valid_lft forever preferred_lft forever

4、安裝docker--(所有節(jié)點(diǎn)執(zhí)行)

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce-18.09.0-3.el7 docker-ce-cli-18.09.0-3.el7      #查看可選擇版本 yum list docker-ce --showduplicates | sort -r
systemctl restart docker && systemctl enable docker

5、安裝kubeadm、kubelet (所有節(jié)點(diǎn)執(zhí)行)

#配置k8s yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
#檢查可安裝版本--(我這選擇安裝1.15.6了)
yum list kubeadm --showduplicates | sort -r
yum list kubelet --showduplicates | sort -r
#安裝1.15.6-0
yum install -y kubelet-1.15.6-0 kubeadm-1.15.6-0
systemctl enable kubelet
——1、初始化第一個(gè)master服務(wù)器
#創(chuàng)建kubeadm-conf.yaml 和 kube-flannel.yml 配置文件,修改配置文件kubeadm-conf.yaml
★ 修改certSANs的 ip 和 對(duì)應(yīng)的 master主機(jī)名
★ etcd 節(jié)點(diǎn)的 ip 改成對(duì)應(yīng)的
★ controlPlaneEndpoint 改成 Vip
★ serviceSubnet: 這個(gè)指的是k8s內(nèi) service 以后要用的 ip 網(wǎng)段
★ podSubnet: 這個(gè)指的是 k8s 內(nèi) pod 以后要用的 ip 網(wǎng)段
cat > kubeadm-config.yaml << EOF
apiServer:
  certSANs:
    - 192.168.2.5
    - 192.168.2.241
    - 192.168.2.242
    - 192.168.2.243
    - 127.0.0.1
    - "2dot241"
    - "2dot242"
    - "2dot243"
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.2.5:6443"
controllerManager: {}
dns: 
  type: CoreDNS
etcd:
  external:
    endpoints:
    - https://192.168.2.241:2379
    - https://192.168.2.242:2379
    - https://192.168.2.243:2379
    caFile: /etc/etcd/ssl/ca.pem
    certFile: /etc/etcd/ssl/etcd.pem
    keyFile: /etc/etcd/ssl/etcd-key.pem
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.15.6
networking: 
  dnsDomain: cluster.local  
  podSubnet: 172.20.0.0/16
  serviceSubnet: 172.21.0.0/16
scheduler: {}
EOF
#修改配置文件kube-flannel.yml 
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml  -O kube-flannel.yml 
#如果下載不下來:鏈接:https://pan.baidu.com/s/1U8GR2B1InUxVP2RiBMCxFw  提取碼:1234 
修改其中net-conf這個(gè)參數(shù),網(wǎng)段和kubeadm-config.yaml中podSubnet: 要一致,其他就不用動(dòng)了,如下圖
kube-flannel.yml
#初始化服務(wù)器
[root@2dot241 ~]# kubeadm init --config kubeadm-config.yaml
初始化服務(wù)器
#按照提示1、配置kubeconf----使用kubectl命令會(huì)用到  2、應(yīng)用kube-flannel.yml  3、加入集群
[root@2dot241 ~]# mkdir -p $HOME/.kube
[root@2dot241 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@2dot241 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@2dot241 ~]# kubectl create -f kube-flannel.yml      #創(chuàng)建kube-flannel網(wǎng)絡(luò)容器
#拷貝加入集群所需密鑰至其他master服務(wù)器
scp -r /etc/kubernetes/pki/ root@192.168.2.242:/etc/kubernetes/
scp -r /etc/kubernetes/pki/ root@192.168.2.243:/etc/kubernetes/
——2、兩臺(tái)master執(zhí)行加入集群命令
[root@2dot242 ~]#  kubeadm join 192.168.2.5:6443 --token 65qmsf.rzppklymqfmhap8z  --discovery-token-ca-cert-hash sha256:bf4be7e565602368290302310a05da70e58021eaddfef2967defa4408e8a765a  --control-plane 
[root@2dot243 ~]#  kubeadm join 192.168.2.5:6443 --token 65qmsf.rzppklymqfmhap8z  --discovery-token-ca-cert-hash sha256:bf4be7e565602368290302310a05da70e58021eaddfef2967defa4408e8a765a  --control-plane
#按提示運(yùn)行使用kubectl命令配置,這樣其他兩個(gè)服務(wù)器也可以使用kubectl命令
#隨意一臺(tái)集群中master服務(wù)器執(zhí)行查看加入集群結(jié)果
[root@2dot242 ~]# kubectl get node
NAME      STATUS   ROLES    AGE     VERSION
2dot241   Ready    master   23h     v1.15.6
2dot242   Ready    master   8m46s   v1.15.6
2dot243   Ready    master   6m46s   v1.15.6
——3、加入k8s-node節(jié)點(diǎn),在node節(jié)點(diǎn)執(zhí)行上面圖片的第三個(gè)紅框框里命令就可以了
#因?yàn)槲疫@做實(shí)驗(yàn)不是實(shí)時(shí),加入命令中token已過期,默認(rèn)24小時(shí)(node節(jié)點(diǎn)加入時(shí)報(bào)錯(cuò)failed to get config map: Unauthorized)。重新在服務(wù)器上獲取
master端執(zhí)行命令:kubeadm token create --print-join-command
[root@2dot241 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.2.5:6443 --token jitt2t.7d2jaobt3j49rrxu --discovery-token-ca-cert-hash sha256:bf4be7e565602368290302310a05da70e58021eaddfef2967defa4408e8a765a 
#粘貼到兩個(gè)node節(jié)點(diǎn)
[root@2dot239 ~]#  kubeadm join 192.168.2.5:6443 --token jitt2t.7d2jaobt3j49rrxu --discovery-token-ca-cert-hash sha256:bf4be7e565602368290302310a05da70e58021eaddfef2967defa4408e8a765a 
[root@2dot240 ~]#  kubeadm join 192.168.2.5:6443 --token jitt2t.7d2jaobt3j49rrxu --discovery-token-ca-cert-hash sha256:bf4be7e565602368290302310a05da70e58021eaddfef2967defa4408e8a765a 
#服務(wù)端運(yùn)行查看節(jié)點(diǎn)命令,node節(jié)點(diǎn)加入完成后大概1分鐘左右會(huì)顯示ready狀態(tài)
[root@2dot241 ~]# kubectl get node
NAME      STATUS   ROLES    AGE     VERSION
2dot239   Ready    <none>   3m3s    v1.15.6
2dot240   Ready    <none>   2m41s   v1.15.6
2dot241   Ready    master   40h     v1.15.6
2dot242   Ready    master   16h     v1.15.6
2dot243   Ready    master   16h     v1.15.6

以后如果有時(shí)間在更新etcd的擴(kuò)展和master擴(kuò)展

k8s-master證書到期更新:http://www.itdecent.cn/p/c4e1396b67cc

感謝兩位大佬的文章
https://blog.csdn.net/qq_31547771/article/details/100699573
https://shenshengkun.github.io/posts/omn700fj.html

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡(jiǎn)書系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容