kubernetes（k8s）二進(jìn)制v14.1HA部署

一、環(huán)境準(zhǔn)備

1.1、角色劃分

10.8.13.80   vip  
10.8.13.81   master01  haproxy、keepalived、etcd、kube-apiserver、kube-controller-manager、kube-scheduler
10.8.13.82   master02  haproxy、keepalived、etcd、kube-apiserver、kube-controller-manager、kube-scheduler
10.8.13.83   master03  haproxy、keepalived、etcd、kube-apiserver、kube-controller-manager、kube-scheduler
10.8.13.84   node01    kubelet、docker、kube_proxy、flanneld
10.8.13.85   node02    kubelet、docker、kube_proxy、flanneld

1.2、各主機(jī)ssh互通

#ssh-keygen
#ssh-copy-id 10.8.13.82(83-85)

1.3、環(huán)境初始化
1.3.1、停止iptables

systemctl stop firewalld.service 
systemctl disable  firewalld.service

1.3.2、關(guān)閉selinux

# cat /etc/selinux/config 
SELINUX=disabled
# setenforce 0

1.3.4、設(shè)置sysctl，開(kāi)啟路由轉(zhuǎn)發(fā)

# cat /etc/sysctl.conf
fs.file-max=1000000
 net.bridge.bridge-nf-call-ip6tables = 1
 net.bridge.bridge-nf-call-iptables = 1
 net.ipv4.ip_forward = 1
 vm.swappiness = 0
 net.ipv4.ip_forward = 1
 net.ipv4.tcp_max_tw_buckets = 6000
 net.ipv4.tcp_sack = 1
 net.ipv4.tcp_window_scaling = 1
 net.ipv4.tcp_rmem = 4096 87380 4194304
 net.ipv4.tcp_wmem = 4096 16384 4194304
 net.ipv4.tcp_max_syn_backlog = 16384
 net.core.netdev_max_backlog = 32768
 net.core.somaxconn = 32768
 net.core.wmem_default = 8388608
 net.core.rmem_default = 8388608
 net.core.rmem_max = 16777216
 net.core.wmem_max = 16777216
 net.ipv4.tcp_timestamps = 1
 net.ipv4.tcp_fin_timeout = 20
 net.ipv4.tcp_synack_retries = 2
 net.ipv4.tcp_syn_retries = 2
 net.ipv4.tcp_syncookies = 1

 net.ipv4.tcp_tw_reuse = 1
 net.ipv4.tcp_mem = 94500000 915000000 927000000
 net.ipv4.tcp_max_orphans = 3276800
 net.ipv4.ip_local_port_range = 1024 65000
 net.nf_conntrack_max = 6553500
 net.netfilter.nf_conntrack_max = 6553500
 net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
 net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
 net.netfilter.nf_conntrack_tcp_timeout_established = 3600

1.3.5、加載ipvs

cat << EOF | tee /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
 modprobe -- ip_vs
 modprobe -- ip_vs_rr
 modprobe -- ip_vs_wrr
 modprobe -- ip_vs_sh
 modprobe -- nf_conntrack_ipv4
 EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

二、集群各功能模塊描述

k8s工作流程.png

Master節(jié)點(diǎn)：

Master節(jié)點(diǎn)上面主要由四個(gè)模塊組成，etcd，APIServer，schedule,controller-manager（haproxy、keepalived高可用后面單獨(dú)說(shuō)）
etcd：

etcd是一個(gè)高可用的鍵值存儲(chǔ)系統(tǒng)，kubernetes使用它來(lái)存儲(chǔ)各個(gè)資源的狀態(tài)，從而實(shí)現(xiàn)了Restful的API。

APIServer:

APIServer負(fù)責(zé)對(duì)外提供Restful的kubernetes API的服務(wù)，它是系統(tǒng)管理指令的統(tǒng)一接口，任何對(duì)資源的增刪該查都要交給APIServer處理后再交給etcd。kubectl(kubernetes提供的客戶端工具，該工具內(nèi)部是對(duì)kubernetes API的調(diào)用）是直接和APIServer交互的。

schedule:

schedule負(fù)責(zé)調(diào)度Pod到合適的Node上，如果把scheduler看成一個(gè)黑匣子，那么它的輸入是pod和由多個(gè)Node組成的列表，輸出是Pod和一個(gè)Node的綁定。 kubernetes目前提供了調(diào)度算法，同樣也保留了接口。用戶根據(jù)自己的需求定義自己的調(diào)度算法。

controller manager:

如果APIServer做的是前臺(tái)的工作的話，那么controller manager就是負(fù)責(zé)后臺(tái)的。每一個(gè)資源都對(duì)應(yīng)一個(gè)控制器。而control manager就是負(fù)責(zé)管理這些控制器的，比如我們通過(guò)APIServer創(chuàng)建了一個(gè)Pod，當(dāng)這個(gè)Pod創(chuàng)建成功后，APIServer的任務(wù)就算完成了。

Node節(jié)點(diǎn)：

每個(gè)Node節(jié)點(diǎn)主要由四個(gè)模板組成：kublet， kube-proxy，docker，flanneld
kube-proxy:

該模塊實(shí)現(xiàn)了kubernetes中的服務(wù)發(fā)現(xiàn)和反向代理功能。kube-proxy支持TCP和UDP連接轉(zhuǎn)發(fā)，默認(rèn)基Round Robin算法將客戶端流量轉(zhuǎn)發(fā)到與service對(duì)應(yīng)的一組后端pod。服務(wù)發(fā)現(xiàn)方面，kube-proxy使用etcd的watch機(jī)制監(jiān)控集群中service和endpoint對(duì)象數(shù)據(jù)的動(dòng)態(tài)變化，并且維護(hù)一個(gè)service到endpoint的映射關(guān)系，從而保證了后端pod的IP變化不會(huì)對(duì)訪問(wèn)者造成影響，另外，kube-proxy還支持session affinity。

kublet：

kublet是Master在每個(gè)Node節(jié)點(diǎn)上面的agent，是Node節(jié)點(diǎn)上面最重要的模塊，它負(fù)責(zé)維護(hù)和管理該Node上的所有容器，但是如果容器不是通過(guò)kubernetes創(chuàng)建的，它并不會(huì)管理。本質(zhì)上，它負(fù)責(zé)使Pod的運(yùn)行狀態(tài)與期望的狀態(tài)一致。

flanneld：

源主機(jī)的flanneld服務(wù)將原本的數(shù)據(jù)內(nèi)容UDP封裝后根據(jù)自己的路由表投遞給目的節(jié)點(diǎn)的flanneld服務(wù)，數(shù)據(jù)到達(dá)以后被解包，然后直接進(jìn)入目的節(jié)點(diǎn)的flannel0虛擬網(wǎng)卡，然后被轉(zhuǎn)發(fā)到目的主機(jī)的docker0虛擬網(wǎng)卡，最后就像本機(jī)容器通信一下的有docker0路由到達(dá)目標(biāo)容器。

docker：

不做贅述

三、下載鏈接

Client Binaries
https://dl.k8s.io/v1.14.1/kubernetes-client-linux-amd64.tar.gz
Server Binaries
https://dl.k8s.io/v1.14.1/kubernetes-server-linux-amd64.tar.gz
Node Binaries
https://dl.k8s.io/v1.14.1/kubernetes-node-linux-amd64.tar.gz
etcd
https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcd-v3.3.11-linux-amd64.tar.gz
flannel
https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

四、Master部署

以下操作都在master01上執(zhí)行，生成證書(shū)之后拷貝到master02和master03

4.1、下載軟件

wget https://dl.k8s.io/v1.14.1/kubernetes-server-linux-amd64.tar.gz
wget https://dl.k8s.io/v1.14.1/kubernetes-client-linux-amd64.tar.gz
wget https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcd-v3.3.11-linux-amd64.tar.gz
wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

4.2、ssl安裝

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

4.3、創(chuàng)建etcd證書(shū)

在所有節(jié)點(diǎn)（master01-03、node01-02）創(chuàng)建此路徑

mkdir /k8s/etcd/{bin,cfg,ssl} -p
mkdir /k8s/kubernetes/{bin,cfg,ssl} -p

1)、etcd ca配置

cd /k8s/etcd/ssl/
cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "etcd": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

2)、etcd ca證書(shū)

cat << EOF | tee ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

3)、etcd server證書(shū)

cat << EOF | tee server-csr.json
{
    "CN": "etcd",
    "hosts": [
    "10.8.13.81",
    "10.8.13.82",
    "10.8.13.83"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

4)、生成etcd ca證書(shū)和私鑰
初始化ca

cfssl gencert -initca ca-csr.json | cfssljson -bare ca 
[root@master01 ssl]# ls
ca-config.json  ca-csr.json  server-csr.json
[root@master01 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca 
2019/05/01 16:13:54 [INFO] generating a new CA key and certificate from CSR
2019/05/01 16:13:54 [INFO] generate received request
2019/05/01 16:13:54 [INFO] received CSR
2019/05/01 16:13:54 [INFO] generating key: rsa-2048
2019/05/01 16:13:54 [INFO] encoded CSR
2019/05/01 16:13:54 [INFO] signed certificate with serial number 144752911121073185391033754516204538929473929443
[root@master01 ssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server-csr.json

生成server證書(shū)

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server
[root@master01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server
2019/05/01 16:18:53 [INFO] generate received request
2019/05/01 16:18:53 [INFO] received CSR
2019/05/01 16:18:53 [INFO] generating key: rsa-2048
2019/05/01 16:18:54 [INFO] encoded CSR
2019/05/01 16:18:54 [INFO] signed certificate with serial number 388122587040599986639159163167557684970159030057
2019/05/01 16:18:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. 
For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master01 ssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem

4.4、etcd安裝

1）解壓縮

tar -zxf etcd-v3.3.11-linux-amd64.tar.gz
cd etcd-v3.3.11-linux-amd64/
cp etcd etcdctl /k8s/etcd/bin/
mkdir /data1/etcd

2）配置etcd主文件

vim /k8s/etcd/cfg/etcd.conf   
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/data1/etcd"
ETCD_LISTEN_PEER_URLS="https://10.8.13.81:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.8.13.81:2379"
 
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.8.13.81:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.8.13.81:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.8.13.81:2380,etcd02=https://10.8.13.82:2380,etcd03=https://10.8.13.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"

3）配置etcd啟動(dòng)文件

vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/data1/etcd/
EnvironmentFile=-/k8s/etcd/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /k8s/etcd/bin/etcd --name=\"${ETCD_NAME}\" --data-dir=\"${ETCD_DATA_DIR}\" --listen-client-urls=\"${ETCD_LISTEN_CLIENT_URLS}\" --listen-peer-urls=\"${ETCD_LISTEN_PEER_URLS}\" --advertise-client-urls=\"${ETCD_ADVERTISE_CLIENT_URLS}\" --initial-cluster-token=\"${ETCD_INITIAL_CLUSTER_TOKEN}\" --initial-cluster=\"${ETCD_INITIAL_CLUSTER}\" --initial-cluster-state=\"${ETCD_INITIAL_CLUSTER_STATE}\" --cert-file=\"${ETCD_CERT_FILE}\" --key-file=\"${ETCD_KEY_FILE}\" --trusted-ca-file=\"${ETCD_TRUSTED_CA_FILE}\" --client-cert-auth=\"${ETCD_CLIENT_CERT_AUTH}\" --peer-cert-file=\"${ETCD_PEER_CERT_FILE}\" --peer-key-file=\"${ETCD_PEER_KEY_FILE}\" --peer-trusted-ca-file=\"${ETCD_PEER_TRUSTED_CA_FILE}\" --peer-client-cert-auth=\"${ETCD_PEER_CLIENT_CERT_AUTH}\""
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

4)、拷貝master01etcd的證書(shū)、配置文件、啟動(dòng)文件到master02和master03對(duì)應(yīng)路徑下

scp /k8s/etcd/ssl/* 10.8.13.82:/k8s/etcd/ssl/
scp /k8s/etcd/ssl/* 10.8.13.83:/k8s/etcd/ssl/
scp /k8s/etcd/cfg/* 10.8.13.82:/k8s/etcd/cfg/
scp /k8s/etcd/cfg/* 10.8.13.83:/k8s/etcd/cfg/
scp /k8s/etcd/bin/* 10.8.13.82:/k8s/etcd/bin/
scp /k8s/etcd/bin/* 10.8.13.83:/k8s/etcd/bin/
scp /usr/lib/systemd/system/etcd.service 10.8.13.82:/usr/lib/systemd/system/etcd.service
scp /usr/lib/systemd/system/etcd.service 10.8.13.83:/usr/lib/systemd/system/etcd.service

5)、修改master02、master03 etcd的conf配置文件
matser02 etcd.conf配置如下：

ssh 10.8.13.82
vim /k8s/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/data1/etcd"
ETCD_LISTEN_PEER_URLS="https://10.8.13.82:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.8.13.82:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.8.13.82:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.8.13.82:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.8.13.81:2380,etcd02=https://10.8.13.82:2380,etcd03=https://10.8.13.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"

matser03 etcd.conf配置如下：

ssh 10.8.13.83
vim /k8s/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/data1/etcd"
ETCD_LISTEN_PEER_URLS="https://10.8.13.83:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.8.13.83:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.8.13.83:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.8.13.83:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.8.13.81:2380,etcd02=https://10.8.13.82:2380,etcd03=https://10.8.13.83:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#[Security]
ETCD_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_PEER_CERT_FILE="/k8s/etcd/ssl/server.pem"
ETCD_PEER_KEY_FILE="/k8s/etcd/ssl/server-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"

6)、啟動(dòng)etcd服務(wù)，并加入開(kāi)機(jī)自啟動(dòng)(master三個(gè)節(jié)點(diǎn)都執(zhí)行)

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

7)、etcd服務(wù)檢查

/k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379" cluster-health
以下為輸出：
member 262d942ab474feaa is healthy: got healthy result from https://10.8.13.82:2379
member 3e95c59733e7d54f is healthy: got healthy result from https://10.8.13.83:2379
member fe03446cb13e0221 is healthy: got healthy result from https://10.8.13.81:2379
cluster is healthy
至此etcd安裝完成。。。

4.5、haproxy安裝配置

1)、master01配置(需要注意的是端口自定義為16443)
yum -y install haproxy
master01、master02、master03都安裝haproxy

vim /etc/haproxy/haproxy.cfg
global
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000


#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
    mode                 tcp
    bind                 *:16443
    option               tcplog
    default_backend      kubernetes-apiserver

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
    mode        tcp
    balance     roundrobin
    server      k8s01 10.8.13.81:6443 check
    server      k8s02 10.8.13.82:6443 check
    server      k8s03 10.8.13.83:6443 check

#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
    bind                 *:1080
    stats auth           admin:awesomePassword
    stats refresh        5s
    stats realm          HAProxy\ Statistics
    stats uri            /admin?stats

2）拷貝master01的haproxy到master02和master03對(duì)應(yīng)路徑下

scp /etc/haproxy/haproxy.cfg 10.8.13.82:/etc/haproxy/haproxy.cfg
scp /etc/haproxy/haproxy.cfg 10.8.13.83:/etc/haproxy/haproxy.cfg

3)啟動(dòng)haproxy服務(wù)，并加入開(kāi)機(jī)自啟動(dòng)(master三個(gè)節(jié)點(diǎn)都執(zhí)行)

systemctl daemon-reload
systemctl enable haproxy
systemctl start haproxy

4.6、keepalived安裝配置

1）master01配置
yum -y install keepalived
master01、master02、master03都安裝keepalived

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.8.13.80
    }
    track_script {
        check_haproxy
    }
}

2）master02配置

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 99
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.8.13.80
    }
    track_script {
        check_haproxy
    }
}

3）master03配置

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
}

vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 3
    weight -2
    fall 10
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens160
    virtual_router_id 51
    priority 98
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.8.13.80
    }
    track_script {
        check_haproxy
    }
}

4）啟動(dòng)keepalived服務(wù)

systemctl daemon-reload
systemctl enable keepalived
systemctl start keepalived
[root@master01 ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2019-05-10 20:33:33 CST; 3 days ago
  Process: 992 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1115 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─1115 /usr/sbin/keepalived -D
           ├─1116 /usr/sbin/keepalived -D
           └─1117 /usr/sbin/keepalived -D

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[root@hwzx-test-cmpmaster01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:90:22:79 brd ff:ff:ff:ff:ff:ff
    inet 10.8.13.81/24 brd 10.8.13.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet 10.8.13.80/32 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::6772:8bb6:b50c:57fe/64 scope link 
       valid_lft forever preferred_lft forever
vip在master01上

5)keepalived配置注意事項(xiàng)

>1.killall -0 根據(jù)進(jìn)程名稱(chēng)檢測(cè)進(jìn)程是否存活，如果服務(wù)器沒(méi)有該命令，請(qǐng)使用yum install psmisc -y安裝
>2.第一個(gè)master節(jié)點(diǎn)的state為MASTER，其他master節(jié)點(diǎn)的state為BACKUP
>3.priority表示各個(gè)節(jié)點(diǎn)的優(yōu)先級(jí)，范圍：0～250（非強(qiáng)制要求）

4.7、生成kubernets證書(shū)與私鑰

1）制作kubernetes ca證書(shū)

cd /k8s/kubernetes/ssl
cat << EOF | tee ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat << EOF | tee ca-csr.json
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@master01 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2019/05/01 09:47:08 [INFO] generating a new CA key and certificate from CSR
2019/05/01 09:47:08 [INFO] generate received request
2019/05/01 09:47:08 [INFO] received CSR
2019/05/01 09:47:08 [INFO] generating key: rsa-2048
2019/05/01 09:47:08 [INFO] encoded CSR
2019/05/01 09:47:08 [INFO] signed certificate with serial number 156611735285008649323551446985295933852737436614
[root@master01 ssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

2）制作apiserver證書(shū)
注意hosts處，所有IP都寫(xiě)進(jìn)去，包括vip

cat << EOF | tee server-csr.json
{
    "CN": "kubernetes",
    "hosts": [
      "10.254.0.1",
      "127.0.0.1",
      "10.8.13.81",
      "10.8.13.82",
      "10.8.13.83",
      "10.8.13.84",
      "10.8.13.85",
      "10.8.13.80",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
[root@master01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
2019/05/01 09:51:56 [INFO] generate received request
2019/05/01 09:51:56 [INFO] received CSR
2019/05/01 09:51:56 [INFO] generating key: rsa-2048
2019/05/01 09:51:56 [INFO] encoded CSR
2019/05/01 09:51:56 [INFO] signed certificate with serial number 399376216731194654868387199081648887334508501005
2019/05/01 09:51:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master01 ssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  server.csr  server-csr.json  server-key.pem  server.pem

3）制作kube-proxy證書(shū)

cat << EOF | tee kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
[root@master01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2019/05/01 09:52:40 [INFO] generate received request
2019/05/01 09:52:40 [INFO] received CSR
2019/05/01 09:52:40 [INFO] generating key: rsa-2048
2019/05/01 09:52:40 [INFO] encoded CSR
2019/05/01 09:52:40 [INFO] signed certificate with serial number 633932731787505365511506755558794469389165123417
2019/05/01 09:52:40 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@master01 ssl]# ls
ca-config.json  ca-csr.json  ca.pem          kube-proxy-csr.json  kube-proxy.pem  server-csr.json  server.pem
ca.csr          ca-key.pem   kube-proxy.csr  kube-proxy-key.pem   server.csr      server-key.pem

4.8部署kubernetes server

kubernetes master 節(jié)點(diǎn)運(yùn)行如下組件：

kube-apiserver
kube-scheduler
kube-controller-manager
kube-scheduler 和 kube-controller-manager 以集群模式運(yùn)行，通過(guò) leader 選舉產(chǎn)生一個(gè)工作進(jìn)程，其它進(jìn)程處于阻塞模式。
1）解壓縮文件

tar -zxf kubernetes-server-linux-amd64.tar.gz 
cd kubernetes/server/bin/
cp kube-scheduler kube-apiserver kube-controller-manager kubectl /k8s/kubernetes/bin/

2）部署kube-apiserver組件
創(chuàng)建TLS Bootstrapping Token

[root@master01 bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
af93a4194e7bcf7f05dc0bab3a6e97cd
 
vim /k8s/kubernetes/cfg/token.csv
af93a4194e7bcf7f05dc0bab3a6e97cd,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

創(chuàng)建Apiserver配置文件
注：--bind-address=當(dāng)前節(jié)點(diǎn)ip
--advertise-address=當(dāng)前節(jié)點(diǎn)ip

vim /k8s/kubernetes/cfg/kube-apiserver 
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379 \
--bind-address=10.8.13.81 \
--secure-port=6443 \
--advertise-address=10.8.13.81 \
--allow-privileged=true \
--service-cluster-ip-range=10.254.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/k8s/kubernetes/ssl/server.pem  \
--tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/k8s/etcd/ssl/server-key.pem"

創(chuàng)建apiserver systemd文件

vim /usr/lib/systemd/system/kube-apiserver.service 
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
 
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver
ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

拷貝master01 kubernetes的證書(shū)、配置文件、啟動(dòng)文件到master02和master03對(duì)應(yīng)路徑下

scp /k8s/kubernetes/ssl/* 10.8.13.82:/k8s/kubernetes/ssl/
scp /k8s/kubernetes/ssl/* 10.8.13.83:/k8s/kubernetes/ssl/
scp /k8s/kubernetes/cfg/* 10.8.13.82:/k8s/kubernetes/cfg/
scp /k8s/kubernetes/cfg/* 10.8.13.83:/k8s/kubernetes/cfg/
scp /k8s/kubernetes/bin/* 10.8.13.82:/k8s/kubernetes/bin/
scp /k8s/kubernetes/bin/* 10.8.13.83:/k8s/kubernetes/bin/
scp /usr/lib/systemd/system/kube-apiserver.service 10.8.13.82:/usr/lib/systemd/system
scp /usr/lib/systemd/system/kube-apiserver.service 10.8.13.83:/usr/lib/systemd/system

5)、修改master02、master03 etcd的conf配置文件
matser02 etcd.conf配置如下：

ssh 10.8.13.82
vim /k8s/kubernetes/cfg/kube-apiserver 
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379 \
--bind-address=10.8.13.82 \
--secure-port=6443 \
--advertise-address=10.8.13.82 \
--allow-privileged=true \
--service-cluster-ip-range=10.254.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/k8s/kubernetes/ssl/server.pem  \
--tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/k8s/etcd/ssl/server-key.pem"

ssh 10.8.13.83
vim /k8s/kubernetes/cfg/kube-apiserver 
KUBE_APISERVER_OPTS="--logtostderr=true \
--v=4 \
--etcd-servers=https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379 \
--bind-address=10.8.13.83 \
--secure-port=6443 \
--advertise-address=10.8.13.83 \
--allow-privileged=true \
--service-cluster-ip-range=10.254.0.0/16 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/k8s/kubernetes/cfg/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/k8s/kubernetes/ssl/server.pem  \
--tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem \
--client-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/k8s/etcd/ssl/ca.pem \
--etcd-certfile=/k8s/etcd/ssl/server.pem \
--etcd-keyfile=/k8s/etcd/ssl/server-key.pem"

啟動(dòng)服務(wù)

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
[root@elasticsearch01 bin]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2019-05-10 20:33:32 CST; 2 days ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 705 (kube-apiserver)
   CGroup: /system.slice/kube-apiserver.service
           └─705 /k8s/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379 --bind-address=10.8.13.81 --secure-port=6443 --advertise-address=10.8.13.81 --allow-privileged=true --s...

5月 13 16:00:43 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:43.495504     705 wrap.go:47] GET /api/v1/namespaces/default/endpoints/kubernetes: (3.700854ms) 200 [kube-apiserver/v1.13.1 (linux/amd64) kubernetes/eec55b9 10.8.13.81:56744]
5月 13 16:00:45 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:45.955530     705 wrap.go:47] GET /api/v1/services?resourceVersion=37540&timeout=6m29s&timeoutSeconds=389&watch=true: (6m29.001574609s) 200 [kube-proxy/v1.13.1 (linux/amd64) kub... 10.8.13.81:56844]
5月 13 16:00:45 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:45.958607     705 get.go:247] Starting watch for /api/v1/services, rv=37540 labels= fields= timeout=8m28s
5月 13 16:00:46 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:46.323978     705 wrap.go:47] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s: (4.410282ms) 200 [kube-scheduler/v1.13.1 (linux/amd64) kubernetes/eec55b9/...n 127.0.0.1:43276]
5月 13 16:00:46 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:46.371766     705 wrap.go:47] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: (3.606335ms) 200 [kube-controller-manager/v1.13.1 (linux/amd64) k...n 127.0.0.1:43776]
5月 13 16:00:46 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:46.376888     705 wrap.go:47] GET /apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=32859&timeout=5m5s&timeoutSeconds=305&watch=true: (5m5.001015872s) 200 [kube-apiser... 10.8.13.81:56744]
5月 13 16:00:46 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:46.377312     705 reflector.go:357] k8s.io/kube-aggregator/pkg/client/informers/internalversion/factory.go:117: Watch close - *apiregistration.APIService total 0 items received
5月 13 16:00:46 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:46.378469     705 get.go:247] Starting watch for /apis/apiregistration.k8s.io/v1/apiservices, rv=32859 labels= fields= timeout=8m12s
5月 13 16:00:49 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:49.206602     705 wrap.go:47] GET /api/v1/namespaces/kube-system/endpoints/kube-controller-manager?timeout=10s: (4.541086ms) 200 [kube-controller-manager/v1.13.1 (linux/amd64) k...n 127.0.0.1:43776]
5月 13 16:00:50 hwzx-test-cmpmaster01 kube-apiserver[705]: I0513 16:00:50.027213     705 wrap.go:47] GET /api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s: (4.418662ms) 200 [kube-scheduler/v1.13.1 (linux/amd64) kubernetes/eec55b9/...n 127.0.0.1:43276]
Hint: Some lines were ellipsized, use -l to show in full.

[root@master01 bin]# ps -ef |grep kube-apiserver
root       705     1  3 5月10 ?       02:35:10 /k8s/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379 --bind-address=10.8.13.81 --secure-port=6443 --advertise-address=10.8.13.81 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction --authorization-mode=RBAC,Node --enable-bootstrap-token-auth --token-auth-file=/k8s/kubernetes/cfg/token.csv --service-node-port-range=30000-50000 --tls-cert-file=/k8s/kubernetes/ssl/server.pem --tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem --client-ca-file=/k8s/kubernetes/ssl/ca.pem --service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem --etcd-cafile=/k8s/etcd/ssl/ca.pem --etcd-certfile=/k8s/etcd/ssl/server.pem --etcd-keyfile=/k8s/etcd/ssl/server-key.pem
root      7098 24767  0 15:57 pts/0    00:00:00 grep --color=auto kube-apiserver
[root@master01 bin]# netstat -tulpn |grep kube-apiserve
tcp        0      0 10.8.13.81:6443         0.0.0.0:*               LISTEN      705/kube-apiserver  
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      705/kube-apiserver

3）部署kube-scheduler組件
創(chuàng)建kube-scheduler配置文件

vim  /k8s/kubernetes/cfg/kube-scheduler 
KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect"

參數(shù)備注：

--address：在 127.0.0.1:10251 端口接收 http /metrics 請(qǐng)求；kube-scheduler 目前還不支持接收 https 請(qǐng)求；
--kubeconfig：指定 kubeconfig 文件路徑，kube-scheduler 使用它連接和驗(yàn)證 kube-apiserver；
--leader-elect=true：集群運(yùn)行模式，啟用選舉功能；被選為 leader 的節(jié)點(diǎn)負(fù)責(zé)處理工作，其它節(jié)點(diǎn)為阻塞狀態(tài)；
創(chuàng)建kube-scheduler systemd文件

vim /usr/lib/systemd/system/kube-scheduler.service 
 
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
 
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler
ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

拷貝master01 kube-scheduler配置文件、啟動(dòng)文件到master02和master03對(duì)應(yīng)路徑下

scp /k8s/kubernetes/cfg/kube-scheduler 10.8.13.82:/k8s/kubernetes/cfg/kube-scheduler
scp /k8s/kubernetes/cfg/kube-scheduler 10.8.13.83:/k8s/kubernetes/cfg/kube-scheduler
scp /usr/lib/systemd/system/kube-scheduler.service 10.8.13.82:/usr/lib/systemd/system/kube-scheduler.service
scp /usr/lib/systemd/system/kube-scheduler.service 10.8.13.83:/usr/lib/systemd/system/kube-scheduler.service

啟動(dòng)服務(wù)

systemctl daemon-reload
systemctl enable kube-scheduler.service 
systemctl start kube-scheduler.service
[root@master01 bin]# systemctl status kube-scheduler.service
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2019-05-10 20:33:32 CST; 2 days ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 693 (kube-scheduler)
   CGroup: /system.slice/kube-scheduler.service
           └─693 /k8s/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect

5月 13 16:10:49 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:49.024121     693 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_7601efea-7319-11e9-8964-0050569059b4 and has not yet expired
5月 13 16:10:49 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:49.024161     693 leaderelection.go:210] failed to acquire lease kube-system/kube-scheduler
5月 13 16:10:51 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:51.151743     693 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_7601efea-7319-11e9-8964-0050569059b4 and has not yet expired
5月 13 16:10:51 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:51.151799     693 leaderelection.go:210] failed to acquire lease kube-system/kube-scheduler
5月 13 16:10:53 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:53.434965     693 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_7601efea-7319-11e9-8964-0050569059b4 and has not yet expired
5月 13 16:10:53 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:53.434999     693 leaderelection.go:210] failed to acquire lease kube-system/kube-scheduler
5月 13 16:10:57 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:57.571674     693 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_7601efea-7319-11e9-8964-0050569059b4 and has not yet expired
5月 13 16:10:57 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:10:57.571707     693 leaderelection.go:210] failed to acquire lease kube-system/kube-scheduler
5月 13 16:11:01 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:11:01.914369     693 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_7601efea-7319-11e9-8964-0050569059b4 and has not yet expired
5月 13 16:11:01 hwzx-test-cmpmaster01 kube-scheduler[693]: I0513 16:11:01.914411     693 leaderelection.go:210] failed to acquire lease kube-system/kube-scheduler

4）部署kube-controller-manager組件
創(chuàng)建kube-controller-manager配置文件

vim /k8s/kubernetes/cfg/kube-controller-manager
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.254.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/k8s/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem"

創(chuàng)建kube-controller-manager systemd文件

vim /usr/lib/systemd/system/kube-controller-manager.service 
 
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
 
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager
ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

拷貝master01 kube-controller-manager配置文件、啟動(dòng)文件到master02和master03對(duì)應(yīng)路徑下

scp /k8s/kubernetes/cfg/kube-controller-manager 10.8.13.82:/k8s/kubernetes/cfg/kube-controller-manager
scp /k8s/kubernetes/cfg/kube-controller-manager 10.8.13.83:/k8s/kubernetes/cfg/kube-controller-manager
scp /usr/lib/systemd/system/kube-controller-manager.service 10.8.13.82:/usr/lib/systemd/system/kube-controller-manager.service
scp /usr/lib/systemd/system/kube-controller-manager.service 10.8.13.83:/usr/lib/systemd/system/kube-controller-manager.service

啟動(dòng)服務(wù)

systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager
[root@master01 bin]# systemctl status kube-controller-manager
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2019-05-10 20:33:32 CST; 2 days ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 685 (kube-controller)
   CGroup: /system.slice/kube-controller-manager.service
           └─685 /k8s/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=true --address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/k8s/kubernetes/ssl/ca...

5月 13 16:16:45 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:45.539102     685 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_823f19e6-7319-11e9-94be-0050569059b4 and has not yet expired
5月 13 16:16:45 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:45.539136     685 leaderelection.go:210] failed to acquire lease kube-system/kube-controller-manager
5月 13 16:16:48 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:48.767187     685 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_823f19e6-7319-11e9-94be-0050569059b4 and has not yet expired
5月 13 16:16:48 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:48.767221     685 leaderelection.go:210] failed to acquire lease kube-system/kube-controller-manager
5月 13 16:16:50 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:50.939294     685 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_823f19e6-7319-11e9-94be-0050569059b4 and has not yet expired
5月 13 16:16:50 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:50.939329     685 leaderelection.go:210] failed to acquire lease kube-system/kube-controller-manager
5月 13 16:16:53 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:53.212185     685 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_823f19e6-7319-11e9-94be-0050569059b4 and has not yet expired
5月 13 16:16:53 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:53.212218     685 leaderelection.go:210] failed to acquire lease kube-system/kube-controller-manager
5月 13 16:16:57 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:57.291399     685 leaderelection.go:289] lock is held by hwzx-test-cmpmaster03_823f19e6-7319-11e9-94be-0050569059b4 and has not yet expired
5月 13 16:16:57 hwzx-test-cmpmaster01 kube-controller-manager[685]: I0513 16:16:57.291430     685 leaderelection.go:210] failed to acquire lease kube-system/kube-controller-manager

4.9、驗(yàn)證kubeserver服務(wù)

設(shè)置環(huán)境變量(所有服務(wù)器都執(zhí)行此步)

vim /etc/profile
PATH=/k8s/kubernetes/bin:$PATH
source /etc/profile

查看master服務(wù)狀態(tài)

[root@master01 ~]# kubectl get cs,nodes
NAME                                 STATUS    MESSAGE             ERROR
componentstatus/scheduler            Healthy   ok                  
componentstatus/controller-manager   Healthy   ok                  
componentstatus/etcd-0               Healthy   {"health":"true"}   
componentstatus/etcd-1               Healthy   {"health":"true"}   
componentstatus/etcd-2               Healthy   {"health":"true"} 
至此master組件安裝完畢

五、Node部署(node01、node02安裝)

kubernetes work 節(jié)點(diǎn)運(yùn)行如下組件：
docker
kubelet
kube-proxy
flannel

5.1 Docker環(huán)境安裝

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort -r
yum install docker-ce -y
systemctl start docker && systemctl enable docker

5.2 部署kubelet組件

kublet 運(yùn)行在每個(gè) worker 節(jié)點(diǎn)上，接收 kube-apiserver 發(fā)送的請(qǐng)求，管理 Pod 容器，執(zhí)行交互式命令，如exec、run、logs 等;
kublet 啟動(dòng)時(shí)自動(dòng)向 kube-apiserver 注冊(cè)節(jié)點(diǎn)信息，內(nèi)置的 cadvisor 統(tǒng)計(jì)和監(jiān)控節(jié)點(diǎn)的資源使用情況;
為確保安全，只開(kāi)啟接收 https 請(qǐng)求的安全端口，對(duì)請(qǐng)求進(jìn)行認(rèn)證和授權(quán)，拒絕未授權(quán)的訪問(wèn)(如apiserver、heapster)。

1)、安裝二進(jìn)制文件

wget https://dl.k8s.io/v1.13.1/kubernetes-node-linux-amd64.tar.gz
tar zxvf kubernetes-node-linux-amd64.tar.gz
cd kubernetes/node/bin/
cp kube-proxy kubelet kubectl /k8s/kubernetes/bin/

2)、從master01復(fù)制相關(guān)證書(shū)到node01和node02節(jié)點(diǎn)

[root@master01 ssl]# cd /k8s/kubernetes/ssl/
[root@master01 ssl]# scp *.pem 10.8.13.84:/k8s/kubernetes/ssl/
root@10.8.13.84's password: 
ca-key.pem                                                                                         100% 1679   914.6KB/s   00:00    
ca.pem                                                                                             100% 1359     1.0MB/s   00:00    
kube-proxy-key.pem                                                                                 100% 1675     1.2MB/s   00:00    
kube-proxy.pem                                                                                     100% 1403     1.1MB/s   00:00    
server-key.pem                                                                                     100% 1679   809.1KB/s   00:00    
server.pem                                                                                         100% 1675     1.2MB/s   00:00
[root@master01 ssl]# scp /k8s/etcd/ssl/* 10.8.13.84:/k8s/etcd/ssl/
[root@master01 ssl]# scp /k8s/etcd/bin/* 10.8.13.84:/k8s/etcd/bin/

[root@master01 ssl]# scp *.pem 10.8.13.85:/k8s/kubernetes/ssl/
root@10.8.13.85's password: 
ca-key.pem                                                                                         100% 1679   914.6KB/s   00:00    
ca.pem                                                                                             100% 1359     1.0MB/s   00:00    
kube-proxy-key.pem                                                                                 100% 1675     1.2MB/s   00:00    
kube-proxy.pem                                                                                     100% 1403     1.1MB/s   00:00    
server-key.pem                                                                                     100% 1679   809.1KB/s   00:00    
server.pem                                                                                         100% 1675     1.2MB/s   00:00
[root@master01 ssl]# scp /k8s/etcd/ssl/* 10.8.13.85:/k8s/etcd/ssl/
[root@master01 ssl]# scp /k8s/etcd/bin/* 10.8.13.85:/k8s/etcd/bin/

3)、創(chuàng)建kubelet bootstrap kubeconfig文件
通過(guò)腳本實(shí)現(xiàn)
KUBE_APISERVER=vip:haproxy中自定義的端口
BOOTSTRAP_TOKEN=部署kube-apiserver中生成的token

vim /k8s/kubernetes/cfg/environment.sh
#!/bin/bash
#創(chuàng)建kubelet bootstrapping kubeconfig 
BOOTSTRAP_TOKEN=af93a4194e7bcf7f05dc0bab3a6e97cd
KUBE_APISERVER="https://10.8.13.80:16443"
#設(shè)置集群參數(shù)
kubectl config set-cluster kubernetes \
  --certificate-authority=/k8s/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig
 
#設(shè)置客戶端認(rèn)證參數(shù)
kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig
 
# 設(shè)置上下文參數(shù)
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig
 
# 設(shè)置默認(rèn)上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
 
#----------------------
 
# 創(chuàng)建kube-proxy kubeconfig文件
 
kubectl config set-cluster kubernetes \
  --certificate-authority=/k8s/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig
 
kubectl config set-credentials kube-proxy \
  --client-certificate=/k8s/kubernetes/ssl/kube-proxy.pem \
  --client-key=/k8s/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig
 
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig
 
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

執(zhí)行腳本

[root@node01 cfg]# cd /k8s/kubernetes/cfg/
[root@node01 cfg]# sh environment.sh 
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
Switched to context "default".
[root@node01 cfg]# ls
bootstrap.kubeconfig  environment.sh  kube-proxy.kubeconfig

4)、創(chuàng)建kubelet參數(shù)配置模板文件
address:node節(jié)點(diǎn)IP

vim /k8s/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 10.8.13.84
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.254.0.10"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true

5)、創(chuàng)建kubelet配置文件
--hostname-override=node節(jié)點(diǎn)IP

vim /k8s/kubernetes/cfg/kubelet
 
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=10.8.13.84 \
--kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/k8s/kubernetes/cfg/kubelet.config \
--cert-dir=/k8s/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

6)、創(chuàng)建kubelet systemd文件

vim /usr/lib/systemd/system/kubelet.service 
 
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service
 
[Service]
EnvironmentFile=/k8s/kubernetes/cfg/kubelet
ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process
 
[Install]
WantedBy=multi-user.target

7)、將kubelet-bootstrap用戶綁定到系統(tǒng)集群角色(在master01執(zhí)行)

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap

注意這個(gè)默認(rèn)連接localhost:8080端口，可以在master上操作

[root@master01 ssl]# kubectl create clusterrolebinding kubelet-bootstrap \
>   --clusterrole=system:node-bootstrapper \
>   --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created

8)、復(fù)制node01kubelet配置和啟動(dòng)服務(wù)文件到node02相對(duì)應(yīng)路徑

scp /k8s/kubernetes/cfg/* 10.8.13.85:/k8s/kubernetes/cfg/
scp /usr/lib/systemd/system/kubelet.service 10.8.13.85:/usr/lib/systemd/system/kubelet.service

9)、修改node02中kubelet.config和kubelet文件中的nodeIP
node02中kubelet.config配置
address:node節(jié)點(diǎn)IP

vim /k8s/kubernetes/cfg/kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 10.8.13.85
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.254.0.10"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true

node02中kubelet配置
--hostname-override=node節(jié)點(diǎn)IP

vim /k8s/kubernetes/cfg/kubelet
 
KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=10.8.13.85 \
--kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
--config=/k8s/kubernetes/cfg/kubelet.config \
--cert-dir=/k8s/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

10)、啟動(dòng)服務(wù)
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet

[root@node01 ~]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-05-10 20:31:30 CST; 3 days ago
 Main PID: 8583 (kubelet)
   Memory: 45.5M
   CGroup: /system.slice/kubelet.service
           └─8583 /k8s/kubernetes/bin/kubelet --logtostderr=true --v=4 --hostname-override=10.8.13.84 --kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig --config=/k8s/kubernetes/cfg/kubelet.config --cer...

11)、Master接受kubelet CSR請(qǐng)求(master01操作，接受兩個(gè)node節(jié)點(diǎn))
可以手動(dòng)或自動(dòng) approve CSR 請(qǐng)求。推薦使用自動(dòng)的方式，因?yàn)閺?v1.8 版本開(kāi)始，可以自動(dòng)輪轉(zhuǎn)approve csr 后生成的證書(shū)，如下是手動(dòng) approve CSR請(qǐng)求操作方法
查看CSR列表

[root@master01 ssl]# kubectl get csr
NAME                                                   AGE    REQUESTOR           CONDITION
node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc   102s   kubelet-bootstrap   Pending

接受node

[root@master01 ssl]# kubectl certificate approve node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc
certificatesigningrequest.certificates.k8s.io/node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc approved

再查看CSR

[root@master01 ssl]# kubectl get csr
NAME                                                   AGE     REQUESTOR           CONDITION
node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc   5m13s   kubelet-bootstrap   Approved,Issued

5.3部署kube-proxy組件(node01執(zhí)行)

kube-proxy 運(yùn)行在所有 node節(jié)點(diǎn)上，它監(jiān)聽(tīng) apiserver 中 service 和 Endpoint 的變化情況，創(chuàng)建路由規(guī)則來(lái)進(jìn)行服務(wù)負(fù)載均衡
1)、創(chuàng)建 kube-proxy 配置文件
--hostname-override=node節(jié)點(diǎn)IP

vim /k8s/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=10.8.13.84 \
--cluster-cidr=10.254.0.0/16 \
--kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"

2)、創(chuàng)建kube-proxy systemd文件

vim /usr/lib/systemd/system/kube-proxy.service 
 
[Unit]
Description=Kubernetes Proxy
After=network.target
 
[Service]
EnvironmentFile=-/k8s/kubernetes/cfg/kube-proxy
ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

3)、復(fù)制node01kube-proxy配置和服務(wù)啟動(dòng)文件到node02相對(duì)應(yīng)路徑

scp /k8s/kubernetes/cfg/kube-proxy 10.8.13.85:/k8s/kubernetes/cfg/kube-proxy
scp /usr/lib/systemd/system/kube-proxy.service 10.8.13.85:/usr/lib/systemd/system/kube-proxy.service

4)、修改node02kube-proxy配置文件如下
--hostname-override=node節(jié)點(diǎn)IP

vim /k8s/kubernetes/cfg/kube-proxy
KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=10.8.13.85 \
--cluster-cidr=10.254.0.0/16 \
--kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"

5)、啟動(dòng)服務(wù)
systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy

[root@node01 ~]# systemctl status kube-proxy.service 
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2019-05-10 20:31:31 CST; 3 days ago
 Main PID: 8669 (kube-proxy)
   Memory: 9.9M
   CGroup: /system.slice/kube-proxy.service
           ? 8669 /k8s/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=10.8.13.84 --cluster-cidr=10.254.0.0/16 --kubeconfig...

May 14 09:07:50 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:50.634641    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:51 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:51.365166    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:52 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:52.647317    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:53 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:53.375833    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:54 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:54.658691    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:55 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:55.387881    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:56 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:56.670562    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:57 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:57.398763    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:58 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:58.682049    8669 config.go:141] Calling handler.OnEndpointsUpdate
May 14 09:07:59 hwzx-test-cmpnode01 kube-proxy[8669]: I0514 09:07:59.411141    8669 config.go:141] Calling handler.OnEndpointsUpdate

6)、查看集群狀態(tài)

[root@master01 ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE     VERSION
10.8.13.84   Ready    <none>   3d13h   v1.14.1
10.8.13.85   Ready    <none>   3d13h   v1.14.1
至此node組件安裝完成

六、Flanneld網(wǎng)絡(luò)部署(以node01為例，node02同樣操作)

默認(rèn)沒(méi)有flanneld網(wǎng)絡(luò)，Node節(jié)點(diǎn)間的pod不能通信，只能Node內(nèi)通信，為了部署步驟簡(jiǎn)潔明了，故flanneld放在后面安裝
flannel服務(wù)需要先于docker啟動(dòng)。flannel服務(wù)啟動(dòng)時(shí)主要做了以下幾步的工作：
從etcd中獲取network的配置信息
劃分subnet，并在etcd中進(jìn)行注冊(cè)
將子網(wǎng)信息記錄到/run/flannel/subnet.env中

6.1 etcd注冊(cè)網(wǎng)段

[root@node01 ~]# /k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379,https://10.8.13.84:2379,https://10.8.13.85:2379"  set /k8s/network/config  '{ "Network": "10.254.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "10.254.0.0/16", "Backend": {"Type": "vxlan"}}

flanneld 當(dāng)前版本 (v0.11.0) 不支持 etcd v3，故使用 etcd v2 API 寫(xiě)入配置 key 和網(wǎng)段數(shù)據(jù)；
寫(xiě)入的 Pod 網(wǎng)段 ${CLUSTER_CIDR} 必須是 /16 段地址，必須與 kube-controller-manager 的 --cluster-cidr 參數(shù)值一致；

6.2 flannel安裝

1)、解壓安裝

tar -zxf flannel-v0.11.0-linux-amd64.tar.gz
mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/

2)、配置flanneld

vim /k8s/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://10.8.13.81:2379,https://10.8.13.82:2379,https://10.8.13.83:2379,https://10.8.13.84:2379,https://10.8.13.85:2379 -etcd-cafile=/k8s/etcd/ssl/ca.pem -etcd-certfile=/k8s/etcd/ssl/server.pem -etcd-keyfile=/k8s/etcd/ssl/server-key.pem -etcd-prefix=/k8s/network"

3)、創(chuàng)建flanneld systemd文件

vim /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
 
[Service]
Type=notify
EnvironmentFile=/k8s/kubernetes/cfg/flanneld
ExecStart=/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

注意

mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網(wǎng)網(wǎng)段信息寫(xiě)入 /run/flannel/docker 文件，后續(xù) docker 啟動(dòng)時(shí) 使用這個(gè)文件中的環(huán)境變量配置 docker0 網(wǎng)橋；
flanneld 使用系統(tǒng)缺省路由所在的接口與其它節(jié)點(diǎn)通信，對(duì)于有多個(gè)網(wǎng)絡(luò)接口（如內(nèi)網(wǎng)和公網(wǎng)）的節(jié)點(diǎn)，可以用 -iface 參數(shù)指定通信接口;
flanneld 運(yùn)行時(shí)需要 root 權(quán)限；

3）配置Docker啟動(dòng)指定子網(wǎng)
添加EnvironmentFile=/run/flannel/subnet.env，修改ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS即可

vim /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
 
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
 
[Install]
WantedBy=multi-user.target

4)、啟動(dòng)服務(wù)
注意啟動(dòng)flannel前要關(guān)閉docker及相關(guān)的kubelet這樣flannel才會(huì)覆蓋docker0網(wǎng)橋

systemctl daemon-reload
systemctl stop docker
systemctl start flanneld
systemctl enable flanneld
systemctl start docker
systemctl restart kubelet
systemctl restart kube-proxy

5)、驗(yàn)證服務(wù)

[root@node01 bin]# cat /run/flannel/subnet.env 
DOCKER_OPT_BIP="--bip=10.254.88.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.254.88.1/24 --ip-masq=false --mtu=1450"

注意查看docker0和flannel是不是屬于同一網(wǎng)段

[root@node01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:90:67:d1 brd ff:ff:ff:ff:ff:ff
    inet 10.8.13.84/24 brd 10.8.13.255 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::802:2c0f:a197:38a7/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:5c:18:5b:93 brd ff:ff:ff:ff:ff:ff
    inet 10.254.88.1/24 brd 10.254.88.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:5cff:fe18:5b93/64 scope link 
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN 
    link/ether 8e:f6:f8:87:47:ee brd ff:ff:ff:ff:ff:ff
    inet 10.254.88.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8cf6:f8ff:fe87:47ee/64 scope link 
       valid_lft forever preferred_lft forever
至此flannel安裝完成

[root@hwzx-test-cmpmaster01 ~]# kubectl get nodes,cs
NAME              STATUS   ROLES    AGE     VERSION
node/10.8.13.84   Ready    <none>   3d13h   v1.14.1
node/10.8.13.85   Ready    <none>   3d13h   v1.14.1

NAME                                 STATUS    MESSAGE             ERROR
componentstatus/controller-manager   Healthy   ok                  
componentstatus/scheduler            Healthy   ok                  
componentstatus/etcd-1               Healthy   {"health":"true"}   
componentstatus/etcd-0               Healthy   {"health":"true"}   
componentstatus/etcd-2               Healthy   {"health":"true"}
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書(shū)系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容