鍛骨境-第4層 k8s生產環(huán)境高可用集群搭建

存儲的高可用

官網地址
項目源碼地址
上一層文章-開發(fā)環(huán)境k8s 高可用master 的搭建

申明：

這里用的loadbalance ，還是上層文章開發(fā)環(huán)境的nginx .

stream {
    server {
        listen 8443;
        proxy_pass kube_apiserver;
    }

    upstream kube_apiserver {
        server 192.168.10.133:6443 weight=50 max_fails=3 fail_timeout=5s;
        server 192.168.10.134:6443 weight=50 max_fails=3 fail_timeout=5s;
        
    }
}

說明

Etcd是一個高可用的鍵值存儲系統(tǒng)，主要用于共享配置和服務發(fā)現(xiàn)，它通過Raft一致性算法處理日志復制以保證強一致性，我們可以理解它為一個高可用強一致性的服務發(fā)現(xiàn)存儲倉庫。

在kubernetes集群中，etcd主要用于配置共享和服務發(fā)現(xiàn)

Etcd主要解決的是分布式系統(tǒng)中數據一致性的問題，而分布式系統(tǒng)中的數據分為控制數據和應用數據，etcd處理的數據類型為控制數據，對于很少量的應用數據也可以進行處理。

Etcd 好處：

  • 簡單。
    使用Go語言編寫部署簡單；使用HTTP作為接口使用簡單；使用Raft算法保證強一致性
    讓用戶易于理解。
  • 數據持久化。
    etcd默認數據一更新就進行持久化。
  • 安全。
    etcd支持SSL客戶端安全認證。

示例部署ETCD高可用集群

上一節(jié)的問題，etcd 沒有集群化

  • 后果就是etcd 意外死亡，那么k8s 將不可用
  • etcd 還需要做的就是，數據的備份

開始搭建etcd 集群

我們完整的結構應該是這樣的：

image.png
  • 我們部署的etcd 是容器化的，并沒有獨立成一個集群
  • 參考官網 可知，在上一步的kubeadm-config.yaml 中沒有配置 etcd 的地址，通過describe 命令查看etcd 也可以知道配置有問題。
    上面基于容器的做法很容易出問題，這次按照生產的標準搭建高可用k8s 集群。

開始

安裝cdssl 證書生成工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

創(chuàng)建 CA 配置文件（ca-config.json）

{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "kubernetes": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
            "client auth"
                ]
            },
            "etcd": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

創(chuàng)建 CA 證書簽名請求（ca-csr.json）

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shenzhen",
      "L": "shenzhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成 CA 證書和私鑰
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

[root@k8s-node1 bin]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/10/29 21:10:59 [INFO] generating a new CA key and certificate from CSR
2019/10/29 21:10:59 [INFO] generate received request
2019/10/29 21:10:59 [INFO] received CSR
2019/10/29 21:10:59 [INFO] generating key: rsa-2048
2019/10/29 21:11:00 [INFO] encoded CSR
2019/10/29 21:11:00 [INFO] signed certificate with serial number 213974363926484894183998850838526320112981116568
[root@k8s-node1 bin]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  cfssl  cfssl-certinfo  cfssljson

創(chuàng)建 kubernetes證書簽名請求（k8s-csr.json），這里注意hosts需要配置kubernetes 自身的域名，否則會導致coreDns 不可用（其實就是認證失敗，調度不了etcd 接口）

{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.10.133",
    "192.168.10.134",
    "192.168.10.135",
    "10.96.0.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"

  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shenzhen",
      "L": "shenzhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

執(zhí)行

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes k8s-csr.json | cfssljson -bare kubernetes


[root@k8s-node1 bin]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  cfssl  cfssl-certinfo  cfssljson  k8s-csr.json  kubernetes.csr  kubernetes-key.pem  kubernetes.pem
[root@k8s-node1 bin]#

復制pem后綴的證書到/etc/kubernetes/ssl下,每個節(jié)點都做
這里說明下，kubeadm reset 命令會刪除 /etc/kubernetes/pki 下所有的文件。所有有些證書，比如etcd 的，我們存在另一個地方，一個是本地的master 主機上跑了etcd 服務需要，還一個就是存?zhèn)€備份。

在后面，可以看到我把這里的etcd 的各種證書，每次 reset 完之后，都會在 /etc/kubernetes/pki 下創(chuàng)建 etcd目錄，然后 cp -r /etc/kubernetes/ssl/*.pem /etc/kubernetes/pki /etcd 。

可以認為。k8s 也是etcd 的一個客戶端，etcd 服務做了tls ,客戶端當然需要證書才能訪問。

# mkdir -p /etc/kubernetes/ssl
# cp *.pem /etc/kubernetes/ssl

去https://github.com/etcd-io/etcd/releases
下載安裝包，我用到最新的，解壓在了 /etc/etcd 目錄下即可。

配置 etcd.serice ，使用systemctl 管理 。配置127.0.0.1 不需要https.

[root@k8s-node1 bin]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.yaml
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /etc/etcd/etcd --name=etcd-0 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://192.168.10.134:2380 --listen-peer-urls=https://192.168.10.134:2380 --listen-client-urls=https://192.168.10.134:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.10.134:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=\"etcd-0=https://192.168.10.133:2380,etcd-1=https://192.168.10.134:2380\" --initial-cluster-state=new --data-dir=/var/lib/etcd"
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

另一個節(jié)點同樣做 證書復制，把etcd 放在了 /etc/etcd 下面，配置修改操作。

啟動etcd

[root@k8s-node1 bin]# systemctl daemon-reload
[root@k8s-node1 bin]# systemctl start etcd
[root@k8s-node1 bin]# ps -ef |grep etcd 
root      44905      1  2 11:37 ?        00:00:00 /etc/etcd/etcd --name=etcd-1 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://192.168.10.134:2380 --listen-peer-urls=https://192.168.10.134:2380 --listen-client-urls=https://192.168.10.134:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.10.134:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=etcd-0=https://192.168.10.133:2380,etcd-1=https://192.168.10.134:2380 --initial-cluster-state=new --data-dir=/var/lib/etcd
root      44918  16456  0 11:38 pts/1    00:00:00 grep --color=auto etcd

查看集群結果


[root@k8s-master k8s]# cd /etc/etcd
[root@k8s-master etcd]# ls
etcd  etcd-0.etcd  etcdctl  etcd.yml  install  nohup.out
[root@k8s-master etcd]# 
[root@k8s-master etcd]# ./etcdctl --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.10.133:2379,https://192.168.10.134:2379  endpoint health 
https://192.168.10.134:2379 is healthy: successfully committed proposal: took = 15.768454ms
https://192.168.10.133:2379 is healthy: successfully committed proposal: took = 17.041281ms

由此可知，etcd 集群完成。

開始配置k8s

初始化第一個節(jié)點：

  • 重置一個節(jié)點
[root@k8s-master k8s]# kubeadm   reset 
[reset] WARNING: changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] are you sure you want to proceed? [y/N]: y
[preflight] running pre-flight checks
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1030 13:25:39.401263   66304 reset.go:213] [reset] Unable to fetch the kubeadm-config ConfigMap, using etcd pod spec as fallback: failed to get config map: Get https://192.168.10.133:8443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
[reset] no etcd config found. Assuming external etcd
[reset] please manually reset etcd to prevent further issues
[reset] stopping the kubelet service
[reset] unmounting mounted directories in "/var/lib/kubelet"
[reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example: 
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

[root@k8s-master k8s]# mkdir -p  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# cp  -r /etc/kubernetes/ssl/*.pem  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]#

初始化：

重點

先修改kubelet 自帶的etcd 配置，這里可惡的竟然是cgroup-driver 設置的是systemd .
刪除自帶的etcd 配置文件（把所有的master 節(jié)點上的這個文件都刪除了）。 這里坑了好久。。。。

[root@k8s-master kubelet.service.d]# 
[root@k8s-master kubelet.service.d]# cat 20-etcd-service-manager.conf 
[Service]
ExecStart=
#  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
[root@k8s-master kubelet.service.d]# 
[root@k8s-master kubelet.service.d]# 
[root@k8s-master kubelet.service.d]# rm -rf 20-etcd-service-manager.conf

復制etcd 證書到etc/kubernetes/pki/etcd/ ，這個目錄kubeadm reset 之后會被情況

[root@k8s-master k8s]# mkdir -p  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# 
[root@k8s-master k8s]# cp  -r /etc/kubernetes/ssl/*.pem  /etc/kubernetes/pki/etcd/

上面這步驟要在所有master 上執(zhí)行！上面這步驟要在所有master 上執(zhí)行！上面這步驟要在所有master 上執(zhí)行！

編寫kubeadm 配置文件,certSANs寫全自己的 master地址：

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "192.168.10.133"
  - "127.0.0.1"
  - "192.168.10.134"
controlPlaneEndpoint: "192.168.10.133:8443"
networking:
  podSubnet: 10.244.0.0/16
etcd:
    external:
        endpoints:
        - https://192.168.10.133:2379
        - https://192.168.10.134:2379
        caFile: /etc/kubernetes/pki/etcd/ca.pem
        certFile: /etc/kubernetes/pki/etcd/kubernetes.pem
        keyFile: /etc/kubernetes/pki/etcd/kubernetes-key.pem

因為前面的etcd 的cfssl 證書都拿來放在了/etc/kubernets/pki/etcd 下面了，直接進行init 操作

[root@k8s-master k8s]# mkdir -p  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# cp  -r /etc/kubernetes/ssl/*.pem  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# kubeadm init --config=kubeadm-config.yaml   -upload-certs --ignore-preflight-errors=all  
I1030 14:02:04.630701   70169 version.go:94] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1030 14:02:04.630785   70169 version.go:95] falling back to the local client version: v1.13.0
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate authority generation
[certs] External etcd mode: Skipping etcd/peer certificate authority generation
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.133 192.168.10.133 192.168.10.133]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.504737 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-master" as an annotation
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: s1ry1w.uedyn8db3jnm1tsg
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.10.133:8443 --token s1ry1w.uedyn8db3jnm1tsg --discovery-token-ca-cert-hash sha256:fdcc767dc0aeebfadaa4faef2ee19642ce6d85462771baaba23f927696c4ef78

安裝flannel 組件服務，同上一步，安裝完成后，節(jié)點變?yōu)?ready 狀態(tài)
見：kube-flannel.yml

[root@k8s-master k8s]# kubectl  apply -f flannel.yaml  
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
[root@k8s-master k8s]# kubectl  get no 

NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   5m12s   v1.13.0

OK ，第一個節(jié)點終于完成了。這時候，這個節(jié)點的證書就可以復制到其他的master 主機上了。
高版本的使用upload-certs 參數的話，證書也不需要每次復制了。

初始化第二個master節(jié)點：
我用的是1.13 版本的k8s , join 命令帶 "--experimental-control-plane " 這個參數，代表加入的是個master 節(jié)點，不帶就是node 。 注意這里也需要刪除kubelete 自帶的etcd 文件。

重點：
首先，就是把第一個master 上的證書拿過來，既把maseter1 上的 /etc/kubernetes/pki 文件夾全部復制到master2 上面的 /etc/kubernetes/pki 。 這個文件放的是各種證書。

開始加入主節(jié)點

[root@k8s-node1 k8s]# kubeadm join 192.168.10.133:8443 --token f2mitj.hbrghf2lpmy4n91f --discovery-token-ca-cert-hash sha256:2cbd8aa7a7822beec6fff001f8b03f54180ef3924314964e97174562d71a2873    --experimental-control-plane  
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[discovery] Trying to connect to API Server "192.168.10.133:8443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.10.133:8443"
[discovery] Requesting info from "https://192.168.10.133:8443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.10.133:8443"
[discovery] Successfully established connection with API Server "192.168.10.133:8443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[join] Running pre-flight checks before initializing the new control plane instance
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[certs] Using the existing "front-proxy-client" certificate and key
[certs] Using the existing "apiserver" certificate and key
[certs] Using the existing "apiserver-kubelet-client" certificate and key
[certs] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node1" as an annotation
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

[root@k8s-node1 k8s]# mkdir -p $HOME/.kube
[root@k8s-node1 k8s]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp：是否覆蓋"/root/.kube/config"？ y
[root@k8s-node1 k8s]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-node1 k8s]# 
[root@k8s-node1 k8s]# 
[root@k8s-node1 k8s]# kubectl get no 
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   28m   v1.13.0
k8s-node1    Ready    master   17m   v1.13.0

此時查看集群的組件狀態(tài)：

[root@k8s-node1 pki]# kubectl  get cs 

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"} 

[root@k8s-node1 pki]# kubectl  get  po --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-86c58d9df4-4px8n             1/1     Running   0          11m
kube-system   coredns-86c58d9df4-qwbql             1/1     Running   0          11m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          10m
kube-system   kube-apiserver-k8s-node1             1/1     Running   0          6m55s
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          10m
kube-system   kube-controller-manager-k8s-node1    1/1     Running   0          6m55s
kube-system   kube-flannel-ds-amd64-nntpq          1/1     Running   0          10m
kube-system   kube-flannel-ds-amd64-tj4sz          1/1     Running   0          6m55s
kube-system   kube-proxy-fm22x                     1/1     Running   0          11m
kube-system   kube-proxy-ngknh                     1/1     Running   0          6m55s
kube-system   kube-scheduler-k8s-master            1/1     Running   0          10m
kube-system   kube-scheduler-k8s-node1             1/1     Running   0          6m55s

[root@k8s-node1 pki]# kubectl  get  no
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   11m     v1.13.0
k8s-node1    Ready    master   7m12s   v1.13.0

至此，k8s 雙master 節(jié)點搭建完成，且etcd 集群外置。 當集群死亡之后，再次重啟可以從etcd 恢復集群狀態(tài)。

注意的是，如果想重新搭建k8s集群，需要清空etcd里存儲的pod 數據。

[root@k8s-node1 etcd]# ./etcdctl --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.10.133:2379,https://192.168.10.134:2379  del / --prefix 
447
[root@k8s-node1 etcd]#

高版本的 master 擴展

在init 第一個master 時候的不同；

kubeadm init --config=kubeadm-config.yaml --upload-certs --ignore-preflight-errors=all

高版本其實會打出兩條join 信息，一個是加入 master 節(jié)點的信息，一個是加入node 節(jié)點的信息，和低版本的join命令類似，不多在加入master 的時候又多了一個 認證的參數，如下：

kubeadm join master:6443 --token uwm1iq.bydvqhg91670w58g \
  --discovery-token-ca-cert-hash sha256:fef3eba9bc05450a0e3c705239775bf5297a82ecf4faf683a7cbb3a38b433ca0 \
  --experimental-control-plane --certificate-key cdad981734e547e60133a044ac72a60f6021038c5c3ca4d25a886e0ce79f9b57 \
 --ignore-preflight-errors=all
 # --ignore-preflight-errors=all表示忽略錯誤，可以不加

附 在機房花了20分鐘安裝k8s v1.16.2 版本安裝完之后的效果：

[root@k8s-master1 master]# kubectl  get no  -o wide 
NAME          STATUS   ROLES    AGE         VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-master1   Ready    master   20m         v1.16.2   192.168.1.110   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-master2   Ready    master   15m         v1.16.2   192.168.1.111   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-node1     Ready    <none>   6m24s       v1.16.2   192.168.1.120   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-node2     Ready    <none>   2m55s       v1.16.2   192.168.1.121   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-node3     Ready    <none>   <invalid>   v1.16.2   192.168.1.122   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
  • 安裝文件kubeadm-conf.yaml：
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
clusterName: kubernetes
apiServer:
   certSANs:
   - "192.168.1.110"
   - "192.168.1.111"
   - "127.0.0.1"
   - "192.168.1.109"
   - "k8s-master1"
   - "k8s-master2"
    
controlPlaneEndpoint: "192.168.1.109:6443"
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: v1.16.0
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.245.0.0/16"
scheduler: {}
controllerManager: {}
etcd:
    external:
        endpoints:
        - https://192.168.1.110:2379
        - https://192.168.1.111:2379
        caFile: /etc/etcd/ssl/ca.pem
        certFile: /etc/etcd/ssl/kubernetes.pem
        keyFile: /etc/etcd/ssl/kubernetes-key.pem
  • master1 初始化成功，看著命令join即可：
[root@k8s-master1 master]# kubeadm init --config=test.yaml  --upload-certs
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR ExternalEtcdVersion]: couldn't load external etcd's certificate and key pair /etc/etcd/ssl/kubernetes.pem, /etc/etcd/ssl/kubernetes-key.pe: open /etc/etcd/ssl/kubernetes-key.pe: no such file or directory
    [ERROR ExternalEtcdClientCertificates]: /etc/etcd/ssl/kubernetes-key.pe doesn't exist
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[root@k8s-master1 master]# vim test.yaml 
[root@k8s-master1 master]# 
[root@k8s-master1 master]# kubeadm init --config=test.yaml  --upload-certs
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-master1 k8s-master2] and IPs [10.245.0.1 192.168.1.110 192.168.1.109 192.168.1.110 192.168.1.111 127.0.0.1 192.168.1.109]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 63.035131 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
96dacef0c81bf971bcb39f347c3d3fd6e2fb2c3d253c6da01ec29f1e39eb2713
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 2sqa85.x8jxuibdr19f9wg0
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.1.109:6443 --token 2sqa85.x8jxuibdr19f9wg0 \
    --discovery-token-ca-cert-hash sha256:f07f424c74f4871639ac3d4131db7d51c550d0965df1856e9bf51d711f79c43b \
    --control-plane --certificate-key 96dacef0c81bf971bcb39f347c3d3fd6e2fb2c3d253c6da01ec29f1e39eb2713

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use 
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.109:6443 --token 2sqa85.x8jxuibdr19f9wg0 \
    --discovery-token-ca-cert-hash sha256:f07f424c74f4871639ac3d4131db7d51c550d0965df1856e9bf51d711f79c43b

默認生成的證書是臨時加入使用的，失效可以用：

kubeadm init phase upload-certs --experimental-upload-certs

node 的加入就簡單了，和原先的一樣，不帶這些主節(jié)點的參數信息如：

kubeadm join master:6443 --token 74fxrx.34uwnbxun31d9cgc \
    --discovery-token-ca-cert-hash sha256:7056fa528a5f0f962f565ee4eb7a96d7ecc5432883bd3c506ca0fa1eab09064c

例如，忘記了join 命令,如下重新生成即可：

[root@k8s-master1 ~]# kubeadm token  create --print-join-command
kubeadm join 192.168.1.109:6443 --token 779f9q.1jm3gryowdvpreff     --discovery-token-ca-cert-hash sha256:d1e5bb20d5b73dd317e5f9e7e699f5f02932b691608c138af032f45f2838e21e 
[root@k8s-master1 ~]# 
[root@k8s-master1 master]# kubeadm init phase upload-certs --upload-certs 
W1108 12:51:43.116233    8127 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1108 12:51:43.116287    8127 version.go:99] falling back to the local client version: v1.15.0
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
3461c4d9b5332796907d1412d9cb82a36869c1f3dd171d2d95b496643366183a

有些東西參考上一層。開發(fā)環(huán)境k8s 高可用master 的搭建

下一層 ： 鍛骨境-第5層 k8s-ETCD集群備份與恢復

最后編輯于 ：
?著作權歸作者所有,轉載或內容合作請聯(lián)系作者
【社區(qū)內容提示】社區(qū)部分內容疑似由AI輔助生成，瀏覽時請結合常識與多方信息審慎甄別。
平臺聲明：文章內容（如有圖片或視頻亦包括在內）由作者上傳并發(fā)布，文章內容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務。

相關閱讀更多精彩內容

友情鏈接更多精彩內容