k8s集群安裝一etcd

本文參考了網(wǎng)易云課堂阿良老師的系列視頻《1天入門Kubernetes/K8S》，學(xué)習(xí)之后所做的筆記。本文不使用kubeadm等自動化工具，而是從官網(wǎng)下載二進(jìn)制包手動部署。

集群環(huán)境規(guī)劃

阿里云非大陸節(jié)點(diǎn)，避免各種墻的問題。3臺機(jī)器，2CPU 2G最低配。系統(tǒng)鏡像：ubuntu_16_04_64_XXXX

角色 IP 組件
master 172.31.173.35 kube-apiserver
kube-controller-manager
kube-scheduler
etcd
node1 172.31.173.36 kubelet
kube-proxy
docker
flannel
etcd
node2 172.31.173.37 kubelet
kube-proxy
docker
flannel
etcd

安裝 docker

$ apt-get update 
$ apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
$ apt-get update
$ apt-get install docker-ce 
$ systemctl start docker
$ systemctl enable docker
# 測試docker
$ docker --version

TLS 證書

請先了解CA，公鑰私鑰概念，加解密過程。
證書有機(jī)構(gòu)簽發(fā)證書，收費(fèi)，被瀏覽器信任。
自簽證書不受信任，但是功能都是一樣的。

組件 證書
etcd ca.pem，server.pem，server-key.pem
kube-apiserver ca.pem，server.pem，server-key.pem
kubelet ca.pem，ca-key.pem
kube-proxy ca.pem，kube-proxy.pem，kube-proxy-key.pem
kubectl ca.pem，admin.pem，admin-key.pem

安裝cfssl

當(dāng)然openssl也可以生成數(shù)字證書，這里用cfssl。

# 參考
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
https://pkg.cfssl.org/

# 下載
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

# 放到/usr/local/bin目錄下，方便使用
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

# 驗(yàn)證
cfssl --help

# 生成證書模板，然后，在模板文件上修改，在模板上修改，在模板上修改，說三遍
cfssl print-defaults config > config.json
cfssl print-defaults csr > csr.json

生成CA證書，需要ca-config.json和ca-csr.json兩個配置文件

# ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    }
  },
  "profiles": {
    "kubernetes": {
      "expiry": "87600h",
      "usages": [
        "signing",
        "key encipherment",
        "server auth",
        "client auth"
      ]
    }
  }
}
# ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# 生成CA證書
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
# 執(zhí)行命令，會生成ca.pem和ca-key.pem兩個文件

生成server證書

# server-csr.json
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "172.31.173.35",
    "172.31.173.36",
    "172.31.173.37",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# 生成server證書
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

# 生成server-key和server.pem

生成admin證書

admin-csr.json

{
  "CN": "admin",
  "hosts": [
    
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

生成admin.pem和admin-key.pem

生成kube-proxy證書

kube-proxy-csr.json

{
  "CN": "System:kube-proxy",
  "hosts": [
    
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "Beijing",
      "ST": "Beijing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

保留*.pem證書，其他文件刪除
ls | grep -v pem | xargs -i rm {}

/root/ssl目錄下有以下文件：

admin-key.pem
admin.pem
ca-key.pem
ca.pem
kube-proxy-key.pem
kube-proxy.pem
server-key.pem
server.pem

部署etcd

首先我們確定下安裝配置文件的目錄

# bin目錄下放腳本
/opt/kubernetes/bin
# cfg目錄下放配置文件
/opt/kubernetes/cfg
# ssl目錄下放證書
/opt/kubernetes/ssl

下載地址：
https://github.com/etcd-io/etcd/releases/download/v3.3.12/etcd-v3.3.12-linux-amd64.tar.gz
將etct-XX解壓后的文件etcd/etcdctl兩個腳本放到/opt/kubernetes/bin目錄下

etcd配置文件

放到/opt/kubernetes/cfg目錄下

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.31.173.35:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.31.173.35:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.31.173.35:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.31.173.35:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.31.173.35:2380,etcd02=https://172.31.173.36:2380,etcd03=https://172.31.173.37:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

用systemd管理etcd進(jìn)程，vi /usr/lib/systemd/system/etcd.service，文件寫入以下內(nèi)容

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/kubernetes/cfg/etcd
ExecStart=/opt/kubernetes/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/kubernetes/ssl/server.pem \
--key-file=/opt/kubernetes/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/ssl/server.pem \
--peer-key-file=/opt/kubernetes/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/ssl/ca.pem \


[Install]
WantedBy=multi-user.target

啟動etcd

# 啟動etcd
systemctl start etcd
systemctl enable etcd

#可以看到etcd進(jìn)程已經(jīng)啟動，并在嘗試連接其他節(jié)點(diǎn)。
#如果啟動失敗，可用以下命令調(diào)試
ps -ef | grep etcd
systemctl status etcd.service
journalctl -u etcd
journalctl -xe
etcdctl cluster-health

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
systemctl restart etcd

# 如果啟動失敗，仔細(xì)檢查參數(shù)配置。

設(shè)置ssh免密碼登錄

ssh-copy-id root@172.31.173.36
ssh-copy-id root@172.31.173.37
輸入密碼回車。。。

從master節(jié)點(diǎn)拷貝文件到node節(jié)點(diǎn)

ssh root@172.31.173.36
ssh root@172.31.173.37
#ssh登錄到node節(jié)點(diǎn)創(chuàng)建目錄
mkdir -p /opt/kubernetes/{bin,cfg,ssl}

# 拷貝文件到node節(jié)點(diǎn)
scp -r /opt/kubernetes/{bin,cfg,ssl} root@172.31.173.36:/opt/kubernetes
scp -r /opt/kubernetes/{bin,cfg,ssl} root@172.31.173.37:/opt/kubernetes

scp /usr/lib/systemd/system/etcd.service root@172.31.173.36:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@172.31.173.37:/usr/lib/systemd/system/

# 修改node節(jié)點(diǎn)下的etcd.conf文件
ETCD_NAME=xxx02 在node2節(jié)點(diǎn)
ETCD_NAME=xxx03 在node3節(jié)點(diǎn)
對應(yīng)的ip也修改下

測試etcd集群狀態(tài)

/opt/kubernetes/bin/etcdctl \
--ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/server.pem \
--key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://172.31.173.35:2379,https://172.31.173.36:2379,https://172.31.173.37:2379" \
cluster-health

見證奇跡的時刻截圖：


image.png

etcd snapshot


# 快照
export ETCDCTL_API=3
etcdctl  --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=127.0.0.1:2379 snapshot save /tmp/snapshot.db

# 從快照恢復(fù)
etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=127.0.0.1:2379 snapshot restore /tmp/snapshot.db --data-dir=/var/lib/etcd/

#  啟動新etcd節(jié)點(diǎn)，指定--data-dir=/var/lib/etcd/

待續(xù)。。。

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容