Kubernetes作為目前最炙手可熱的容器編排軟件，受到越來(lái)越多互聯(lián)網(wǎng)公司的喜愛(ài)，之所以會(huì)出現(xiàn)這樣的現(xiàn)象，主要還是因?yàn)檫@套編排軟件不僅功能強(qiáng)大，還非常方便進(jìn)行二次開(kāi)發(fā)，整個(gè)Kubernetes的生態(tài)圈非?；钴S。有了Kubernetes這套“骨架”，可以在上面運(yùn)行非常多的中間件和應(yīng)用程序。Apiserver，Controller Manager，Scheduler，Proxy可以認(rèn)為是這套“骨架”的組成部分，而Etcd便是這套系統(tǒng)的底座，整個(gè)系統(tǒng)的數(shù)據(jù)都存儲(chǔ)在它當(dāng)中，保證了Etcd的穩(wěn)定性整個(gè)容器編排系統(tǒng)的穩(wěn)定性就有了保證。

在正式開(kāi)始之前，說(shuō)明下實(shí)驗(yàn)環(huán)境，五臺(tái)主機(jī)組成Kubernetes集群。

• 控制平面主機(jī)：p1.xufu.xyz、p2.xufu.xyz、p3.xufu.xyz
• 工作主機(jī)：p4.xufu.xyz、p5.xufu.xyz
• 主機(jī)系統(tǒng)：CentOS Linux release 7.9.2009 (Core)
• Etcd版本：3.5.1

Etcd

• 簡(jiǎn)介

它是一種鍵值數(shù)據(jù)庫(kù)，Kubernetes將自身的狀態(tài)數(shù)據(jù)都存儲(chǔ)在Etcd中，這些數(shù)據(jù)包含deployment，pod，daemonset等各種對(duì)象。整個(gè)集群組件中只有Apiserver會(huì)直接和Etcd通信，其他組件都是直接和Apiserver通信。在生產(chǎn)環(huán)境中，最好以集群的方式運(yùn)行Etcd，官方建議運(yùn)行五個(gè)節(jié)點(diǎn)組成集群，這樣可以在丟失兩個(gè)節(jié)點(diǎn)的情況下， 保證數(shù)據(jù)的可用性。

• 訪(fǎng)問(wèn)

Etcd中存放有不少敏感數(shù)據(jù)，所以對(duì)它的訪(fǎng)問(wèn)需要使用公鑰和私鑰。為了方便訪(fǎng)問(wèn)，在家目錄文件“.bashrc”配置如下命令別名。

<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">alias ectl='etcdctl --endpoints p1.xufu.xyz:2379 \    --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key \    --cacert=/etc/kubernetes/pki/etcd/ca.crt'</pre>

顯示當(dāng)前Etcd存儲(chǔ)的所有Key。

<pre class="prettyprint hljs vim" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ectl get / --prefix --keys-only</pre>

• 備份

使用kubeadm部署的集群，Etcd默認(rèn)將數(shù)據(jù)存放在宿主機(jī)的目錄/var/lib/etcd下。

Etcd數(shù)據(jù)目錄

Etcd的版本2和3差別很大，因筆者使用的是3版本，備份命令是這樣（在任一控制平面主機(jī)操作均可），命令執(zhí)行不用對(duì)Etcd做關(guān)閉操作。

<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ectl snapshot save snapshot-$(date +%F)</pre>

查看備份的快照。

快照信息

• 恢復(fù)

恢復(fù)操作相比較備份操作有稍許麻煩。下面步驟需要在三臺(tái)控制平面主機(jī)上都操作。

1，關(guān)閉kubelet，否則它會(huì)自動(dòng)重啟apiserver。

<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">systemctl stop kubelet.service</pre>

2，關(guān)閉apiserver容器，使用docker命令找到容器，直接stop。

<pre class="prettyprint hljs nginx" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">docker stop apiservername</pre>

3，重命名目錄/var/lib/etcd，創(chuàng)建新的，并修改目錄權(quán)限。

<pre class="prettyprint hljs awk" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">mv /var/lib/etcd /var/lib/etcd_oldmkdir /var/lib/etcdchmod 700 /var/lib/etcd</pre>

4，使用備份的快照，恢復(fù)數(shù)據(jù)。

<pre class="prettyprint hljs awk" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">ectl snapshot restore /data/etcd_backup/etcd-snapshot-2022-08-03.db \  --name p1.xufu.xyz \  --initial-cluster "etcd-0=https://p1.xufu.xyz:2380,etcd-1=https://p2.xufu.xyz:2380,etcd-2=https://p3.xufu.xyz:2380"  \  --initial-advertise-peer-urls https://p1.xufu.xyz:2380 \  --data-dir=/var/lib/etcd</pre>

p2和p3節(jié)點(diǎn)類(lèi)似。

5，啟動(dòng)三臺(tái)etcd，確認(rèn)下集群是否健康。

<pre class="prettyprint hljs vim" style="padding: 0.5em; font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; color: rgb(68, 68, 68); border-radius: 4px; display: block; margin: 0px 0px 1.5em; font-size: 14px; line-height: 1.5em; word-break: break-all; overflow-wrap: break-word; white-space: pre; background-color: rgb(246, 246, 246); border: none; overflow-x: auto; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">[root@p1 etcd]# ectl --endpoints=https://p1.xufu.xyz:2379,https://p2.xufu.xyz:2379,https://p3.xufu.xyz:2379 endpoint healthhttps://p2.xufu.xyz:2379 is healthy: successfully committed proposal: took = 100.193247msp1.xufu.xyz:2379 is healthy: successfully committed proposal: took = 111.234607mshttps://p1.xufu.xyz:2379 is healthy: successfully committed proposal: took = 113.788721mshttps://p3.xufu.xyz:2379 is healthy: successfully committed proposal: took = 118.403445ms</pre>

確認(rèn)沒(méi)問(wèn)題后，啟動(dòng)apiserver容器。

至此，整個(gè)恢復(fù)備份過(guò)程操作完畢。