1. k8s安裝部署介紹
1.1 部署工具
- 使用批量部署工具, 如(ansible/saltstack)
- 手動二進制
- kubeadm
- apt/yum等方式安裝, 以守護進程的方式在宿主機上, 類似于nginx一樣, 使用service腳本啟動
1.2 kubeadm方式部署
官方文檔: https://kubernetes.io/zh/docs/setup/independent/create-cluster-kubeadm/
1.11版本開始逐漸穩(wěn)定, 可以在生產(chǎn)環(huán)境使用. 目前最新版為1.25
使?k8s官?提供的部署?具kubeadm?動安裝，需要在master和node節(jié)點上安裝docker等組件，然后初始化，把管理端的控制服務(wù)和node上的服務(wù)都以pod的?式運行
以pod方式運行的服務(wù):
master節(jié)點: kube-controller-manager, kube-scheduler, kube-apiserver
node節(jié)點: kube-proxy
以二進制方式直接運行:
node節(jié)點: kubelet
master節(jié)點:
不運行業(yè)務(wù)容器, 僅負責進群調(diào)度管理
node節(jié)點:
運行業(yè)務(wù)容器
2. kubeadm介紹
https://kubernetes.io/zh/docs/reference/setup-tools/kubeadm/kubeadm/
2.1 安裝前注意事項
禁?swap，selinux(CentOS)，iptables，并優(yōu)化內(nèi)核參數(shù)及資源限制參數(shù)
2.2 部署過程
1. 基礎(chǔ)環(huán)境準備: 禁?swap，selinux(CentOS)，iptables，并優(yōu)化內(nèi)核參數(shù)及資源限制參數(shù)
2. 部署harbor及haproxy+keepalived?可?反向代理
3. 在master節(jié)點和node節(jié)點安裝docker
4. 在所有master安裝指定版本的kubeadm 、kubelet、kubectl、docker
5. 在所有node節(jié)點安裝指定版本的kubeadm 、kubelet、docker，在node節(jié)點kubectl為可選安裝
看是否需要在node執(zhí)?kubectl命令進?集群管理及pod管理等操作
6. master節(jié)點運?kubeadm init初始化命令
7. 驗證master節(jié)點狀態(tài)
8. 在node節(jié)點使?kubeadm命令將??加?k8s master(需要使?master?成的token和ca公鑰進行認證)
9. 驗證node節(jié)點狀態(tài)
10. 創(chuàng)建pod并測試?絡(luò)通信
11. 部署web服務(wù)Dashboard
12. k8s集群升級案例
3. 基礎(chǔ)環(huán)境準備
3.1 服務(wù)器環(huán)境
最?化安裝基礎(chǔ)系統(tǒng)，關(guān)閉防?墻, selinux和swap，更新軟件源、時間同步、安裝常?命令，重啟后驗證基礎(chǔ)配置，Centos 推薦使?Centos 7.5及以上的系統(tǒng)，Ubuntu推薦18.04及以上穩(wěn)定版.
master節(jié)點: 提供api-server, 需要實現(xiàn)高可用, 通過負載均衡器調(diào)度, 一般使用3個節(jié)點即可. k8s的master不像redis等其他的集群服務(wù)的master節(jié)點有數(shù)量要求.
使用kubeadm部署k8s時, master節(jié)點最多只能down一個. 測試環(huán)境如果想節(jié)約資源, 可以使用一個master節(jié)點進行控制, 出現(xiàn)問題及時修復(fù)即可.
node節(jié)點: node節(jié)點包含兩個組件, kube-proxy和kubelet, 這兩個組件也是通過負載均衡器去訪問master節(jié)點的api-server.
LB: 負載均衡器可以搭建兩個, 一個給管理員提供api-server訪問, 一個給node節(jié)點提供api-server訪問.
實際工作中:
node節(jié)點: 負責運行業(yè)務(wù)鏡像, 所以配置要求較高, 最好是物理機, 一般256G+2核以上配置
master節(jié)點, harbor和負載均衡: 可以是虛擬機, 也可以是物理機, 但最好還是物理機
本文實驗環(huán)境:
master節(jié)點:
master-19: 10.0.0.19
master-29: 10.0.0.29
master-39: 10.0.0.39
HAproxy:
ha-1: 10.0.0.49
ha-2: 10.0.0.59
Harbor:
harbor: 10.0.0.69
node節(jié)點:
node-79: 10.0.0.79
node-89: 10.0.0.89
node-99: 10.0.0.99
3.1.1 禁用交換分區(qū)
k8s不支持交換分區(qū), 一旦不禁用, 那么k8s是無法部署的. 雖然可以跳過交換分區(qū)檢測, 但是不推薦. 如果內(nèi)存不夠, 就加內(nèi)存.
如果大量使用交換分區(qū), 那么數(shù)據(jù)會寫到磁盤上, 這時服務(wù)器的性能就會降低.
在master和node節(jié)點禁用swap分區(qū)
#1. 禁用交換分區(qū): 全部主機
ansible all -m shell -a "sed -r -i.bak '/swap/s@(.*)@#\1@' /etc/fstab"
vim /etc/fstab
# swap was on /dev/sda5 during installation
#UUID=ba6d2bf8-da50-4041-9920-b2b1e6f46626 none swap sw 0 0
reboot
3.1.2 內(nèi)核參數(shù)調(diào)整
master以及node節(jié)點都需要執(zhí)行
#2. 內(nèi)核參數(shù)調(diào)整: 全部主機
ansible all -m copy -a "src=limits.conf dest=/etc/security/limits.conf backup=yes"
ansible all -m copy -a "src=sysctl.conf dest=/etc/sysctl.conf backup=yes"
內(nèi)核參數(shù)調(diào)整:
# 資源限制調(diào)整
vim /etc/security/limits.conf
* soft core unlimited
* hard core unlimited
* soft nproc 1000000
* hard nproc 1000000
* soft nofile 1000000
* hard nofile 1000000
* soft memlock 32000
* hard memlock 32000
* soft msgqueue 8192000
* hard msgqueue 8192000
# 內(nèi)核參數(shù)調(diào)整
# 開啟宿主機的ipv4_forward, 讓宿主機開啟路由轉(zhuǎn)發(fā)功能, 否則容器只能在宿主機內(nèi)部通信.
vim /etc/sysctl.conf
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.ip_forward = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
# Controls the default maxmimum size of a mesage queue
kernel.msgmnb = 65536
# # Controls the maximum size of a message, in bytes
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# # Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# TCP kernel paramater
net.ipv4.tcp_mem = 786432 1048576 1572864
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
# socket buffer
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 20480
net.core.optmem_max = 81920
# TCP conn
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_syn_retries = 3
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
# tcp conn reuse
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_max_tw_buckets = 20000
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_timestamps = 1 #?
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syncookies = 1
# keepalive conn
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.ip_local_port_range = 10001 65000
# swap
vm.overcommit_memory = 0
vm.swappiness = 10
#net.ipv4.conf.eth1.rp_filter = 0
#net.ipv4.conf.lo.arp_ignore = 1
#net.ipv4.conf.lo.arp_announce = 2
#net.ipv4.conf.all.arp_ignore = 1
#net.ipv4.conf.all.arp_announce = 2
禁?swap，selinux，iptables，并優(yōu)化內(nèi)核參數(shù)及資源限制參數(shù), 全部修改完, 重啟服務(wù)器生效
#3. 重啟生效
ansible all -m shell -a "reboot"
reboot
3.2 haproxy+keepalived部署
如果公司本身就有haproxy+keepalived, 不想再重新部署, 那么可以使用已有的. 但是, 建議最好配置單獨的負載均衡, 不要多個業(yè)務(wù)共有一個負載均衡
這里僅在ha1-49-10.0.0.49上部署
root@ha1-49:~# apt -y install haproxy keepalived
vip: 10.0.0.188
node節(jié)點通過10.0.0.188訪問master節(jié)點
haproxy監(jiān)聽在10.0.0.188的tcp端口, 這里不能用http代理, 因為通信過程中會有很多證書驗證
image.png
3.2.1 配置keepalived的vip
root@ha1-49:~# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
garp_master_delay 10
smtp_alert
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0.0.188 dev eth0 label eth0:0
}
}
root@ha1-49:~# systemctl restart keepalived
root@ha1-49:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.49 netmask 255.255.255.0 broadcast 10.0.0.255
inet6 fe80::20c:29ff:fe49:4ce4 prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:49:4c:e4 txqueuelen 1000 (Ethernet)
RX packets 4676 bytes 3176356 (3.1 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2672 bytes 552760 (552.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.188 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:49:4c:e4 txqueuelen 1000 (Ethernet)
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 203 bytes 16161 (16.1 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 203 bytes 16161 (16.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
3.2.2 配置haproxy的監(jiān)聽
root@ha1-49:~# vim /etc/haproxy/haproxy.cfg
listen k8s-6443
bind 10.0.0.188:6443
mode tcp
server 10.0.0.19 10.0.0.19:6443 check inter 3s fall 3 rise 5
server 10.0.0.29 10.0.0.29:6443 check inter 3s fall 3 rise 5
server 10.0.0.39 10.0.0.39:6443 check inter 3s fall 3 rise 5
# 默認輪訓(xùn)即可. api請求都是http無狀態(tài)的, 不需要源地址哈希, session共享或者session保持
root@ha1-49:~# systemctl restart haproxy
root@ha1-49:~# ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 2000 10.0.0.188:6443 0.0.0.0:*
LISTEN 0 128 127.0.0.53%lo:53 0.0.0.0:*
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
本文僅配置一組haproxy+keepalived, 正常情況還要進行haproxy的高可用
3.3 docker安裝
基于docker運行k8s時, 對docker版本有要求
官方會對k8s和docker的兼容版本進行測試驗證, 并且在k8s各版本的CHANGELOG中發(fā)布
以k8s-v1.17.16為例
image.png
image.png
本文部署:
docker-v19.03.14: master節(jié)點和node節(jié)點
k8s-v1.19.1 后續(xù)會演示如果升級到1.19.6
kubeadm 1.19.1-00
kubectl 1.19.1-00: 系統(tǒng)管理客戶端命令行, 在管理節(jié)點安裝即可
kubelet 1.19.1-00
kubeproxy: 是以容器方式, 自動運行, 無需手動安裝
在master節(jié)點和node節(jié)點安裝docker
ansible master_k8s:node_k8s:harbor -m script -a "docker_install.sh"
root@master-19:/data/scripts# vim docker_install.sh
#!/bin/bash
apt update
apt -y install \
ca-certificates \
curl \
gnupg \
lsb-release
mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
VERSION_STRING=5:19.03.14~3-0~ubuntu-bionic
apt -y install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING
root@master-19:/data/scripts# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-10-31 15:10:51 AEDT; 51s ago
...
3.4 harbor部署
基于kubeadmin部署k8s時, 會從docker hub下載很多鏡像, 建議把這些鏡像全部下載到本地, 修改tag后上傳到本地的harbor. 這樣后續(xù)部署k8s時, 就可以從本地的harbor拉取鏡像加快速度, 同時也可以避免萬一官方哪天不再提供該版本的鏡像了, 就沒法下載了
4. 部署kubeadm組件
kubeadm組件需要在master和node節(jié)點都安裝. 其可用于在任意master節(jié)點初始化集群, 也用于把其余的master和所有node節(jié)點加入到k8s集群
master節(jié)點: 利用kubeadmin去初始化集群, 初始化集群在任意一個master執(zhí)行即可
初始化完成后, 需要在其余的master節(jié)點執(zhí)行kubeadm, 把自己加入到k8s集群
node節(jié)點: 把node節(jié)點加入到k8s集群, 也是用的kubeadm工具
kubelet是node節(jié)點必須安裝的, master也需要安裝, 因為master節(jié)點也需要啟一些容器去運行
kubectl安裝在管理節(jié)點上, 需要在哪個節(jié)點管理k8s, 就在哪個節(jié)點安裝kubectl
4.1 master節(jié)點和node節(jié)點安裝kubeadm
kubeadm, kubelet, kubectl這三個組件的版本要和k8s一致, 也就是說, 需要裝哪個版本的k8s, 就要按照對應(yīng)版本的組件
master節(jié)點安裝kubeadm, kubelet, kubectl
#!/bin/bash
apt update
apt install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | \
tee /etc/apt/sources.list.d/kubernetes.list
apt update
apt install -y kubelet=1.19.1-00 kubeadm=1.19.1-00 kubectl=1.19.1-00
node節(jié)點安裝kubeadm, kubelet
#!/bin/bash
apt update
apt install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | \
tee /etc/apt/sources.list.d/kubernetes.list
apt update
apt install -y kubelet=1.19.1-00 kubeadm=1.19.1-00
此時, 因為系統(tǒng)還沒有生成一些認證文件, 所以kubelet會報錯, 不過不影響, 之后創(chuàng)建了認證文件即可修復(fù)
image.png
image.png
4.2 kubeadm命令使用
kubeadm命令:
Usage:
kubeadm [command]
Available Commands:
alpha Kubeadm experimental sub-commands # kubeadm處于測試階段的命令
completion Output shell completion code for the specified shell (bash or zsh)
config Manage configuration for a kubeadm cluster persisted in a ConfigMap in the cluster
help Help about any command
init Run this command in order to set up the Kubernetes control plane
join Run this on any machine you wish to join an existing cluster
reset Performs a best effort revert of changes made to this host by 'kubeadm init' or 'kubeadm join'
token Manage bootstrap tokens
upgrade Upgrade your cluster smoothly to a newer version with this command
version Print the version of kubeadm
Flags:
--add-dir-header If true, adds the file directory to the header of the log messages
-h, --help help for kubeadm
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--rootfs string [EXPERIMENTAL] The path to the 'real' host root filesystem.
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
-v, --v Level number for the log level verbosity
Use "kubeadm [command] --help" for more information about a command.
- alpha: kubeadm處于測試階段的命令
在alpha命令集中, 提供k8s證書管理命令
用kubeadm初始化的k8s, 內(nèi)部默認的證書有效期為1年時間
一旦過了一年, 證書過期, k8s將無法再使用, 屆時, 就要重新初始化證書, 就要利用到alpha命令集中的renew命令, renew會更新集群的所有證書
所以要定期檢查證書有效期時間, 在一年到期前, 進行renew
check-expiration命令用來檢查證書的有效時間
- completion: 用于kubeadm的bash命令補全, 需要安裝bash-completion.
需要在所有master和node節(jié)點執(zhí)行, 因為master和node節(jié)點都安裝了kubeadm
mkdir /data/scripts -p
# kubeadm支持bash和zsh補全, 不過一般都是用bash
# 使用補全之前, 需要把補全的參和變量追加到一個腳本里
kubeadm completion bash > /data/scripts/kubeadm.sh # 腳本名字可以自定義
chmod a+x /data/scripts/kubeadm.sh
source /data/scripts/kubeadm.sh
echo 'source /data/scripts/kubeadm.sh' >> /etc/profile # 開機自動生效
- config: 管理kubeadm集群的配置，該配置保留在集群的ConfigMap中
執(zhí)行命令 kubeadm config print init-defaults 會生成一個配置文件
該配置文件包含一系列的參數(shù), 利用這個配置文件, 可以用來初始化k8s集群
- help: Help about any command
kubeadm help
- init: 初始化集群
在任意一臺master節(jié)點執(zhí)行即可, 本案例在master1-10.0.0.19執(zhí)行
join: 將節(jié)點加?到已經(jīng)存在的k8s master
reset: 還原使用, 撤銷kubeadm init或者kubeadm join對系統(tǒng)產(chǎn)?的環(huán)境變化. 一般用于物理機, 把kubeadm所做的操作全部還原
kubeadm init初始化只需要在一個master節(jié)點執(zhí)行一次即可
kubeadm join需要在其與所有的master節(jié)點和node節(jié)點都執(zhí)行
- token: 管理token
kubeadm init完成集群初始化后, 執(zhí)行init的master節(jié)點, 會生成一個集群的token
后續(xù)將節(jié)點join到k8s集群時, 需要做一個認證, 該認證就是通過初始化集群階段生成的token進行認證的
token默認的有效期是24小時, 所以join時必須確保token是有效的
集群中, 任意master節(jié)點生成的token都是有效的, 因為這個token會存放到etcd里. 如果失效了, 可以登錄任意master節(jié)點重新生成
- upgrade: 升級k8s版本
k8s更新速度非常快, 所以一點當前版本功能無法滿足現(xiàn)有需求, 就需要升級k8s版本
- version: 查看版本信息
root@master-19:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:24:31Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
4.3 kubeadm init集群初始化準備工作
在三臺master中任意?臺master 進?集群初始化，?且集群初始化只需要初始化?次. 這里在master-19-10.0.0.19進行初始化
4.3.1 kubeadm init命令
root@master-19:~# kubeadm init --help
****** --apiserver-advertise-address string The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.
# apiserver監(jiān)聽的地址, 如果沒有設(shè)置, 那么就用默認的地址. 該地址可以不設(shè)置, 那么就用本地的網(wǎng)絡(luò).
# 一般建議設(shè)置一個專門的監(jiān)聽地址, "x.x.x.x" string類型, 在哪個master節(jié)點進行的初始化, 就寫哪個節(jié)點的ip地址, 這里就是10.0.0.19
****** --apiserver-bind-port int32 Port for the API Server to bind to. (default 6443)
# apiserver監(jiān)聽的端口號, 默認6443
# 以下四項是k8s證書信息, kubeadm初始化時會自動完成證書的配置, 所以如果不用外部的證書的話, 就無須配置
#--apiserver-cert-extra-sans stringSlice # 可選的證書額外信息，?于指定API Server的服
務(wù)器證書?？梢允荌P地址也可以是DNS名稱。
--cert-dir string # 證書的存儲路徑，缺省路徑為 /etc/kubernetes/pki
--certificate-key string # 定義?個?于加密kubeadm-certs Secret中的控制平臺證書的密鑰
--config string # kubeadm配置?件的路徑
****** --control-plane-endpoint string # 為控制平臺指定?個穩(wěn)定的IP地址或DNS名稱，即配置?個可以?期使?且是?可?的VIP或者域名，k8s 多master?可?基于此參數(shù)實現(xiàn). 這個vip就是負載均衡的vip地址. 如果使用單master集群, 那么就無需這項配置
--cri-socket string # 要連接的CRI(容器運?時接?，Container Runtime Interface, 簡稱CRI)套接字的路徑，如果為空，則kubeadm將嘗試?動檢測此值(使用docker的socket)，"僅當安裝了多個CRI或具有?標準CRI插槽時，才使?此選項". k8s-1.19.1還是使用的docker運行時, 所以此項無需配置, 會默認調(diào)用docker的socket, 從v1.24開始, 使用container.d作為運行時
--dry-run # 不要應(yīng)?任何更改，只是輸出將要執(zhí)?的操作，其實就是測試運?
--experimental-kustomize string # ?于存儲kustomize為靜態(tài)pod清單所提供的補丁的路徑
--feature-gates string # ?組?來描述各種功能特性的鍵值（key=value）對，選項是：IPv6DualStack=true|false (ALPHA - default=false)
****** --ignore-preflight-errors strings # 可以忽略檢查過程中出現(xiàn)的錯誤信息，?如忽略swap，如果為all就忽略所有. 如果開啟了swap, 那么這項一定要加上, 不過不建議開啟swap分區(qū)
****** --image-repository string # 設(shè)置?個鏡像倉庫，默認為k8s.gcr.io
****** --kubernetes-version string # 指定安裝k8s版本，默認為stable-1
****** --node-name string # 指定node節(jié)點名稱, 系統(tǒng)會自動使用node節(jié)點的主機名稱, 所以無需執(zhí)行, 但是要保證每個node節(jié)點的主機名不同, master節(jié)點主機名也要不同
****** --pod-network-cidr # 設(shè)置pod ip地址范圍, pod網(wǎng)絡(luò)的大地址段, 每個pod會被分配一個來自該地址段的小子網(wǎng). 這個大的地址段一定要大(比如, 10.200.0.0/16), 確保未來能滿足所有pod的需求. 注意不要和其他的網(wǎng)段沖突.
****** --service-cidr # 設(shè)置service?絡(luò)地址范圍, service的網(wǎng)絡(luò)地址不要和pod網(wǎng)絡(luò)段相同(比如, 10.100.0.0/16)
****** --service-dns-domain string # 設(shè)置k8s內(nèi)部域名，默認為cluster.local，會由相應(yīng)的DNS服務(wù)(kube-dns/coredns)解析?成域名記錄。k8s中, 每創(chuàng)建一個服務(wù), 都會給該服務(wù)起一個后綴名, 一般公司會用一個統(tǒng)計的后綴名, 比如project.online(生產(chǎn)環(huán)境), project.test(測試環(huán)境)
# ip地址, DNS, 域名一定要初始化階段就規(guī)劃好, 否則一旦集群運行起來, 再做修改很麻煩
--skip-certificate-key-print # 不打印?于加密的key信息. 這項不要加, 因為一定要打印加密key信息
--skip-phases strings # 要跳過哪些階段. 這些不要加, 整個初始化過程都是必要的, 不要跳過
--skip-token-print # 跳過打印token信息. token不指定的話無法把節(jié)點加入集群, 所以不要指定
--token # 指定token, token可以單獨執(zhí)行. 如果不指定, 那么會根據(jù)默認的格式生成token. The format is [a-z0-9]{6}\.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
--token-ttl # 指定token過期時間，默認為24?時，0為永不過期. token一般24小時有效期就夠了
--upload-certs #更新證書
#全局可選項：
--add-dir-header # 如果為true，在?志頭部添加?志?錄
--log-file string # 如果不為空，將使?此?志?件, 如果不指定, 那么使用系統(tǒng)默認的/var/log/syslog文件
--log-file-max-size uint # 設(shè)置?志?件的最???，單位為兆，默認為1800兆，0為沒有限制
--rootfs # 宿主機的根路徑，也就是絕對路徑
--skip-headers # 如果為true，在log?志??不顯示標題前綴
--skip-log-headers # 如果為true，在log?志??不顯示標題
4.3.2 準備鏡像
kubeadmin初始化k8s時, 會用到一些鏡像, 可以通過kubeadm config images list命令查看
不同的k8s版本, 所需要的鏡像是不同的
# 先通過kubeadm config images list獲取鏡像列表
root@master-19:~# kubeadm config images list --kubernetes-version v1.19.1 # 指定k8s版本
W1031 22:45:06.928798 28927 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.19.1 # 以容器運行
k8s.gcr.io/kube-controller-manager:v1.19.1 # 以容器運行
k8s.gcr.io/kube-scheduler:v1.19.1 # 以容器運行
k8s.gcr.io/kube-proxy:v1.19.1 # 以容器運行
k8s.gcr.io/pause:3.2 # 一個pod通常只啟動一個容器, 但是也可以啟動多個. 多個容器封裝到一個pod中, 那么容器的網(wǎng)絡(luò)也需要封裝到pod中, 而pause容器就是實現(xiàn)網(wǎng)絡(luò)封裝的
k8s.gcr.io/etcd:3.4.13-0
k8s.gcr.io/coredns:1.7.0
運行在同一個pod中的不同的容器, 可以運行的是不同的rootfs, 比如一個是CentOS, 另一個是Ubuntu. 通過namespace, 將兩個容器的rootfs進行隔離. 但是不同的容器使用的網(wǎng)絡(luò)確是一樣的, 不同的容器都是使用相同的socket, 每個容器都封裝一個pause, 這樣在不同的容器上, 都可以看到該網(wǎng)絡(luò)內(nèi)ip和端口使用情況. 比如, 在php這個容器內(nèi), 可以看到本網(wǎng)絡(luò)的80端口被占用, 并且雙方都可以通過127.0.0.1去相互訪問. 這樣就解決了同一個pod中, 不同容器的網(wǎng)絡(luò)通信問題.
image.png
4.3.3 master節(jié)點拉取鏡像
建議提前在master節(jié)點下載鏡像以減少安裝等待時間，避免后期因鏡像下載異常?導(dǎo)致k8s部署異常
master節(jié)點需要全部七個鏡像, 而node節(jié)點只需要pause和kube-proxy即可
root@master-19:/data/scripts# vim images_pull.sh
#!/bin/bash
docker pull k8s.gcr.io/kube-apiserver:v1.19.1
docker pull k8s.gcr.io/kube-controller-manager:v1.19.1
docker pull k8s.gcr.io/kube-scheduler:v1.19.1
docker pull k8s.gcr.io/kube-proxy:v1.19.1
docker pull k8s.gcr.io/pause:3.2
docker pull k8s.gcr.io/etcd:3.4.13-0
docker pull k8s.gcr.io/coredns:1.7.0
root@master-19:/data/scripts# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.19.1 33c60812eab8 2 years ago 118MB
k8s.gcr.io/kube-apiserver v1.19.1 ce0df89806bb 2 years ago 119MB
k8s.gcr.io/kube-controller-manager v1.19.1 538929063f23 2 years ago 111MB
k8s.gcr.io/kube-scheduler v1.19.1 49eb8a235d05 2 years ago 45.7MB
k8s.gcr.io/etcd 3.4.13-0 0369cf4303ff 2 years ago 253MB
k8s.gcr.io/coredns 1.7.0 bfe3a36ebd25 2 years ago 45.2MB
k8s.gcr.io/pause 3.2 80d28bedfe5d 2 years ago 683kB
5. 單master節(jié)點初始化
image.png
如果是測試環(huán)境、開發(fā)環(huán)境等??產(chǎn)環(huán)境，可以使?單master節(jié)點，?產(chǎn)環(huán)境要使?多master節(jié)點，以保證k8s的?可?
5.1 在單master節(jié)點執(zhí)行初始化命令
# apiserver監(jiān)聽地址: 10.0.0.19
# 監(jiān)聽端口: 6443
# k8s版本: v1.19.1
# service網(wǎng)段: 10.100.0.0/16
# pod網(wǎng)段: 10.200.0.0/16
root@master-19:~# kubeadm init --apiserver-advertise-address=10.0.0.19 --apiserver-bind-port=6443 --kubernetes-version=v1.19.1 --pod-network-cidr=10.200.0.0/16 \
--service-cidr=10.100.0.0/16 --service-dns-domain=david.local --ignore-preflight-errors=swap
W1101 22:44:53.127197 13597 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.1
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.david.local master-19.k8s] and IPs [10.100.0.1 10.0.0.19]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master-19.k8s] and IPs [10.0.0.19 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master-19.k8s] and IPs [10.0.0.19 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.503222 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master-19.k8s as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master-19.k8s as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: g3l8wx.9yaprg4pgdi1k2vy
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully! # 初始化成功
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 10.0.0.19:6443 --token g3l8wx.9yaprg4pgdi1k2vy \
--discovery-token-ca-cert-hash sha256:64924aa9fdfd8e6f47f9957bbf6b4b13751a21dfd28deb57dd72e802106aa719
5.2 配置kubectl命令認證
kubectl可以用來管理k8s, 但是必須要先配置認證才能進行后續(xù)管理
認證文件: /etc/kubernetes/admin.conf, 包括k8s服務(wù)器地址, 以及證書信息, 千萬不要泄漏
server: https://10.0.0.19:6443 - kubectl在管理k8s時, 會向這個api地址發(fā)起請求, 同時會攜帶這個文件里的認證信息.
如果該文件泄漏, 那么其他人也可通過這個認證文件去管理k8s
需要在哪個節(jié)點管理k8s, 就在哪個節(jié)點安裝kubectl, 并且配置認證. 這里在master-19-10.0.0.19配置
To start using your cluster, you need to run the following as a regular user:
root@master-19:~# mkdir -p $HOME/.kube
root@master-19:~# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@master-19:~# sudo chown $(id -u):$(id -g) $HOME/.kube/config
配置了kubectl認證后, 可以先通過kubectl get node驗證認證成功
# 此時節(jié)點狀態(tài)是NotReady, 是因為網(wǎng)絡(luò)組件還沒啟動, 所以node節(jié)點是無法啟動的
root@master-19:~# kubectl get node
NAME STATUS ROLES AGE VERSION
master-19.k8s NotReady master 103s v1.19.1
5.3 部署網(wǎng)絡(luò)組件
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
官方提供了二進制包和yaml文件兩種部署方式, 對于Kubernetes v1.17+以上版本, 可以使用yaml文件部署, 這里演示yaml文件部署, 在執(zhí)行初始化的master節(jié)點部署flannel
- 先獲取到y(tǒng)ml文件
root@master-19:/data/scripts# mkdir yaml
root@master-19:/data/scripts# cd yaml/
root@master-19:/data/scripts/yaml# wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
- 修改kube-flannel.yml文件配置
# 修改pod地址段
net-conf.json: | # net-conf.json指定pod地址段
{
# "Network": "10.244.0.0/16", # 244為官方默認生成的pod地址段地址. 必須改成初始化時指定了pod地址段, 這里為200
"Network": "10.200.0.0/16",
"Backend": {
"Type": "vxlan"
}
# 修改鏡像下載地址: 兩個地址保持一致, 不要用rc版本
initContainers:
- name: install-cni-plugin
#image: flannelcni/flannel-cni-plugin:v1.1.0 for ppc64le and mips64le (dockerhub limitations may apply)
# 必須實現(xiàn)確保該地址是本地可以訪問的, 因為會從該地址拉取鏡像, 可以先手動docker pull測試一下, 如果失敗, 就要換到其他的地址
# 換到其他地址后, 可以docker pull到本地, 然后重新打上tag號, 上傳到本地harbor. 之后, 修改該鏡像地址為本地harbor地址, 這樣之后就從本地harbor拉取該鏡像了
# 注意不要使用rc版本
image: docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
containers:
- name: kube-flannel
#image: flannelcni/flannel:v0.20.0 for ppc64le and mips64le (dockerhub limitations may apply)
image: docker.io/rancher/mirrored-flannelcni-flannel:v0.20.0
root@master-19:~# docker pull docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
v1.1.0: Pulling from rancher/mirrored-flannelcni-flannel-cni-plugin
6097bfa160c1: Pull complete
d10987c60bb3: Pull complete
Digest: sha256:28d3a6be9f450282bf42e4dad143d41da23e3d91f66f19c01ee7fd21fd17cb2b
Status: Downloaded newer image for rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
root@master-19:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rancher/mirrored-flannelcni-flannel-cni-plugin v1.1.0 fcecffc7ad4a 5 months ago 8.09MB
- 執(zhí)行yml文件
root@master-19:/data/scripts/yaml# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
kubectl部署flannel時, 會先去~/.kube/config文件中, 查找apiserver的地址以及認證信息, 把yml文件上傳到apiserver. 然后, apiserver再根據(jù)yml文件中的配置信息, 再去創(chuàng)建flannel的pod
- 驗證pod和node節(jié)點啟動
root@master-19:/data/scripts/yaml# kubectl get node
NAME STATUS ROLES AGE VERSION
master-19.k8s Ready master 5m15s v1.19.1
root@master-19:/data/scripts/yaml# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-h9t89 1/1 Running 0 56s
kube-system coredns-f9fd979d6-2mmkj 1/1 Running 0 5m7s
kube-system coredns-f9fd979d6-j8vxt 1/1 Running 0 5m7s
kube-system etcd-master-19.k8s 1/1 Running 0 5m23s
kube-system kube-apiserver-master-19.k8s 1/1 Running 0 5m22s
kube-system kube-controller-manager-master-19.k8s 1/1 Running 0 5m23s
kube-system kube-proxy-fr4th 1/1 Running 0 5m7s
kube-system kube-scheduler-master-19.k8s 1/1 Running 0 5m23s
到此, master單節(jié)點就初始化成功了
5.4 添加其余node節(jié)點
如果忘記了taken或者ca證書公鑰的哈希值, 可以在管理節(jié)點通過以下命令獲取
# 獲取token, 注意, 必須是有效的token
root@master-19:~# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
vt01rg.ykssroge3yadxi3a 23h 2022-11-03T12:46:01+11:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
# 獲取ca證書公鑰的哈希值
root@master-19:~# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
000259f02dc208616aa8118508c6c6aa6ad769187521a8fefb7910b9cb3f922c
# 在所有node節(jié)點執(zhí)行
Then you can join any number of worker nodes by running the following on each as root:
# token默認24小時過期, 而證書的sha256哈希值是不會過期的
kubeadm join 10.0.0.19:6443 --token g3l8wx.9yaprg4pgdi1k2vy \
--discovery-token-ca-cert-hash sha256:64924aa9fdfd8e6f47f9957bbf6b4b13751a21dfd28deb57dd72e802106aa719
root@node1-79:~# kubeadm join 10.0.0.19:6443 --token g3l8wx.9yaprg4pgdi1k2vy --discovery-token-ca-cert-hash sha256:64924aa9fdfd8e6f47f9957bbf6b4b13751a21dfd28deb57dd72e802106aa719
# node節(jié)點執(zhí)行后, 也會進行初始化, 下載flannel, 等到全部初始化完畢, node節(jié)點狀態(tài)也會變成ready, 可以在master節(jié)點查看
root@node1-79:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rancher/mirrored-flannelcni-flannel v0.20.0 fd14f6e39753 2 weeks ago 59.4MB
rancher/mirrored-flannelcni-flannel-cni-plugin v1.1.0 fcecffc7ad4a 5 months ago 8.09MB
k8s.gcr.io/kube-proxy v1.19.1 33c60812eab8 2 years ago 118MB
k8s.gcr.io/pause 3.2 80d28bedfe5d 2 years ago 683kB
root@master-19:~# kubectl get node
NAME STATUS ROLES AGE VERSION
master-19.k8s Ready master 8m29s v1.19.1
node1-79.k8s Ready <none> 82s v1.19.1
node2-89.k8s Ready <none> 80s v1.19.1
node3-99.k8s Ready <none> 78s v1.19.1
到此, 一個單master節(jié)點, 3個node節(jié)點的集群就搭建完畢, 可以運行服務(wù)了
5.5 kubeadm reset還原
在master節(jié)點(10.0.0.19)和其余三個node節(jié)點執(zhí)行reset, 恢復(fù). 清空后, 進行多節(jié)點master的配置
root@master-19:~# kubeadm reset
root@master-19:~# rm -rf $HOME/.kube/config # node節(jié)點如果沒有配置認證, 那么可以不執(zhí)行
root@node1-79:~# kubeadm reset
root@node1-79:~# rm -rf $HOME/.kube/config
root@node2-89:~# kubeadm reset
root@node2-89:~# rm -rf $HOME/.kube/config
root@node3-99:~# kubeadm reset
root@node3-99:~# rm -rf $HOME/.kube/config
6. 多節(jié)點master初始化
單master節(jié)點缺點:
- master故障后, 集群就無法使用了, 沒有提供高可用
- 如果想在master節(jié)點上運行業(yè)務(wù)服務(wù)pod時, 還需要額外配置
kubectl taint nodes --all node-role.kubernetes.io/master-, 否則在master節(jié)點上是無法運行除了管理端以外的pod的
