k3s初探

k3s是rancher?出品的一個(gè)簡化、輕量的k8s，本篇博客記錄k3s的安裝及踩的部分坑。
從名字上也能看出，k3s比k8s少了些東西，詳情可見其官網(wǎng)k3s.io，本地試驗(yàn)可參考官網(wǎng)離線安裝教程

k3s官網(wǎng)

安裝步驟

準(zhǔn)備工作

首先去其github的releases頁下載主可執(zhí)行文件k3s、離線安裝包k3s-airgap-images-amd64.tar和安裝腳本
我用的是v1.28.3+k3s2版本，其于2023年11月8日發(fā)布。
增加可執(zhí)行文件和腳本的可執(zhí)行權(quán)限

wget https://get.k3s.io -O install-k3s.sh
chmod +x install-k3s.sh

需要有/usr/local/bin/k3s，可考慮軟連接

sudo ln -s /home/dev/program/k3s /usr/local/bin/k3s

復(fù)制離線安裝包tar文件到/var/lib/rancher/k3s/agent/images

sudo mkdir -p /var/lib/rancher/k3s/agent/images
sudo cp k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images

定制一些變量

先設(shè)置變量如下：

export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_EXEC="--docker --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 666"

逐個(gè)解釋一下：

  1. INSTALL_K3S_SKIP_DOWNLOAD=true效果為不去下載k3s可執(zhí)行文件
  2. INSTALL_K3S_EXEC="（略）"效果為啟動(dòng)k3s服務(wù)時(shí)使用的額外參數(shù)
  3. --docker效果為使用docker而不是默認(rèn)的containerd
  4. --write-kubeconfig-mode 666效果為將配置文件權(quán)限改為非所有者也可讀可寫，進(jìn)而使kubectl命令無需root或sudo
  5. --write-kubeconfig ~/.kube/config效果為將配置文件寫到k8s默認(rèn)會(huì)用的位置，而不是k3s默認(rèn)的位置/etc/rancher/k3s/k3s.yaml。后者會(huì)導(dǎo)致istio、helm需要額外設(shè)置或無法運(yùn)行。

官網(wǎng)教程-安裝選項(xiàng)中還有其他可用的選項(xiàng)

執(zhí)行安裝腳本

$ ./install-k3s.sh
[INFO]  Skipping k3s download and verify
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

執(zhí)行k3s命令看看效果

$ k3s
NAME:
   k3s - Kubernetes, but small and simple

USAGE:
   k3s [global options] command [command options] [arguments...]

VERSION:
   v1.28.3+k3s2 (bbafb86e)

COMMANDS:
   server           Run management server
   agent            Run node agent
   kubectl          Run kubectl
   crictl           Run crictl
   ctr              Run ctr
   check-config     Run config check
   token            Manage bootstrap tokens
   etcd-snapshot    
   secrets-encrypt  Control secrets encryption and keys rotation
   certificate      Manage K3s certificates
   completion       Install shell completion script
   help, h          Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug                     (logging) Turn on debug logs [$K3S_DEBUG]
   --data-dir value, -d value  (data) Folder to hold state (default: /var/lib/rancher/k3s or ${HOME}/.rancher/k3s if not root)
   --help, -h                  show help
   --version, -v               print the version

還有k3s kubectl和kubectl

$ k3s kubectl get all --all-namespaces
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-66f496764-mkwjv      1/1     Running   0          5m9s
kube-system   pod/helm-install-traefik-t4xlj   1/1     Running   0          5m8s


NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-system   service/kube-dns     ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m27s
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                  5m25s


NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   1/1     1            1           5m27s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-66f496764   1         1         1       5m9s



NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   0/1           5m8s       5m25s

訪問kubernetes服務(wù)

由于k3s默認(rèn)沒有提供dashboard作為web ui，先訪問k8s的rest

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP

會(huì)要求輸入用戶名密碼，在~/.kube/config中有訪問其的用戶名密碼，內(nèi)容類似如下：

users:
- name: default
  user:
    password: ec2fb0ab4401d7f2525d480fd08e908d
    username: admin

文件位置默認(rèn)為/etc/rancher/k3s/k3s.yaml，但是前述步驟中通過--write-kubeconfig ~/.kube/config修改
認(rèn)證似乎是www basic（對(duì)k8s還沒了解到這種程度，此處存疑）
也可kubectl version或隨便kubectl run測試一下

若干問題

如何卸載

見install.sh的回顯，其中有uninstall-script：

[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh

離線安裝包

如果沒有復(fù)制k3s-airgap-images-amd64.tar，會(huì)卡著

$ k3s kubectl get all --all-namespaces
NAMESPACE     NAME                             READY   STATUS              RESTARTS   AGE
kube-system   pod/helm-install-traefik-t4xlj   0/1     ContainerCreating   0          4m42s
kube-system   pod/coredns-66f496764-mkwjv      0/1     ContainerCreating   0          4m43s


NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-system   service/kube-dns     ClusterIP   10.43.0.10   <none>        53/UDP,53/TCP,9153/TCP   5m1s
default       service/kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP                  4m59s


NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/1     1            0           5m1s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-66f496764   1         1         0       4m43s



NAMESPACE     NAME                             COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik   0/1           4m42s      4m59s

復(fù)制后，安裝過程繼續(xù)

拉不下鏡像

可能因?yàn)樯独R像失敗，可通過kubectl describe pod coredns-57d8bbb86-mndrr -n kube-system查看events：

Events:
  Type     Reason                  Age               From                      Message
  ----     ------                  ----              ----                      -------
  Warning  FailedScheduling        <unknown>         default-scheduler         0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled               <unknown>         default-scheduler         Successfully assigned kube-system/coredns-57d8bbb86-mndrr to dk-aspire-5943g
  Warning  FailedCreatePodSandBox  3s (x4 over 89s)  kubelet, dk-aspire-5943g  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

可見是由于拉不下鏡像k8s.gcr.io/pause:3.1，于是從阿里云拉下鏡像，再tag

$ docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
3.1: Pulling from google_containers/pause
cf9202429979: Pull complete 
Digest: sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca
Status: Downloaded newer image for registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
$ docker images
REPOSITORY                                                  TAG                 IMAGE ID            CREATED             SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/pause   3.1                 da86e6ba6ca1        22 months ago       742kB
$ docker tag da86e6ba6ca1 k8s.gcr.io/pause:3.1
$ docker images
REPOSITORY                                                  TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/pause                                            3.1                 da86e6ba6ca1        22 months ago       742kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause   3.1                 da86e6ba6ca1        22 months ago       742kB

kubectl需要root權(quán)限

前已述及，在安裝前設(shè)置若干變量，其中有針對(duì)這個(gè)問題的

$ kubectl get all
WARN[2019-10-20T22:58:52.068331383+08:00] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions 
error: Error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

/etc/rancher/k3s/k3s.yaml的默認(rèn)權(quán)限為-rw-------即600，所有者root root
根據(jù)提示，在啟動(dòng)時(shí)需要帶有--write-kubeconfig-mode *新權(quán)限*，經(jīng)試驗(yàn)，666可以起到讓kubectl無需root權(quán)限的效果
此外，v1.17.0+k3s.1的文檔中提到一個(gè)選項(xiàng)：

   --rootless                                 (experimental) Run rootless

但是試驗(yàn)不成功，service k3s啟動(dòng)失敗
定制環(huán)境變量如下：

export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_EXEC="--docker --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 666"

啟動(dòng)失敗日志片段如下：

$ ./install-k3s.sh 
（略）
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xe" for details.
$ journalctl -xe
（略）
Jan 09 15:31:40 dk-mi13 k3s[4490]: time="2020-01-09T15:31:40.488024565+08:00" level=fatal msg="resolving : determining current user: $HOME is not defined"
Jan 09 15:31:40 dk-mi13 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit k3s.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Jan 09 15:31:40 dk-mi13 systemd[1]: k3s.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Jan 09 15:31:40 dk-mi13 systemd[1]: Failed to start Lightweight Kubernetes.
-- Subject: A start job for unit k3s.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit k3s.service has finished with a failure.
-- 
-- The job identifier is 16669 and the job result is failed.

//TODO
這個(gè)問題暫未解決，報(bào)錯(cuò)中似乎比較關(guān)鍵的

level=fatal msg="resolving : determining current user: $HOME is not defined"

也有問題，無論是自己的賬戶還是su用root賬戶，查看$HOME變量均可得到值

$ echo $HOME
/home/dk
$ env | grep HOME
HOME=/home/dk
（略）
$ su
Password: 
# echo $HOME
/root
# env | grep HOME
HOME=/root
（略）

KUBECONFIG位置

配置文件默認(rèn)位置給其他處帶來一些不便，例如使用helm需要如下額外參數(shù)以指定配置文件的位置

--kubeconfig /etc/rancher/k3s/k3s.yaml

想改為更為通用的~/.kube/config，使用參數(shù)--write-kubeconfig ~/.kube/config
此外，在v1.17.0+k3s.1版本中，使用kubectl -v 6可見其對(duì)配置文件的處理：

$ kubectl get all -v 6
I0109 11:27:11.815808   20876 loader.go:375] Config loaded from file:  /etc/rancher/k3s/k3s.yaml

依然讀取/etc/rancher/k3s/k3s.yaml，但這個(gè)文件實(shí)際上鏈接到了~/.kube/config：

$ ll /etc/rancher/k3s/k3s.yaml 
lrwxrwxrwx 1 root root 21 Jan  9 15:48 /etc/rancher/k3s/k3s.yaml -> /home/dk/.kube/config
$ ll ~/.kube/config
-rw-rw-rw- 1 root root 1052 Jan  9 15:48 /home/dk/.kube/config

kubectl get all 耗時(shí)長

用v1.17.0+k3s.1執(zhí)行kubectl get all耗時(shí)較長（v1.18.6+k3s1中問題依舊），但是kubectl get pod等查看一種資源的命令耗時(shí)并不較長，增加-v 6查看更詳細(xì)日志：

$ kubectl get all -v 6
（略）
I0109 11:27:11.824426   20876 round_trippers.go:443] GET https://127.0.0.1:6443/api?timeout=32s 200 OK in 8 milliseconds
I0109 11:27:11.824977   20876 round_trippers.go:443] GET https://127.0.0.1:6443/apis?timeout=32s 200 OK in 0 milliseconds
I0109 11:27:11.825346   20876 cached_discovery.go:130] failed to write cache to /home/dk/.kube/cache/discovery/127.0.0.1_6443/servergroups.json due to mkdir /home/dk/.kube/cache: permission denied
I0109 11:27:11.828528   20876 round_trippers.go:443] GET https://127.0.0.1:6443/api/v1?timeout=32s 200 OK in 2 milliseconds
I0109 11:27:11.829574   20876 cached_discovery.go:87] failed to write cache to /home/dk/.kube/cache/discovery/127.0.0.1_6443/v1/serverresources.json due to mkdir /home/dk/.kube/cache: permission denied
（略）

可知，原因是向~/.kube/cache文件夾下寫時(shí)無權(quán)限，處理大量錯(cuò)誤耗費(fèi)了時(shí)間。默認(rèn)無此文件夾，上層.kube文件夾所有者root root，權(quán)限755

$ ll ~ | grep .kube
drwxr-xr-x  2 root root  4096 Jan  9 11:30  .kube/

若使用sudo kubectl get all沒有此耗時(shí)問題。
修正方法，將此文件夾權(quán)限改為其他用戶可寫；或者新建cache和http-cache兩文件夾，并更改所有者為當(dāng)前用戶。后一種方法例：

$ sudo mkdir cache http-cache
$ sudo chown dk:dk cache http-cache

至此，解決了kubectl get all等命令耗時(shí)太長問題

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容