1. k8s 鏡像實現(xiàn)升級與回滾

1.1 pod版本更新流程

構(gòu)建請求
k8s集群使用REST API進(jìn)行通信, 請求會以HTTP的方式發(fā)給 kube-apiserver 進(jìn)行處理。當(dāng)我們是在shell中輸入“kubectl apply -f deployment.yml”時，本地的 kubectl 會解析yaml中的配置根據(jù)其內(nèi)容構(gòu)造相應(yīng)對象的 HTTP 請求參數(shù)。首先 kubectl 會檢查有沒有語法錯誤（比如創(chuàng)建不支持的資源或使用格式錯誤的鏡像名稱），出現(xiàn)錯誤后會直接返回不會發(fā)送到 kube-apiserver 以節(jié)省網(wǎng)絡(luò)負(fù)載。通過檢查后 kubectl 就會構(gòu)造出HTTP請求發(fā)送給 kube-apiserver。

認(rèn)證，鑒權(quán)，準(zhǔn)入控制
現(xiàn)在請求已經(jīng)發(fā)送給了 kube-apiserver，kube-apiserver接下來會判斷這個請求的發(fā)起者是否合法，即請求發(fā)起者對應(yīng)的用戶信息是否存儲k8s集群中，此過程稱為認(rèn)證（Authentication）。k8s提供了多種認(rèn)證方式，這里我們不做過多的討論，如果認(rèn)證沒有通過則會直接返回失敗的錯誤信息，通過了就會進(jìn)入一步鑒權(quán)。

雖然我們的身份已經(jīng)得到了k8s的認(rèn)可，但是身份 (identity) 和許可 (permission) 并不是一個概念，就像mysql賬號有的有讀寫權(quán)限而有的只有讀取權(quán)限一樣。此時kube-apiserver會檢查用戶的權(quán)限是否可以進(jìn)行相應(yīng)的操作，對應(yīng)我們文章中的命令就是創(chuàng)建deployment的權(quán)限，這里k8s也提供了多種方式進(jìn)行鑒權(quán)不再贅述。

好了 kube-apiserver 確認(rèn)請求發(fā)起者有相應(yīng)的權(quán)限，這樣就可以執(zhí)行創(chuàng)建deployment的動作了嗎。很“不幸”還有最后一步，準(zhǔn)入控制。Kubernetes 準(zhǔn)入控制器是控制和強制使用集群的一種插件。我們可以把它看作是攔截（已認(rèn)證）API 請求的攔截器，它可以更改請求對象，甚至完全拒絕請求。這是可以配置的插件，也就是說你通過這套機制自己開發(fā)一套插件部署在集群中來控制請求的行為。k8s官方提供了很多“內(nèi)置”的準(zhǔn)入控制器。

etcd
終于我們的請求被驗證通過，kube-apiserver會在etcd（服務(wù)發(fā)現(xiàn)的后端，存儲了集群的狀態(tài)及其配置）中創(chuàng)建我們的 Deployment 對象，創(chuàng)建過程中出現(xiàn)的任何錯誤都會被捕獲，最后kube-apiserver會構(gòu)造HTTP響應(yīng)返回給客戶端，我們在輸入完命下回車之后看到的信息就是kubectl 得到HTTP響應(yīng)解析后的信息。注意此時我們部署的 Deployment 對象現(xiàn)在雖然保存在于 etcd 中，但是它還沒有被部署到真正的 Node 上。

控制循環(huán) （Control loops）
接下來的步驟對于請求調(diào)用者來說都是異步執(zhí)行的，因為請求的響應(yīng)已經(jīng)在上一步得到了。

我們已經(jīng)創(chuàng)建了Deployment，但是并沒有創(chuàng)建涉及 Deployment 所依賴的資源拓?fù)洌ù死又芯褪荝eplicaSet 和Pod ）這其實是k8s通過內(nèi)置控制器（Controller）自動幫我們創(chuàng)建的。

Controller 是一個用于將系統(tǒng)狀態(tài)從當(dāng)前狀態(tài)調(diào)諧到期望狀態(tài)的異步腳本。所有內(nèi)置的 Controller 都通過組件 kube-controller-manager 并行運行，每種 Controller 都負(fù)責(zé)一種具體的控制流程。

比如我們本次使用到的 Deployment Controller

當(dāng)k8s在etcd中新創(chuàng)建了一個Deployment對象， Deployment Controller會監(jiān)聽（ListAndWatch）到這個事件之后然后檢查Deployment這個對象的期望狀態(tài)，和實際狀態(tài)作對比，比如這次檢查到相關(guān)聯(lián)的對象ReplicaSet（因為本質(zhì)上Deployment是通過控制ReplicaSet來控制Pod的）沒有被創(chuàng)建，Deployment Controller就會創(chuàng)建關(guān)聯(lián)的ReplicaSet ，創(chuàng)建ReplicaSet之后Deployment Controller的并不會檢查對應(yīng)管理的Pod，這是ReplicaSet Controller的工作。

ReplicaSet Controller 和 Deployment Controller 工作類似，ReplicaSet Controller監(jiān)聽的是ReplicaSet 這個對象，當(dāng)ReplicaSet 被創(chuàng)建時就會檢查這個ReplicaSet 對象對應(yīng)的期望狀態(tài)，創(chuàng)建Pod對象。

這里也可以看出Deployment并不是直接管理Pod，而是通過ReplicaSet，即Deployment管理ReplicaSet， ReplicaSet管理Pod。

實際上 Control loops 的細(xì)節(jié)有很多，包括實現(xiàn)監(jiān)聽的Informer機制，內(nèi)部工作隊列WorkQueue，本地緩存等等，如果全部展開如要大量的篇幅，而且作者也并沒有完全掌握內(nèi)部細(xì)節(jié)，我會在后續(xù)系列文章再次總結(jié)。

而此時我們也只是在etcd中創(chuàng)建了Deployment，ReplicaSet， Pod這3個對象，還沒有在實際Node 中部署。

調(diào)度（Scheduler）
當(dāng)所有的 Controller 正常運行后，etcd 中就會保存一個 Deployment、一個 ReplicaSet 和三個 Pod，并且可以通過 kube-apiserver 查看到。這時如果你在shell里 get pod查看剛才的pod狀態(tài) 你會看到Pending狀態(tài)（調(diào)度中，即它們還沒有被調(diào)度到集群中合適的 Node 上）。

k8s是依靠Scheduler這個組件完成調(diào)度操作的。Scheduler 組件運行在集群控制平面上，工作方式與其他 Controller 相同：監(jiān)聽事件并調(diào)諧狀態(tài)。具體來說， Scheduler 的作用是過濾 PodSpec 中 NodeName 字段為空的 Pod 并嘗試將其調(diào)度到合適的節(jié)點。Scheduler會經(jīng)過一系列的比如資源限制（cpu，內(nèi)存）等算法首先選出一批符合條件的 Node，然后通過第二輪算法（列如負(fù)載均衡情況）給Node打分，將Pod 調(diào)度最高分的Node上，調(diào)度器就會將Pod對象的nodeName字段的值，修改為上述Node的名字。度器對一個Pod調(diào)度成功，實際上就是將它的spec.nodeName字段填上調(diào)度結(jié)果的節(jié)點名字。

Kubelet
總結(jié)一下已經(jīng)完成的任務(wù)：
HTTP 請求通過了認(rèn)證、授權(quán)和準(zhǔn)入控制階段；
一個 Deployment、ReplicaSet 和三個 Pod 被持久化到 etcd；
最后每個 Pod 都被調(diào)度到合適的節(jié)點。
到目前為止，所有的工作僅僅只是針對保存在 etcd 中的資源對象，接下來的步驟涉及到在工作節(jié)點之間運行具體的容器，這是分布式系統(tǒng) Kubernetes 的關(guān)鍵因素。這些事情都是由 Kubelet 完成的。

在 Kubernetes 集群中，每個 Node 節(jié)點上都會啟動一個 Kubelet 服務(wù)進(jìn)程，該進(jìn)程用于處理 Scheduler 下發(fā)到本節(jié)點的 Pod 并管理其生命周期。這意味著它將處理 Pod 與 Container Runtime 之間所有的轉(zhuǎn)換邏輯，包括掛載卷、容器日志、垃圾回收等操作。

我們可以把 Kubelet 當(dāng)成一種特殊的 Controller，它每隔 20 秒（可以自定義）向 kube-apiserver 查詢 Pod，過濾 NodeName 與自身所在節(jié)點匹配的 Pod 列表。

當(dāng)檢測到新的pod對象還沒有在Node上創(chuàng)建時，Kubelet進(jìn)行一些前置操作，然后通過CRI（Container Runtime Interface）創(chuàng)建pause容器，通過CNI (Container Network Interface)為 Pod 設(shè)置網(wǎng)絡(luò)，最后通過CRI拉取我們文件定義中的nginx鏡像，創(chuàng)建并啟動起來！

image.png

1.2 命令行實現(xiàn)升級與回滾

deployment支持兩種鏡像更新策略：重建更新和滾動更新（默認(rèn)），可以通過strategy選項進(jìn)行配置

strategy：指定新的pod替換舊的pod的策略，支持兩個屬性：
  type：指定策略類型，支持兩種策略
    Recreate：在創(chuàng)建出新的pod之前會先殺掉所有已存在的pod
    RollingUpdate：滾動更新，就是殺死一部分，就啟動一部分，在更新過程中，存在兩個版本pod
  rollingUpdate：當(dāng)type為RollingUpdate時生效，用于為RollingUpdate設(shè)置參數(shù)，支持兩個屬性
    maxUnavailable：用來指定在升級過程中不可用pod的最大數(shù)量，默認(rèn)為25%
    maxSurge：用來指定在升級過程中可以超過期望的pod的最大數(shù)量，默認(rèn)為25%

1.2.1 重建更新

先創(chuàng)建，nginx-deployment，設(shè)置更新策略

root@k8s-ansible-client:~/yaml/20211024/01# cat nginx-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
     app: nginx-pod
  template:
    metadata:
      labels:
        app: nginx-pod
    spec:
      containers:
      - name: nginx
        image: nginx:1.17.1
  strategy: #策略
    type: Recreate  #重建更新策略

root@k8s-ansible-client:~/yaml/20211024/01# kubectl apply -f nginx-deployment.yamldeployment.apps/nginx-deployment created

root@k8s-ansible-client:~/yaml/20211024/01# kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-deployment-5d9c9b97bb-52p64   1/1     Running   0             43s
nginx-deployment-5d9c9b97bb-htv84   1/1     Running   0             43s
nginx-deployment-5d9c9b97bb-sh9qv   1/1     Running   0             43s

驗證

# 首先記錄原本的pod名
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-deployment-5d9c9b97bb-52p64   1/1     Running   0             80s
nginx-deployment-5d9c9b97bb-htv84   1/1     Running   0             80s
nginx-deployment-5d9c9b97bb-sh9qv   1/1     Running   0             80s
zookeeper1-cdbb7fbc-5pgdg           1/1     Running   1 (17d ago)   17d
zookeeper2-f4944446d-2xnjd          1/1     Running   0             17d
zookeeper3-589f6bc7-2mnz6           1/1     Running   0             17d

# 更改pod鏡像
root@k8s-ansible-client:~/yaml/20211024/01# kubectl set image deploy nginx-deployment nginx=nginx:1.17.2
deployment.apps/nginx-deployment image updated

root@k8s-ansible-client:~/yaml/20211024/01# kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-deployment-7c7477c7ff-2jvq9   1/1     Running   0             17s
nginx-deployment-7c7477c7ff-475pt   1/1     Running   0             17s
nginx-deployment-7c7477c7ff-ndtl9   1/1     Running   0             17s

root@k8s-ansible-client:~/yaml/20211024/01# kubectl describe pod nginx-deployment-7c7477c7ff-2jvq9
Name:         nginx-deployment-7c7477c7ff-2jvq9
Namespace:    default
Priority:     0
Node:         192.168.20.236/192.168.20.236
Start Time:   Wed, 03 Nov 2021 22:53:40 +0800
Labels:       app=nginx-pod
              pod-template-hash=7c7477c7ff
Annotations:  <none>
Status:       Running
IP:           172.20.108.97
IPs:
  IP:           172.20.108.97
Controlled By:  ReplicaSet/nginx-deployment-7c7477c7ff
Containers:
  nginx:
    Container ID:   docker://a2603aa066a31def432092288e78d7da6340674a01d563771dc104944e487657
    Image:          nginx:1.17.2
    Image ID:       docker-pullable://nginx@sha256:5411d8897c3da841a1f45f895b43ad4526eb62d3393c3287124a56be49962d41
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 03 Nov 2021 22:53:49 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4s94z (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-4s94z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  114s  default-scheduler  Successfully assigned default/nginx-deployment-7c7477c7ff-2jvq9 to 192.168.20.236
  Normal  Pulling    114s  kubelet            Pulling image "nginx:1.17.2"
  Normal  Pulled     107s  kubelet            Successfully pulled image "nginx:1.17.2" in 6.950730761s
  Normal  Created    106s  kubelet            Created container nginx
  Normal  Started    106s  kubelet            Started container nginx

注：Recreate 的升級策略就是完全刪除再重建，這樣會導(dǎo)致服務(wù)一段時間不可用

1.2.1 滾動更新

創(chuàng)建，ru-nginx-deployment，設(shè)置更新策略

root@k8s-ansible-client:~/yaml/20211024/01# cat ru-nginx-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
     app: nginx-pod
  template:
    metadata:
      labels:
        app: nginx-pod
    spec:
      containers:
      - name: nginx
        image: nginx:1.17.1
  strategy:
    type: RollingUpdate #滾動更新策略
    rollingUpdate:
      maxUnavailable: 25%  #用來指定在升級過程中不可用pod的最大數(shù)量，默認(rèn)為25%
      maxSurge: 25% #用來指定在升級過程中可以超過期望的pod的最大數(shù)量，默認(rèn)為25%

root@k8s-ansible-client:~/yaml/20211024/01# kubectl apply -f ru-nginx-deployment.yaml 
deployment.apps/nginx-deployment created
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get pods
NAME                                READY   STATUS    RESTARTS      AGE
nginx-deployment-5d9c9b97bb-26m8w   1/1     Running   0             12s
nginx-deployment-5d9c9b97bb-kvn9m   1/1     Running   0             12s
nginx-deployment-5d9c9b97bb-wsjph   1/1     Running   0             12s

驗證

# 滾動更新前(nginx為1.17.2)
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get deploy,rs,pod -o wide
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                SELECTOR
deployment.apps/nginx-deployment   3/3     3            3           74s   nginx        nginx:1.17.1                                          app=nginx-pod

NAME                                          DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                SELECTOR
replicaset.apps/nginx-deployment-5d9c9b97bb   3         3         3       74s   nginx        nginx:1.17.1                                          app=nginx-pod,pod-template-hash=5d9c9b97bb

NAME                                    READY   STATUS    RESTARTS      AGE   IP              NODE             NOMINATED NODE   READINESS GATES
pod/nginx-deployment-5d9c9b97bb-26m8w   1/1     Running   0             74s   172.20.213.58   192.168.20.253   <none>           <none>
pod/nginx-deployment-5d9c9b97bb-kvn9m   1/1     Running   0             74s   172.20.108.98   192.168.20.236   <none>           <none>
pod/nginx-deployment-5d9c9b97bb-wsjph   1/1     Running   0             74s   172.20.108.99   192.168.20.236   <none>           <none>

# 更新鏡像
root@k8s-ansible-client:~/yaml/20211024/01# kubectl set image deploy nginx-deployment nginx=nginx:1.17.3
deployment.apps/nginx-deployment image updated
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get deploy,rs,pod -o wide
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                SELECTOR
deployment.apps/nginx-deployment   3/3     3            3           3m22s   nginx        nginx:1.17.3                                          app=nginx-pod

NAME                                          DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                                                SELECTOR
replicaset.apps/nginx-deployment-5d9c9b97bb   0         0         0       3m22s   nginx        nginx:1.17.1                                          app=nginx-pod,pod-template-hash=5d9c9b97bb
replicaset.apps/nginx-deployment-76fd8c7f84   3         3         3       33s     nginx        nginx:1.17.3                                          app=nginx-pod,pod-template-hash=76fd8c7f84

NAME                                    READY   STATUS    RESTARTS      AGE   IP               NODE             NOMINATED NODE   READINESS GATES
pod/nginx-deployment-76fd8c7f84-9k5xc   1/1     Running   0             24s   172.20.108.100   192.168.20.236   <none>           <none>
pod/nginx-deployment-76fd8c7f84-f8fmc   1/1     Running   0             17s   172.20.213.60    192.168.20.253   <none>           <none>
pod/nginx-deployment-76fd8c7f84-zlx6z   1/1     Running   0             33s   172.20.213.59    192.168.20.253   <none>           <none>

# 查看更新后鏡像版本（nginx為1.17.3）

滾動更新的過程

image.png

1.2.3 回滾

deployment支持版本升級過程中的暫停，繼續(xù)功能以及版本回退等諸多功能，如下

kubectl rollout：版本升級相關(guān)功能，支持下面的選項：

status：顯示當(dāng)前升級狀態(tài)
history：顯示升級歷史記錄
pause：暫停版本升級過程
resume：繼續(xù)已經(jīng)暫停的版本升級過程
restart：重啟版本升級過程
undo：回滾到上一級版本（可以使用--to-revision回滾到指定版本）

# 當(dāng)前nginx版本
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get deploy -o wide
NAME               READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                SELECTOR
nginx-deployment   3/3     3            3           9m31s   nginx        nginx:1.17.3                                          app=nginx-pod

# 查看升級狀態(tài)
root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout status deploy nginx-deployment
deployment "nginx-deployment" successfully rolled out

# 查看升級歷史
root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout history deploy nginx-deployment
deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

# 查看版本詳情
root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout history deployment nginx-deployment --revision=2
deployment.apps/nginx-deployment with revision #2
Pod Template:
  Labels:   app=nginx-pod
    pod-template-hash=76fd8c7f84
  Containers:
   nginx:
    Image:  nginx:1.17.3
    Port:   <none>
    Host Port:  <none>
    Environment:    <none>
    Mounts: <none>
  Volumes:  <none>

# 版本回滾這里使用--to-revision=1回滾到1版本，如果省略這個選項，則會回退到上個版本
root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout undo deployment nginx-deployment --to-revision=1
deployment.apps/nginx-deployment rolled back

# 查看當(dāng)前版本（nginx為1.17.1）說明回退成功
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get deployments -o wide
NAME               READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                SELECTOR
nginx-deployment   3/3     3            3           12m   nginx        nginx:1.17.1                                          app=nginx-pod

1.3 金絲雀發(fā)布（灰度發(fā)布）

deployment支持更新過程中的控制，如"暫停（pause）"或"繼續(xù)（resume）"更新操作，觀察新服務(wù)實際運行狀況再決定操作

金絲雀發(fā)布（可以理解為灰度發(fā)布）：比如有一批新的pod資源創(chuàng)建完成后立即暫停更新過程，此時，僅存在一部分新版本的應(yīng)用，主體部分還是舊的版本。然后，再篩選一小部分的用戶請求路由到新的pod應(yīng)用，繼續(xù)觀察能否穩(wěn)定地按期望的方式運行。確定沒問題之后再繼續(xù)完成余下的pod資源滾動更新，否則立即回滾更新操作。這就是所謂的金絲雀發(fā)布。

# 更新deployment版本，并配置暫停deployment
root@k8s-ansible-client:~/yaml/20211024/01# kubectl set image deploy nginx-deployment nginx=nginx:1.17.3 && kubectl rollout pause deploy nginx-deployment
deployment.apps/nginx-deployment image updated
deployment.apps/nginx-deployment paused

# 查看rs，發(fā)現(xiàn)老版本rs沒有減少，新版本rs增加一個，狀態(tài)為更新中....
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get rs,deploy
NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-5d9c9b97bb   3         3         3       14m
replicaset.apps/nginx-deployment-76fd8c7f84   1         1         1       11m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   4/3     1            4           14m

root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout status deployment nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...

# 繼續(xù)deploy的更新
root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout resume deployment nginx-deployment 
deployment.apps/nginx-deployment resumed
root@k8s-ansible-client:~/yaml/20211024/01# kubectl rollout status deployment nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out

# 查看rs更新結(jié)果，發(fā)現(xiàn)老版本均停止，新版本已經(jīng)創(chuàng)建好
root@k8s-ansible-client:~/yaml/20211024/01# kubectl get rs,deploy
NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-5d9c9b97bb   0         0         0       16m
replicaset.apps/nginx-deployment-76fd8c7f84   3         3         3       14m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   3/3     3            3           16m

2. 通過Jenkins與gitlab實現(xiàn)代碼升級與回滾

2.1 部署Jenkins

安裝 Java

root@ubuntu-template:~# apt update -y
root@ubuntu-template:~# apt install openjdk-11-jdk -y

驗證Java版本

root@ubuntu-template:~# java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

安裝 Jenkins
從清華鏡像源下載Jenkins包

root@ubuntu-template:~# wget https://mirrors.tuna.tsinghua.edu.cn/jenkins/debian-stable/jenkins_2.303.2_all.deb

安裝

root@ubuntu-template:~# dpkg -i jenkins_2.303.2_all.deb
root@ubuntu-template:~# apt -f install

在安裝完成后，Jenkins 服務(wù)將會被自動啟動。你可以通過打印服務(wù)狀態(tài)來驗證它：

root@ubuntu-template:~# systemctl status jenkins
● jenkins.service - LSB: Start Jenkins at boot time
     Loaded: loaded (/etc/init.d/jenkins; generated)
     Active: active (exited) since Wed 2021-11-03 23:51:13 CST; 59s ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 0 (limit: 2278)
     Memory: 0B
     CGroup: /system.slice/jenkins.service

Nov 03 23:51:12 ubuntu-template systemd[1]: Starting LSB: Start Jenkins at boot time...
Nov 03 23:51:12 ubuntu-template jenkins[5046]: Correct java version found
Nov 03 23:51:12 ubuntu-template jenkins[5046]:  * Starting Jenkins Automation Server jenkins
Nov 03 23:51:12 ubuntu-template su[5100]: (to jenkins) root on none
Nov 03 23:51:12 ubuntu-template su[5100]: pam_unix(su-l:session): session opened for user jenkins by (uid=0)
Nov 03 23:51:12 ubuntu-template su[5100]: pam_unix(su-l:session): session closed for user jenkins
Nov 03 23:51:13 ubuntu-template jenkins[5046]:    ...done.
Nov 03 23:51:13 ubuntu-template systemd[1]: Started LSB: Start Jenkins at boot time.

瀏覽器訪問http://192.168.20.176:8080

image.png

注：如果出現(xiàn)離線模式，修改/var/lib/jenkins/updates/default.json，將google換成baidu

image.png

安裝完成

image.png

2.2 部署gitlab

安裝相關(guān)依賴包

root@ubuntu-template:~# apt update -y
root@ubuntu-template:~# apt install ca-certificates curl openssh-server postfix -y

設(shè)置存儲庫&安裝Gitlab

root@ubuntu-template:~# cd /tmp && curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh
root@ubuntu-template:/tmp# bash /tmp/script.deb.sh
root@ubuntu-template:/tmp# apt install gitlab-ce -y

直接下載deb包安裝

#下載地址
root@ubuntu-template:/usr/local/src# wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/apt/packages.gitlab.com/gitlab/gitlab-ce/ubuntu/pool/focal/main/g/gitlab-ce/gitlab-ce_14.4.1-ce.0_amd64.deb
root@ubuntu-template:/usr/local/src# dpkg -i gitlab-ce_14.4.1-ce.0_amd64.deb

配置文件

# 修改配置文件
root@ubuntu-template:/tmp# vim /etc/gitlab/gitlab.rb

# 設(shè)置訪問鏈接，關(guān)鍵詞external_url，修改url為你本地的IP
external_url 'http://{你的IP}'

# 設(shè)置發(fā)送郵箱
#綁定郵箱
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'QQ郵箱'          #xxx@qq.com
gitlab_rails['gitlab_email_display_name'] = '郵箱名稱'  #xxx
 
#配置SMTP
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.qq.com"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_user_name'] = "QQ郵箱"
gitlab_rails['smtp_password'] = "郵箱授權(quán)碼"
gitlab_rails['smtp_domain'] = "smtp.qq.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = true

# 修改端口
nginx['listen_port'] = 8099

# 修改端口后為了保持統(tǒng)一，需要將訪問鏈接修改成加上端口號的url
external_url 'http://{你的IP}:{你的端口號}'

常用命令

命令	作用
gitlab-ctl reconfigure	修改gitlab.rb文件之后重新加載配置
gitlab-ctl status	查看 GitLab 狀態(tài)
gitlab-ctl start	啟動 GitLab
gitlab-ctl stop	停止 GitLab
gitlab-ctl restart	重啟 GitLab
gitlab-ctl tail	查看所有日志
gitlab-ctl tail nginx/gitlab_acces.log	查看 nginx 訪問日志
gitlab-ctl tail postgresql	查看 postgresql 日志

針對上述配置，執(zhí)行完成后，需要重新配置

root@ubuntu-template:/tmp# gitlab-ctl reconfigure
root@ubuntu-template:/tmp# gitlab-ctl restart

初始化
如果想訪問配置鏈接登錄，需要先對賬號進(jìn)行初始化。
進(jìn)入GitLab控制臺

root@ubuntu-template:/tmp# gitlab-rails console

輸入以下命令，修改root用戶密碼。

user = User.where(id:1).first
user.password = {新密碼}
user.password_confirmation = {新密碼}
user.save!
exit

驗證
/etc/gitlab/initial_root_password
密碼：XGfZGJGgjQn/+YTmLThGobFfxLZjhJQQvn+noYWhMYc=

image.png

2.3 使用Jenkins與gitlab代碼升級和回退

2.3.1 Jenkins服務(wù)器設(shè)置ssh密鑰拉取gitlab代碼

jenkins服務(wù)器生成密鑰
如果使用Jenkins啟動的服務(wù)器，使用Jenkins用戶生成密鑰

root@ubuntu-template:~# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:obi+GG0NFiB88Zl/iN4TrDlWJC1qMq27c18o2e6BbJ8 root@ubuntu-template
The key's randomart image is:
+---[RSA 3072]----+
|o ...            |
| o o. +          |
|  . .* o.        |
|  . .oB...       |
| o ++..*S.       |
|  *o=+* o        |
| ..*+X.+         |
| .o*= = .        |
| o= =E           |
+----[SHA256]-----+

root@ubuntu-template:~# cat .ssh/id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEYxAEAysfDwOb0OW9allSrrsAwETprXTrlqHEKFqmhoT86VBUb+38rlHMwQxrIvTHqXIFdfptzPs7UWomk2gUN/Ge0Q60PLvv74byHDyzNJ3GQHrhTdDIuENToaDdEucNPWkWrMT/uZaTVa/sHk9gVGI2imVOmpeprx4/Bj5QgFGcYKUg66aLLnJIYYnojly/7v21ihJTqpgiT8uF6nOxAODR+x0E0j5uJeHDVotSsXDmT8w002GAhRuAfdV9Wh64BXMf4sgcNIOTJhh/1+facyAwMBhAHfzY1zo/OUS6AVzLYvq22NQ4hPlb7Ttv0kvvdNzf708HvE6WKvNkrwVJgww01EOGZvfdptBIcqJQ7QWn0GkB157EPn3t0U94r2VRP1dSszXw8QkwCRvSfN+Y+5MPARZBRotbZjklIJvxYSJbgTIMXup0A61dou9diCK/pMVkTxD/oCohtd+H5DfrjhMJT6/3TqEpUUmvbMp1ooinfGY1En3+jEvw2ROVNLU= root@ubuntu-template

將公鑰復(fù)制到gitlab上

image.png

驗證拉取代碼

root@ubuntu-template:/data# git clone git@192.168.20.216:pop/app1.git .
Cloning into '.'...
The authenticity of host '192.168.20.216 (192.168.20.216)' can't be established.
ECDSA key fingerprint is SHA256:mEyjijhaO0Gy5tMg1uTqQXjWOPiW1tf491hnlWUT3Wk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.20.216' (ECDSA) to the list of known hosts.
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (6/6), done.
root@ubuntu-template:/data# ls
index.html  README.md

2.3.2 配置Jenkins

創(chuàng)建一個任務(wù)，名稱：pop-test-job1

image.png

編輯任務(wù)配置，設(shè)置構(gòu)建參數(shù)

image.png

設(shè)置構(gòu)建執(zhí)行shell

image.png

shell腳本放在Jenkins服務(wù)器上，代碼如下：

root@ubuntu-template:/data/scripts# cat pop-test-job1.sh 
#!/bin/bash

#記錄腳本開始執(zhí)行時間
starttime=`date +'%Y-%m-%d %H:%M:%S'`

#變量
SHELL_DIR="/root/scripts"
SHELL_NAME="$0"
K8S_CONTROLLER1="192.168.20.250"
# K8S_CONTROLLER2="172.31.7.102"
DATE=`date +%Y-%m-%d_%H_%M_%S`
METHOD=$1
Branch=$2


if test -z $Branch;then
  Branch=develop
fi


function Code_Clone(){
  Git_URL="git@192.168.20.216:pop/app1.git"
  DIR_NAME=`echo ${Git_URL} |awk -F "/" '{print $2}' | awk -F "." '{print $1}'`
  DATA_DIR="/data/gitdata/pop"
  Git_Dir="${DATA_DIR}/${DIR_NAME}"
  cd ${DATA_DIR} &&  echo "即將清空上一版本代碼并獲取當(dāng)前分支最新代碼" && sleep 1 && rm -rf ${DIR_NAME}
  echo "即將開始從分支${Branch} 獲取代碼" && sleep 1
  git clone -b ${Branch} ${Git_URL} 
  echo "分支${Branch} 克隆完成，即將進(jìn)行代碼編譯!" && sleep 1
  #cd ${Git_Dir} && mvn clean package
  #echo "代碼編譯完成，即將開始將IP地址等信息替換為測試環(huán)境"
  #####################################################
  sleep 1
  cd ${Git_Dir}
  tar czf ${DIR_NAME}.tar.gz  ./*
}

#將打包好的壓縮文件拷貝到k8s 控制端服務(wù)器
function Copy_File(){
  echo "壓縮文件打包完成，即將拷貝到k8s 控制端服務(wù)器${K8S_CONTROLLER1}" && sleep 1
  scp ${Git_Dir}/${DIR_NAME}.tar.gz ${K8S_CONTROLLER1}:/data/k8s-data/dockerfile/web/pop/tomcat-app1
  echo "壓縮文件拷貝完成,服務(wù)器${K8S_CONTROLLER1}即將開始制作Docker 鏡像!" && sleep 1
}

#到控制端執(zhí)行腳本制作并上傳鏡像
function Make_Image(){
  echo "開始制作Docker鏡像并上傳到Harbor服務(wù)器" && sleep 1
  ssh ${K8S_CONTROLLER1} "cd /data/k8s-data/dockerfile/web/pop/tomcat-app1 && bash build-command.sh ${DATE}"
  echo "Docker鏡像制作完成并已經(jīng)上傳到harbor服務(wù)器" && sleep 1
}

#到控制端更新k8s yaml文件中的鏡像版本號,從而保持yaml文件中的鏡像版本號和k8s中版本號一致
function Update_k8s_yaml(){
  echo "即將更新k8s yaml文件中鏡像版本" && sleep 1
  ssh ${K8S_CONTROLLER1} "cd /data/k8s-data/yaml/pop/tomcat-app1 && sed -i 's/image: harbor.openscp.*/image: harbor.openscp.com\/base\/tomcat-app1:${DATE}/g' tomcat-app1.yaml"
  echo "k8s yaml文件鏡像版本更新完成,即將開始更新容器中鏡像版本" && sleep 1
}

#到控制端更新k8s中容器的版本號,有兩種更新辦法，一是指定鏡像版本更新，二是apply執(zhí)行修改過的yaml文件
function Update_k8s_container(){
  #第一種方法
  ssh ${K8S_CONTROLLER1} "kubectl set image deployment/pop-tomcat-app1-deployment  pop-tomcat-app1-container=harbor.openscp.com/base/tomcat-app1:${DATE} -n pop" 
  #第二種方法,推薦使用第一種
  #ssh root@${K8S_CONTROLLER1} "cd  /opt/k8s-data/yaml/magedu/tomcat-app1  && kubectl  apply -f tomcat-app1.yaml --record" 
  echo "k8s 鏡像更新完成" && sleep 1
  echo "當(dāng)前業(yè)務(wù)鏡像版本: harbor.openscp.com/base/tomcat-app1:${DATE}"
  #計算腳本累計執(zhí)行時間，如果不需要的話可以去掉下面四行
  endtime=`date +'%Y-%m-%d %H:%M:%S'`
  start_seconds=$(date --date="$starttime" +%s);
  end_seconds=$(date --date="$endtime" +%s);
  echo "本次業(yè)務(wù)鏡像更新總計耗時："$((end_seconds-start_seconds))"s"
}

#基于k8s 內(nèi)置版本管理回滾到上一個版本
function rollback_last_version(){
  echo "即將回滾之上一個版本"
  ssh ${K8S_CONTROLLER1}  "kubectl rollout undo deployment/pop-tomcat-app1-deployment  -n pop"
  sleep 1
  echo "已執(zhí)行回滾至上一個版本"
}

#使用幫助
usage(){
  echo "部署使用方法為 ${SHELL_DIR}/${SHELL_NAME} deploy "
  echo "回滾到上一版本使用方法為 ${SHELL_DIR}/${SHELL_NAME} rollback_last_version"
}

#主函數(shù)
main(){
  case ${METHOD}  in
  deploy)
    Code_Clone;
    Copy_File;
    Make_Image; 
    Update_k8s_yaml;
    Update_k8s_container;
  ;;
  rollback)
    rollback_last_version;
  ;;
  *)
    usage;
  esac;
}

main $1 $2

注：設(shè)置好Jenkins（如果是Jenkins啟動的服務(wù)器，Jenkins用戶免密登錄，并且目錄加好權(quán)限）到k8s deploy client 免密登錄

2.3.3 使用Jenkins代碼更新

image.png

驗證

root@k8s-ansible-client:/data/k8s-data/yaml/pop/tomcat-app1# kubectl get deploy,rs,pod -o wide -n pop
NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                  IMAGES                                                    SELECTOR
deployment.apps/pop-tomcat-app1-deployment   2/2     2            2           16m   pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:2021-11-06_00_35_09   app=pop-tomcat-app1-selector

NAME                                                    DESIRED   CURRENT   READY   AGE     CONTAINERS                  IMAGES                                                    SELECTOR
replicaset.apps/pop-tomcat-app1-deployment-54bb9d8f8c   0         0         0       16m     pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:v1                    app=pop-tomcat-app1-selector,pod-template-hash=54bb9d8f8c
replicaset.apps/pop-tomcat-app1-deployment-64c4689fd7   2         2         2       2m22s   pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:2021-11-06_00_35_09   app=pop-tomcat-app1-selector,pod-template-hash=64c4689fd7

NAME                                              READY   STATUS    RESTARTS      AGE     IP               NODE             NOMINATED NODE   READINESS GATES
pod/pop-tomcat-app1-deployment-64c4689fd7-m74wn   1/1     Running   0             2m22s   172.20.191.48    192.168.20.147   <none>           <none>
pod/pop-tomcat-app1-deployment-64c4689fd7-shhqc   1/1     Running   0             2m20s   172.20.108.104   192.168.20.236   <none>           <none>

2.3.4 使用Jenkins代碼回滾

執(zhí)行任務(wù)，method選擇rollback

image.png

驗證

root@k8s-ansible-client:/data/k8s-data/yaml/pop/tomcat-app1# kubectl get deploy,rs,pod -o wide -n pop
NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                  IMAGES                                   SELECTOR
deployment.apps/pop-tomcat-app1-deployment   2/2     2            2           27m   pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:v1   app=pop-tomcat-app1-selector

NAME                                                    DESIRED   CURRENT   READY   AGE    CONTAINERS                  IMAGES                                                    SELECTOR
replicaset.apps/pop-tomcat-app1-deployment-54bb9d8f8c   2         2         2       27m    pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:v1                    app=pop-tomcat-app1-selector,pod-template-hash=54bb9d8f8c
replicaset.apps/pop-tomcat-app1-deployment-64c4689fd7   0         0         0       13m    pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:2021-11-06_00_35_09   app=pop-tomcat-app1-selector,pod-template-hash=64c4689fd7
replicaset.apps/pop-tomcat-app1-deployment-749864754d   0         0         0       117s   pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:2021-11-06_00_46_40   app=pop-tomcat-app1-selector,pod-template-hash=749864754d

NAME                                              READY   STATUS        RESTARTS      AGE    IP               NODE             NOMINATED NODE   READINESS GATES
pod/pop-tomcat-app1-deployment-54bb9d8f8c-l9tcs   1/1     Running       0             31s    172.20.213.8     192.168.20.253   <none>           <none>
pod/pop-tomcat-app1-deployment-54bb9d8f8c-vg6td   1/1     Running       0             29s    172.20.191.51    192.168.20.147   <none>           <none>

3. k8s結(jié)合ELK日志收集

3.1 部署zookpeer

主機名稱	IP地址	配置
zookpeer-1	192.168.20.60	2核2G
zookpeer-2	192.168.20.247	2核2G
zookpeer-3	192.168.20.204	2核2G

下載zookpeer安裝包, 并且安裝

root@ubuntu-template:/usr/local/src# wget https://dlcdn.apache.org/zookeeper/zookeeper-3.5.9/apache-zookeeper-3.5.9-bin.tar.gz
root@ubuntu-template:~# apt update -y
root@ubuntu-template:~# apt install openjdk-8-jdk -y
root@ubuntu-template:/usr/local/src# tar -zxvf apache-zookeeper-3.5.9-bin.tar.gz -C /data/

# 編輯配置文件
root@ubuntu-template:/data/apache-zookeeper-3.5.9-bin/conf# cat zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper
clientPort=2181
maxClientCnxns=60
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
server.1=192.168.20.60:2888:3888
server.2=192.168.20.247:2888:3888
server.3=192.168.20.204:2888:3888

# 添加myid
192.168.20.60
root@ubuntu-template:/data/apache-zookeeper-3.5.9-bin/conf# echo 1 > /data/zookeeper/myid
192.168.20.247
root@ubuntu-template:/data/apache-zookeeper-3.5.9-bin/conf# echo 2 > /data/zookeeper/myid
192.168.20.204
root@ubuntu-template:/data/apache-zookeeper-3.5.9-bin/conf# echo 3 > /data/zookeeper/myid

啟動服務(wù)

root@ubuntu-template:/data/apache-zookeeper-3.5.9-bin# ./bin/zkServer.sh start

root@ubuntu-template:/data/apache-zookeeper-3.5.9-bin# ./bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/apache-zookeeper-3.5.9-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader

3.2 部署kafka

和zookeeper放在相同的服務(wù)器上
下載kafka安裝包，并安裝

root@ubuntu-template:/usr/local/src# wget https://archive.apache.org/dist/kafka/2.4.1/kafka_2.13-2.4.1.tgz
root@ubuntu-template:/usr/local/src# tar -zxvf kafka_2.13-2.4.1.tgz -C /data/

編輯配置文件

root@ubuntu-template:/data/kafka_2.13-2.4.1/config# vim server.properties
...
broker.id=101  # 三臺機器id要不一樣
...
listeners=PLAINTEXT://192.168.20.60:9092  # 設(shè)置本機ip
...
log.dirs=/data/kafka-logs
...
zookeeper.connect=192.168.20.60:2181,192.168.20.247:2181,192.168.20.204:2181
...

啟動服務(wù)

root@ubuntu-template:/data/kafka_2.13-2.4.1/bin# /data/kafka_2.13-2.4.1/bin/kafka-server-start.sh -daemon /data/kafka_2.13-2.4.1/config/server.properties

3.3 Elasticsearch集群部署

主機名稱	IP地址	配置
elasticsearch-1	192.168.20.239	2核2G
elasticsearch-2	192.168.20.121	2核2G
elasticsearch-3	192.168.20.213	2核2G

下載elasticsearch安裝包，并安裝

root@ubuntu-template:/usr/local/src# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-amd64.deb

root@ubuntu-template:/usr/local/src# dpkg -i elasticsearch-7.6.2-amd64.deb 
Selecting previously unselected package elasticsearch.
(Reading database ... 76333 files and directories currently installed.)
Preparing to unpack elasticsearch-7.6.2-amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (7.6.2) ...
Setting up elasticsearch (7.6.2) ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (245.4-4ubuntu3.11) ...

修改配置文件

# node1
root@ubuntu-template:/etc/elasticsearch# grep -v "^#" elasticsearch.yml |grep -v "^$"
cluster.name: my-pop-cluster01
node.name: node1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.20.239
http.port: 9200
discovery.seed_hosts: ["192.168.20.239", "192.168.20.121", "192.168.20.213"]
cluster.initial_master_nodes: ["192.168.20.239", "192.168.20.121", "192.168.20.213"]
gateway.recover_after_nodes: 2
action.destructive_requires_name: true

# node2
root@ubuntu-template:/etc/elasticsearch# grep -v "^#" elasticsearch.yml |grep -v "^$"
cluster.name: my-pop-cluster01
node.name: node2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.20.121
http.port: 9200
discovery.seed_hosts: ["192.168.20.239", "192.168.20.121", "192.168.20.213"]
cluster.initial_master_nodes: ["192.168.20.239", "192.168.20.121", "192.168.20.213"]
gateway.recover_after_nodes: 2
action.destructive_requires_name: true

# node3
root@ubuntu-template:/etc/elasticsearch# grep -v "^#" elasticsearch.yml |grep -v "^$"
cluster.name: my-pop-cluster01
node.name: node3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.20.213
http.port: 9200
discovery.seed_hosts: ["192.168.20.239", "192.168.20.121", "192.168.20.213"]
cluster.initial_master_nodes: ["192.168.20.239", "192.168.20.121", "192.168.20.213"]
gateway.recover_after_nodes: 2
action.destructive_requires_name: true

啟動服務(wù)

root@ubuntu-template:/etc/elasticsearch# systemctl restart elasticsearch

瀏覽器插件elasticsearch-head

image.png

訪問elasticsearch

image.png

3.4 配置filebeat

首先，鏡像里面需要filebet客戶端，配置文件如下：

root@k8s-ansible-client:/data/k8s-data/dockerfile/web/pop/tomcat-app1# cat filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /apps/tomcat/logs/catalina.out
  fields:
    type: tomcat-catalina
- type: log
  enabled: true
  paths:
    - /apps/tomcat/logs/localhost_access_log.*.txt
  fields:
    type: tomcat-accesslog
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.kafka:
  hosts: ["192.168.20.60:9092", "192.168.20.247:9092", "192.168.20.204:9092"]
  topic: 'pop-test-app1'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

將filebeat配置文件打入到業(yè)務(wù)鏡像(也可以在這步，安裝filebeat客戶端)，如下：

root@k8s-ansible-client:/data/k8s-data/dockerfile/web/pop/tomcat-app1# cat Dockerfile 
#tomcat web1
FROM harbor.openscp.com/base/tomcat-base:v8.5.43

ADD catalina.sh /apps/tomcat/bin/catalina.sh
ADD server.xml /apps/tomcat/conf/server.xml
ADD app1.tar.gz /data/tomcat/webapps/myapp/
ADD run_tomcat.sh /apps/tomcat/bin/run_tomcat.sh
ADD filebeat.yml /etc/filebeat/filebeat.yml
RUN chown  -R nginx.nginx /data/ /apps/

EXPOSE 8080 8443

CMD ["/apps/tomcat/bin/run_tomcat.sh"]

啟動腳本, 設(shè)置filebeat啟動

root@k8s-ansible-client:/data/k8s-data/dockerfile/web/pop/tomcat-app1# cat run_tomcat.sh 
#!/bin/bash

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat &
su - nginx -c "/apps/tomcat/bin/catalina.sh start"
tail -f /etc/hosts

重新構(gòu)建業(yè)務(wù)鏡像，替換Deployment文件里面鏡像地址，啟動業(yè)務(wù)pod

root@k8s-ansible-client:/data/k8s-data/yaml/pop/tomcat-app1# kubectl apply -f tomcat-app1.yaml 
deployment.apps/pop-tomcat-app1-deployment created
service/pop-tomcat-app1-service created
root@k8s-ansible-client:/data/k8s-data/yaml/pop/tomcat-app1# kubectl get deploy,rs,pod -o wide -n pop
NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS                  IMAGES                                   SELECTOR
deployment.apps/pop-tomcat-app1-deployment   1/1     1            1           5s    pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:v3   app=pop-tomcat-app1-selector

NAME                                                    DESIRED   CURRENT   READY   AGE   CONTAINERS                  IMAGES                                   SELECTOR
replicaset.apps/pop-tomcat-app1-deployment-849dcff94f   1         1         1       5s    pop-tomcat-app1-container   harbor.openscp.com/base/tomcat-app1:v3   app=pop-tomcat-app1-selector,pod-template-hash=849dcff94f

NAME                                              READY   STATUS    RESTARTS      AGE   IP              NODE             NOMINATED NODE   READINESS GATES
pod/pop-tomcat-app1-deployment-849dcff94f-dtrqd   1/1     Running   0             5s    172.20.191.52   192.168.20.147   <none>           <none>

3.5 部署logstash

放在elasticsearch集群服務(wù)器上
下載logstash安裝包, 并安裝

root@ubuntu-template:/usr/local/src# wget https://artifacts.elastic.co/downloads/logstash/logstash-7.6.2.deb
root@ubuntu-template:~# apt update -y
root@ubuntu-template:~# apt install openjdk-8-jdk -y
root@ubuntu-template:/usr/local/src# dpkg -i logstash-7.6.2.deb

編輯配置文件，講kafka數(shù)據(jù)導(dǎo)出到es里面

root@ubuntu-template:/etc/logstash/conf.d# cat kafka-to-es.yml 
input {
  kafka {
    bootstrap_servers => "192.168.20.60:9092,192.168.20.247:9092,192.168.20.204:9092"
    topics => ["pop-test-app1"]
    codec => "json"
  }
}




output {
  if [fields][type] == "tomcat-accesslog" {
    elasticsearch {
      hosts => ["192.168.20.239:9200","192.168.20.121:9200","192.168.20.213:9200"]
      index => "pop-test-app1-accesslog-%{+YYYY.MM.dd}"
    }
  }

  if [fields][type] == "tomcat-catalina" {
    elasticsearch {
      hosts => ["192.168.20.239:9200","192.168.20.121:9200","192.168.20.213:9200"]
      index => "pop-test-app1-catalinalog-%{+YYYY.MM.dd}"
    }
  }

#  stdout { 
#    codec => rubydebug
#  }
}

啟動logstash

root@ubuntu-template:~# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/kafka-to-es.yml > /dev/null &

驗證，查看es頁面

image.png

3.6 部署kibana

下載kibana軟件包，并安裝

root@ubuntu-template:/usr/local/src# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.6.2-amd64.deb
root@ubuntu-template:/usr/local/src# dpkg -i kibana-7.6.2-amd64.deb 
Selecting previously unselected package kibana.
(Reading database ... 108897 files and directories currently installed.)
Preparing to unpack kibana-7.6.2-amd64.deb ...
Unpacking kibana (7.6.2) ...
Setting up kibana (7.6.2) ...
Processing triggers for systemd (245.4-4ubuntu3.11) ...

修改配置文件

root@ubuntu-template:/usr/local/src# vim /etc/kibana/kibana.yml
...
server.host: "0.0.0.0"
...
elasticsearch.hosts: ["http://192.168.20.239:9200", "http://192.168.20.121:9200","http://192.168.20.213:9200"]
...
i18n.locale: "en"

啟動kibana服務(wù)

# 啟動成功以后 會監(jiān)聽 5601端口
root@ubuntu-template:/usr/local/src# /usr/share/kibana/bin/kibana --allow-root &

配置可視化
登錄kibana后臺

image.png

創(chuàng)建新索引模式

image.png

已經(jīng)創(chuàng)建好兩個新索引模式

image.png

然后查看pod的日志，可以根據(jù)條件搜索需要的日志

image.png

注：可以根據(jù)需要，寫一些聚合數(shù)據(jù)定制一些可視化圖表進(jìn)行展示

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av

k8s鏡像升級與回滾&基于ELK收集分析日志

k8s鏡像升級與回滾&基于ELK收集分析日志

1. k8s 鏡像實現(xiàn)升級與回滾

1.1 pod版本更新流程

1.2 命令行實現(xiàn)升級與回滾

1.2.1 重建更新

1.2.1 滾動更新

1.2.3 回滾

1.3 金絲雀發(fā)布（灰度發(fā)布）

2. 通過Jenkins與gitlab實現(xiàn)代碼升級與回滾

2.1 部署Jenkins

2.2 部署gitlab

2.3 使用Jenkins與gitlab代碼升級和回退

2.3.1 Jenkins服務(wù)器設(shè)置ssh密鑰拉取gitlab代碼

2.3.2 配置Jenkins

2.3.3 使用Jenkins代碼更新

2.3.4 使用Jenkins代碼回滾

3. k8s結(jié)合ELK日志收集

3.1 部署zookpeer

3.2 部署kafka

3.3 Elasticsearch集群部署

3.4 配置filebeat

3.5 部署logstash

3.6 部署kibana

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九 欧美,1769亚洲,黄色成人av

k8s鏡像升級與回滾&基于ELK收集分析日志

1. k8s 鏡像實現(xiàn)升級與回滾

1.1 pod版本更新流程

1.2 命令行實現(xiàn)升級與回滾

1.2.1 重建更新

1.2.1 滾動更新

1.2.3 回滾

1.3 金絲雀發(fā)布（灰度發(fā)布）

2. 通過Jenkins與gitlab實現(xiàn)代碼升級與回滾

2.1 部署Jenkins

2.2 部署gitlab

2.3 使用Jenkins與gitlab代碼升級和回退

2.3.1 Jenkins服務(wù)器設(shè)置ssh密鑰拉取gitlab代碼

2.3.2 配置Jenkins

2.3.3 使用Jenkins代碼更新

2.3.4 使用Jenkins代碼回滾

3. k8s結(jié)合ELK日志收集

3.1 部署zookpeer

3.2 部署kafka

3.3 Elasticsearch集群部署

3.4 配置filebeat

3.5 部署logstash

3.6 部署kibana

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容

色偷偷精品伊人,欧洲久久精品,欧美综合婷婷骚逼,国产AV主播,国产最新探花在线,九色在线视频一区,伊人大交九欧美,1769亚洲,黄色成人av