k8s-Jenkins（實現(xiàn)NFS動態(tài)存儲）

1、在K8s中 部署Jenkins優(yōu)點和缺點問題，簡單介紹下：

1.傳統(tǒng)Jenkins集群架構(gòu)一些問題

  • Master發(fā)生故障時，整個流程都不可用
  • Slave集群的環(huán)境配置不一樣，來完成不同語言的編譯打包，但是這些差異化的配置導(dǎo)致管理起來不方便，維護(hù)麻煩
  • 資源分配不均衡，有的slave要運行的job出現(xiàn)排隊等待，而有的salve處于空閑狀態(tài)
  • 資源浪費，每臺slave可能是物理機或者虛擬機，當(dāng)slave處于空閑狀態(tài)時，也不能完全釋放掉資源

2.K8s中Jenkins集群架構(gòu)優(yōu)點

  • 當(dāng)Jenkins Master接受到Build請求后，會根據(jù)配置的Label動態(tài)創(chuàng)建一個運行在Pod中的Jenkins Slave并注冊到Master上，當(dāng)運行完Job后，這個Slave會被注銷并且這個Pod也會自動刪除，恢復(fù)到最初的狀態(tài)(這個策略可以設(shè)置)
  • 服務(wù)高可用，當(dāng)Jenkins Master出現(xiàn)故障時，Kubernetes會自動創(chuàng)建一個新的Jenkins Master容器，并且將Volume分配給新創(chuàng)建的容器，保證數(shù)據(jù)不丟失，從而達(dá)到集群服務(wù)高可用的作用
  • 動態(tài)伸縮，合理使用資源，每次運行Job時，會自動創(chuàng)建一個Jenkins Slave，Job完成后，Slave自動注銷并刪除容器，資源自動釋放，并且Kubernetes會根據(jù)每個資源的使用情況，動態(tài)分配slave到空閑的節(jié)點上創(chuàng)建，降低出現(xiàn)因某節(jié)點資源利用率高，降低出現(xiàn)因某節(jié)點利用率高出現(xiàn)排隊的情況
  • 擴展性好，當(dāng)Kubernetes集群的資源嚴(yán)重不足導(dǎo)致Job排隊等待時，可以很容器的添加一個Kubernetes Node到集群，從而實現(xiàn)擴展

2、集群環(huán)境

image.png

3、使用Deployment和StatefulSet，兩個控制器方式部署Jenkins

Jenkins-Deployment 控制器方式

1)k8s-node1部署NFS服務(wù)端配置

# 所有服務(wù)端節(jié)點安裝nfs 
yum install -y nfs-utils
systemctl enable nfs-server rpcbind --now

# 創(chuàng)建nfs共享目錄，授權(quán)
mkdir -p /data/k8s
chown -R 777 /data/k8s

# 寫入/etc/exports文件
cat > /etc/exports << EOF
echo "/data/k8s 192.168.56.0/24(rw,no_root_squash,sync)" >/etc/exports
EOF

# 重啟
systemctl reload nfs-server

# 使用如下命令驗證：
showmout -e 192.168.56.11
....

創(chuàng)建Jenkins集群所需的YAML文件

1)創(chuàng)建命名空間和存放Jenkins的YAML目錄

kubectl create namespace devops
mkdir -p /opt/jenkins

2)為Jenkins數(shù)據(jù)持久化存儲創(chuàng)建一個PV

cat >/opt/jenkins/jenkins_pv.yaml <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: opspv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Delete
  nfs:
    server: 192.168.56.11
    path: /data/k8s

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: opspvc
  namespace: devops
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
EOF

3)創(chuàng)建Jenkins集群權(quán)限serviceAccount文件

cat >/opt/jenkins/jenkins_rbac.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: devops

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jenkins
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: devops
EOF

4)創(chuàng)建Jenkins Deployment

cat jenkins_deployment.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
  namespace: devops
spec:
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccount: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080                #外部訪問端口
          name: web
          protocol: TCP
        - containerPort: 50000              #jenkins save發(fā)現(xiàn)端口
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60          #容器初始化完成后，等待60秒進(jìn)行探針檢查
          timeoutSeconds: 5
          failureThreshold: 12          #當(dāng)Pod成功啟動且檢查失敗時，Kubernetes將在放棄之前嘗試failureThreshold次。放棄生存檢查意味著重新啟動Pod。而放棄就緒檢查，Pod將被標(biāo)記為未就緒。默認(rèn)為3.最小值為1
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:                       #需要將jenkins_home目錄掛載出來
        - name: jenkinshome
          subPath: jenkins
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
      volumes:
      - name: jenkinshome
        persistentVolumeClaim:
          claimName: opspvc

5)創(chuàng)建Jenkins SVC

cat >/opt/jenkins/jenkins_svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: demon
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 30002
  - name: agent
    port: 50000
    targetPort: agent
EOF

6)依次創(chuàng)建

[root@k8s-node1 jenkins]# ls
jenkins_deployment.yaml jenkins_pv.yaml jenkins_rbac.yaml jenkins_svc.yaml

[root@k8s-node1 jenkins]# kubectl apply -f ./

7)查看結(jié)果

[root@k8s-node1 jenkins]# kubectl get pv,pvc,pod,svc -n devops
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM          STORAGECLASS   REASON    AGE
persistentvolume/opspv   10Gi       RWX            Delete           Bound     devops/opspvc                            1h

NAME                           STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/opspvc   Bound     opspv     10Gi       RWX                           1h

NAME                           READY     STATUS    RESTARTS   AGE
pod/jenkins-6d7bc49b74-d9jxc   1/1       Running   0          1h

NAME              TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                          AGE
service/jenkins   NodePort   10.1.148.201   <none>        8080:30002/TCP,50000:26723/TCP   1h

8080：端口為我們jenkins訪問端口
50000：端口為jenkins save發(fā)現(xiàn)端口

Jenkins-StatefulSet 控制器方式

1.創(chuàng)建兩個目錄

mkdir -pv /data/nfs-client/ /data/jenkins

2.nfs服務(wù)端

# 所有服務(wù)端節(jié)點安裝nfs 
yum install -y nfs-utils
systemctl enable nfs-server rpcbind --now

# 創(chuàng)建nfs共享目錄，授權(quán)
mkdir -p /data/k8s && chown -R 777 /data/k8s

# 寫入/etc/exports文件
cat > /etc/exports << EOF
echo "/data/k8s 192.168.56.0/24(rw,no_root_squash,sync)" >/etc/exports
EOF

# 重啟
systemctl reload nfs-server

# 使用如下命令驗證：
showmout -e 192.168.56.11
....

部署nfs-client所有認(rèn)證（rbac、class、）

1.創(chuàng)建rbac.yaml文件

cat > rbac.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io
EOF

4.創(chuàng)建class.yaml文件

cat > class.yaml << EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"
EOF

5.創(chuàng)建deployment.yaml

cat > deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: fuseim.pri/ifs
            - name: NFS_SERVER
              value: 192.168.1.201  # 修改為自己NFS服務(wù)IP
            - name: NFS_PATH
              value: /data/k8s    # 修改為共享目錄
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.201  # 修改為自己NFS服務(wù)IP
            path: /data/k8s   # 修改為共享目錄
EOF

創(chuàng)建Jenkins yaml文件

1)創(chuàng)建jenkins_rbac.yaml

cat > jenkins_rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: jenkins
  namespace: devops
rules:
  - apiGroups: ["extensions", "apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: devops

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: jenkinsClusterRole
  namespace: devops
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["create","delete","get","list","patch","update","watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkinsClusterRoleBinding
  namespace: devops
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jenkinsClusterRole
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: devops
EOF

2)創(chuàng)建jenkins_serviceaccount.yaml

cat > jenkins_serviceaccount.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: devops
EOF

3)創(chuàng)建jenkins_StatefulSet.yaml

cat > jenkins_StatefulSet.yaml << EOF
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: jenkins
  namespace: devops01
  labels:
    name: jenkins 
spec:
  serviceName: jenkins
  selector:
    matchLabels:
      app: jenkins
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: jenkins
      labels:
        app: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      containers:
      - name: jenkins
        image: jenkins/jenkins:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080                #外部訪問端口
          name: web
          protocol: TCP
        - containerPort: 50000              #jenkins save發(fā)現(xiàn)端口
          name: agent
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60          #容器初始化完成后，等待60秒進(jìn)行探針檢查
          timeoutSeconds: 5
          failureThreshold: 12    #當(dāng)Pod成功啟動且檢查失敗時，Kubernetes將在放棄之前嘗試failureThreshold次。放棄生存檢查意味著重新啟動Pod。而放棄就緒檢查，Pod將被標(biāo)記為未就緒。默認(rèn)為3.最小值為1
        readinessProbe:
          httpGet:
            path: /login
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        volumeMounts:                       #需要將jenkins_home目錄掛載出來
        - name: jenkins-home
          subPath: jenkins
          mountPath: /var/jenkins_home
        env:
        - name: LIMITS_MEMORY
          valueFrom:
            resourceFieldRef:
              resource: limits.memory
              divisor: 1Mi
        - name: JAVA_OPTS
          value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai
      securityContext:
        fsGroup: 1000
  volumeClaimTemplates: 
  - metadata:
      name: jenkins-home
    spec: 
      storageClassName: "managed-nfs-storage"
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
EOF

4)創(chuàng)建jenkins_Service.yaml

cat > jenkins_Service.yaml << EOF
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: devops01
  labels:
    app: jenkins
spec:
  selector:
    app: jenkins
  type: NodePort
  ports:
  - name: web
    port: 8080
    targetPort: web
    nodePort: 8081
  - name: agent
    port: 50000
    targetPort: agent
EOF

5)依次創(chuàng)建

[root@k8s-node1 jenkins]# ls
jenkins_rbac.yaml jenkins_Service.yaml jenkins_StatefulSet.yaml jenkins_rbac.yaml

[root@k8s-node1 jenkins]# kubectl apply -f ./

7)查看結(jié)果
.....

Jenkins部署OK后，可以通過瀏覽器訪問集群任意IP的svc端口

管理員密碼路徑：持久化在/data/k8s下，所以jenkins的所有配置都在這下面

cat /data/k8s/jenkins/secrets/initialAdminPassword

1)直接推薦安裝即可

image.png

2)安裝完成后我們進(jìn)入jenkins主頁面

image.png

3)Jenkins–>插件–>安裝插件Kubernetes

image.png

Jenkins中配置k8s

1)系統(tǒng)管理->系統(tǒng)配置

image.png

2)配置拉到最下面找到Kubernetes插件

Name            # 配置的名稱
Kubernetes URL   # 這里的URL是K8s內(nèi)部的URL，實際上就是`svcname` = `https://kubernetes.default.svc.cluster.local`
Kubernetes Namespace k8s的命名空間 # 實際上就是Jenkins所在的命名空間
image.png

3)Jenkins URL配置

Jenkins URL  # 這里的URL是jenkins的svc名稱加上命名空間，實際上就是在k8s集群內(nèi)部訪問jenkins的一個方法，這里也不需要修改
http://jenkins.demon.svc.cluster.local:8080
image.png

4)配置添 Jenkins Slave Pod模板

Name = Pod 名稱 Namespave = Pod命名空間 Labels = Pod標(biāo)簽

5)容器的模板配置

image.png

6)創(chuàng)建volume的配置

Jenkins Master收到Build請求時，會根據(jù)配置的Label動態(tài)創(chuàng)建一個運行在Pod中的Jenkins Slave并注冊到Master上，當(dāng)Job運行完，這個Slave會被注銷并且這個Pod也會自動刪除，恢復(fù)到最初狀態(tài)


image.png

7)測試驗證

新建Job選擇流水線

image.png

流水線Pipeline

def label = "jenkins-slave"
podTemplate(label: label, cloud: 'kubernetes')
{
node(label) {
        stage('pull code') {
            echo "拉取代碼"
        }
        stage('build') {
            echo "代碼編譯"
        }
        stage('SonarQube') {
            echo "質(zhì)量掃描"
        }
    }
}

執(zhí)行效果

image.png
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容