使用kubeadm安裝kubernetes_v1.18.x

準備工作

• 參考：https://kuboard.cn/install/history-k8s/install-k8s-1.18.x.html

• 參考：node節(jié)點數(shù)量和pod數(shù)量優(yōu)勢劣勢對比https://blog.csdn.net/weixin_38754564/article/details/100917081

• 至少2臺 2核4G 的服務器

• Cent OS 7.6 / 7.7 / 7.8

安裝后的軟件版本為

• Kubernetes v1.18.9
• calico 3.13.1
• nginx-ingress 1.5.5
• Docker 19.03.8

服務器配置

IPAddress | Role | 配置
---|---|---|---
10.100.5.116 | master | 16核32G
10.100.5.126 | node1 | 16核32G
10.100.5.161 | node2 | 16核32G

• 關(guān)閉并禁用防火墻
• 關(guān)閉并禁用selinux
• 關(guān)閉swap

安裝常用工具

[root@localhost ~]# yum install -y lrzsz zip unzip openssl openssl-devel patch wget lsof ntp ntpdate net-tools vim tree

檢查 centos / hostname

# 在 master 節(jié)點和 worker 節(jié)點都要執(zhí)行
cat /etc/redhat-release

# 此處 hostname 的輸出將會是該機器在 Kubernetes 集群中的節(jié)點名字
# 不能使用 localhost 作為節(jié)點的名字
hostname

# 請使用 lscpu 命令，核對 CPU 信息
# Architecture: x86_64    本安裝文檔不支持 arm 架構(gòu)
# CPU(s):       2         CPU 內(nèi)核數(shù)量不能低于 2
lscpu
 

操作系統(tǒng)兼容性

CentOS 版本| 本文檔是否兼容| 備注
--- | --- | --- | ---
7.8| ??| 已驗證
7.7| ??| 已驗證
7.6| ??| 已驗證
7.5| ??| 已證實會出現(xiàn) kubelet 無法啟動的問題
7.4| ??| 已證實會出現(xiàn) kubelet 無法啟動的問題
7.3| ??| 已證實會出現(xiàn) kubelet 無法啟動的問題
7.2| ??| 已證實會出現(xiàn) kubelet 無法啟動的問題

配置hosts

如果您需要修改 hostname，可執(zhí)行如下指令：

# 修改 hostname
hostnamectl set-hostname your-new-host-name
# 查看修改結(jié)果
hostnamectl status
# 設(shè)置 hostname 解析
echo "127.0.0.1   $(hostname)" >> /etc/hosts

檢查網(wǎng)絡(luò)

在所有節(jié)點執(zhí)行命令

# ip route show
# ip address

kubelet使用的IP地址

ip route show 命令中，可以知道機器的默認網(wǎng)卡，通常是 eth0，如 default via 172.21.0.23 dev eth0
ip address 命令中，可顯示默認網(wǎng)卡的 IP 地址，Kubernetes 將使用此 IP 地址與集群內(nèi)的其他節(jié)點通信，如 172.17.216.80
所有節(jié)點上 Kubernetes 所使用的 IP 地址必須可以互通（無需 NAT 映射、無安全組或防火墻隔離）

如果是多個網(wǎng)卡，例如virtualbox虛擬機或者公有云場景，網(wǎng)卡1為host-only，網(wǎng)卡2為nat出公網(wǎng)，需要做如下配置操作，將host-only（ifcfg-enp0s3）修改為DEFROUTE=no

image.png

image.png

image.png

安裝docker及kubelet

使用 root 身份在所有節(jié)點執(zhí)行如下代碼，以安裝軟件：

• docker
• nfs-utils
• kubectl / kubeadm / kubelet

請將腳本最后的 1.18.9 替換成您需要的版本號， 腳本中間的 v1.18.x 不要替換

master 節(jié)點和 worker 節(jié)點都要執(zhí)行

[root@localhost ~]# sh install_kubelet.sh

以下是install_kubelet.sh腳本：

#!/bin/bash

# 在 master 節(jié)點和 worker 節(jié)點都要執(zhí)行
# 最后一個參數(shù) 1.18.9 用于指定 kubenetes 版本，支持所有 1.18.x 版本的安裝
# 騰訊云 docker hub 鏡像
# export REGISTRY_MIRROR="https://mirror.ccs.tencentyun.com"
# DaoCloud 鏡像
# export REGISTRY_MIRROR="http://f1361db2.m.daocloud.io"
# 華為云鏡像
# export REGISTRY_MIRROR="https://05f073ad3c0010ea0f4bc00b7105ec20.mirror.swr.myhuaweicloud.com"
# 阿里云 docker hub 鏡像
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com

# DOCKER_VERSION：安裝docker的版本號
# 安裝需求修改版本號
DOCKER_VERSION="19.03.8"

# K8S_VERSION：安裝kubelet、kubeadm、kubectl的版本號
# 安裝需求修改版本號
K8S_VERSION="1.18.9"

# 安裝 docker
# 參考文檔如下
# https://docs.docker.com/install/linux/docker-ce/centos/ 
# https://docs.docker.com/install/linux/linux-postinstall/

# 卸載舊版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-ce-cli \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine

# 設(shè)置 yum repository
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

# 安裝并啟動 docker
yum install -y docker-ce-${DOCKER_VERSION}  docker-ce-cli-${DOCKER_VERSION} containerd.io
systemctl enable docker
systemctl start docker

# 安裝 nfs-utils
# 必須先安裝 nfs-utils 才能掛載 nfs 網(wǎng)絡(luò)存儲
yum install -y nfs-utils
yum install -y wget

# 關(guān)閉 防火墻
systemctl stop firewalld
systemctl disable firewalld

# 關(guān)閉 SeLinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

# 關(guān)閉 swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab

# 修改 /etc/sysctl.conf
# 如果有配置，則修改
sed -i "s#^net.ipv4.ip_forward.*#net.ipv4.ip_forward=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-ip6tables.*#net.bridge.bridge-nf-call-ip6tables=1#g"  /etc/sysctl.conf
sed -i "s#^net.bridge.bridge-nf-call-iptables.*#net.bridge.bridge-nf-call-iptables=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.disable_ipv6.*#net.ipv6.conf.all.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.default.disable_ipv6.*#net.ipv6.conf.default.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.lo.disable_ipv6.*#net.ipv6.conf.lo.disable_ipv6=1#g"  /etc/sysctl.conf
sed -i "s#^net.ipv6.conf.all.forwarding.*#net.ipv6.conf.all.forwarding=1#g"  /etc/sysctl.conf
# 可能沒有，追加
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" >> /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding = 1"  >> /etc/sysctl.conf
# 執(zhí)行命令以應用
sysctl -p

# 配置K8S的yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 卸載舊版本
yum remove -y kubelet kubeadm kubectl

# 安裝kubelet、kubeadm、kubectl
# 將 ${1} 替換為 kubernetes 版本號，例如 1.17.2
yum install -y kubelet-${K8S_VERSION} kubeadm-${K8S_VERSION} kubectl-${K8S_VERSION}

# 修改docker Cgroup Driver為systemd
# # 將/usr/lib/systemd/system/docker.service文件中的這一行 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
# # 修改為 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd
# 如果不修改，在添加 worker 節(jié)點時可能會碰到如下錯誤
# [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". 
# Please follow the guide at https://kubernetes.io/docs/setup/cri/
sed -i "s#^ExecStart=/usr/bin/dockerd.*#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd#g" /usr/lib/systemd/system/docker.service

# 設(shè)置 docker 鏡像，提高 docker 鏡像下載速度和穩(wěn)定性
# 如果您訪問 https://hub.docker.io 速度非常穩(wěn)定，亦可以跳過這個步驟
curl -sSL https://kuboard.cn/install-script/set_mirror.sh | sh -s ${REGISTRY_MIRROR}

# 重啟 docker，并啟動 kubelet
systemctl daemon-reload
systemctl restart docker
systemctl enable kubelet && systemctl start kubelet

docker version

如果報錯：
Transaction check error:
  file /usr/bin/kubectl from install of kubectl-1.18.9-0.x86_64 conflicts with file from package kubernetes-client-1.5.2-0.7.git269f928.el7.x86_64

則卸載已安裝組件，然后重新執(zhí)行安裝腳本：
[root@local ~]# yum remove -y  kubernetes-client-1.5.2-0.7.git269f928.el7.x86_64

初始化 master 節(jié)點

APISERVER_NAME 不能是 master 的 hostname
APISERVER_NAME 必須全為小寫字母、數(shù)字、小數(shù)點，不能包含減號
POD_SUBNET 所使用的網(wǎng)段不能與 master節(jié)點/worker節(jié)點 所在的網(wǎng)段重疊。該字段的取值為一個 CIDR 值，如果您對 CIDR 這個概念還不熟悉，請仍然執(zhí)行 export POD_SUBNET=10.100.0.1/16 命令，不做修改

只在 master 節(jié)點執(zhí)行

[root@localhost ~]# sh init_master.sh

以下是init_master.sh腳本：

#!/bin/bash

# 只在 master 節(jié)點執(zhí)行
# 替換 x.x.x.x 為 master 節(jié)點實際 IP（請使用內(nèi)網(wǎng) IP）
# export 命令只在當前 shell 會話中有效，開啟新的 shell 窗口后，如果要繼續(xù)安裝過程，請重新執(zhí)行此處的 export 命令
export MASTER_IP=10.100.5.116
# 替換 apiserver.demo 為 您想要的 dnsName
export APISERVER_NAME=apiserver.ops
# Kubernetes 容器組所在的網(wǎng)段，該網(wǎng)段安裝完成后，由 kubernetes 創(chuàng)建，事先并不存在于您的物理網(wǎng)絡(luò)中
export POD_SUBNET=10.244.0.0/16
export SERVICE_SUBNET=10.96.0.0/16
echo "${MASTER_IP}    ${APISERVER_NAME}" >> /etc/hosts

# K8S_VERSION：安裝kubelet、kubeadm、kubectl的版本號
# 安裝需求修改版本號
K8S_VERSION="1.18.9"

# 腳本出錯時終止執(zhí)行
set -e

if [ ${#POD_SUBNET} -eq 0 ] || [ ${#APISERVER_NAME} -eq 0 ]; then
  echo -e "\033[31;1m請確保您已經(jīng)設(shè)置了環(huán)境變量 POD_SUBNET 和 APISERVER_NAME \033[0m"
  echo 當前POD_SUBNET=$POD_SUBNET
  echo 當前APISERVER_NAME=$APISERVER_NAME
  exit 1
fi


# 查看完整配置選項 https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
rm -f ./kubeadm-config.yaml
cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v${K8S_VERSION}
imageRepository: registry.aliyuncs.com/k8sxio
controlPlaneEndpoint: "${APISERVER_NAME}:6443"
networking:
  serviceSubnet: "${SERVICE_SUBNET}"
  podSubnet: "${POD_SUBNET}"
  dnsDomain: "cluster.local"
EOF

# kubeadm init
# 根據(jù)您服務器網(wǎng)速的情況，您需要等候 3 - 10 分鐘
kubeadm init --config=kubeadm-config.yaml --upload-certs

# 配置 kubectl
rm -rf /ops/k8s/.kube/
mkdir -p /ops/k8s/.kube/
cp -i /etc/kubernetes/admin.conf /ops/k8s/.kube/config

# 安裝 calico 網(wǎng)絡(luò)插件
# 參考文檔 https://docs.projectcalico.org/v3.13/getting-started/kubernetes/self-managed-onprem/onpremises
# echo "安裝calico-3.13.1"
# rm -f calico-3.13.1.yaml
# wget https://kuboard.cn/install-script/calico/calico-3.13.1.yaml
# kubectl apply -f calico-3.13.1.yaml

如果是多網(wǎng)卡網(wǎng)絡(luò)檢查etc，apiserver監(jiān)聽網(wǎng)卡ip，替換

sed -i 's#10.0.3.15#192.168.56.3#g' /etc/kubernetes/manifests/*
#pods會自動重啟

安裝kube-flannel網(wǎng)絡(luò)

[root@localhost ~]# kubectl apply -f kube-flannel.yml

如果提示：
The connection to the server localhost:8080 was refused - did you specify the right host or port?

需要編輯文件設(shè)置
vim /etc/profile
在底部增加新的環(huán)境變量 export KUBECONFIG=/etc/kubernetes/admin.conf

以下是kube-flannel.yml腳本：

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - cp
        args:
        - -f
        - /etc/kube-flannel/cni-conf.json
        - /etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.14.0
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        #多網(wǎng)卡指定flannel網(wǎng)卡
        #- --iface=eth1
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run/flannel
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: /run/flannel
      - name: cni
        hostPath:
          path: /etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

檢查初始化結(jié)果

# 只在 master 節(jié)點執(zhí)行

# 執(zhí)行如下命令，等待 3-10 分鐘，直到所有的容器組處于 Running 狀態(tài)
[root@localhost ~]# watch kubectl get pod -n kube-system -o wide

# 查看 master 節(jié)點初始化結(jié)果
kubectl get nodes -o wide

[root@localhost ~]# kubectl get pod -n kube-system -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
coredns-66db54ff7f-kx5r8            1/1     Running   0          30m   10.244.0.3     gdc-ops-3   <none>           <none>
coredns-66db54ff7f-x952p            1/1     Running   0          30m   10.244.0.2     gdc-ops-3   <none>           <none>
etcd-gdc-ops-3                      1/1     Running   0          30m   10.100.5.116   gdc-ops-3   <none>           <none>
kube-apiserver-gdc-ops-3            1/1     Running   0          30m   10.100.5.116   gdc-ops-3   <none>           <none>
kube-controller-manager-gdc-ops-3   1/1     Running   0          30m   10.100.5.116   gdc-ops-3   <none>           <none>
kube-flannel-ds-4495k               1/1     Running   0          11m   10.100.5.116   gdc-ops-3   <none>           <none>
kube-proxy-hq52k                    1/1     Running   0          30m   10.100.5.116   gdc-ops-3   <none>           <none>
kube-scheduler-gdc-ops-3            1/1     Running   0          30m   10.100.5.116   gdc-ops-3   <none>           <none>

初始化 worker節(jié)點

只在 master 節(jié)點執(zhí)行

獲得 join命令參數(shù)
[root@localhost ~]# kubeadm token create --print-join-command
W0521 18:15:25.331475   31928 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join apiserver.ops:6443 --token 1v62wv.dpof3wnsplpq6v1d     --discovery-token-ca-cert-hash sha256:3a79f96da88718093be461623c5a1511f2b4389e7e05cff76cc0f5c4c8778a5c

該 token 的有效時間為 2 個小時，2小時內(nèi)，您可以使用此 token 初始化任意數(shù)量的 worker 節(jié)點。

針對所有的 worker 節(jié)點執(zhí)行

[root@localhost ~]# init_worker.sh

以下是init_worker.sh腳本：

#!/bin/bash
# 只在 worker 節(jié)點執(zhí)行
# 替換 x.x.x.x 為 master 節(jié)點的內(nèi)網(wǎng) IP
export MASTER_IP=192.168.2.15
# 替換 apiserver.demo 為初始化 master 節(jié)點時所使用的 APISERVER_NAME
export APISERVER_NAME=apiserver.ops
echo "${MASTER_IP}    ${APISERVER_NAME}" >> /etc/hosts

# 替換為 master 節(jié)點上 kubeadm token create 命令的輸出
kubeadm join apiserver.ops:6443 --token 1v62wv.dpof3wnsplpq6v1d     --discovery-token-ca-cert-hash sha256:3a79f96da88718093be461623c5a1511f2b4389e7e05cff76cc0f5c4c8778a5c
 

檢查初始化結(jié)果

# 只在 master 節(jié)點執(zhí)行

[root@localhost ~]# kubectl get nodes -o wide
NAME        STATUS   ROLES    AGE    VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
gdc-ops-1   Ready    <none>   102s   v1.18.9   10.100.5.126   <none>        CentOS Linux 7 (Core)   3.10.0-1062.18.1.el7.x86_64   docker://19.3.8
gdc-ops-2   Ready    <none>   119s   v1.18.9   10.100.5.161   <none>        CentOS Linux 7 (Core)   3.10.0-1062.18.1.el7.x86_64   docker://19.3.8
gdc-ops-3   Ready    master   69m    v1.18.9   10.100.5.116   <none>        CentOS Linux 7 (Core)   3.10.0-1127.19.1.el7.x86_64   docker://19.3.8

安裝dashboard

在master上執(zhí)行

[root@localhost ~]# kubectl apply -f recommended.yaml

以下是recommended.yaml腳本：

# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort # 新增
  ports:
    - name: https
      port: 443
      targetPort: 8443
      nodePort: 30001 # 新增
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin # kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.2.0
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            #增加http的healthcheck  
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
    spec:
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.6
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

檢查dashboard安裝結(jié)果

[root@localhost ~]# kubectl get pod,svc -n kubernetes-dashboard -o wide
NAME                                             READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
pod/dashboard-metrics-scraper-78f5d9f487-jsrpk   1/1     Running   0          15m   10.244.1.29   gdc-ops-2   <none>           <none>
pod/kubernetes-dashboard-67b4b54bc5-zll58        1/1     Running   0          15m   10.244.1.28   gdc-ops-2   <none>           <none>

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE   SELECTOR
service/dashboard-metrics-scraper   ClusterIP   10.96.146.51    <none>        8000/TCP                       15m   k8s-app=dashboard-metrics-scraper
service/kubernetes-dashboard        NodePort    10.96.238.225   <none>        443:30001/TCP,9090:30742/TCP   15m   k8s-app=kubernetes-dashboard

配置dashboard域名HTTPS證書

• 上傳https證書到k8s master節(jié)點
• 配置*.xxx.com.cn泛域名的SSL證書

[root@GDC-OPS-3 ~]# cd /OPS/secret
[root@GDC-OPS-3 secret]# kubectl create secret tls xxx-com-cn-ingress-secret --key xxx.com.cn.key --cert xxx.com.cn.crt --namespace kubernetes-dashboard

配置dashboard的ingress

kubectl apply -f ingress-nginx-dashboard_deploy.yaml

以下是ingress-nginx-dashboard.yaml

apiVersion: extensions/v1beta1      #api版本
kind: Ingress       #清單類型
metadata:           #元數(shù)據(jù)
  name: ingress-nginx    #ingress的名稱
  namespace: kubernetes-dashboard     #所屬名稱空間
  annotations:           #注解信息
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:      #規(guī)格
  tls:
  - hosts:
    - "*.xxx.com.cn"
    secretName: xxx-com-cn-ingress-secret
  rules:   #定義后端轉(zhuǎn)發(fā)的規(guī)則
  - host: ops-k8s.xxx.com.cn    #通過域名進行轉(zhuǎn)發(fā)
    http:
      paths:       
      - path:       #配置訪問路徑，如果通過url進行轉(zhuǎn)發(fā)，需要修改；空默認為訪問的路徑為"/"
        backend:    #配置后端服務
          serviceName: kubernetes-dashboard
          servicePort: 443

然后創(chuàng)建賬戶，獲取token

#創(chuàng)建賬戶
[root@localhost ~]# kubectl create serviceaccount dashboard-admin -n kubernetes-dashboard
#授權(quán)
[root@localhost ~]# kubectl create clusterrolebinding dashboard-admin-rb --clusterrole=cluster-admin --serviceaccount=kubernetes-dashboard:dashboard-admin
#獲取賬號token
[root@localhost ~]# kubectl get secrets -n kubernetes-dashboard |grep dashboard-admin
dashboard-admin-token-hq9qp        kubernetes.io/service-account-token   3      7m33s
[root@localhost ~]# kubectl describe secrets dashboard-admin-token-hq9qp -n kubernetes-dashboard
Name:         dashboard-admin-token-hq9qp
Namespace:    kubernetes-dashboard
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 2621a8ee-18a1-4001-abec-72d1de82dc11

Type:  kubernetes.io/service-account-token

Data
====
namespace:  20 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6ImVkNDFwVzdjamN2ZFhTYXhVb0pucF9zUnlfUUxPMm1QYzVLYWVTM2oxY0kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4taHE5cXAiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMjYyMWE4ZWUtMThhMS00MDAxLWFiZWMtNzJkMWRlODJkYzExIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.NMTNKTfzeeybQQp0Z76jlT4hSlxCZRocvIiEW-c-WOkRA8fMiOahbmgTM0MOaRdz0DjpgJu2j5ISBC9nCqgLabH3TbnX70Kkm1wQASlBntM33y7xiD9Gakq6nhoZ2UOfVl0gIveux2cThEwbsCfA7HDpgCpECd16nkoG14Kmku0USjj3six_GlyaAQyzcHYKmG_vw70jALUCwKd7J8YqFQNy2IjaJ5gQ2nCXwtICvLuBr_l8_xpNYLqTIIRX0R625JLuIUI_0jp_tihgCu1ul9HoPKhnvlh0mcC_Sf_gmdiE7cX5MQ2LScar0B3ysDeLbKFcq4aVH5yYoNBrKkVgsg
ca.crt:     1025 bytes

• 配置WEB頁面出口
  做負載均衡映射：公網(wǎng)ip:443 --> 10.100.5.116:443
  做負載均衡映射：公網(wǎng)ip:80 --> 10.100.5.116:80

• 配置域名解析
  ops-k8s.xxx.com.cn --> 公網(wǎng)ip

web登錄

瀏覽器訪問https://域名：端口
谷歌瀏覽器因?qū)o認證頁面的處理機制，是訪問不了的
所以這里使用火狐瀏覽器進行訪問：

dashboard地址：https://ops-k8s.xxx.com.cn/

出現(xiàn)輸入token登錄，輸入token

eyJhbGciOiJSUzI1NiIsImtpZCI6ImVkNDFwVzdjamN2ZFhTYXhVb0pucF9zUnlfUUxPMm1QYzVLYWVTM2oxY0kifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4taHE5cXAiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiMjYyMWE4ZWUtMThhMS00MDAxLWFiZWMtNzJkMWRlODJkYzExIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmVybmV0ZXMtZGFzaGJvYXJkOmRhc2hib2FyZC1hZG1pbiJ9.NMTNKTfzeeybQQp0Z76jlT4hSlxCZRocvIiEW-c-WOkRA8fMiOahbmgTM0MOaRdz0DjpgJu2j5ISBC9nCqgLabH3TbnX70Kkm1wQASlBntM33y7xiD9Gakq6nhoZ2UOfVl0gIveux2cThEwbsCfA7HDpgCpECd16nkoG14Kmku0USjj3six_GlyaAQyzcHYKmG_vw70jALUCwKd7J8YqFQNy2IjaJ5gQ2nCXwtICvLuBr_l8_xpNYLqTIIRX0R625JLuIUI_0jp_tihgCu1ul9HoPKhnvlh0mcC_Sf_gmdiE7cX5MQ2LScar0B3ysDeLbKFcq4aVH5yYoNBrKkVgsg

引入外部MySQL

創(chuàng)建mysql-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: mysql-cmdb
spec:
  ports:
    - port: 3306

創(chuàng)建mysql-endpoints.yaml

kind: Endpoints
apiVersion: v1
metadata:
  name: mysql-cmdb
  namespace: default
subsets:
  - addresses:
      - ip: 10.100.10.38
    ports:
      - port: 3306

master執(zhí)行mysql-service.yaml和mysql-endpoints.yaml

kubectl apply -f mysql-service.yaml
kubectl apply -f mysql-endpoints.yaml
kubectl describe svc mysql-cmdb -n default

配置Ingress

配置HTTPS證書

• 上傳https證書到k8s master節(jié)點
• 配置*.xxx.com.cn泛域名的SSL證書

[root@GDC-OPS-3 secret]# cd /OPS/secret
[root@GDC-OPS-3 secret]# kubectl create secret tls xxx-com-cn-ingress-secret --key xxx.com.cn.key --cert xxx.com.cn.crt

配置Ingress-controller

• 在k8s管理頁面，創(chuàng)建新資源，ingress-nginx_deploy.yaml文件。
• 或者 在master上執(zhí)行：

kubectl apply -f ingress-nginx_deploy.yaml

• 1、下載：https://github.com/kubernetes/ingress-nginx/archive/refs/tags/controller-v0.46.0.zip
• 2、解壓controller-v0.46.0.zip，在目錄ingress-nginx-controller-v0.46.0\deploy\static\provider\baremetal\deploy.yaml
• 3、將文件deploy.yaml修改為ingress-nginx_deploy.yaml
• 4、修改ingress-nginx_deploy.yaml配置：

1、修改image，注釋原有k8s.gcr.io的image，新增pollyduan的image（在docker hub可以找到ingress-nginx鏡像）: 
#image: k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
image: pollyduan/ingress-nginx-controller:v0.46.0
2、增加Service: ingress-nginx-controller的30080和30443端口

以下是ingress-nginx_deploy.yaml

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: default
automountServiceAccountToken: true
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: default
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: default
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader-nginx
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: default
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: default
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: default
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
      nodePort: 30080
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
      nodePort: 30443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: default
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          #image: k8s.gcr.io/ingress-nginx/controller:v0.46.0@sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
          image: pollyduan/ingress-nginx-controller:v0.46.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    matchPolicy: Equivalent
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1beta1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      service:
        namespace: default
        name: ingress-nginx-controller-admission
        path: /networking/v1beta1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: default
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: default
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: default
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: default
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: default
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-3.30.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.46.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: create
          image: docker.io/jettech/kube-webhook-certgen:v1.5.1
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
            - --namespace=$(POD_NAMESPACE)
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.30.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.46.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: default
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-3.30.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.46.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      containers:
        - name: patch
          image: docker.io/jettech/kube-webhook-certgen:v1.5.1
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=$(POD_NAMESPACE)
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000