視頻教程連接：kubernetes快速入門(mén)
寫(xiě)在前面
本章介紹kubernetes系列教程的ingress概念，在kubernetes中對(duì)外暴露服務(wù)的方式有兩種：service（NodePort或者外部LoadBalancer）和ingress，其中service是提供四層的負(fù)載均衡，通過(guò)iptables DNAT或lvs nat模式實(shí)現(xiàn)后端Pod的代理請(qǐng)求。如需實(shí)現(xiàn)http，域名，URI，證書(shū)等請(qǐng)求方式，service是無(wú)法實(shí)現(xiàn)的，需要借助于ingress來(lái)來(lái)實(shí)現(xiàn)，本文將來(lái)介紹ingress相關(guān)的內(nèi)容。
1. Ingress簡(jiǎn)介
An API object that manages external access to the services in a cluster, typically HTTP.
Ingress can provide load balancing, SSL termination and name-based virtual hosting.
引用官方關(guān)于ingress的介紹我們可以得知，ingress是一種通過(guò)http協(xié)議暴露kubernetes內(nèi)部服務(wù)的api對(duì)象，即充當(dāng)Edge Router邊界路由器的角色對(duì)外基于七層的負(fù)載均衡調(diào)度機(jī)制，能夠提供以下幾個(gè)功能：
- 負(fù)載均衡，將請(qǐng)求自動(dòng)負(fù)載均衡到后端的Pod上；
- SSL加密，客戶(hù)端到Ingress Controller為https加密，到后端Pod為明文的http；
- 基于名稱(chēng)的虛擬主機(jī)，提供基于域名或URI更靈活的路由方式
實(shí)現(xiàn)Ingress包含的組件有：
- Ingress，客戶(hù)端，負(fù)責(zé)定義ingress配置，將請(qǐng)求轉(zhuǎn)發(fā)給Ingress Controller；
- Ingress Controller，Ingress控制器，實(shí)現(xiàn)七層轉(zhuǎn)發(fā)的Edge Router，通過(guò)調(diào)用k8s的api動(dòng)態(tài)感知集群中Pod的變化而動(dòng)態(tài)更新配置文件并重載， Controller需要部署在k8s集群中以實(shí)現(xiàn)和集群中的pod通信，通常以DaemonSets或Deployments的形式部署，并對(duì)外暴露80和443端口，對(duì)于DaemonSets來(lái)說(shuō)，一般是以hostNetwork或者h(yuǎn)ostPort的形式暴露，Deployments則以NodePort的方式暴露，控制器的多個(gè)節(jié)點(diǎn)則借助外部負(fù)載均衡ExternalLB以實(shí)現(xiàn)統(tǒng)一接入；
- Ingress配置規(guī)則，Controller控制器通過(guò)service服務(wù)發(fā)現(xiàn)機(jī)制動(dòng)態(tài)實(shí)現(xiàn)后端Pod路由轉(zhuǎn)發(fā)規(guī)則的實(shí)現(xiàn)；
- Service，kuberntes中四層的負(fù)載均衡調(diào)度機(jī)制，Ingress借助service的服務(wù)發(fā)現(xiàn)機(jī)制實(shí)現(xiàn)集群中Pod資源的動(dòng)態(tài)感知；
- Pod，后端實(shí)際負(fù)責(zé)響應(yīng)請(qǐng)求容器，由控制器如Deployment創(chuàng)建，通過(guò)標(biāo)簽Labels和service關(guān)聯(lián)，服務(wù)發(fā)現(xiàn)。
簡(jiǎn)而言之，ingress控制器借助service的服務(wù)發(fā)現(xiàn)機(jī)制實(shí)現(xiàn)配置的動(dòng)態(tài)更新以實(shí)現(xiàn)Pod的負(fù)載均衡機(jī)制實(shí)現(xiàn)，由于涉及到Ingress Controller的動(dòng)態(tài)更新，目前社區(qū)Ingress Controller大體包含兩種類(lèi)型的控制器：
- 傳統(tǒng)的七層負(fù)載均衡如Nginx，HAproxy，開(kāi)發(fā)了適應(yīng)微服務(wù)應(yīng)用的插件，具有成熟，高性能等優(yōu)點(diǎn)；
- 新型微服務(wù)負(fù)載均衡如Traefik，Envoy，Istio，專(zhuān)門(mén)適用于微服務(wù)+容器化應(yīng)用場(chǎng)景，具有動(dòng)態(tài)更新特點(diǎn)；
| 類(lèi)型 | 常見(jiàn)類(lèi)型 | 優(yōu)點(diǎn) | 缺點(diǎn) |
|---|---|---|---|
| 傳統(tǒng)負(fù)載均衡 | nginx，haproxy | 成熟，穩(wěn)定，高性能 | 動(dòng)態(tài)更新需reload配置文件 |
| 微服務(wù)負(fù)載均衡 | Traefik，Envoy，Istio | 天生為微服務(wù)而生，動(dòng)態(tài)更新 | 性能還有待提升 |
2. Nginx Ingress
2.1 Nginx ingress介紹
By default, pods of Kubernetes services are not accessible from the external network, but only by other pods within the Kubernetes cluster. Kubernetes has a built?in configuration for HTTP load balancing, called Ingress, that defines rules for external connectivity to Kubernetes services. Users who need to provide external access to their Kubernetes services create an Ingress resource that defines rules, including the URI path, backing service name, and other information. The Ingress controller can then automatically program a frontend load balancer to enable Ingress configuration. The NGINX Ingress Controller for Kubernetes is what enables Kubernetes to configure NGINX and NGINX Plus for load balancing Kubernetes services.
Nginx Ingress Controller是實(shí)現(xiàn)ingress的具體實(shí)現(xiàn)，包含有兩個(gè)版本：Ngnix OSS和Nginx Plus版，后者是商業(yè)化增強(qiáng)版，支持更多的功能，詳情參考官方文檔介紹https://www.nginx.com/products/nginx/kubernetes-ingress-controller#compare-versions
2.2 Nginx ingress安裝
首先需要安裝Nginx Ingress Controller控制器，控制器安裝方式包含兩種：DaemonSets和Deployments。
- DaemonSets通過(guò)hostPort的方式暴露80和443端口，可通過(guò)Node的調(diào)度由專(zhuān)門(mén)的節(jié)點(diǎn)實(shí)現(xiàn)部署
- Deployments則通過(guò)NodePort的方式實(shí)現(xiàn)控制器端口的暴露，借助外部負(fù)載均衡實(shí)現(xiàn)高可用負(fù)載均衡
除此之外，還需要部署Namespace，ServiceAccount，RBAC，Secrets，Custom Resource Definitions等資源，如下開(kāi)始部署。
2.2.1 基礎(chǔ)依賴(lài)環(huán)境準(zhǔn)備
1、github中下載源碼包,安裝部署文件在kubernetes-ingress/deployments/目錄下
[root@node-1 ~]# git clone https://github.com/nginxinc/kubernetes-ingress.git
[root@node-1 ~]# tree kubernetes-ingress/deployments/
kubernetes-ingress/deployments/
├── common
│ ├── custom-resource-definitions.yaml 自定義資源
│ ├── default-server-secret.yaml Secrets
│ ├── nginx-config.yaml
│ └── ns-and-sa.yaml Namspace+ServiceAccount
├── daemon-set
│ ├── nginx-ingress.yaml DaemonSets控制器
│ └── nginx-plus-ingress.yaml
├── deployment
│ ├── nginx-ingress.yaml Deployments控制器
│ └── nginx-plus-ingress.yaml
├── helm-chart Helm安裝包
│ ├── chart-icon.png
│ ├── Chart.yaml
│ ├── README.md
│ ├── templates
│ │ ├── controller-configmap.yaml
│ │ ├── controller-custom-resources.yaml
│ │ ├── controller-daemonset.yaml
│ │ ├── controller-deployment.yaml
│ │ ├── controller-leader-election-configmap.yaml
│ │ ├── controller-secret.yaml
│ │ ├── controller-serviceaccount.yaml
│ │ ├── controller-service.yaml
│ │ ├── controller-wildcard-secret.yaml
│ │ ├── _helpers.tpl
│ │ ├── NOTES.txt
│ │ └── rbac.yaml
│ ├── values-icp.yaml
│ ├── values-plus.yaml
│ └── values.yaml
├── rbac RBAC認(rèn)證授權(quán)
│ └── rbac.yaml
├── README.md
└── service Service定義
├── loadbalancer-aws-elb.yaml
├── loadbalancer.yaml DaemonSets暴露服務(wù)方式
└── nodeport.yaml Deployments暴露服務(wù)方式
2、創(chuàng)建Namespace和ServiceAccount, kubectl apply -f common/default-server-secret.yaml
apiVersion: v1
kind: Namespace
metadata:
name: nginx-ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress
namespace: nginx-ingress
3、創(chuàng)建Secrets自簽名證書(shū)，kubectl apply -f common/default-server-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: default-server-secret
namespace: nginx-ingress
type: Opaque
data:
tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN2akNDQWFZQ0NRREFPRjl0THNhWFhEQU5CZ2txaGtpRzl3MEJBUXNGQURBaE1SOHdIUVlEVlFRRERCWk8KUjBsT1dFbHVaM0psYzNORGIyNTBjbTlzYkdWeU1CNFhEVEU0TURreE1qRTRNRE16TlZvWERUSXpNRGt4TVRFNApNRE16TlZvd0lURWZNQjBHQTFVRUF3d1dUa2RKVGxoSmJtZHlaWE56UTI5dWRISnZiR3hsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUwvN2hIUEtFWGRMdjNyaUM3QlBrMTNpWkt5eTlyQ08KR2xZUXYyK2EzUDF0azIrS3YwVGF5aGRCbDRrcnNUcTZzZm8vWUk1Y2Vhbkw4WGM3U1pyQkVRYm9EN2REbWs1Qgo4eDZLS2xHWU5IWlg0Rm5UZ0VPaStlM2ptTFFxRlBSY1kzVnNPazFFeUZBL0JnWlJVbkNHZUtGeERSN0tQdGhyCmtqSXVuektURXUyaDU4Tlp0S21ScUJHdDEwcTNRYzhZT3ExM2FnbmovUWRjc0ZYYTJnMjB1K1lYZDdoZ3krZksKWk4vVUkxQUQ0YzZyM1lma1ZWUmVHd1lxQVp1WXN2V0RKbW1GNWRwdEMzN011cDBPRUxVTExSakZJOTZXNXIwSAo1TmdPc25NWFJNV1hYVlpiNWRxT3R0SmRtS3FhZ25TZ1JQQVpQN2MwQjFQU2FqYzZjNGZRVXpNQ0F3RUFBVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQWpLb2tRdGRPcEsrTzhibWVPc3lySmdJSXJycVFVY2ZOUitjb0hZVUoKdGhrYnhITFMzR3VBTWI5dm15VExPY2xxeC9aYzJPblEwMEJCLzlTb0swcitFZ1U2UlVrRWtWcitTTFA3NTdUWgozZWI4dmdPdEduMS9ienM3bzNBaS9kclkrcUI5Q2k1S3lPc3FHTG1US2xFaUtOYkcyR1ZyTWxjS0ZYQU80YTY3Cklnc1hzYktNbTQwV1U3cG9mcGltU1ZmaXFSdkV5YmN3N0NYODF6cFErUyt1eHRYK2VBZ3V0NHh3VlI5d2IyVXYKelhuZk9HbWhWNThDd1dIQnNKa0kxNXhaa2VUWXdSN0diaEFMSkZUUkk3dkhvQXprTWIzbjAxQjQyWjNrN3RXNQpJUDFmTlpIOFUvOWxiUHNoT21FRFZkdjF5ZytVRVJxbStGSis2R0oxeFJGcGZnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdi91RWM4b1JkMHUvZXVJTHNFK1RYZUprckxMMnNJNGFWaEMvYjVyYy9XMlRiNHEvClJOcktGMEdYaVN1eE9ycXgrajlnamx4NXFjdnhkenRKbXNFUkJ1Z1B0ME9hVGtIekhvb3FVWmcwZGxmZ1dkT0EKUTZMNTdlT1l0Q29VOUZ4amRXdzZUVVRJVUQ4R0JsRlNjSVo0b1hFTkhzbysyR3VTTWk2Zk1wTVM3YUhudzFtMApxWkdvRWEzWFNyZEJ6eGc2clhkcUNlUDlCMXl3VmRyYURiUzc1aGQzdUdETDU4cGszOVFqVUFQaHpxdmRoK1JWClZGNGJCaW9CbTVpeTlZTW1hWVhsMm0wTGZzeTZuUTRRdFFzdEdNVWozcGJtdlFmazJBNnljeGRFeFpkZFZsdmwKMm82MjBsMllxcHFDZEtCRThCay90elFIVTlKcU56cHpoOUJUTXdJREFRQUJBb0lCQVFDZklHbXowOHhRVmorNwpLZnZJUXQwQ0YzR2MxNld6eDhVNml4MHg4Mm15d1kxUUNlL3BzWE9LZlRxT1h1SENyUlp5TnUvZ2IvUUQ4bUFOCmxOMjRZTWl0TWRJODg5TEZoTkp3QU5OODJDeTczckM5bzVvUDlkazAvYzRIbjAzSkVYNzZ5QjgzQm9rR1FvYksKMjhMNk0rdHUzUmFqNjd6Vmc2d2szaEhrU0pXSzBwV1YrSjdrUkRWYmhDYUZhNk5nMUZNRWxhTlozVDhhUUtyQgpDUDNDeEFTdjYxWTk5TEI4KzNXWVFIK3NYaTVGM01pYVNBZ1BkQUk3WEh1dXFET1lvMU5PL0JoSGt1aVg2QnRtCnorNTZud2pZMy8yUytSRmNBc3JMTnIwMDJZZi9oY0IraVlDNzVWYmcydVd6WTY3TWdOTGQ5VW9RU3BDRkYrVm4KM0cyUnhybnhBb0dCQU40U3M0ZVlPU2huMVpQQjdhTUZsY0k2RHR2S2ErTGZTTXFyY2pOZjJlSEpZNnhubmxKdgpGenpGL2RiVWVTbWxSekR0WkdlcXZXaHFISy9iTjIyeWJhOU1WMDlRQ0JFTk5jNmtWajJTVHpUWkJVbEx4QzYrCk93Z0wyZHhKendWelU0VC84ajdHalRUN05BZVpFS2FvRHFyRG5BYWkyaW5oZU1JVWZHRXFGKzJyQW9HQkFOMVAKK0tZL0lsS3RWRzRKSklQNzBjUis3RmpyeXJpY05iWCtQVzUvOXFHaWxnY2grZ3l4b25BWlBpd2NpeDN3QVpGdwpaZC96ZFB2aTBkWEppc1BSZjRMazg5b2pCUmpiRmRmc2l5UmJYbyt3TFU4NUhRU2NGMnN5aUFPaTVBRHdVU0FkCm45YWFweUNweEFkREtERHdObit3ZFhtaTZ0OHRpSFRkK3RoVDhkaVpBb0dCQUt6Wis1bG9OOTBtYlF4VVh5YUwKMjFSUm9tMGJjcndsTmVCaWNFSmlzaEhYa2xpSVVxZ3hSZklNM2hhUVRUcklKZENFaHFsV01aV0xPb2I2NTNyZgo3aFlMSXM1ZUtka3o0aFRVdnpldm9TMHVXcm9CV2xOVHlGanIrSWhKZnZUc0hpOGdsU3FkbXgySkJhZUFVWUNXCndNdlQ4NmNLclNyNkQrZG8wS05FZzFsL0FvR0FlMkFVdHVFbFNqLzBmRzgrV3hHc1RFV1JqclRNUzRSUjhRWXQKeXdjdFA4aDZxTGxKUTRCWGxQU05rMXZLTmtOUkxIb2pZT2pCQTViYjhibXNVU1BlV09NNENoaFJ4QnlHbmR2eAphYkJDRkFwY0IvbEg4d1R0alVZYlN5T294ZGt5OEp0ek90ajJhS0FiZHd6NlArWDZDODhjZmxYVFo5MWpYL3RMCjF3TmRKS2tDZ1lCbyt0UzB5TzJ2SWFmK2UwSkN5TGhzVDQ5cTN3Zis2QWVqWGx2WDJ1VnRYejN5QTZnbXo5aCsKcDNlK2JMRUxwb3B0WFhNdUFRR0xhUkcrYlNNcjR5dERYbE5ZSndUeThXczNKY3dlSTdqZVp2b0ZpbmNvVlVIMwphdmxoTUVCRGYxSjltSDB5cDBwWUNaS2ROdHNvZEZtQktzVEtQMjJhTmtsVVhCS3gyZzR6cFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
4、創(chuàng)建ConfigMap自定義配置文件，kubectl apply -f common/nginx-config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:
5、為虛擬云主機(jī)和虛擬云主機(jī)路由定義自定義資源，支持自定義虛擬主機(jī)和虛擬路由，kubectl apply -f common/custom-resource-definitions.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualservers.k8s.nginx.org
spec:
group: k8s.nginx.org
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: virtualservers
singular: virtualserver
kind: VirtualServer
shortNames:
- vs
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: virtualserverroutes.k8s.nginx.org
spec:
group: k8s.nginx.org
versions:
- name: v1
served: true
storage: true
scope: Namespaced
names:
plural: virtualserverroutes
singular: virtualserverroute
kind: VirtualServerRoute
shortNames:
- vsr
6、配置RBAC認(rèn)證授權(quán)，實(shí)現(xiàn)ingress控制器訪問(wèn)集群中的其他資源，kubectl apply -f rbac/rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nginx-ingress
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- list
- watch
- update
- create
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- list
- watch
- get
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- k8s.nginx.org
resources:
- virtualservers
- virtualserverroutes
verbs:
- list
- watch
- get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: nginx-ingress
subjects:
- kind: ServiceAccount
name: nginx-ingress
namespace: nginx-ingress
roleRef:
kind: ClusterRole
name: nginx-ingress
apiGroup: rbac.authorization.k8s.io
2.2.2 部署Ingress控制器
1、 部署控制器，控制器可以DaemonSets和Deployment的形式部署，如下是DaemonSets的配置文件
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: Always
name: nginx-ingress
ports:
- name: http
containerPort: 80
hostPort: 80 #通過(guò)hostPort的方式暴露端口
- name: https
containerPort: 443
hostPort: 443
#- name: prometheus
#containerPort: 9113
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-leader-election
#- -enable-prometheus-metrics
Deployments的配置文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress
namespace: nginx-ingress
spec:
replicas: 1 #副本的個(gè)數(shù)
selector:
matchLabels:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
#annotations:
#prometheus.io/scrape: "true"
#prometheus.io/port: "9113"
spec:
serviceAccountName: nginx-ingress
containers:
- image: nginx/nginx-ingress:edge
imagePullPolicy: Always
name: nginx-ingress
ports: #內(nèi)部暴露的服務(wù)端口，需要通過(guò)NodePort的方式暴露給外部
- name: http
containerPort: 80
- name: https
containerPort: 443
#- name: prometheus
#containerPort: 9113
securityContext:
allowPrivilegeEscalation: true
runAsUser: 101 #nginx
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-leader-election
#- -enable-prometheus-metrics
2、我們以DaemonSets的方式部署，DaemonSet部署集群中各個(gè)節(jié)點(diǎn)都是對(duì)等，如果有外部LoadBalancer則通過(guò)外部負(fù)載均衡路由至Ingress中
[root@node-1 deployments]# kubectl apply -f daemon-set/nginx-ingress.yaml
daemonset.apps/nginx-ingress created
[root@node-1 deployments]# kubectl get daemonsets -n nginx-ingress
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
nginx-ingress 3 3 3 3 3 <none> 15s
[root@node-1 ~]# kubectl get pods -n nginx-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-ingress-7mpfc 1/1 Running 0 2m44s 10.244.0.50 node-1 <none> <none>
nginx-ingress-l2rtj 1/1 Running 0 2m44s 10.244.1.144 node-2 <none> <none>
nginx-ingress-tgf6r 1/1 Running 0 2m44s 10.244.2.160 node-3 <none> <none>
3、校驗(yàn)Nginx Ingress安裝情況，此時(shí)三個(gè)節(jié)點(diǎn)均是對(duì)等，即訪問(wèn)任意一個(gè)節(jié)點(diǎn)均能實(shí)現(xiàn)相同的效果，統(tǒng)一入口則通過(guò)外部負(fù)載均衡，如果在云環(huán)境下執(zhí)行kubectl apply -f service/loadbalancer.yaml創(chuàng)建外部負(fù)載均衡實(shí)現(xiàn)入口調(diào)度，自建的可以通過(guò)lvs或nginx等負(fù)載均衡實(shí)現(xiàn)接入，本文不再贅述，讀者可以自行研究。
備注說(shuō)明：如果以Deployments的方式部署，則需要執(zhí)行service/nodeport.yaml創(chuàng)建NodePort類(lèi)型的Service，實(shí)現(xiàn)的效果和DaemonSets類(lèi)似。
3. Ingress資源定義
上面的章節(jié)已安裝了一個(gè)Nginx Ingress Controller控制器，有了Ingress控制器后，我們就可以定義Ingress資源來(lái)實(shí)現(xiàn)七層負(fù)載轉(zhuǎn)發(fā)了，大體上Ingress支持三種使用方式：1. 基于虛擬主機(jī)轉(zhuǎn)發(fā)，2. 基于虛擬機(jī)主機(jī)URI轉(zhuǎn)發(fā)，3. 支持TLS加密轉(zhuǎn)發(fā)。
3.1 Ingress定義
1、環(huán)境準(zhǔn)備，先創(chuàng)建一個(gè)nginx的Deployment應(yīng)用，包含2個(gè)副本
[root@node-1 ~]# kubectl run ingress-demo --image=nginx:1.7.9 --port=80 --replicas=2
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-demo 2/2 2 2 116s
2、以service方式暴露服務(wù)端口
[root@node-1 ~]# kubectl expose deployment ingress-demo --port=80 --protocol=TCP --target-port=80
service/ingress-demo exposed
[root@node-1 ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-demo ClusterIP 10.109.33.91 <none> 80/TCP 2m15s
3、上述兩個(gè)步驟已創(chuàng)建了一個(gè)service，如下我們定義一個(gè)ingress對(duì)象將起轉(zhuǎn)發(fā)至ingress-demo這個(gè)service，通過(guò)ingress.class指定控制器的類(lèi)型為nginx
[root@node-1 nginx-ingress]# cat nginx-ingress-demo.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
rules:
- host: www.happylau.cn
http:
paths:
- path: /
backend:
serviceName: ingress-demo
servicePort: 80
4、創(chuàng)建ingress對(duì)象
[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-demo.yaml
ingress.extensions/nginx-ingress-demo created
查看ingress資源列表
[root@node-1 nginx-ingress]# kubectl get ingresses
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-demo www.happylau.cn 80 4m4s
5、查看ingress詳情，可以在Rules規(guī)則中看到后端Pod的列表，自動(dòng)發(fā)現(xiàn)和關(guān)聯(lián)相關(guān)Pod
[root@node-1 ~]# kubectl describe ingresses nginx-ingress-demo
Name: nginx-ingress-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
www.happylau.cn
/ ingress-demo:80 (10.244.1.146:80,10.244.2.162:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-demo","namespace":"default"},"spec":{"rules":[{"host":"www.happylaulab.cn","http":{"paths":[{"backend":{"serviceName":"ingress-demo","servicePort":80},"path":"/"}]}}]}}
kubernets.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 9m7s nginx-ingress-controller Configuration for default/nginx-ingress-demo was added or updated
Normal AddedOrUpdated 9m7s nginx-ingress-controller Configuration for default/nginx-ingress-demo was added or updated
Normal AddedOrUpdated 9m7s nginx-ingress-controller Configuration for default/nginx-ingress-demo was added or updated
6、測(cè)試驗(yàn)證，ingress規(guī)則的配置信息已注入到Ingress Controller中，環(huán)境中Ingress Controller是以DaemonSets的方式部署在集群中，如果有外部的負(fù)載均衡，則將www.happylau.cn域名的地址解析為負(fù)載均衡VIP。由于測(cè)試環(huán)境沒(méi)有搭建負(fù)載均衡，將hosts解析執(zhí)行node-1，node-2或者node-3任意一個(gè)IP都能實(shí)現(xiàn)相同的功能。
上述測(cè)試解析正常，當(dāng)然也可以解析為node-1和node-2的IP，如下：
[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.101
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 24 Dec 2019 10:32:22 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
ETag: "54999765-264"
Accept-Ranges: bytes
[root@node-1 ~]# curl -I http://www.happylau.cn --resolve www.happylau.cn:80:10.254.100.102
HTTP/1.1 200 OK
Server: nginx/1.17.6
Date: Tue, 24 Dec 2019 10:32:24 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Last-Modified: Tue, 23 Dec 2014 16:25:09 GMT
ETag: "54999765-264"
Accept-Ranges: bytes
3.2 Ingress動(dòng)態(tài)配置
上面的章節(jié)介紹了ingress資源對(duì)象的申明配置，在這個(gè)章節(jié)中我們探究一下Nginx Ingress Controller的實(shí)現(xiàn)機(jī)制和動(dòng)態(tài)配置更新機(jī)制，以方便了解Ingress控制器的工作機(jī)制。
1、 查看Nginx Controller控制器的配置文件，在nginx-ingress pod中存儲(chǔ)著ingress的配置文件
[root@node-1 ~]# kubectl get pods -n nginx-ingress
NAME READY STATUS RESTARTS AGE
nginx-ingress-7mpfc 1/1 Running 0 6h15m
nginx-ingress-l2rtj 1/1 Running 0 6h15m
nginx-ingress-tgf6r 1/1 Running 0 6h15m
#查看配置文件，每個(gè)ingress生成一個(gè)配置文件，文件名為：命名空間-ingres名稱(chēng).conf
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- ls -l /etc/nginx/conf.d
total 4
-rw-r--r-- 1 nginx nginx 1005 Dec 24 10:06 default-nginx-ingress-demo.conf
#查看配置文件
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress -- cat /etc/nginx/conf.d/default-nginx-ingress-demo.conf
# configuration for default/nginx-ingress-demo
#upstream的配置，會(huì)用least_conn算法，通過(guò)service服務(wù)發(fā)現(xiàn)機(jī)制動(dòng)態(tài)識(shí)別到后端的Pod
upstream default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 {
zone default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80 256k;
random two least_conn;
server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name www.happylau.cn;
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-demo-www.happylau.cn-ingress-demo-80; #調(diào)用upstream實(shí)現(xiàn)代理
}
}
通過(guò)上述查看配置文件可得知，Nginx Ingress Controller實(shí)際是根據(jù)ingress規(guī)則生成對(duì)應(yīng)的nginx配置文件，以實(shí)現(xiàn)代理轉(zhuǎn)發(fā)的功能，加入Deployments的副本數(shù)變更后nginx的配置文件會(huì)發(fā)生什么改變呢？
2、更新控制器的副本數(shù)，由2個(gè)Pod副本擴(kuò)容至3個(gè)
[root@node-1 ~]# kubectl scale --replicas=3 deployment ingress-demo
deployment.extensions/ingress-demo scaled
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-demo 3/3 3 3 123m
3、再次查看nginx的配置文件，ingress借助于service的服務(wù)發(fā)現(xiàn)機(jī)制，將加入的Pod自動(dòng)加入到nginx upstream中
4、查看nginx pod的日志（kubectl logs nginx-ingress-7mpfc -n nginx-ingress），有reload優(yōu)雅重啟的記錄，即通過(guò)更新配置文件+reload實(shí)現(xiàn)配置動(dòng)態(tài)更新。
通過(guò)上述的配置可知，ingress調(diào)用kubernetes api去感知kubernetes集群中的變化情況，Pod的增加或減少這些變化，然后動(dòng)態(tài)更新nginx ingress controller的配置文件，并重新載入配置。當(dāng)集群規(guī)模越大時(shí)，會(huì)頻繁涉及到配置文件的變動(dòng)和重載，因此nginx這方面會(huì)存在先天的劣勢(shì)，專(zhuān)門(mén)為微服務(wù)負(fù)載均衡應(yīng)運(yùn)而生，如Traefik，Envoy，Istio，這些負(fù)載均衡工具能夠提供大規(guī)模，頻繁動(dòng)態(tài)更新的場(chǎng)景，但性能相比Nginx，HAproxy還存在一定的劣勢(shì)。往后的章節(jié)中，我們?cè)賹?duì)其他的Ingress控制器做介紹。
3.3 Ingress路徑轉(zhuǎn)發(fā)
Ingress支持URI格式的轉(zhuǎn)發(fā)方式，同時(shí)支持URL重寫(xiě)，如下以?xún)蓚€(gè)service為例演示，service-1安裝nginx，service-2安裝httpd，分別用http://demo.happylau.cn/news和http://demo.happylau.cn/sports轉(zhuǎn)發(fā)到兩個(gè)不同的service
1、環(huán)境準(zhǔn)備，創(chuàng)建兩個(gè)應(yīng)用并實(shí)現(xiàn)service暴露，創(chuàng)建deployments時(shí)指定--explose創(chuàng)建service
[root@node-1 ~]# kubectl run service-1 --image=nginx:1.7.9 --port=80 --replicas=1 --expose=true
service/service-1 created
deployment.apps/service-1 created
[root@node-1 ~]# kubectl run service-2 --image=httpd --port=80 --replicas=1 --expose=true
service/service-2 created
deployment.apps/service-2 created
查看deployment狀態(tài)
[root@node-1 ~]# kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
ingress-demo 4/4 4 4 4h36m
service-1 1/1 1 1 65s
service-2 1/1 1 1 52s
查看service狀態(tài)，服務(wù)已經(jīng)正常
[root@node-1 ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-demo ClusterIP 10.109.33.91 <none> 80/TCP 4h36m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 101d
service-1 ClusterIP 10.106.245.71 <none> 80/TCP 68s
service-2 ClusterIP 10.104.204.158 <none> 80/TCP 55s
2、創(chuàng)建ingress對(duì)象，通過(guò)一個(gè)域名將請(qǐng)求轉(zhuǎn)發(fā)至后端兩個(gè)service
[root@node-1 nginx-ingress]# cat nginx-ingress-uri-demo.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-uri-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: demo.happylau.cn
http:
paths:
- path: /news
backend:
serviceName: service-1
servicePort: 80
- path: /sports
backend:
serviceName: service-2
servicePort: 80
3、創(chuàng)建ingress規(guī)則，查看詳情
[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-uri-demo.yaml
ingress.extensions/nginx-ingress-uri-demo created
#查看詳情
[root@node-1 nginx-ingress]# kubectl get ingresses.
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-demo www.happylau.cn 80 4h35m
nginx-ingress-uri-demo demo.happylau.cn 80 4s
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-uri-demo
Name: nginx-ingress-uri-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules: #對(duì)應(yīng)的轉(zhuǎn)發(fā)url規(guī)則
Host Path Backends
---- ---- --------
demo.happylau.cn
/news service-1:80 (10.244.2.163:80)
/sports service-2:80 (10.244.1.148:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/rewrite-target":"/"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-uri-demo","namespace":"default"},"spec":{"rules":[{"host":"demo.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/news"},{"backend":{"serviceName":"service-2","servicePort":80},"path":"/sports"}]}}]}}
kubernets.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 11s nginx-ingress-controller Configuration for default/nginx-ingress-uri-demo was added or updated
Normal AddedOrUpdated 11s nginx-ingress-controller Configuration for default/nginx-ingress-uri-demo was added or updated
Normal AddedOrUpdated 11s nginx-ingress-controller Configuration for default/nginx-ingress-uri-demo was added or updated
4、準(zhǔn)備測(cè)試，站點(diǎn)中創(chuàng)建對(duì)應(yīng)的路徑
[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
root@service-1-7b66bf758f-xj9jh:/# echo "service-1 website page" >/usr/share/nginx/html/news
[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash
root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "service-2 website page" >/usr/local/apache2/htdocs/sports
5、測(cè)試驗(yàn)證
[root@node-1 ~]# curl http://demo.happylau.cn/news --resolve demo.happylau.cn:80:10.254.100.101
service-1 website page
[root@node-1 ~]# curl http://demo.happylau.cn/sports --resolve demo.happylau.cn:80:10.254.100.101
service-2 website page
6、通過(guò)上述的驗(yàn)證測(cè)試可以得知，ingress支持URI的路由方式轉(zhuǎn)發(fā)，其對(duì)應(yīng)在ingress中的配置文件內(nèi)容是怎樣的呢，我們看下ingress controller生成對(duì)應(yīng)的nginx配置文件內(nèi)容，實(shí)際是通過(guò)ingress的location來(lái)實(shí)現(xiàn)，將不同的localtion轉(zhuǎn)發(fā)至不同的upstream以實(shí)現(xiàn)service的關(guān)聯(lián)，配置文件如下：
[root@node-1 ~]# kubectl exec -it nginx-ingress-7mpfc -n nginx-ingress /bin/bash
nginx@nginx-ingress-7mpfc:/$ cat /etc/nginx/conf.d/default-nginx-ingress-uri-demo.conf |grep -v "^$"
# configuration for default/nginx-ingress-uri-demo
#定義兩個(gè)upstream和后端的service關(guān)聯(lián)
upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 {
zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80 256k;
random two least_conn;
server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 {
zone default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80 256k;
random two least_conn;
server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name demo.happylau.cn;
#定義location實(shí)現(xiàn)代理，通過(guò)proxy_pass和后端的service關(guān)聯(lián)
location /news {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-1-80;
}
location /sports {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-uri-demo-demo.happylau.cn-service-2-80;
}
}
3.4 Ingress虛擬主機(jī)
ingress支持基于名稱(chēng)的虛擬主機(jī)，實(shí)現(xiàn)單個(gè)IP多個(gè)域名轉(zhuǎn)發(fā)的需求，通過(guò)請(qǐng)求頭部攜帶主機(jī)名方式區(qū)分開(kāi)，將上個(gè)章節(jié)的ingress刪除，使用service-1和service-2兩個(gè)service來(lái)做演示。
1、創(chuàng)建ingress規(guī)則，通過(guò)主機(jī)名實(shí)現(xiàn)轉(zhuǎn)發(fā)規(guī)則
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-virtualname-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
rules:
- host: news.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-1
servicePort: 80
- host: sports.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-2
servicePort: 80
2、生成ingress規(guī)則并查看詳情，一個(gè)ingress對(duì)應(yīng)兩個(gè)HOSTS
[root@node-1 nginx-ingress]# kubectl apply -f nginx-ingress-virtualname.yaml
ingress.extensions/nginx-ingress-virtualname-demo created
#查看列表
[root@node-1 nginx-ingress]# kubectl get ingresses nginx-ingress-virtualname-demo
NAME HOSTS ADDRESS PORTS AGE
nginx-ingress-virtualname-demo news.happylau.cn,sports.happylau.cn 80 12s
#查看詳情
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-virtualname-demo
Name: nginx-ingress-virtualname-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
news.happylau.cn
/ service-1:80 (10.244.2.163:80)
sports.happylau.cn
/ service-2:80 (10.244.1.148:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-virtualname-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}]}}
kubernets.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 28s nginx-ingress-controller Configuration for default/nginx-ingress-virtualname-demo was added or updated
Normal AddedOrUpdated 28s nginx-ingress-controller Configuration for default/nginx-ingress-virtualname-demo was added or updated
Normal AddedOrUpdated 28s nginx-ingress-controller Configuration for default/nginx-ingress-virtualname-demo was added or updated
3、準(zhǔn)備測(cè)試數(shù)據(jù)并測(cè)試
[root@node-1 ~]# kubectl exec -it service-1-7b66bf758f-xj9jh /bin/bash
root@service-1-7b66bf758f-xj9jh:/# echo "news demo" >/usr/share/nginx/html/index.html
[root@node-1 ~]# kubectl exec -it service-2-7c7444684d-w9cv9 /bin/bash
root@service-2-7c7444684d-w9cv9:/usr/local/apache2# echo "sports demo" >/usr/local/apache2/htdocs/index.html
測(cè)試：
[root@node-1 ~]# curl http://news.happylau.cn --resolve news.happylau.cn:80:10.254.100.102
news demo
[root@node-1 ~]# curl http://sports.happylau.cn --resolve sports.happylau.cn:80:10.254.100.102
sports demo
4、查看nginx的配置文件內(nèi)容，通過(guò)在server中定義不同的server_name以區(qū)分，代理到不同的upstream以實(shí)現(xiàn)service的代理。
# configuration for default/nginx-ingress-virtualname-demo
upstream default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 {
zone default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80 256k;
random two least_conn;
server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 {
zone default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80 256k;
random two least_conn;
server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
server_tokens on;
server_name news.happylau.cn;
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-virtualname-demo-news.happylau.cn-service-1-80;
}
}
server {
listen 80;
server_tokens on;
server_name sports.happylau.cn;
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-virtualname-demo-sports.happylau.cn-service-2-80;
}
}
3.5 Ingress TLS加密
四層的負(fù)載均衡無(wú)法支持https請(qǐng)求，當(dāng)前大部分業(yè)務(wù)都要求以https方式接入，Ingress能支持https的方式接入，通過(guò)Secrets存儲(chǔ)證書(shū)+私鑰，實(shí)現(xiàn)https接入，同時(shí)還能支持http跳轉(zhuǎn)功能。對(duì)于用戶(hù)的請(qǐng)求流量來(lái)說(shuō)，客戶(hù)端到ingress controller是https流量，ingress controller到后端service則是http，提高用戶(hù)訪問(wèn)性能，如下介紹ingress TLS功能實(shí)現(xiàn)步驟。
1、生成自簽名證書(shū)和私鑰
[root@node-1 ~]# openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout tls.key -out tls.crt
Generating a 2048 bit RSA private key
....................................................+++
........................................+++
writing new private key to 'tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN #國(guó)家
State or Province Name (full name) []:GD #省份
Locality Name (eg, city) [Default City]:ShenZhen #城市
Organization Name (eg, company) [Default Company Ltd]:Tencent #公司
Organizational Unit Name (eg, section) []:HappyLau #組織
Common Name (eg, your name or your server's hostname) []:www.happylau.cn #域名
Email Address []:573302346@qq.com #郵箱地址
#tls.crt為證書(shū)，tls.key為私鑰
[root@node-1 ~]# ls tls.* -l
-rw-r--r-- 1 root root 1428 12月 26 13:21 tls.crt
-rw-r--r-- 1 root root 1708 12月 26 13:21 tls.key
2、配置Secrets，將證書(shū)和私鑰配置到Secrets中
[root@node-1 ~]# kubectl create secret tls happylau-sslkey --cert=tls.crt --key=tls.key
secret/happylau-sslkey created
查看Secrets詳情,證書(shū)和私要包含在data中，文件名為兩個(gè)不同的key：tls.crt和tls.key
[root@node-1 ~]# kubectl describe secrets happylau-sslkey
Name: happylau-sslkey
Namespace: default
Labels: <none>
Annotations: <none>
Type: kubernetes.io/tls
Data
====
tls.crt: 1428 bytes
tls.key: 1708 bytes
3、配置ingress調(diào)用Secrets實(shí)現(xiàn)SSL證書(shū)加密
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-tls-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
spec:
tls:
- hosts:
- news.happylau.cn
- sports.happylau.cn
secretName: happylau-sslkey
rules:
- host: news.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-1
servicePort: 80
- host: sports.happylau.cn
http:
paths:
- path: /
backend:
serviceName: service-2
servicePort: 80
4、創(chuàng)建ingress并查看ingress詳情
[root@node-1 nginx-ingress]# kubectl describe ingresses nginx-ingress-tls-demo
Name: nginx-ingress-tls-demo
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
happylau-sslkey terminates news.happylau.cn,sports.happylau.cn
Rules:
Host Path Backends
---- ---- --------
news.happylau.cn
/ service-1:80 (10.244.2.163:80)
sports.happylau.cn
/ service-2:80 (10.244.1.148:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernets.io/ingress.class":"nginx"},"labels":{"ingres-controller":"nginx"},"name":"nginx-ingress-tls-demo","namespace":"default"},"spec":{"rules":[{"host":"news.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-1","servicePort":80},"path":"/"}]}},{"host":"sports.happylau.cn","http":{"paths":[{"backend":{"serviceName":"service-2","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["news.happylau.cn","sports.happylau.cn"],"secretName":"happylau-sslkey"}]}}
kubernets.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 22s nginx-ingress-controller Configuration for default/nginx-ingress-tls-demo was added or updated
Normal AddedOrUpdated 22s nginx-ingress-controller Configuration for default/nginx-ingress-tls-demo was added or updated
Normal AddedOrUpdated 22s nginx-ingress-controller Configuration for default/nginx-ingress-tls-demo was added or updated
5、 將news.happylau.cn和sports.happylau.cn寫(xiě)入到hosts文件中，并通過(guò)https://news.happylau.cn 的方式訪問(wèn)，瀏覽器訪問(wèn)內(nèi)容提示證書(shū)如下，信任證書(shū)即可訪問(wèn)到站點(diǎn)內(nèi)容。
查看證書(shū)詳情，正是我們制作的自簽名證書(shū)，生產(chǎn)實(shí)際使用時(shí)，推薦使用CA機(jī)構(gòu)頒發(fā)簽名證書(shū)。
6、接下來(lái)查看一下tls配置https的nginx配置文件內(nèi)容，可以看到在server塊啟用了https并配置證書(shū)，同時(shí)配置了http跳轉(zhuǎn)，因此直接訪問(wèn)http也能夠?qū)崿F(xiàn)自動(dòng)跳轉(zhuǎn)到https功能。
# configuration for default/nginx-ingress-tls-demo
upstream default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 {
zone default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80 256k;
random two least_conn;
server 10.244.2.163:80 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 {
zone default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80 256k;
random two least_conn;
server 10.244.1.148:80 max_fails=1 fail_timeout=10s max_conns=0;
}
server {
listen 80;
listen 443 ssl; #https監(jiān)聽(tīng)端口，證書(shū)和key，實(shí)現(xiàn)和Secrets關(guān)聯(lián)
ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;
server_tokens on;
server_name news.happylau.cn;
#http跳轉(zhuǎn)功能，即訪問(wèn)http會(huì)自動(dòng)跳轉(zhuǎn)至https
if ($scheme = http) {
return 301 https://$host:443$request_uri;
}
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-tls-demo-news.happylau.cn-service-1-80;
}
}
server {
listen 80;
listen 443 ssl;
ssl_certificate /etc/nginx/secrets/default-happylau-sslkey;
ssl_certificate_key /etc/nginx/secrets/default-happylau-sslkey;
server_tokens on;
server_name sports.happylau.cn;
if ($scheme = http) {
return 301 https://$host:443$request_uri;
}
location / {
proxy_http_version 1.1;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 1m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffering on;
proxy_pass http://default-nginx-ingress-tls-demo-sports.happylau.cn-service-2-80;
}
}
4. Nginx Ingress高級(jí)功能
4.1 定制化參數(shù)
ingress controller提供了基礎(chǔ)反向代理的功能，如果需要定制化nginx的特性或參數(shù)，需要通過(guò)ConfigMap和Annotations來(lái)實(shí)現(xiàn)，兩者實(shí)現(xiàn)的方式有所不同，ConfigMap用于指定整個(gè)ingress集群資源的基本參數(shù)，修改后會(huì)被所有的ingress對(duì)象所繼承；Annotations則被某個(gè)具體的ingress對(duì)象所使用，修改只會(huì)影響某個(gè)具體的ingress資源，沖突時(shí)其優(yōu)先級(jí)高于ConfigMap。
4.1.1 ConfigMap自定義參數(shù)
安裝nginx ingress controller時(shí)默認(rèn)會(huì)包含一個(gè)空的ConfigMap，可以通過(guò)ConfigMap來(lái)自定義nginx controller的默認(rèn)參數(shù)，如下以修改一些參數(shù)為例：
1、 定義ConfigMap參數(shù)
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-config
namespace: nginx-ingress
data:
proxy-connect-timeout: "10s"
proxy-read-timeout: "10s"
proxy-send-timeout: "10"
client-max-body-size: "3m"
2、 應(yīng)用配置并查看ConfigMap配置
[root@node-1 ~]# kubectl get configmaps -n nginx-ingress nginx-config -o yaml
apiVersion: v1
data:
client-max-body-size: 3m
proxy-connect-timeout: 10s
proxy-read-timeout: 10s
proxy-send-timeout: 10s
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"client-max-body-size":"3m","proxy-connect-timeout":"10s","proxy-read-timeout":"10s","proxy-send-timeout":"10"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"nginx-config","namespace":"nginx-ingress"}}
creationTimestamp: "2019-12-24T04:39:23Z"
name: nginx-config
namespace: nginx-ingress
resourceVersion: "13845543"
selfLink: /api/v1/namespaces/nginx-ingress/configmaps/nginx-config
uid: 9313ae47-a0f0-463e-a25a-1658f1ca0d57
3 、此時(shí)，ConfigMap定義的配置參數(shù)會(huì)被集群中所有的Ingress資源繼承（除了annotations定義之外）
有很多參數(shù)可以定義，詳情配置可參考方文檔說(shuō)明：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/configmap-and-annotations.md#Summary-of-ConfigMap-and-Annotations
4.1.2 Annotations自定義參數(shù)
ConfigMap定義的是全局的配置參數(shù)，修改后所有的配置都會(huì)受影響，如果想針對(duì)某個(gè)具體的ingress資源自定義參數(shù)，則可以通過(guò)Annotations來(lái)實(shí)現(xiàn)，下面開(kāi)始以實(shí)際的例子演示Annotations的使用。
1、修改ingress資源，添加annotations的定義,通過(guò)nginx.org組修改了一些參數(shù)，如proxy-connect-timeout，調(diào)度算法為round_robin（默認(rèn)為least _conn）
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-ingress-demo
labels:
ingres-controller: nginx
annotations:
kubernets.io/ingress.class: nginx
nginx.org/proxy-connect-timeout: "30s"
nginx.org/proxy-send-timeout: "20s"
nginx.org/proxy-read-timeout: "20s"
nginx.org/client-max-body-size: "2m"
nginx.org/fail-timeout: "5s"
nginx.org/lb-method: "round_robin"
spec:
rules:
- host: www.happylau.cn
http:
paths:
- path: /
backend:
serviceName: ingress-demo
servicePort: 80
2、 重新應(yīng)用ingress對(duì)象并查看參數(shù)配置情況
由上面的演示可得知，Annotations的優(yōu)先級(jí)高于ConfigMapMap，Annotations修改參數(shù)只會(huì)影響到某一個(gè)具體的ingress資源，其定義的方法和ConfigMap相相近似，但又有差別，部分ConfigMap的參數(shù)Annotations無(wú)法支持，反過(guò)來(lái)Annotations定義的參數(shù)ConfigMap也不一定支持，下圖列舉一下常規(guī)支持參數(shù)情況：
ConfigMap和Annotations詳細(xì)支持說(shuō)明：鏈接說(shuō)明
4.2 虛擬主機(jī)和路由
安裝nginx ingress時(shí)我們安裝了一個(gè)customresourcedefinitions自定義資源，其能夠提供除了默認(rèn)ingress功能之外的一些高級(jí)特性如
- 虛擬主機(jī)VirtualServer
- 虛擬路由VirtualServerRoute
- 健康檢查Healthcheck
- 流量切割Split
- 會(huì)話保持SessionCookie
- 重定向Redirect
這些功能大部分依賴(lài)于Nginx Plus高級(jí)版本的支持，社區(qū)版本僅支持部分，對(duì)于企業(yè)級(jí)開(kāi)發(fā)而言，豐富更多的功能可以購(gòu)買(mǎi)企業(yè)級(jí)Nginx Plus版本。如下以通過(guò)VirtualServer和VirtualServerRoute定義upstream配置為例演示功能使用。
1、定義VirtualServer資源,其配置和ingress資源對(duì)象類(lèi)似，能支持的功能會(huì)更豐富一點(diǎn)
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: cafe
spec:
host: cafe.example.com
tls:
secret: cafe-secret
upstreams:
- name: tea
service: tea-svc
port: 80
name: tea
service: ingress-demo
subselector:
version: canary
lb-method: round_robin
fail-timeout: 10s
max-fails: 1
max-conns: 32
keepalive: 32
connect-timeout: 30s
read-timeout: 30s
send-timeout: 30s
next-upstream: "error timeout non_idempotent"
next-upstream-timeout: 5s
next-upstream-tries: 10
client-max-body-size: 2m
tls:
enable: true
routes:
- path: /tea
action:
pass: tea
2、 應(yīng)用資源并查看VirtualServer資源列表
[root@node-1 ~]# kubectl apply -f vs.yaml
virtualserver.k8s.nginx.org/cafe unchanged
[root@node-1 ~]# kubectl get virtualserver
NAME AGE
cafe 2m52s
3、檢查ingress控制器的配置文件情況,生成的配置和upstream定義一致
nginx@nginx-ingress-7mpfc:/etc/nginx/conf.d$ cat vs_default_cafe.conf
upstream vs_default_cafe_tea {
zone vs_default_cafe_tea 256k;
server 10.244.0.51:80 max_fails=1 fail_timeout=10s max_conns=32;
server 10.244.1.146:80 max_fails=1 fail_timeout=10s max_conns=32;
server 10.244.1.147:80 max_fails=1 fail_timeout=10s max_conns=32;
server 10.244.2.162:80 max_fails=1 fail_timeout=10s max_conns=32;
keepalive 32;
}
server {
listen 80;
server_name cafe.example.com;
listen 443 ssl;
ssl_certificate /etc/nginx/secrets/default;
ssl_certificate_key /etc/nginx/secrets/default;
ssl_ciphers NULL;
server_tokens "on";
location /tea {
proxy_connect_timeout 30s;
proxy_read_timeout 30s;
proxy_send_timeout 30s;
client_max_body_size 2m;
proxy_max_temp_file_size 1024m;
proxy_buffering on;
proxy_http_version 1.1;
set $default_connection_header "";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://vs_default_cafe_tea;
proxy_next_upstream error timeout non_idempotent;
proxy_next_upstream_timeout 5s;
proxy_next_upstream_tries 10;
}
}
寫(xiě)在最后
本文詳細(xì)介紹了基于nginx的ingress實(shí)現(xiàn)，通過(guò)實(shí)際的案例演示ingress的安裝部署，基于虛擬主機(jī)的配置，基于TLS加密實(shí)現(xiàn)https，高級(jí)章節(jié)中介紹了負(fù)載均衡參數(shù)定制，自定義資源虛擬主機(jī)和虛擬路由的實(shí)現(xiàn)，通過(guò)該章節(jié)相信能加深對(duì)ingress服務(wù)暴露機(jī)制的理解。實(shí)現(xiàn)ingress controller的方式有多種，下一個(gè)章節(jié)我們將介紹基于HAproxy和TKE Ingress控制器的實(shí)現(xiàn)。
參考文獻(xiàn)
Ingress配置：https://kubernetes.io/docs/concepts/services-networking/ingress/
Ingress控制器：https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
Nginx ingress安裝文檔：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md
Nginx ingress文檔說(shuō)明：https://github.com/nginxinc/kubernetes-ingress/tree/master/docs
虛擬主機(jī)和路由：https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/virtualserver-and-virtualserverroute.md
當(dāng)你的才華撐不起你的野心時(shí)，你就應(yīng)該靜下心來(lái)學(xué)習(xí)
?
如果覺(jué)得文章對(duì)您有幫助，請(qǐng)訂閱專(zhuān)欄，分享給有需要的朋友吧??
關(guān)于作者 劉海平（HappyLau ）云計(jì)算高級(jí)顧問(wèn) 目前在騰訊云從事公有云相關(guān)工作，曾就職于酷狗，EasyStack，擁有多年公有云+私有云計(jì)算架構(gòu)設(shè)計(jì)，運(yùn)維，交付相關(guān)經(jīng)驗(yàn)，參與了酷狗，南方電網(wǎng)，國(guó)泰君安等大型私有云平臺(tái)建設(shè)，精通Linux，Kubernetes，OpenStack，Ceph等開(kāi)源技術(shù)，在云計(jì)算領(lǐng)域具有豐富實(shí)戰(zhàn)經(jīng)驗(yàn)，擁有RHCA/OpenStack/Linux授課經(jīng)驗(yàn)。
