環(huán)境信息

* 集群1: Uk8s    1.20.6      Node節(jié)點 4核8G    iptable(ucloud vpc cni) 
* 集群2: Uk8s    1.20.6      Node節(jié)點 4核8G    ipvs (ucloud vpc cni)
* 集群3: K8S     1.20.6      Node節(jié)點 4核8G    ebpf (cilium/ipvlan)   
* 集群4: ACK.    1.20        Node節(jié)點 4核8G    ebpf (Terway/ipvlan)       

測試工具

sirot/netperf-latest 鏡像，包含netperf/iperf工具

• 準備測試環(huán)境

# kubectl run -it --rm --restart=Never busybox --image=busybox sh
# kubectl get nodes --show-labels

kubectl  delete svc --all -n network-bench
kubectl  delete deploy --all -n network-bench
kubectl  delete pods --all -n network-bench

kubectl create ns network-bench

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: netperf-server
  namespace: network-bench
  labels:
    app: netperf-server
    role: local
spec:
  containers:
  - image: sirot/netperf-latest
    command: ["/bin/sh","-c","netserver -p 4444 -4; iperf3 -s -i 1;"] 
    imagePullPolicy: IfNotPresent
    name: netperf
    ports:
    - name: netperf-port
      containerPort: 4444
    - name: iperf-port
      containerPort: 5210
  restartPolicy: Always
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: role
            operator: In
            values:
            - local
        topologyKey: kubernetes.io/hostname
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: role
            operator: In
            values:
            - remote
        topologyKey: kubernetes.io/hostname
---
apiVersion: v1
kind: Service
metadata:
  name: netperf-headless-svc
  namespace: network-bench
  labels:
    app: netperf-headless-svc
spec:
  ports:
  - name: netperf-port
    port: 4444
    targetPort: 4444
  - name: iperf-port
    port: 5201
    targetPort: 5201
  clusterIP: None
  selector:
    app: netperf-server
---
apiVersion: v1
kind: Pod
metadata:
  name: netperf-client
  namespace: network-bench
  labels:
    app: netperf-client
    role: local
spec:
  containers:
  - image: sirot/netperf-latest
    command:
      - sleep
      - "7200"
    imagePullPolicy: IfNotPresent
    name: netperf
  restartPolicy: Always
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: role
            operator: In
            values:
            - local
        topologyKey: kubernetes.io/hostname
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: role
            operator: In
            values:
            - remote
        topologyKey: kubernetes.io/hostname
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netperf-deploy
  namespace: network-bench
  labels:
    app: netperf-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      role: remote
  template:
    metadata:
      labels:
        app: netperf-remote
        role: remote
    spec:
      containers:
      - name: netperf-remote
        image: sirot/netperf-latest
        command: ["/bin/sh","-c","netserver -p 4444 -4; iperf3 -s -i 1;"] 
        ports:
        - name: netperf-port
          containerPort: 4444
        - name: iperf-port
          containerPort: 5210
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: role
                operator: In
                values:
                - local
            topologyKey: kubernetes.io/hostname
---
apiVersion: v1
kind: Service
metadata:
  name: netperf-remote-svc
  namespace: network-bench
spec:
  selector:
    role: remote
  type: ClusterIP
  ports:
  - name: netperf-port
    port: 4444
    targetPort: 4444
  - name: iperf-port
    port: 5201
    targetPort: 5201
EOF

• 執(zhí)行測試腳本

#！/bin/bash

echo "# POD NetWork Test Result"
  echo "## iperf_tcp_pod_to_pod: "
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'iperf3 -c netperf-headless-svc -t 10' | tail -n 5
  
  echo "## iperf_udp_pod_to_pod: "
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'iperf3 -u -c netperf-headless-svc -t 10' | tail -n 5

  echo "## netperf_tcp_rr_pod_to_pod: "
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'netperf -t TCP_RR -H netperf-headless-svc -p 4444 -l 10' | tail -n 5

  echo "## netperf_tcp_crr_pod_to_pod:"
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'netperf -t TCP_CRR -H netperf-headless-svc -p 4444 -l 10' | tail -n 5


  netperf_pod=`kubectl  get pods -n network-bench | grep deploy | awk  '{print $1}'`

  echo "## iperf_tcp_pod_to_pod_over_node:"
  kubectl exec -t -i ${netperf_pod} -n network-bench -- sh -c 'iperf3 -c netperf-headless-svc -t 10' | tail -n 5

  echo "## iperf_udp_pod_to_pod_over_node"
  kubectl exec -t -i ${netperf_pod} -n network-bench -- sh -c 'iperf3 -u -c netperf-headless-svc -t 10' | tail -n 5

  echo "## netperf_tcp_rr_pod_to_pod_over_node: "
  kubectl exec -t -i ${netperf_pod} -n network-bench -- sh -c 'netperf -t TCP_RR -H netperf-headless-svc -p 4444 -l 10' | tail -n 5

  echo "## netperf_tcp_crr_pod_to_pod_over_node:"
  kubectl exec -t -i ${netperf_pod} -n network-bench -- sh -c 'netperf -t TCP_CRR -H netperf-headless-svc -p 4444 -l 10' | tail -n 5

  echo "## iperf3_tcp_pod_to_remote_svc" 
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'iperf3 -c netperf-remote-svc -t 10'  | tail -n 6

  echo "## iperf3_udp_pod_to_remote_svc" 
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'iperf3 -u -c netperf-remote-svc -t 10' | tail -n 6

  echo "## netperf_tcp_rr_pod_to_remote_svc"
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'netperf -t TCP_RR -H netperf-remote-svc -p 4444 -l 10' | tail -n 6

  echo "## netperf_tcp_crr_pod_to_remote_svc"
  kubectl exec -t -i netperf-client -n network-bench -- sh -c 'netperf -t TCP_CRR -H netperf-remote-svc -p 4444 -l 10' | tail -n 6

TestCase

1. iperf 參考測試用例

服務端: iperf3 -s -i 1
客戶端(TCP請求): iperf3 -c iperf3_server_ip -t 60
客戶端(UDP請求): iperf3 -u -c iperf3_server_ip -t 60

2. netperf 參考測試用例

服務端: netserver -p 4444 -D -4
客戶端(TCP_RR方式): netperf -t TCP_RR -H netperf_server_ip -p 4444
客戶端(TCP_CRR方式): netperf -t TCP_CRR -H netperf_server_ip -p 4444

測試場景

1. 同Node pod 網絡性能測試
2. 跨Node pod 網絡性能測試
3. Pod to ClusterIP 網絡性能測試

測試結果

云廠商K8S集群對比

結論與分析

1. cilium cni

• 能實現 同node POD間網絡最大帶寬吞吐，和tcp_rr測試項的最大交易量
• 在跨節(jié)點 pod 網絡吞吐性能低于Ucloud Vpc CNI，
• 在跨節(jié)點 pod 網絡tcp_crr只能達到Ucloud Vpc CNI的一半

2. UcloudVpc cni iptable 轉發(fā)模式下能達到最大的跨節(jié)點 pod 網絡帶寬
3. UcloudVpc cni ipvs 轉發(fā)模式下能達到最大的跨節(jié)點 tcp_rr，tcp_crr測試的最大交易量
4. 阿里云(Terway/ipvlan) POD網絡帶寬最高，無論同節(jié)點POD網絡帶寬，還是跨節(jié)點POD網絡帶寬，接近宿主Node網絡的性能極限

以上只是對 k8s集群pod網絡的基準測試

參考

1.【山外筆記-工具框架】Netperf網絡性能測試工具詳解教程
https://www.cnblogs.com/davidesun/p/12726006.html
2.【網絡性能測試方法 】https://help.aliyun.com/knowledge_detail/55757.html#HFXbx