RPS和RFS網(wǎng)卡多隊列性能調(diào)優(yōu)實踐

前言

為了解決LVS ksoftirqd CPU使用率100%導致網(wǎng)卡軟中斷丟包，我和同事們一起搜索了大量的資料去分析問題，特別是感謝美團技術(shù)團隊的分享幫助我們快速梳理優(yōu)化思路，最后明確了如何重構(gòu)RPS和RFS網(wǎng)卡多隊列的優(yōu)化腳本。個人認為這是一個大家可能普遍會遇到的問題，文章內(nèi)的分析思路和解決方案未必是最優(yōu)解，也歡迎各位分享自己的解決方法。

RPS和RFS網(wǎng)卡多隊列性能調(diào)優(yōu)實踐

更新歷史

2019年07月03日 - 初稿

閱讀原文 - https://wsgzao.github.io/post/rps/

擴展閱讀

Redis 高負載下的中斷優(yōu)化 - https://tech.meituan.com/2018/03/16/redis-high-concurrency-optimization.html

事故復盤

我們遇到的問題屬于計劃外的incident，現(xiàn)象是某產(chǎn)品用戶在線率突然降低，LVS Master同時收到CPU High Load告警，檢查發(fā)現(xiàn)該節(jié)點出現(xiàn)網(wǎng)卡大量斷開重連和丟包情況，應(yīng)急切換到LVS Slave也出現(xiàn)上述問題，在排除掉流量異常和外部攻擊后選擇切換DNS到背后的Nginx Real Servers后服務(wù)逐步恢復。

復盤核心原因在于系統(tǒng)初始化時rps優(yōu)化腳本沒有成功執(zhí)行，這個腳本起初是因為早期DBA團隊遇到過CPU負載較高導致網(wǎng)卡異常，這個優(yōu)化腳本也一直傳承至今，卻已經(jīng)沒有人知道為什么添加?，F(xiàn)在大多數(shù)服務(wù)器沒有執(zhí)行成功而被大家一直所忽視顯然也是post check沒有做到位。在早期大家都停留在Bash Shell運維的階段，沒有專職的團隊來管理確實容易失控，好在現(xiàn)在可以基于Ansible來做初始化和檢查，運維的壓力也減輕了一部分。

通過Google搜索相關(guān)知識的過程中，我們也發(fā)現(xiàn)在不少人都會遇到這樣類似的問題。比如這篇文章提到lvs/irq

lvs 的性能問題，軟中斷耗盡 CPU 單核后到達處理極限

  • 瓶頸現(xiàn)象：當壓力較大時，Lvs 服務(wù)器 CPU 的其中一個核使用率達到 100%（處理軟中斷）。
    • 當 Lvs 服務(wù)器處理軟中斷的那個核使用率達到 100%，就到達系統(tǒng)處理上限。
    • 占用 CPU 的是進程 “ksoftirqd”，它未能使用到多核。
  • 做雙網(wǎng)卡綁定，然后調(diào)試內(nèi)核 SMP，中斷主要是來自網(wǎng)卡的，不是 LVS 本身。需要把 2 個網(wǎng)卡來的 IRQ 均衡在雙核 CPU 上面。

和華為的工程師們在交換經(jīng)驗的時候?qū)Ψ椒窒砹艘粋€關(guān)于RSS和RPS關(guān)系圖，之后的內(nèi)容還會引用美團技術(shù)團隊的分析


image

我們遇到的情況是缺少可用服務(wù)器資源選擇把用戶外部請求流量和Codis Cache Cluster內(nèi)部流量臨時混在了同一個LVS上，雖然看上去CPU和traffic的整體壓力都不算高，但是CPU的處理壓力可能恰好集中在了和外網(wǎng)Bond1網(wǎng)卡相同的Core上最后引起了ksoftirqd軟中斷，而內(nèi)網(wǎng)Bond0網(wǎng)卡就沒有監(jiān)控到任何丟包。雖然我們也有正常開啟irqbalance，但不清楚是不是因為受到cpupower performance和NUMA的影響最后也沒能阻止事故的發(fā)生，最終的優(yōu)化方案主要是手動開啟RPS和RFS，大致步驟如下：

  1. set cpupower cpupower frequency-set -g performance，CPU 優(yōu)化建議使用 cpupower 設(shè)置 CPU Performance 模式
  2. activate rps/rfs by script: rps.sh
  3. double ring buffer size: ethtool -G p1p1 [rx|tx] 4096, check ethtool -g p1p1
  4. double NAPI poll budget: sysctl -w net.core.netdev_budget=600
  5. add zabbix monitor on net.if.in[eth0,errors,dropped,overruns]
#!/bin/bash
# chkconfig: 2345 90 60
### BEGIN INIT INFO
# Provides:          rps
# Required-Start:    $local_fs $remote_fs $network $syslog
# Required-Stop:     $local_fs $remote_fs $network $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: enable rps config for ubuntu
# Description:       enabele rps which is a kernel tweak for network performance
### END INIT INFO

NAME=rps
DESC=rps

# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# cpupower frequency-set -g performance
# activate rps/rfs by script: https://gist.github.com/wsgzao/18828f69147635f3e38a14690a633daf
# double ring buffer size: ethtool -G p1p1 [rx|tx] 4096, ethtool -g p1p1
# double NAPI poll budget: sysctl -w net.core.netdev_budget=600

rps() {
  net_interface=`ip link show | grep "state UP" | awk '{print $2}' | egrep -v '^docker|^veth' | tr ":\n" " "`

  for em in ${net_interface[@]}
  do
      rq_count=`ls /sys/class/net/$em/queues/rx-* -d | wc -l`
      rps_flow_cnt_value=`expr 32768 / $rq_count`

      for ((i=0; i< $rq_count; i++))
      do
          echo $rps_flow_cnt_value > /sys/class/net/$em/queues/rx-$i/rps_flow_cnt
      done

      flag=0
      while [ -f /sys/class/net/$em/queues/rx-$flag/rps_cpus ]
      do
          echo `cat  /sys/class/net/$em/queues/rx-$flag/rps_cpus | sed 's/0/f/g' ` >  /sys/class/net/$em/queues/rx-$flag/rps_cpus
          flag=$(($flag+1))
      done
  done
  echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
  sysctl -p
}

check_rps() {
  ni_list=`ip link show | grep "state UP" | awk '{print $2}' | egrep -v "^docker|^veth" | tr ":\n" " "`
  for n in $ni_list
  do
      rx_queues=`ls /sys/class/net/$n/queues/ | grep "rx-[0-9]"`
      for q in $rx_queues
      do
          rps_cpus=`cat /sys/class/net/$n/queues/$q/rps_cpus`
          rps_flow_cnt=`cat /sys/class/net/$n/queues/$q/rps_flow_cnt`

          echo "[$n]" $q "--> rps_cpus =" $rps_cpus ", rps_flow_cnt =" $rps_flow_cnt
      done
  done
  rps_sock_flow_entries=`cat /proc/sys/net/core/rps_sock_flow_entries`
  echo "rps_sock_flow_entries =" $rps_sock_flow_entries
}

case "$1" in
  start)
        echo -n "Starting $DESC: "
        rps
        check_rps
        ;;
  stop)
        echo -n "Stop is not supported. "
        ;;
  restart|reload|force-reload)
        echo -n "Restart is not supported. "
        ;;
  status)
        check_rps
        ;;
  *)
        echo "Usage: $0 [start|status]"
        ;;
esac

exit 0

Scaling in the Linux Networking Stack

了解RSS、RPS、RFS等基礎(chǔ)知識很重要

This document describes a set of complementary techniques in the Linux
networking stack to increase parallelism and improve performance for
multi-processor systems.

The following technologies are described:

  • RSS: Receive Side Scaling
  • RPS: Receive Packet Steering
  • RFS: Receive Flow Steering
  • Accelerated Receive Flow Steering
  • XPS: Transmit Packet Steering

https://www.kernel.org/doc/Documentation/networking/scaling.txt

RPS

RECEIVE PACKET STEERING (RPS)

Receive Packet Steering (RPS) is similar to RSS in that it is used to direct packets to specific CPUs for processing. However, RPS is implemented at the software level, and helps to prevent the hardware queue of a single network interface card from becoming a bottleneck in network traffic.

RPS has several advantages over hardware-based RSS:

  • RPS can be used with any network interface card.

  • It is easy to add software filters to RPS to deal with new protocols.

  • RPS does not increase the hardware interrupt rate of the network device. However, it does introduce inter-processor interrupts.

RPS is configured per network device and receive queue, in the /sys/class/net/*device*/queues/*rx-queue*/rps_cpus file, where device is the name of the network device (such as eth0) and rx-queue is the name of the appropriate receive queue (such as rx-0).

The default value of the rps_cpus file is zero. This disables RPS, so the CPU that handles the network interrupt also processes the packet.

To enable RPS, configure the appropriate rps_cpus file with the CPUs that should process packets from the specified network device and receive queue.

The rps_cpus files use comma-delimited CPU bitmaps. Therefore, to allow a CPU to handle interrupts for the receive queue on an interface, set the value of their positions in the bitmap to 1. For example, to handle interrupts with CPUs 0, 1, 2, and 3, set the value of rps_cpus to 00001111 (1+2+4+8), or f (the hexadecimal value for 15).

For network devices with single transmit queues, best performance can be achieved by configuring RPS to use CPUs in the same memory domain. On non-NUMA systems, this means that all available CPUs can be used. If the network interrupt rate is extremely high, excluding the CPU that handles network interrupts may also improve performance.

For network devices with multiple queues, there is typically no benefit to configuring both RPS and RSS, as RSS is configured to map a CPU to each receive queue by default. However, RPS may still be beneficial if there are fewer hardware queues than CPUs, and RPS is configured to use CPUs in the same memory domain.

RFS

RECEIVE FLOW STEERING (RFS)

Receive Flow Steering (RFS) extends RPS behavior to increase the CPU cache hit rate and thereby reduce network latency. Where RPS forwards packets based solely on queue length, RFS uses the RPS backend to calculate the most appropriate CPU, then forwards packets based on the location of the application consuming the packet. This increases CPU cache efficiency.

RFS is disabled by default. To enable RFS, you must edit two files:

/proc/sys/net/core/rps_sock_flow_entries

Set the value of this file to the maximum expected number of concurrently active connections. We recommend a value of 32768 for moderate server loads. All values entered are rounded up to the nearest power of 2 in practice.

/sys/class/net/*device*/queues/*rx-queue*/rps_flow_cnt

Replace device with the name of the network device you wish to configure (for example, eth0), and rx-queue with the receive queue you wish to configure (for example, rx-0).

Set the value of this file to the value of rps_sock_flow_entries divided by N, where N is the number of receive queues on a device. For example, if rps_flow_entries is set to 32768 and there are 16 configured receive queues, rps_flow_cnt should be set to 2048. For single-queue devices, the value of rps_flow_cnt is the same as the value of rps_sock_flow_entries.

Data received from a single sender is not sent to more than one CPU. If the amount of data received from a single sender is greater than a single CPU can handle, configure a larger frame size to reduce the number of interrupts and therefore the amount of processing work for the CPU. Alternatively, consider NIC offload options or faster CPUs.

Consider using numactl or taskset in conjunction with RFS to pin applications to specific cores, sockets, or NUMA nodes. This can help prevent packets from being processed out of order.

了解接收數(shù)據(jù)包的流程

這里摘取了美團技術(shù)團隊的分析，在此表示感謝

接收數(shù)據(jù)包是一個復雜的過程，涉及很多底層的技術(shù)細節(jié)，但大致需要以下幾個步驟：

  1. 網(wǎng)卡收到數(shù)據(jù)包。
  2. 將數(shù)據(jù)包從網(wǎng)卡硬件緩存轉(zhuǎn)移到服務(wù)器內(nèi)存中。
  3. 通知內(nèi)核處理。
  4. 經(jīng)過 TCP/IP 協(xié)議逐層處理。
  5. 應(yīng)用程序通過 read() 從 socket buffer 讀取數(shù)據(jù)。
image

將網(wǎng)卡收到的數(shù)據(jù)包轉(zhuǎn)移到主機內(nèi)存（NIC 與驅(qū)動交互）

NIC 在接收到數(shù)據(jù)包之后，首先需要將數(shù)據(jù)同步到內(nèi)核中，這中間的橋梁是 rx ring buffer。它是由 NIC 和驅(qū)動程序共享的一片區(qū)域，事實上，rx ring buffer 存儲的并不是實際的 packet 數(shù)據(jù)，而是一個描述符，這個描述符指向了它真正的存儲地址，具體流程如下：

  1. 驅(qū)動在內(nèi)存中分配一片緩沖區(qū)用來接收數(shù)據(jù)包，叫做 sk_buffer；
  2. 將上述緩沖區(qū)的地址和大?。唇邮彰枋龇?，加入到 rx ring buffer。描述符中的緩沖區(qū)地址是 DMA 使用的物理地址；
  3. 驅(qū)動通知網(wǎng)卡有一個新的描述符；
  4. 網(wǎng)卡從 rx ring buffer 中取出描述符，從而獲知緩沖區(qū)的地址和大??；
  5. 網(wǎng)卡收到新的數(shù)據(jù)包；
  6. 網(wǎng)卡將新數(shù)據(jù)包通過 DMA 直接寫到 sk_buffer 中。
image

當驅(qū)動處理速度跟不上網(wǎng)卡收包速度時，驅(qū)動來不及分配緩沖區(qū)，NIC 接收到的數(shù)據(jù)包無法及時寫到 sk_buffer，就會產(chǎn)生堆積，當 NIC 內(nèi)部緩沖區(qū)寫滿后，就會丟棄部分數(shù)據(jù)，引起丟包。這部分丟包為 rx_fifo_errors，在 /proc/net/dev 中體現(xiàn)為 fifo 字段增長，在 ifconfig 中體現(xiàn)為 overruns 指標增長。

通知系統(tǒng)內(nèi)核處理（驅(qū)動與 Linux 內(nèi)核交互）

這個時候，數(shù)據(jù)包已經(jīng)被轉(zhuǎn)移到了 sk_buffer 中。前文提到，這是驅(qū)動程序在內(nèi)存中分配的一片緩沖區(qū)，并且是通過 DMA 寫入的，這種方式不依賴 CPU 直接將數(shù)據(jù)寫到了內(nèi)存中，意味著對內(nèi)核來說，其實并不知道已經(jīng)有新數(shù)據(jù)到了內(nèi)存中。那么如何讓內(nèi)核知道有新數(shù)據(jù)進來了呢？答案就是中斷，通過中斷告訴內(nèi)核有新數(shù)據(jù)進來了，并需要進行后續(xù)處理。

提到中斷，就涉及到硬中斷和軟中斷，首先需要簡單了解一下它們的區(qū)別：

  • 硬中斷： 由硬件自己生成，具有隨機性，硬中斷被 CPU 接收后，觸發(fā)執(zhí)行中斷處理程序。中斷處理程序只會處理關(guān)鍵性的、短時間內(nèi)可以處理完的工作，剩余耗時較長工作，會放到中斷之后，由軟中斷來完成。硬中斷也被稱為上半部分。
  • 軟中斷： 由硬中斷對應(yīng)的中斷處理程序生成，往往是預先在代碼里實現(xiàn)好的，不具有隨機性。（除此之外，也有應(yīng)用程序觸發(fā)的軟中斷，與本文討論的網(wǎng)卡收包無關(guān)。）也被稱為下半部分。

當 NIC 把數(shù)據(jù)包通過 DMA 復制到內(nèi)核緩沖區(qū) sk_buffer 后，NIC 立即發(fā)起一個硬件中斷。CPU 接收后，首先進入上半部分，網(wǎng)卡中斷對應(yīng)的中斷處理程序是網(wǎng)卡驅(qū)動程序的一部分，之后由它發(fā)起軟中斷，進入下半部分，開始消費 sk_buffer 中的數(shù)據(jù)，交給內(nèi)核協(xié)議棧處理。

image

通過中斷，能夠快速及時地響應(yīng)網(wǎng)卡數(shù)據(jù)請求，但如果數(shù)據(jù)量大，那么會產(chǎn)生大量中斷請求，CPU 大部分時間都忙于處理中斷，效率很低。為了解決這個問題，現(xiàn)在的內(nèi)核及驅(qū)動都采用一種叫 NAPI（new API）的方式進行數(shù)據(jù)處理，其原理可以簡單理解為 中斷 + 輪詢，在數(shù)據(jù)量大時，一次中斷后通過輪詢接收一定數(shù)量包再返回，避免產(chǎn)生多次中斷。

什么是中斷？

由于接收來自外圍硬件 (相對于 CPU 和內(nèi)存) 的異步信號或者來自軟件的同步信號，而進行相應(yīng)的硬件、軟件處理；發(fā)出這樣的信號稱為進行中斷請求 (interrupt request, IRQ)

硬中斷與軟中斷？

  • 硬中斷：外圍硬件發(fā)給 CPU 或者內(nèi)存的異步信號就稱之為硬中斷
  • 軟中斷：由軟件系統(tǒng)本身發(fā)給操作系統(tǒng)內(nèi)核的中斷信號，稱之為軟中斷。通常是由硬中斷處理程序或進程調(diào)度程序?qū)Σ僮飨到y(tǒng)內(nèi)核的中斷，也就是我們常說的系統(tǒng)調(diào)用 (System Call)

硬中斷與軟中斷之區(qū)別與聯(lián)系？

  1. 硬中斷是有外設(shè)硬件發(fā)出的，需要有中斷控制器之參與。其過程是外設(shè)偵測到變化，告知中斷控制器，中斷控制器通過 CPU 或內(nèi)存的中斷腳通知 CPU，然后硬件進行程序計數(shù)器及堆棧寄存器之現(xiàn)場保存工作（引發(fā)上下文切換），并根據(jù)中斷向量調(diào)用硬中斷處理程序進行中斷處理
  2. 軟中斷則通常是由硬中斷處理程序或者進程調(diào)度程序等軟件程序發(fā)出的中斷信號，無需中斷控制器之參與，直接以一個 CPU 指令之形式指示 CPU 進行程序計數(shù)器及堆棧寄存器之現(xiàn)場保存工作 (亦會引發(fā)上下文切換)，并調(diào)用相應(yīng)的軟中斷處理程序進行中斷處理 (即我們通常所言之系統(tǒng)調(diào)用)
  3. 硬中斷直接以硬件的方式引發(fā)，處理速度快。軟中斷以軟件指令之方式適合于對響應(yīng)速度要求不是特別嚴格的場景
  4. 硬中斷通過設(shè)置 CPU 的屏蔽位可進行屏蔽，軟中斷則由于是指令之方式給出，不能屏蔽
  5. 硬中斷發(fā)生后，通常會在硬中斷處理程序中調(diào)用一個軟中斷來進行后續(xù)工作的處理
  6. 硬中斷和軟中斷均會引起上下文切換 (進程 / 線程之切換)，進程切換的過程是差不多的

查看中斷情況

1.top 按下數(shù)字鍵 1

  • hi : time spent servicing hardware interrupts
  • si : time spent servicing software interrupts

2.mpstat -P ALL 2

  • irq 為硬中斷
  • soft 為軟中斷

mpstat使用介紹和輸出參數(shù)詳解 - https://wsgzao.github.io/post/mpstat/

[root@localhost wangao]# top
top - 19:17:57 up 7 days,  7:11,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 338 total,   1 running, 337 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  2.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
%Cpu8  :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu9  :  0.0 us,  0.0 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  3.0 si,  0.0 st
%Cpu10 :  0.0 us,  0.0 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  1.3 si,  0.0 st
%Cpu11 :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu12 :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu13 :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu14 :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu15 :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu16 :  0.0 us,  0.0 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  2.7 si,  0.0 st
%Cpu17 :  0.0 us,  0.0 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
%Cpu18 :  0.3 us,  0.7 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
%Cpu19 :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu20 :  0.0 us,  0.0 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  1.0 si,  0.0 st
%Cpu21 :  0.0 us,  0.0 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  3.0 si,  0.0 st
%Cpu22 :  0.0 us,  0.0 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
%Cpu23 :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu24 :  0.0 us,  0.0 sy,  0.0 ni, 97.7 id,  0.0 wa,  0.0 hi,  2.3 si,  0.0 st
%Cpu25 :  0.0 us,  0.0 sy,  0.0 ni, 97.0 id,  0.0 wa,  0.0 hi,  3.0 si,  0.0 st
%Cpu26 :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu27 :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu28 :  0.0 us,  0.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  3.3 si,  0.0 st
%Cpu29 :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
%Cpu30 :  0.0 us,  0.0 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu31 :  0.0 us,  0.0 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  2.6 si,  0.0 st
KiB Mem : 65688788 total, 62127224 free,  1742128 used,  1819436 buff/cache
KiB Swap: 66060284 total, 66060284 free,        0 used. 63291844 avail Mem


[root@localhost wangao]# mpstat -P ALL 2
Linux 3.10.0-957.21.3.el7.x86_64 (localhost.localdomain)    07/04/2019  _x86_64_    (32 CPU)

07:18:35 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
07:18:37 PM  all    0.02    0.00    0.02    0.00    0.00    2.14    0.00    0.00    0.00   97.83
07:18:37 PM    0    0.00    0.00    0.00    0.00    0.00    2.50    0.00    0.00    0.00   97.50
07:18:37 PM    1    0.00    0.00    0.00    0.00    0.00    4.41    0.00    0.00    0.00   95.59
07:18:37 PM    2    0.00    0.00    0.00    0.00    0.00    2.51    0.00    0.00    0.00   97.49
07:18:37 PM    3    0.00    0.00    0.00    0.00    0.00    2.96    0.00    0.00    0.00   97.04
07:18:37 PM    4    0.00    0.00    0.00    0.00    0.00    1.01    0.00    0.00    0.00   98.99
07:18:37 PM    5    0.00    0.00    0.00    0.00    0.00    2.99    0.00    0.00    0.00   97.01
07:18:37 PM    6    0.00    0.00    0.00    0.00    0.00    1.01    0.00    0.00    0.00   98.99
07:18:37 PM    7    0.00    0.00    0.00    0.00    0.00    2.99    0.00    0.00    0.00   97.01
07:18:37 PM    8    0.00    0.00    0.00    0.00    0.00    1.50    0.00    0.00    0.00   98.50
07:18:37 PM    9    0.00    0.00    0.00    0.00    0.00    2.01    0.00    0.00    0.00   97.99
07:18:37 PM   10    0.00    0.00    0.00    0.00    0.00    1.50    0.00    0.00    0.00   98.50
07:18:37 PM   11    0.00    0.00    0.00    0.00    0.00    1.50    0.00    0.00    0.00   98.50
07:18:37 PM   12    0.00    0.00    0.00    0.00    0.00    1.01    0.00    0.00    0.00   98.99
07:18:37 PM   13    0.00    0.00    0.00    0.00    0.00    2.49    0.00    0.00    0.00   97.51
07:18:37 PM   14    0.00    0.00    0.00    0.00    0.00    1.01    0.00    0.00    0.00   98.99
07:18:37 PM   15    0.00    0.00    0.00    0.00    0.00    3.94    0.00    0.00    0.00   96.06
07:18:37 PM   16    0.00    0.00    0.00    0.00    0.00    2.50    0.00    0.00    0.00   97.50
07:18:37 PM   17    0.00    0.00    0.00    0.00    0.00    2.97    0.00    0.00    0.00   97.03
07:18:37 PM   18    0.00    0.00    0.00    0.00    0.00    2.02    0.00    0.00    0.00   97.98
07:18:37 PM   19    0.00    0.00    0.00    0.00    0.00    1.49    0.00    0.00    0.00   98.51
07:18:37 PM   20    0.00    0.00    0.00    0.00    0.00    1.01    0.00    0.00    0.00   98.99
07:18:37 PM   21    0.00    0.00    0.00    0.00    0.00    2.50    0.00    0.00    0.00   97.50
07:18:37 PM   22    0.00    0.00    0.00    0.00    0.00    1.50    0.00    0.00    0.00   98.50
07:18:37 PM   23    0.00    0.00    0.00    0.00    0.00    3.48    0.00    0.00    0.00   96.52
07:18:37 PM   24    0.00    0.00    0.00    0.00    0.00    0.51    0.00    0.00    0.00   99.49
07:18:37 PM   25    0.00    0.00    0.00    0.00    0.00    2.97    0.00    0.00    0.00   97.03
07:18:37 PM   26    0.00    0.00    0.00    0.00    0.00    1.99    0.00    0.00    0.00   98.01
07:18:37 PM   27    0.00    0.00    0.00    0.00    0.00    0.51    0.00    0.00    0.00   99.49
07:18:37 PM   28    0.00    0.00    0.00    0.00    0.00    1.51    0.00    0.00    0.00   98.49
07:18:37 PM   29    0.00    0.00    0.00    0.00    0.00    3.45    0.00    0.00    0.00   96.55
07:18:37 PM   30    0.00    0.00    0.00    0.00    0.00    1.50    0.00    0.00    0.00   98.50
07:18:37 PM   31    0.00    0.00    0.00    0.00    0.00    2.97    0.00    0.00    0.00   97.03

優(yōu)化建議和效果演示

  1. 查看網(wǎng)卡支持多隊列的情況，ethtool -l eth0
  2. 開啟 irqbalance 服務(wù)，讓系統(tǒng)自動調(diào)整網(wǎng)絡(luò)中斷在多個 CPU 核上的分配, systemctl start irqbalance
  3. 開啟 RPS/RFS 特性，軟件方式實現(xiàn) CPU 均衡，接收包中斷的優(yōu)化

查看RPS/RFS優(yōu)化前后變化

# 優(yōu)化前
[root@localhost wangao]# service rps status
[em1] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[em1] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[em1] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[em1] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-16 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-17 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-18 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-19 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-20 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-21 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-22 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-23 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-24 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-25 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-26 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-27 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-28 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-29 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-30 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-31 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p1] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-16 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-17 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-18 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-19 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-20 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-21 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-22 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-23 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-24 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-25 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-26 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-27 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-28 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-29 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-30 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-31 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[p1p2] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-0 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-1 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-10 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-11 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-12 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-13 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-14 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-15 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-2 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-3 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-4 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-5 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-6 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-7 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-8 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
[bond0] rx-9 --> rps_cpus = 00000000,00000000,00000000,00000000,00000000,00000000 , rps_flow_cnt = 0
rps_sock_flow_entries = 0

# 優(yōu)化后
[root@localhost wangao]# service rps status
[em1] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[em1] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[em1] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[em1] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 8192
[p1p1] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-16 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-17 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-18 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-19 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-20 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-21 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-22 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-23 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-24 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-25 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-26 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-27 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-28 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-29 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-30 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-31 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p1] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-16 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-17 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-18 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-19 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-20 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-21 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-22 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-23 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-24 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-25 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-26 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-27 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-28 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-29 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-30 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-31 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[p1p2] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 1024
[bond0] rx-0 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-1 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-10 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-11 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-12 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-13 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-14 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-15 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-2 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-3 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-4 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-5 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-6 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-7 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-8 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
[bond0] rx-9 --> rps_cpus = 0000,00000000,00000000,00000000,ffffffff , rps_flow_cnt = 2048
rps_sock_flow_entries = 32768

查看軟中斷變化

# 優(yōu)化前
[root@localhost wangao]# egrep 'cpu|p1p1' /proc/interrupts
  68:   14737707          0       7843          0     571920          0     265157          0          0          0       1610          0      15028          0     314078          0      54904          0     449958          0     126160          0     338804          0     668880          0     122459          0     174803          0      26147          0  IR-PCI-MSI-edge      p1p1-TxRx-0
  69:      56960          0     115829          0    2669071          0      61425          0      25161          0    4573083          0      72772          0      30975          0      61705          0     189408          0      13134          0     212159          0      49345          0     104657          0    1475007          0      82145          0  IR-PCI-MSI-edge      p1p1-TxRx-1
  70:      73053          0     118657          0      95018          0    8645414          0      60962          0      83141          0      53948          0       7465          0      99615          0     119273          0      26191          0      14160          0      45703          0     335306          0    4014649          0      94692          0  IR-PCI-MSI-edge      p1p1-TxRx-2
  71:      46293          0      75419          0      57967          0       6088          0      44337          0      88437          0      22859          0      51370          0      45510          0     120119          0      53276          0    1105728          0      92556          0    7627687          0     374344          0      79394          0  IR-PCI-MSI-edge      p1p1-TxRx-3
  74:      55261          0     738486          0      14860          0      76202          0     173074          0     215494          0    6385081          0    2691319          0     167374          0      60585          0      17804          0      63837          0    1267656          0      25655          0      94509          0     115839          0  IR-PCI-MSI-edge      p1p1-TxRx-4
  75:      59182          0      58481          0      89362          0      49916          0      53663          0      40476          0      81847          0      48582          0    7661643          0      74951          0      81414          0      37247          0      58610          0     244719          0    1449948          0      53745          0  IR-PCI-MSI-edge      p1p1-TxRx-5
  76:      46993          0     290826          0      58661          0      31146          0     112260          0      51531          0     124002          0      47309          0      74370          0    5474439          0    8948212          0      97511          0     184151          0     141408          0      60222          0      19645          0  IR-PCI-MSI-edge      p1p1-TxRx-6
  77:      44716          0      30601          0     111528          0      61197          0      29883          0    1262486          0      44401          0      19157          0      55785          0      80069          0     143110          0      67065          0    8581759          0      75214          0     201489          0     116415          0  IR-PCI-MSI-edge      p1p1-TxRx-7
  78:      68542          0      46913          0      60848          0      61158          0      66591          0    8619198          0    1284598          0      68180          0      88156          0    3904413          0      32320          0     158036          0      74480          0      36877          0      91269          0      66078          0  IR-PCI-MSI-edge      p1p1-TxRx-8
  79:      45256          0      70914          0      57025          0      58231          0      70398          0      24195          0      29084          0      55756          0    8559005          0      91255          0     105623          0      66336          0      34652          0     161266          0    1555160          0      63201          0  IR-PCI-MSI-edge      p1p1-TxRx-9
  80:      81984          0      71120          0    8853871          0    4829736          0      38789          0      66915          0      37601          0      55277          0      52352          0     226681          0      64967          0      68163          0     100074          0      48442          0     106219          0     120024          0  IR-PCI-MSI-edge      p1p1-TxRx-10
  81:      69573          0      55072          0      98255          0      49468          0   11084926          0      50300          0      67870          0      91907          0      61667          0      69887          0      83711          0      80970          0      87310          0      47258          0    1438136          0      80672          0  IR-PCI-MSI-edge      p1p1-TxRx-11
  82:      41874          0     141338          0      40619          0    9095415          0    2744017          0      63918          0     203991          0     314563          0    1205451          0     124044          0     112602          0     308959          0      43454          0     128518          0      46659          0      80356          0  IR-PCI-MSI-edge      p1p1-TxRx-12
  83:      33080          0    8793595          0      53807          0      46637          0      62925          0      54755          0      72271          0      60842          0      93343          0      47785          0      82287          0      73409          0     745000          0     832425          0     102907          0      90783          0  IR-PCI-MSI-edge      p1p1-TxRx-13
  84:      52045          0      52518          0      83353          0      64534          0      32414          0      35972          0      45190          0      36925          0      24817          0      79178          0    1443964          0    3712758          0    9480150          0     109740          0      82856          0      58387          0  IR-PCI-MSI-edge      p1p1-TxRx-14
  85:      63117          0      22871          0     220711          0      75206          0      19498          0      70962          0      52117          0      97241          0     123345          0      67058          0     115891          0    7140767          0    1527076          0     230527          0      52956          0      36916          0  IR-PCI-MSI-edge      p1p1-TxRx-15
  86:     341182          0      25306          0     436494          0      79810          0     336125          0      24769          0    1015036          0      56332          0     996708          0      46704          0      44728          0     171661          0      73625          0     112345          0     595442          0    1249696          0  IR-PCI-MSI-edge      p1p1-TxRx-16
  87:      27329          0      55574          0     348849          0      43122          0     333589          0      68221          0     146119          0     264597          0      42572          0      84245          0     217853          0     446305          0      47318          0     202998          0     423085          0     961201          0  IR-PCI-MSI-edge      p1p1-TxRx-17
  88:     270648          0      46600          0     217253          0      31500          0      38469          0    3095476          0      84290          0      52591          0      26967          0      34871          0      38145          0      87283          0      67059          0     270541          0     210215          0    3993171          0  IR-PCI-MSI-edge      p1p1-TxRx-18
  89:     148447          0      48653          0     326406          0      53179          0     123863          0      66748          0     391198          0     159050          0      98765          0      54322          0     141506          0     173276          0    1397478          0     133225          0      78736          0     119593          0  IR-PCI-MSI-edge      p1p1-TxRx-19
  90:      56503          0     135502          0     687704          0     380045          0     469732          0      37274          0     281894          0    1552891          0     121443          0      47009          0     173177          0     650010          0     104931          0     131578          0     217383          0     633766          0  IR-PCI-MSI-edge      p1p1-TxRx-20
  91:     214823          0      61146          0     339177          0      49928          0     327002          0      64010          0     318068          0     541504          0      45756          0      20992          0     176829          0      57846          0     148528          0    1162846          0      94836          0     433683          0  IR-PCI-MSI-edge      p1p1-TxRx-21
  92:     106435          0      26816          0     555204          0      31009          0     391325          0     125873          0     290335          0     567300          0      40470          0     341570          0      61671          0     192984          0     221575          0    3245714          0      80733          0     596397          0  IR-PCI-MSI-edge      p1p1-TxRx-22
  93:     115951          0      70761          0     381334          0     143441          0     872170          0     118644          0     517816          0      60193          0     100746          0      41460          0      70866          0     102305          0     134298          0      79193          0      79264          0    1835037          0  IR-PCI-MSI-edge      p1p1-TxRx-23
  94:     146101          0      35816          0     733832          0      29613          0     313268          0     202986          0     340662          0     100681          0    1993051          0     335538          0      71931          0      23162          0     246612          0     556509          0     224855          0     559908          0  IR-PCI-MSI-edge      p1p1-TxRx-24
  95:     150668          0      36783          0     251091          0      36610          0     452604          0     118424          0     197907          0     471898          0     916471          0      25373          0      43497          0     170329          0     504482          0     504129          0      61751          0     349911          0  IR-PCI-MSI-edge      p1p1-TxRx-25
  96:     175244          0      85547          0     149382          0      25817          0      27945          0      55507          0     297689          0    1877839          0      72682          0     294648          0      34613          0      54712          0     184015          0    5097111          0      62440          0     893488          0  IR-PCI-MSI-edge      p1p1-TxRx-26
  97:     188255          0      79222          0     196770          0      93331          0      62542          0     197921          0     499815          0    1489952          0     685652          0     127194          0      39956          0      48082          0      46078          0     444262          0      89668          0     197128          0  IR-PCI-MSI-edge      p1p1-TxRx-27
  98:     113119          0      44914          0     418302          0      17345          0      51941          0     187466          0      37948          0     572466          0      54668          0      27553          0     580539          0     366022          0    2375605          0      49572          0      73136          0    1132828          0  IR-PCI-MSI-edge      p1p1-TxRx-28
  99:      94404          0      56968          0     407066          0      45791          0     302591          0     106122          0     261980          0     583169          0      32802          0      50997          0      47888          0      99704          0     129685          0     315057          0      38119          0    1544062          0  IR-PCI-MSI-edge      p1p1-TxRx-29
 100:      98958          0     178434          0     354708          0     111329          0     159026          0     156721          0     224555          0     188416          0     224497          0      30343          0     497511          0     121221          0     280827          0     354135          0      84543          0    1650501          0  IR-PCI-MSI-edge      p1p1-TxRx-30
 101:     199414          0      34177          0     506856          0      92717          0     246991          0     118922          0     123314          0     261694          0     145339          0     730269          0     397150          0      26237          0     541393          0     357194          0     174133          0     520104          0  IR-PCI-MSI-edge      p1p1-TxRx-31
 102:          5          0          0          0          0          0        130          0          0          0          4          0          5          0          2          0          1          0          0          0          2          0          0          0          0          0          0          0          0          0         57          0  IR-PCI-MSI-edge      p1p1

# 優(yōu)化后
[root@localhost wangao]# egrep 'cpu|enp175s0f0' /proc/interrupts
 545:          0          0          0          0          0          0          0          0   78728441   66114965   52520832   41158901   65698825   55803106   20888551   23716200          0          0          0          0          0          0          0          0   41296940   19654307   42367407   68035795   31971316   73083489  109360661   60151057  IR-PCI-MSI-edge      enp175s0f0-TxRx-0
 546:          0          0          0          0          0          0          0          0  293334701   57784003   57947637  279907263   83003310   59328994   14582717   10594077          0          0          0          0          0          0          0          0   33844346   53425312  104463040  121835272   48115201   11875829  149517303   27888367  IR-PCI-MSI-edge      enp175s0f0-TxRx-1
 547:          0          0          0          0          0          0          0          0   65087834   74714070   70617455   50729188   50629957   53222216   29320338   43600286          0          0          0          0          0          0          0          0   78790098   21664768   53133919  104396382   28219169   98109898  106275780    6075631  IR-PCI-MSI-edge      enp175s0f0-TxRx-2
 548:          0          0          0          0          0          0          0          0  108290552   62484073   55417923   64030173   55771760   74127972   63180417   62512564          0          0          0          0          0          0          0          0  127950399   63170356   20177522  106969671   21966002  107295156  138414635    3139534  IR-PCI-MSI-edge      enp175s0f0-TxRx-3
 549:          0          0          0          0          0          0          0          0  175701987  134359768  118283034   93078181   56579608   43699849   83211249   72178065          0          0          0          0          0          0          0          0   67170073   21723454   56826677  103798934   14026848   39506854  136483492   11700838  IR-PCI-MSI-edge      enp175s0f0-TxRx-4
 550:          0          0          0          0          0          0          0          0  161298050   70517371   30581739  186350377  170422634  226218127   55471402   95255735          0          0          0          0          0          0          0          0  158450530   36931783   14121926   68432292   26086904    5561980   46129837      10039  IR-PCI-MSI-edge      enp175s0f0-TxRx-5
 551:          0          0          0          0          0          0          0          0   59810450   85668650   74748733   66956510   53509915   67475372   37312571   27187643          0          0          0          0          0          0          0          0   42998302   18434014   26932131   54190593   16723043   46515600   59603049   11897912  IR-PCI-MSI-edge      enp175s0f0-TxRx-6
 552:          0          0          0          0          0          0          0          0   37068253  137459908  126379700  132069647   44577185   92132779   18951865   48826631          0          0          0          0          0          0          0          0  168471567   18399908    9609939  156723119   45300251   47356473   71094409   13049049  IR-PCI-MSI-edge      enp175s0f0-TxRx-7
 553:          0          0          0          0          0          0          0          0    9148701   15294255  337628836   28379485   21729246   18839950    5383431    5064436          0          0          0          0          0          0          0          0   45773035    6898143    3359206    4497486  569915849    5649581  659950071  561102436  IR-PCI-MSI-edge      enp175s0f0-TxRx-8
 554:          0          0          0          0          0          0          0          0   45919755  648235162   90862075   82511196  121536855  775131101    5059996    2245523          0          0          0          0          0          0          0          0   39913983  123180793    4250458  117911846   13217963    4291001   72759615   11901382  IR-PCI-MSI-edge      enp175s0f0-TxRx-9
 555:          0          0          0          0          0          0          0          0   17252329   12804726  202960565  226950817  436490841   62135664  286074659    2405684          0          0          0          0          0          0          0          0    1937801    2774288  198336045   26598791    3834288  620409019    7956742          0  IR-PCI-MSI-edge      enp175s0f0-TxRx-10
 556:          0          0          0          0          0          0          0          0    2543207    1589549  378184536    5537639    2432570   10278492  370935530    2557214          0          0          0          0          0          0          0          0    2084807    2817750    3214825    3352447 1633147902    3067434    3704468          0  IR-PCI-MSI-edge      enp175s0f0-TxRx-11
 557:          0          0          0          0          0          0          0          0   35578276  274609699  253038113    4759061   62653759    2988138   24435129    2729652          0          0          0          0          0          0          0          0 1276874078    3470696  522229642    4623985    3795035    3779577    4484373      26368  IR-PCI-MSI-edge      enp175s0f0-TxRx-12
 558:          0          0          0          0          0          0          0          0    3282953    3151370   13102970   15841447    2290018  191758517  253118172    3139260          0          0          0          0          0          0          0          0    2245766    6169610 1545045398   30596340    4355187  163581051    4176977          0  IR-PCI-MSI-edge      enp175s0f0-TxRx-13
 559:          0          0          0          0          0          0          0          0   88746890    2665364   81746473  477713921    4323711   87193204   74776878    3650446          0          0          0          0          0          0          0          0    2540382  403859013    3537468  997197460  189897196    4592593    5480200          0  IR-PCI-MSI-edge      enp175s0f0-TxRx-14
 560:          0          0          0          0          0          0          0          0    6646487    5048477    4430038    8497796   18754540  225111260  231266854 1499236783          0          0          0          0          0          0          0          0  138682422    4968757   27628949   22916070    3702479   28557972   20003328          0  IR-PCI-MSI-edge      enp175s0f0-TxRx-15
 561:          0          0          0          0          0          0          0          0      60867      79129      72520      56561      47863      58682      37248      30413          0          0          0          0          0          0          0          0      31560      22821      36684      41662      23009      40489      40755   10139346  IR-PCI-MSI-edge      enp175s0f0-TxRx-16
 562:          0          0          0          0          0          0          0          0     704280    2485475    1162836     815515     762812     681074     684065     578558          0          0          0          0          0          0          0          0     384355     706857     392981     350292     339121     403073     201587      19983  IR-PCI-MSI-edge      enp175s0f0-TxRx-17
 563:          0          0          0          0          0          0          0          0     629722    2106933    1485145    1179751     579723     864401     771639     621684          0          0          0          0          0          0          0          0     357663     563356     465863     413412     351007     341288     208065      25611  IR-PCI-MSI-edge      enp175s0f0-TxRx-18
 564:          0          0          0          0          0          0          0          0     823889     835158     898125    1042075     479060     649698     488703     431781          0          0          0          0          0          0          0          0    3198859     368537     361450     268475     279292     335527     175329      21693  IR-PCI-MSI-edge      enp175s0f0-TxRx-19
 565:          0          0          0          0          0          0          0          0     676732    1450852    1388628     704602     644689     423264     404306     365596          0          0          0          0          0          0          0          0    3038526     391864     405727     263275     191222     248226     136003      21112  IR-PCI-MSI-edge      enp175s0f0-TxRx-20
 566:          0          0          0          0          0          0          0          0     566595    1175664    2031641     958319     714165     319767     349309     318957          0          0          0          0          0          0          0          0    2958683     218495     239817     214993     179935     191555     106269      22241  IR-PCI-MSI-edge      enp175s0f0-TxRx-21
 567:          0          0          0          0          0          0          0          0     832674    1176655    1652999     789321     736090     853359     748998     577413          0          0          0          0          0          0          0          0     498610     570634     823144     451361     350612     336046     200696      23342  IR-PCI-MSI-edge      enp175s0f0-TxRx-22
 568:          0          0          0          0          0          0          0          0     486851     946241    1375282     525162     582995     609945     696507     440446          0          0          0          0          0          0          0          0    2986687     390852     513487     317316     283314     320359     191080      21907  IR-PCI-MSI-edge      enp175s0f0-TxRx-23
 569:          0          0          0          0          0          0          0          0   51211317   79773036   82956204   46567489   42244838   62523416   34798214   31871168          0          0          0          0          0          0          0          0   89233552   34741833   45475248   56079776   44626874  112466556  127468577   26730942  IR-PCI-MSI-edge      enp175s0f0-TxRx-24
 570:          0          0          0          0          0          0          0          0   84554805   62689462  101496218   40518441   34792544  140921853   28422396   29428544          0          0          0          0          0          0          0          0   23860917   73164249   50084153   72011910   24525127  548979217  332764098   50908113  IR-PCI-MSI-edge      enp175s0f0-TxRx-25
 571:          0          0          0          0          0          0          0          0   69909137   55776777   94776131   63355410   42321451   74321693   45528982   50604702          0          0          0          0          0          0          0          0   56618040   31516738   58332108   69357737   20168247   58852410  137885013   28088790  IR-PCI-MSI-edge      enp175s0f0-TxRx-26
 572:          0          0          0          0          0          0          0          0   97305670   87729287   88586433   59983480   55828246   79305722   51064394   44409948          0          0          0          0          0          0          0          0   41248135   60491280   43010548   99696137   32513292   64373823   74114831   49645455  IR-PCI-MSI-edge      enp175s0f0-TxRx-27
 573:          0          0          0          0          0          0          0          0   72108897   76540055   69546704   69542158   46167801   57170600   48419068   58439907          0          0          0          0          0          0          0          0   68912196   27371642   58904860   82050006   79475074  107799031  109654468   81082896  IR-PCI-MSI-edge      enp175s0f0-TxRx-28
 574:          0          0          0          0          0          0          0          0   69014002   80647341   62399685   97216594   87591440  103529815   72314579   45731155          0          0          0          0          0          0          0          0   89557655   50919154   68754628  109583064   33891553   76526249  188283182   29596358  IR-PCI-MSI-edge      enp175s0f0-TxRx-29
 575:          0          0          0          0          0          0          0          0   70240109   84468547   61070305   57381753  122569467   76404910   51871561   47071056          0          0          0          0          0          0          0          0   97676445   22451436   44986642   93854970   51734947  141860090   97976950   63786565  IR-PCI-MSI-edge      enp175s0f0-TxRx-30
 576:          0          0          0          0          0          0          0          0   66395628   99188539   63355633   82424649  110498163  119343763  129927329   49811047          0          0          0          0          0          0          0          0   76096869   78511976   32199289  122850446  108523248   49641666  346741057  101551776  IR-PCI-MSI-edge      enp175s0f0-TxRx-31
 577:          0          0          0          0          0          0          0          0        307        422        349        309        268        317        222        167          0          0          0          0          0          0          0          0        139        126        215        193         95        182        231       7349  IR-PCI-MSI-edge      enp175s0f0

[圖片上傳失敗...(image-e35a2f-1562317749637)]


image

[圖片上傳失敗...(image-b5dd50-1562317749637)]

參考文章

Redis 高負載下的中斷優(yōu)化
網(wǎng)卡軟中斷過高問題優(yōu)化總結(jié)
Linux 網(wǎng)絡(luò)協(xié)議棧收消息過程 - TCP Protocol Layer
Monitoring and Tuning the Linux Networking Stack: Receiving Data
Performance Tuning Guide

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容