情景再現(xiàn)

寫了一個前端的小程序，然后需要依托于Nginx才能在服務端部署，為了部署的便利性直接打成了Docker的鏡像，這個鏡像部署到服務器上后發(fā)現(xiàn)Nginx中的 $remote_addr這個獲取到的IP一直是Docker網橋的IP。

配置文件差不多就是正常配置

server {
    listen 32100;

    gzip on;
    gzip_min_length 1k;
    gzip_comp_level 9;
    gzip_types text/plain text/css text/javascript application/json application/javascript ;
    gzip_vary on;
    gzip_disable "MSIE [1-6]\.";

    client_body_buffer_size 200m;
    client_max_body_size 200m;

    proxy_connect_timeout 800s;
    proxy_send_timeout 800s;
    proxy_read_timeout 800s;

    root /app;

    location ~.*\.(html|htm)$ {
        add_header Cache-Control no-cache;
    }

    location ^~/static/ {
        add_header Cache-Control no-cache;
    }

    location / {
        try_files $uri $uri/ /index.html;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header REMOTE-HOST $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

請求后發(fā)現(xiàn)日志全是網關的的IP 158.40.0.1 但是我的IP為 192.168.1.142

158.40.0.1 - - [03/Aug/2023:11:42:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:37 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"

解決過程

排查這個問題有點麻煩，先是換了一臺新的服務器去試，還是有問題，開始懷疑是Nginx的配置寫的有問題，網上各種找，后來在WSL中調試的時候發(fā)現(xiàn)WSL中是好的，才從服務器中下手。網上的那些亂七八糟的解決方案都試完了，又是Host，又是給docker0加入內部網絡，又是加各種些的協(xié)議，亂七八糟的方案，除了Host那個是有效的，其他均沒屌用。

后來認為是服務器內核太老了執(zhí)行了 yum update -y 你猜怎么滴，還是沒屌用。直到我在Github中花了4個小時的仔細研究，在一些Issues中找到了一絲線索。最后定位到就是防火墻的問題(主要是NAT模塊的問題)。

然后將這個玩意開啟之后一切正常

firewall-cmd --zone=public --add-masquerade --permanent
firewall-cmd --reload

開啟之后效果如下

158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
158.40.0.1 - - [03/Aug/2023:11:42:38 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:48 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:49 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:49 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"
192.168.1.142 - - [03/Aug/2023:11:53:49 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" "-"

我以為我已經搞定了，然后依葫蘆畫瓢，在另外一臺服務器上面進行如此操作，你猜怎么著，屌用沒有。
繼續(xù)對比兩臺服務器，發(fā)現(xiàn)了問題所在。

[root@local app]# firewall-cmd --version
0.5.3

防火墻版本兩邊沒對上，一個是執(zhí)行了yum update -y的0.6.3,一個是默認的0.5.3
直接升級防火墻版本解決

yum update firewalld -y

總結

這種情況一般出現(xiàn)在Centos7上面，或者各種Centos7的變體上面(各種國產麒麟系統(tǒng))。
如下兩步，搞定

yum update firewalld -y #升級防火墻版本到0.6.3
firewall-cmd --zone=public --add-masquerade --permanent # 防火墻添加 masquerade 規(guī)則
firewall-cmd --reload # 重啟防火墻

我多測試了一種情況，如果Docker配置文件中的"iptables": false 也是會造成這個問題的。

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "50"
  },
  "bip": "158.40.0.1/24",
  "iptables": false,   // 這玩意是false也會出現(xiàn)，修改為true就對了
  "data-root": "/data/docker"
}