前言
這篇文章主要解釋進(jìn)入docker進(jìn)程的數(shù)據(jù)包是如何經(jīng)過(guò)iptables的4表5鏈最終到達(dá)Docker容器內(nèi)部，以及docker是如何配置iptables實(shí)現(xiàn)路由以及其他的一些功能(如訪問控制等)的。
本文假設(shè)你已經(jīng)知道docker的使用和iptables相關(guān)知識(shí)，這里不再贅述。
先說(shuō)結(jié)論
由于數(shù)據(jù)包進(jìn)入netfilter內(nèi)核有2個(gè)入口(分別是主機(jī)外進(jìn)入和主機(jī)內(nèi)進(jìn)程發(fā)起，參考最經(jīng)典的iptables processing flowchart圖，下面有)，所以我們對(duì)從這2個(gè)入口進(jìn)來(lái)的數(shù)據(jù)包分別說(shuō)明。
主機(jī)外進(jìn)入：
| 順序 | 經(jīng)過(guò)表 | 經(jīng)過(guò)鏈 | 作用 |
|---|---|---|---|
| 1 | nat | prerouting | DNAT,目的地址端口轉(zhuǎn)換為容器ip和端口 |
| 2 | filter | forward | 自定義DOCKER-USER，DOCKER-ISOLATION-STAGE，DOCKER這3條鏈 |
| 3 | nat | postrouting | SNAT,利用MASQUERADE轉(zhuǎn)換源地址 |
主機(jī)內(nèi)進(jìn)入：
| 順序 | 經(jīng)過(guò)表 | 經(jīng)過(guò)鏈 | 作用 |
|---|---|---|---|
| 2 | nat | output | DNAT,目的地址端口轉(zhuǎn)換為容器ip和端口 |
| 3 | nat | postrouting | SNAT,利用MASQUERADE轉(zhuǎn)換源地址 |
其中
DOCKER-USER鏈為Docker定義給用戶用來(lái)添加自定義規(guī)則來(lái)限制訪問策略，這里官網(wǎng)有說(shuō)明
DOCKER-ISOLATION-STAGE鏈用來(lái)實(shí)現(xiàn)Docker多network中的容器互相隔離，不能進(jìn)行互通
DOCKER鏈用來(lái)實(shí)現(xiàn)用戶配置的端口映射策略，如192.168.1.2:80映射到容器80端口
以上數(shù)據(jù)包流轉(zhuǎn)過(guò)程也解釋了為什么如果按往常使用iptables在filter表input鏈配置docker應(yīng)用的訪問控制的話會(huì)不起作用，因?yàn)樵L問docker應(yīng)用的根本不會(huì)走filter[nat]，而且2個(gè)入口都不會(huì)走！
可以看到docker做的無(wú)非就是設(shè)置轉(zhuǎn)換地址之類的規(guī)則，因?yàn)镈ocker在這之前已經(jīng)為每個(gè)容器網(wǎng)絡(luò)建立了獨(dú)立的網(wǎng)橋，配置了路由等等，不過(guò)這里篇幅有限就不討論這些了。下面我們就進(jìn)行實(shí)驗(yàn)，看如何得出這些結(jié)論的。
開始實(shí)驗(yàn)
實(shí)驗(yàn)準(zhǔn)備工作
- 我們?cè)谝慌_(tái)新的機(jī)器(centos)上關(guān)閉防護(hù)墻，安裝好docker，并啟動(dòng)一個(gè)nginx容器，映射為80端口。
systemctl stop firewalld
systemctl start docker
docker run -d -p 80:80 nginx
現(xiàn)在我們查看iptables的nat和filter表，有如下輸出：
[root@localhost ~]# iptables -nvL -t nat
Chain PREROUTING (policy ACCEPT 2 packets, 289 bytes)
pkts bytes target prot opt in out source destination
7 508 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 2 packets, 289 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 34 packets, 2399 bytes)
pkts bytes target prot opt in out source destination
4 240 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 42 packets, 2879 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
0 0 MASQUERADE tcp -- * * 172.17.0.2 172.17.0.2 tcp dpt:80
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
8 480 DNAT tcp -- !docker0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.17.0.2:80
[root@localhost ~]# iptables -nvL -t filter
Chain INPUT (policy ACCEPT 4553 packets, 902K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
44 4498 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
44 4498 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
20 2105 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 240 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
20 2153 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 6081 packets, 6094K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
4 240 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
20 2153 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
44 4498 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
20 2153 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
44 4498 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
可以看到，docker分別在nat表的prerouting output postrouting這3個(gè)鏈和filter表的forward鏈添加了內(nèi)容，后面的實(shí)驗(yàn)我們只需要關(guān)注這幾條鏈即可。
- 要想看到數(shù)據(jù)包流經(jīng)了哪些表和鏈，還需要啟用
ipt_LOG內(nèi)核模塊來(lái)追蹤數(shù)據(jù)包并打印日志，使用以下命令來(lái)跟蹤到達(dá)本機(jī)80端口的數(shù)據(jù)包
modprobe nf_log_ipv4
sysctl net.netfilter.nf_log.2=nf_log_ipv4
iptables -t raw -A PREROUTING -p tcp --dport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --dport 80 -j TRACE
-
同時(shí)打開最經(jīng)典的iptables數(shù)據(jù)流圖參考
netfilter數(shù)據(jù)流圖
再打開一個(gè)終端查看日志tail -n0 -f /var/log/messages
嗯，至此準(zhǔn)備工作就完成了。
分析數(shù)據(jù)包
還是分2種情況，數(shù)據(jù)包從主機(jī)外部進(jìn)入和數(shù)據(jù)包從主機(jī)內(nèi)進(jìn)程發(fā)出(當(dāng)然都是發(fā)給容器的)
直接在本機(jī)之外的機(jī)器打一個(gè)請(qǐng)求過(guò)來(lái)
curl http://192.168.233.147/
立即查看剛剛打開終端的日志/var/log/messages
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:PREROUTING:rule:1 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:DOCKER:rule:2 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:4 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: nat:POSTROUTING:policy:3 IN= OUT=docker0 SRC=192.168.233.1 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=42847 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534489 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B4010303080402080A0B14495E00000000)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42848 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=130 TOS=0x00 PREC=0x00 TTL=64 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=130 TOS=0x00 PREC=0x00 TTL=63 ID=42850 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534490 ACK=3232349856 WINDOW=514 RES=0x00 ACK PSH URGP=0 OPT (0101080A0B144960013BBE96)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42851 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42853 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534568 ACK=3232350709 WINDOW=511 RES=0x00 ACK FIN URGP=0 OPT (0101080A0B144965013BBE99)
Mar 16 00:11:29 localhost kernel: TRACE: raw:PREROUTING:policy:3 IN=ens33 OUT= MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-USER:return:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:2 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: filter:FORWARD:rule:3 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
Mar 16 00:11:29 localhost kernel: TRACE: security:FORWARD:policy:1 IN=ens33 OUT=docker0 MAC=00:0c:29:9a:60:a6:00:50:56:c0:00:08:08:00 SRC=192.168.233.1 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=42854 DF PROTO=TCP SPT=49395 DPT=80 SEQ=1916534569 ACK=3232350710 WINDOW=511 RES=0x00 ACK URGP=0 OPT (0101080A0B144966013BBE9C)
有很多重復(fù)，應(yīng)該是一個(gè)http請(qǐng)求通過(guò)tcp連接傳輸多個(gè)包，每個(gè)包都需要走一次iptables，我們只看前幾個(gè)不重復(fù)的即可。
可以看到，數(shù)據(jù)包經(jīng)過(guò)的順序?yàn)?/p>
raw:PREROUTING -> nat:PREROUTING -> filter:FORWARD -> security:FORWARD -> nat:POSTROUTING
其余帶Docker字樣的鏈為各個(gè)鏈中的自定義鏈，看到這里在對(duì)比數(shù)據(jù)流圖，才發(fā)現(xiàn)這張圖真的誠(chéng)不欺我，數(shù)據(jù)包流向和圖中一模一樣(終于實(shí)踐驗(yàn)證理論)。
圖上大概長(zhǎng)這樣，可以看到是走轉(zhuǎn)發(fā)這條路，并不是直接到達(dá)本地進(jìn)程，這也是為什么安裝Docker時(shí)，要啟用內(nèi)核數(shù)據(jù)包轉(zhuǎn)發(fā)！
而且一般在filter:INPUT鏈添加安全策略的做法對(duì)docker也是不生效的，因?yàn)樵L問容器的數(shù)據(jù)包根本不走filter:INPUT。
image.png
那本機(jī)進(jìn)程發(fā)出的數(shù)據(jù)包走向如何呢？
直接在本機(jī)打一個(gè)請(qǐng)求過(guò)來(lái)測(cè)試
curl http://192.168.233.147/
查看日志
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:OUTPUT:rule:1 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:DOCKER:rule:2 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: nat:POSTROUTING:policy:3 IN= OUT=docker0 SRC=192.168.233.147 DST=172.17.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49913 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932905 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A016E02D00000000001030307) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49914 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=131 TOS=0x00 PREC=0x00 TTL=64 ID=49915 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932906 ACK=2741273735 WINDOW=342 RES=0x00 ACK PSH URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49916 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741273973 WINDOW=350 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49917 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D0016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49918 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932985 ACK=2741274588 WINDOW=360 RES=0x00 ACK FIN URGP=0 OPT (0101080A016E02D4016E02D0) UID=0 GID=0
Mar 16 01:06:23 localhost kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=192.168.233.147 DST=192.168.233.147 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
Mar 16 01:06:23 localhost kernel: TRACE: filter:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
Mar 16 01:06:23 localhost kernel: TRACE: security:OUTPUT:policy:1 IN= OUT=lo SRC=192.168.233.147 DST=172.17.0.2 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=49919 DF PROTO=TCP SPT=60998 DPT=80 SEQ=1424932986 ACK=2741274589 WINDOW=360 RES=0x00 ACK URGP=0 OPT (0101080A016E02D4016E02D4)
可以看到，數(shù)據(jù)包經(jīng)過(guò)的順序?yàn)?/p>
raw:OUTPUT -> nat:OUTPUT -> filter:OUTPUT -> security:OUTPUT
圖上大概長(zhǎng)這樣
image.png
在
nat:OUTPUT鏈做了DNAT和nat:POSTROUTING鏈做了SNAT至此，數(shù)據(jù)包流向就分析完了。
做完實(shí)驗(yàn)收獲最大的應(yīng)該還是如何跟蹤數(shù)據(jù)包走向，畢竟有了這項(xiàng)技能，網(wǎng)絡(luò)不通也可以debug找原因了。
感謝互聯(lián)網(wǎng)的前輩們，他們?cè)缫寻芽硬绕?，我們只需要找到他們的腳印。
參考
https://backreference.org/2010/06/11/iptables-debugging/
https://docs.docker.com/network/packet-filtering-firewalls/#add-iptables-policies-before-dockers-rules
