問題概述
本文主要為了解決,在使用Docker部署Spring Boot應(yīng)用,Spring Boot在向Eureka注冊時(shí),如何配置正確IP的問題。
解決方案配置
先把最終解決方案的配置貼出來:
- Gitlab Runners,該機(jī)器的網(wǎng)卡和IP信息如下??梢钥吹奖緳C(jī)eth0網(wǎng)卡的IP為
10.16.180.7,docker0網(wǎng)卡的IP為172.17.0.1
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
ether 02:42:50:e1:74:da txqueuelen 0 (Ethernet)
RX packets 114151657 bytes 17800379440 (16.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 217089263 bytes 288568000281 (268.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker_gwbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.18.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
ether 02:42:99:31:8f:ad txqueuelen 0 (Ethernet)
RX packets 182761 bytes 11494613 (10.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 139451 bytes 9795778 (9.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.16.180.7 netmask 255.255.240.0 broadcast 10.16.191.255
ether fa:16:3e:0e:3d:fd txqueuelen 1000 (Ethernet)
RX packets 442851873 bytes 307687290719 (286.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 356404292 bytes 51936607569 (48.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 0 (Local Loopback)
RX packets 649114 bytes 47094870 (44.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 649114 bytes 47094870 (44.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth0133ad0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 96:6f:6d:89:29:7b txqueuelen 0 (Ethernet)
RX packets 945694 bytes 2541883907 (2.3 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1978551 bytes 2484265627 (2.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
...
在注冊Gitlab Runner時(shí),Description填上該機(jī)器的IP地址:10.16.180.7

- 采用Gitlab CI發(fā)布Spring Boot應(yīng)用,.gitlab-ci.yml文件如下:
image: docker-registry.qiyi.virtual/ads-bi/docker:dind
stages:
- build
- deploy
cache:
paths:
- .m2
job-build:
image: docker-registry.qiyi.virtual/ads-bi/maven:3.5-jdk-8-slim
tags:
- test-env
stage: build
script:
- source ~/.bashrc && mvn -Dmaven.repo.local=.m2 -Duser.timezone=GMT+08 clean install test sonar:sonar
artifacts:
name: "$CI_PROJECT_NAME"
paths:
- target/*.jar
job-deploy:
image: docker-registry.qiyi.virtual/ads-bi/docker:dind
environment: production
tags:
- test-env
stage: deploy
before_script:
- SERVER_PORT=`awk -v min=10000 -v max=20000 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'`
script:
- docker build -t docker-registry.qiyi.virtual/ads-bi/${CI_PROJECT_NAME}:prod
--build-arg spring_profiles_active=prod
--build-arg server_port=${SERVER_PORT}
--build-arg spring_application_name=${CI_PROJECT_NAME}
--build-arg ip_address=${CI_RUNNER_DESCRIPTION}
.
- docker push docker-registry.qiyi.virtual/ads-bi/${CI_PROJECT_NAME}:prod
- sh ./scripts/start_job.sh ${CI_PROJECT_NAME} prod ${SERVER_PORT}
- docker rmi --force $(docker images | grep " <none>" | awk '{print $3}') || true
- Spring Boot的啟動(dòng)shell腳本
#! /bin/bash
CI_PROJECT_NAME=$1
ENV=$2
SERVER_PORT=$3
TIMESTAMP=`date +%s`
CID=`docker ps | grep ${CI_PROJECT_NAME} | awk '{print $1}'`
if [ ! -z ${CID} ]; then
docker stop ${CID} | xargs docker rm
fi
docker run -d -p ${SERVER_PORT}:${SERVER_PORT} -v /var/log:/var/log --name ${CI_PROJECT_NAME}-${TIMESTAMP} docker-registry.qiyi.virtual/ads-bi/${CI_PROJECT_NAME}:${ENV}
- Dockerfile
FROM docker-registry.qiyi.virtual/ads-bi/maven:3.5-jdk-8-slim
VOLUME /tmp
ADD target/*.jar /app.jar
ARG spring_profiles_active
ARG server_port
ARG spring_application_name
ARG ip_address
ENV SPRING_PROFILES_ACTIVE=${spring_profiles_active}
ENV SERVER_PORT=${server_port}
ENV SPRING_APPLICATION_NAME=${spring_application_name}
ENV EUREKA_INSTANCE_IP-ADDRESS=${ip_address}
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar"]
詳細(xì)解讀
這里需要先介紹一下我們面臨的環(huán)境。首先,我們配置了一個(gè)Gitlab的Specific Runner,作為Spring Boot構(gòu)建的實(shí)際機(jī)器。在這臺(tái)機(jī)器上,需要安裝Docker環(huán)境,那么這臺(tái)機(jī)器就有了至少兩個(gè)IP地址,分別為:
- 本機(jī)網(wǎng)卡eth0,ip地址為10.16.180.7
- Docker網(wǎng)卡docker0,ip地址為172.17.0.1
這里我們最終想在Eureka中注冊的地址,是本機(jī)網(wǎng)卡的地址。
因?yàn)镾pring Cloud Admin是通過Eureka中注冊的地址,來判斷Spring Boot應(yīng)用是否健康的,如果注冊使用的是docker0的地址,這個(gè)地址并不能通過http訪問,Spring Cloud Admin會(huì)認(rèn)為該應(yīng)用是Down的,雖然它實(shí)際的狀態(tài)是UP的。
在網(wǎng)上,我們能找到不少介紹Spring Cloud Eureka的多網(wǎng)卡選擇的文章,例如這篇文章介紹的就很好:
https://blog.csdn.net/xichenguan/article/details/76632033
這里提到了幾個(gè)重要的Spring Cloud配置:
- eureka.instance.prefer-ip-address: true/false
- spring.cloud.inetutils.preferred-networks: list<ip>
- spring.cloud.inetutils.ignored-interfaces: list<ip>
關(guān)于這幾個(gè)配置項(xiàng)的含義,可以參考Spring Cloud的官方文檔,介紹的更加準(zhǔn)確:
https://cloud.spring.io/spring-cloud-static/spring-cloud.html#ignore-network-interfaces
這幾個(gè)配置的目的,就是Spring Boot在啟動(dòng)時(shí),會(huì)在多網(wǎng)卡的環(huán)境,根據(jù)配置項(xiàng)的信息,進(jìn)行網(wǎng)卡的選擇。找到這里,我們認(rèn)為可能通過這項(xiàng)配置就可以選擇到想要的ip地址了。
但實(shí)際并不是這樣的,在經(jīng)過無數(shù)次的嘗試之后,我們發(fā)現(xiàn)在Eureka中每次注冊的ip地址始終都是:

這個(gè)問題我們使用百思不得其解,在翻看這部分的源碼時(shí),我們發(fā)現(xiàn)Spring Boot使用的是InetUtils類進(jìn)行ip選擇的。這部分的源碼在遍歷每個(gè)可用的網(wǎng)絡(luò)之后,結(jié)合我們配置的preferred和ignored信息,選擇index最小的這個(gè)ip作為最終的返回ip。
public InetAddress findFirstNonLoopbackAddress() {
InetAddress result = null;
try {
int lowest = Integer.MAX_VALUE;
for (Enumeration < NetworkInterface > nics = NetworkInterface
.getNetworkInterfaces(); nics.hasMoreElements();) {
NetworkInterface ifc = nics.nextElement();
if (ifc.isUp()) {
log.trace("Testing interface: " + ifc.getDisplayName());
if (ifc.getIndex() < lowest || result == null) {
lowest = ifc.getIndex();
} else if (result != null) {
continue;
}
// @formatter:off
if (!ignoreInterface(ifc.getDisplayName())) {
for (Enumeration < InetAddress > addrs = ifc
.getInetAddresses(); addrs.hasMoreElements();) {
InetAddress address = addrs.nextElement();
if (address instanceof Inet4Address &&
!address.isLoopbackAddress() &&
!ignoreAddress(address)) {
log.trace("Found non-loopback interface: " +
ifc.getDisplayName());
result = address;
}
}
}
// @formatter:on
}
}
} catch (IOException ex) {
log.error("Cannot get first non-loopback address", ex);
}
if (result != null) {
return result;
}
try {
return InetAddress.getLocalHost();
} catch (UnknownHostException e) {
log.warn("Unable to retrieve localhost");
}
return null;
}
這里我們發(fā)現(xiàn),可以通過調(diào)整log的打印等級,將選擇過程中trace等級的日志打印出來。在調(diào)整之后,查看服務(wù)器端這部分的日志如下:
2018-05-04 06:43:00.610 [main] DEBUG org.elasticsearch.common.network -configuration:
lo
inet 127.0.0.1 netmask:255.0.0.0 scope:host
inet6 ::1 prefixlen:128 scope:host
UP LOOPBACK mtu:65536 index:1
eth0
inet 172.17.0.5 netmask:255.255.0.0 broadcast:0.0.0.0 scope:site
inet6 fe80::42:acff:fe11:3 prefixlen:64 scope:link
hardware 02:42:AC:11:00:03
UP MULTICAST mtu:1500 index:4899
...
2018-05-04 06:43:02.332 [main] TRACE org.springframework.cloud.commons.util.InetUtils -Testing interface: eth0
2018-05-04 06:43:02.332 [main] TRACE org.springframework.cloud.commons.util.InetUtils -Found non-loopback interface: eth0
2018-05-04 06:43:02.332 [main] TRACE org.springframework.cloud.commons.util.InetUtils -Testing interface: lo
這時(shí)我們才明白,原來我們采用的是dind的這個(gè)image,進(jìn)行Spring Boot Application Docker鏡像的生成,發(fā)布和啟動(dòng)。那么在構(gòu)建時(shí),會(huì)生成一個(gè)dind的container,這個(gè)container的網(wǎng)卡信息如上所示。所以我們配置時(shí),所有可選的網(wǎng)卡并不屬于Gitlab Runner這臺(tái)機(jī)器,而是Docker container的,那么之前關(guān)于多網(wǎng)卡的配置無論如何都是無用的了。
在發(fā)現(xiàn)問題的原因之后,我們發(fā)現(xiàn)這里需要做的,是把Docker所在的主機(jī)的網(wǎng)卡ip信息,傳遞到container中。由于這兩個(gè)環(huán)境是隔離的,所以并不現(xiàn)實(shí)。
那么最終如何才能把Docker所在主機(jī)的ip,順利的傳遞到container中呢,我們利用了Gitlab Runner注冊時(shí)填寫的信息。我們在Gitlab Runner注冊時(shí),將該機(jī)器的ip,填寫到Description中。在.gitlab-ci.yml中,可以利用內(nèi)置參數(shù)${CI_RUNNER_DESCRIPTION},將此ip傳遞到Dockerfile中。在Dockerfile中,通過設(shè)置系統(tǒng)環(huán)境變量的方式,使Spring Boot獲取到這個(gè)環(huán)境變量。
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'servletConfigInitParams'
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'servletContextInitParams'
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'systemProperties'
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'systemEnvironment'
2018-05-04 06:37:34.611 [main] DEBUG o.s.core.env.SystemEnvironmentPropertySource -PropertySource 'systemEnvironment' does not contain property 'eureka.instance.ip-address', but found equivalent 'EUREKA_INSTANCE_IP-ADDRESS'
2018-05-04 06:37:34.611 [main] DEBUG o.s.core.env.PropertySourcesPropertyResolver -Found key 'eureka.instance.ip-address' in PropertySource 'systemEnvironment' with value of type String
這樣就實(shí)現(xiàn)了Spring Boot的ip指定,在Eureka中注冊的我們想要的Gitlab Runner主機(jī)ip,最終結(jié)果圖如下:
Spring Cloud Eureka:


Spring Cloud Admin:

啟示
- 使用Docker部署應(yīng)用時(shí),需要理清楚各個(gè)環(huán)境之間的關(guān)系
- 在遇到難解的問題,通過查看源碼,并且分析打印日志的方式,定位問題原因