Spring Cloud:Eureka的多網(wǎng)卡IP選擇問題

問題概述

本文主要為了解決,在使用Docker部署Spring Boot應(yīng)用,Spring Boot在向Eureka注冊時(shí),如何配置正確IP的問題。

解決方案配置

先把最終解決方案的配置貼出來:

  • Gitlab Runners,該機(jī)器的網(wǎng)卡和IP信息如下??梢钥吹奖緳C(jī)eth0網(wǎng)卡的IP為10.16.180.7,docker0網(wǎng)卡的IP為172.17.0.1
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:50:e1:74:da  txqueuelen 0  (Ethernet)
        RX packets 114151657  bytes 17800379440 (16.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 217089263  bytes 288568000281 (268.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker_gwbridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:99:31:8f:ad  txqueuelen 0  (Ethernet)
        RX packets 182761  bytes 11494613 (10.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 139451  bytes 9795778 (9.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.16.180.7  netmask 255.255.240.0  broadcast 10.16.191.255
        ether fa:16:3e:0e:3d:fd  txqueuelen 1000  (Ethernet)
        RX packets 442851873  bytes 307687290719 (286.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 356404292  bytes 51936607569 (48.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 0  (Local Loopback)
        RX packets 649114  bytes 47094870 (44.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 649114  bytes 47094870 (44.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth0133ad0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 96:6f:6d:89:29:7b  txqueuelen 0  (Ethernet)
        RX packets 945694  bytes 2541883907 (2.3 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1978551  bytes 2484265627 (2.3 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

...

在注冊Gitlab Runner時(shí),Description填上該機(jī)器的IP地址:10.16.180.7

Gitlab Runner配置

  • 采用Gitlab CI發(fā)布Spring Boot應(yīng)用,.gitlab-ci.yml文件如下:
image: docker-registry.qiyi.virtual/ads-bi/docker:dind
stages:
  - build
  - deploy

cache:
  paths:
    - .m2

job-build:
  image: docker-registry.qiyi.virtual/ads-bi/maven:3.5-jdk-8-slim
  tags:
    - test-env
  stage: build
  script:
    - source ~/.bashrc && mvn -Dmaven.repo.local=.m2 -Duser.timezone=GMT+08 clean install test sonar:sonar
  artifacts:
    name: "$CI_PROJECT_NAME"
    paths:
      - target/*.jar

job-deploy:
  image: docker-registry.qiyi.virtual/ads-bi/docker:dind
  environment: production
  tags:
    - test-env
  stage: deploy
  before_script:
    - SERVER_PORT=`awk -v min=10000 -v max=20000 'BEGIN{srand(); print int(min+rand()*(max-min+1))}'`
  script:
    - docker build -t docker-registry.qiyi.virtual/ads-bi/${CI_PROJECT_NAME}:prod
        --build-arg spring_profiles_active=prod
        --build-arg server_port=${SERVER_PORT}
        --build-arg spring_application_name=${CI_PROJECT_NAME}
        --build-arg ip_address=${CI_RUNNER_DESCRIPTION}
        .
    - docker push docker-registry.qiyi.virtual/ads-bi/${CI_PROJECT_NAME}:prod
    - sh ./scripts/start_job.sh ${CI_PROJECT_NAME} prod ${SERVER_PORT}
    - docker rmi --force $(docker images | grep " <none>" | awk '{print $3}') || true
  • Spring Boot的啟動(dòng)shell腳本
#! /bin/bash
CI_PROJECT_NAME=$1
ENV=$2
SERVER_PORT=$3
TIMESTAMP=`date +%s`
CID=`docker ps | grep ${CI_PROJECT_NAME} | awk '{print $1}'`
if [ ! -z ${CID} ]; then
   docker stop ${CID} | xargs docker rm
fi
docker run -d -p ${SERVER_PORT}:${SERVER_PORT} -v /var/log:/var/log --name ${CI_PROJECT_NAME}-${TIMESTAMP} docker-registry.qiyi.virtual/ads-bi/${CI_PROJECT_NAME}:${ENV}

  • Dockerfile
FROM docker-registry.qiyi.virtual/ads-bi/maven:3.5-jdk-8-slim
VOLUME /tmp
ADD target/*.jar /app.jar
ARG spring_profiles_active
ARG server_port
ARG spring_application_name
ARG ip_address
ENV SPRING_PROFILES_ACTIVE=${spring_profiles_active}
ENV SERVER_PORT=${server_port}
ENV SPRING_APPLICATION_NAME=${spring_application_name}
ENV EUREKA_INSTANCE_IP-ADDRESS=${ip_address}
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar", "/app.jar"]

詳細(xì)解讀

這里需要先介紹一下我們面臨的環(huán)境。首先,我們配置了一個(gè)Gitlab的Specific Runner,作為Spring Boot構(gòu)建的實(shí)際機(jī)器。在這臺(tái)機(jī)器上,需要安裝Docker環(huán)境,那么這臺(tái)機(jī)器就有了至少兩個(gè)IP地址,分別為:

  • 本機(jī)網(wǎng)卡eth0,ip地址為10.16.180.7
  • Docker網(wǎng)卡docker0,ip地址為172.17.0.1

這里我們最終想在Eureka中注冊的地址,是本機(jī)網(wǎng)卡的地址。
因?yàn)镾pring Cloud Admin是通過Eureka中注冊的地址,來判斷Spring Boot應(yīng)用是否健康的,如果注冊使用的是docker0的地址,這個(gè)地址并不能通過http訪問,Spring Cloud Admin會(huì)認(rèn)為該應(yīng)用是Down的,雖然它實(shí)際的狀態(tài)是UP的。

在網(wǎng)上,我們能找到不少介紹Spring Cloud Eureka的多網(wǎng)卡選擇的文章,例如這篇文章介紹的就很好:
https://blog.csdn.net/xichenguan/article/details/76632033

這里提到了幾個(gè)重要的Spring Cloud配置:

  • eureka.instance.prefer-ip-address: true/false
  • spring.cloud.inetutils.preferred-networks: list<ip>
  • spring.cloud.inetutils.ignored-interfaces: list<ip>

關(guān)于這幾個(gè)配置項(xiàng)的含義,可以參考Spring Cloud的官方文檔,介紹的更加準(zhǔn)確:
https://cloud.spring.io/spring-cloud-static/spring-cloud.html#ignore-network-interfaces

這幾個(gè)配置的目的,就是Spring Boot在啟動(dòng)時(shí),會(huì)在多網(wǎng)卡的環(huán)境,根據(jù)配置項(xiàng)的信息,進(jìn)行網(wǎng)卡的選擇。找到這里,我們認(rèn)為可能通過這項(xiàng)配置就可以選擇到想要的ip地址了。

但實(shí)際并不是這樣的,在經(jīng)過無數(shù)次的嘗試之后,我們發(fā)現(xiàn)在Eureka中每次注冊的ip地址始終都是:


image.png

這個(gè)問題我們使用百思不得其解,在翻看這部分的源碼時(shí),我們發(fā)現(xiàn)Spring Boot使用的是InetUtils類進(jìn)行ip選擇的。這部分的源碼在遍歷每個(gè)可用的網(wǎng)絡(luò)之后,結(jié)合我們配置的preferred和ignored信息,選擇index最小的這個(gè)ip作為最終的返回ip。

public InetAddress findFirstNonLoopbackAddress() {
    InetAddress result = null;
    try {
        int lowest = Integer.MAX_VALUE;
        for (Enumeration < NetworkInterface > nics = NetworkInterface
            .getNetworkInterfaces(); nics.hasMoreElements();) {
            NetworkInterface ifc = nics.nextElement();
            if (ifc.isUp()) {
                log.trace("Testing interface: " + ifc.getDisplayName());
                if (ifc.getIndex() < lowest || result == null) {
                    lowest = ifc.getIndex();
                } else if (result != null) {
                    continue;
                }

                // @formatter:off
                if (!ignoreInterface(ifc.getDisplayName())) {
                    for (Enumeration < InetAddress > addrs = ifc
                        .getInetAddresses(); addrs.hasMoreElements();) {
                        InetAddress address = addrs.nextElement();
                        if (address instanceof Inet4Address &&
                            !address.isLoopbackAddress() &&
                            !ignoreAddress(address)) {
                            log.trace("Found non-loopback interface: " +
                                ifc.getDisplayName());
                            result = address;
                        }
                    }
                }
                // @formatter:on
            }
        }
    } catch (IOException ex) {
        log.error("Cannot get first non-loopback address", ex);
    }

    if (result != null) {
        return result;
    }

    try {
        return InetAddress.getLocalHost();
    } catch (UnknownHostException e) {
        log.warn("Unable to retrieve localhost");
    }

    return null;
}

這里我們發(fā)現(xiàn),可以通過調(diào)整log的打印等級,將選擇過程中trace等級的日志打印出來。在調(diào)整之后,查看服務(wù)器端這部分的日志如下:

2018-05-04 06:43:00.610 [main] DEBUG org.elasticsearch.common.network -configuration:

lo
        inet 127.0.0.1 netmask:255.0.0.0 scope:host
        inet6 ::1 prefixlen:128 scope:host
        UP LOOPBACK mtu:65536 index:1

eth0
        inet 172.17.0.5 netmask:255.255.0.0 broadcast:0.0.0.0 scope:site
        inet6 fe80::42:acff:fe11:3 prefixlen:64 scope:link
        hardware 02:42:AC:11:00:03
        UP MULTICAST mtu:1500 index:4899

...
2018-05-04 06:43:02.332 [main] TRACE org.springframework.cloud.commons.util.InetUtils -Testing interface: eth0
2018-05-04 06:43:02.332 [main] TRACE org.springframework.cloud.commons.util.InetUtils -Found non-loopback interface: eth0
2018-05-04 06:43:02.332 [main] TRACE org.springframework.cloud.commons.util.InetUtils -Testing interface: lo

這時(shí)我們才明白,原來我們采用的是dind的這個(gè)image,進(jìn)行Spring Boot Application Docker鏡像的生成,發(fā)布和啟動(dòng)。那么在構(gòu)建時(shí),會(huì)生成一個(gè)dind的container,這個(gè)container的網(wǎng)卡信息如上所示。所以我們配置時(shí),所有可選的網(wǎng)卡并不屬于Gitlab Runner這臺(tái)機(jī)器,而是Docker container的,那么之前關(guān)于多網(wǎng)卡的配置無論如何都是無用的了。

在發(fā)現(xiàn)問題的原因之后,我們發(fā)現(xiàn)這里需要做的,是把Docker所在的主機(jī)的網(wǎng)卡ip信息,傳遞到container中。由于這兩個(gè)環(huán)境是隔離的,所以并不現(xiàn)實(shí)。

那么最終如何才能把Docker所在主機(jī)的ip,順利的傳遞到container中呢,我們利用了Gitlab Runner注冊時(shí)填寫的信息。我們在Gitlab Runner注冊時(shí),將該機(jī)器的ip,填寫到Description中。在.gitlab-ci.yml中,可以利用內(nèi)置參數(shù)${CI_RUNNER_DESCRIPTION},將此ip傳遞到Dockerfile中。在Dockerfile中,通過設(shè)置系統(tǒng)環(huán)境變量的方式,使Spring Boot獲取到這個(gè)環(huán)境變量。

2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'servletConfigInitParams'
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'servletContextInitParams'
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'systemProperties'
2018-05-04 06:37:34.611 [main] TRACE o.s.core.env.PropertySourcesPropertyResolver -Searching for key 'eureka.instance.ip-address' in PropertySource 'systemEnvironment'
2018-05-04 06:37:34.611 [main] DEBUG o.s.core.env.SystemEnvironmentPropertySource -PropertySource 'systemEnvironment' does not contain property 'eureka.instance.ip-address', but found equivalent 'EUREKA_INSTANCE_IP-ADDRESS'
2018-05-04 06:37:34.611 [main] DEBUG o.s.core.env.PropertySourcesPropertyResolver -Found key 'eureka.instance.ip-address' in PropertySource 'systemEnvironment' with value of type String

這樣就實(shí)現(xiàn)了Spring Boot的ip指定,在Eureka中注冊的我們想要的Gitlab Runner主機(jī)ip,最終結(jié)果圖如下:
Spring Cloud Eureka:


Spring Cloud Eureka

Spring Cloud Eureka URL

Spring Cloud Admin:


Spring Cloud Admin

啟示

  • 使用Docker部署應(yīng)用時(shí),需要理清楚各個(gè)環(huán)境之間的關(guān)系
  • 在遇到難解的問題,通過查看源碼,并且分析打印日志的方式,定位問題原因
最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡書系信息發(fā)布平臺(tái),僅提供信息存儲(chǔ)服務(wù)。

友情鏈接更多精彩內(nèi)容