1. 前言

轉(zhuǎn)載請(qǐng)說(shuō)明原文出處, 尊重他人勞動(dòng)成果!

源碼位置: https://github.com/nicktming/flannel
分支: tming-v0.10.0 (基于v0.10.0版本)

flannel
1. [docker 網(wǎng)絡(luò)][flannel] 配置安裝測(cè)試
2. [docker 網(wǎng)絡(luò)][flannel] 背后操作
3. [docker 網(wǎng)絡(luò)][flannel] 源碼簡(jiǎn)單分析

前面兩篇文章 [docker 網(wǎng)絡(luò)][flannel] 配置安裝測(cè)試 和 [docker 網(wǎng)絡(luò)][flannel] 背后操作 已經(jīng)測(cè)試了flannel vxlan實(shí)現(xiàn)跨主機(jī)容器之間的訪問(wèn), 本文將從源碼角度簡(jiǎn)單分析flannel是如何實(shí)現(xiàn)以及管理的, 主要追蹤主要邏輯, 不會(huì)涉及特別多的細(xì)節(jié).

所以將沿著Main方法主線進(jìn)行分析.

2. 查找ExternalInterface

對(duì)外出口的設(shè)備

// backend/common.go
type ExternalInterface struct {
    Iface     *net.Interface
    IfaceAddr net.IP
    ExtAddr   net.IP
}

// main.go
func main() {
    ...
    var extIface *backend.ExternalInterface
    var err error
    // Check the default interface only if no interfaces are specified
    if len(opts.iface) == 0 && len(opts.ifaceRegex) == 0 {
        // 因?yàn)闆](méi)有指定iface 所以直接從eth0里面找
        extIface, err = LookupExtIface("", "")
        if err != nil {
            log.Error("Failed to find any valid interface to use: ", err)
            os.Exit(1)
        }
    } else {
        // 按指定iface找
        // Check explicitly specified interfaces
        for _, iface := range opts.iface {
            extIface, err = LookupExtIface(iface, "")
            if err != nil {
                log.Infof("Could not find valid interface matching %s: %s", iface, err)
            }

            if extIface != nil {
                break
            }
        }
        ...
    }
    log.Infof("======>IfaceAddr:%v, ExtAddr:%v, Iface.Name:%s, Iface.Index:%d, Iface.MTU:%d", extIface.IfaceAddr, extIface.ExtAddr,
        extIface.Iface.Name, extIface.Iface.Index, extIface.Iface.MTU)
    ...
}

該段落的主要目的是找到該節(jié)點(diǎn)與外部網(wǎng)絡(luò)通信的設(shè)備是哪個(gè). 都是調(diào)用的同一個(gè)方法LookupExtIface.

如果沒(méi)有指定iface或者ifaceRegex(通配符), 則尋找默認(rèn)設(shè)備eth0.
如果指定了iface就直接按照iface名字去找到該設(shè)備, 比如eth0. 如果沒(méi)有找到則嘗試通過(guò)ifaceRegex(通配符)去找.

運(yùn)行

[root@master flannel]# pwd
/root/go/src/github.com/coreos/flannel
[root@master flannel]# go build .
[root@master flannel]# ./flannel --etcd-endpoints="http://172.21.0.16:2379"
I1103 10:51:59.247205    6879 main.go:480] Determining IP address of default interface
I1103 10:51:59.247563    6879 main.go:493] Using interface with name eth0 and address 172.21.0.16
I1103 10:51:59.247575    6879 main.go:510] Defaulting external address to interface address (172.21.0.16)
I1103 10:51:59.247584    6879 main.go:232] ======>IfaceAddr:172.21.0.16, ExtAddr:172.21.0.16, Iface.Name:eth0, Iface.Index:2, Iface.MTU:1500
...

可以看到找到了默認(rèn)設(shè)備eth0.

[root@master flannel]# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.21.0.16  netmask 255.255.240.0  broadcast 172.21.15.255
        inet6 fe80::5054:ff:fed5:4f7e  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:d5:4f:7e  txqueuelen 1000  (Ethernet)
        RX packets 10578209  bytes 2187448866 (2.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10211912  bytes 2268572896 (2.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@master flannel]# 

3. 子網(wǎng)管理

這個(gè)是已經(jīng)存到etcd中的子網(wǎng)配置信息.

[root@master flannel]# etcdctl get /coreos.com/network/config
{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}
[root@master flannel]# 

創(chuàng)建子網(wǎng)管理

===> main.go
func main() {
...
sm, err := newSubnetManager()
...
}
func newSubnetManager() (subnet.Manager, error) {
    if opts.kubeSubnetMgr {
        return kube.NewSubnetManager(opts.kubeApiUrl, opts.kubeConfigFile)
    }

    cfg := &etcdv2.EtcdConfig{
        Endpoints: strings.Split(opts.etcdEndpoints, ","),
        Keyfile:   opts.etcdKeyfile,
        Certfile:  opts.etcdCertfile,
        CAFile:    opts.etcdCAFile,
        Prefix:    opts.etcdPrefix,
        Username:  opts.etcdUsername,
        Password:  opts.etcdPassword,
    }

    prevSubnet := ReadSubnetFromSubnetFile(opts.subnetFile)
    log.Infof("======>prevSubnet:%s\n", prevSubnet)
    return etcdv2.NewLocalManager(cfg, prevSubnet)
}
// 從/run/flannel/subnet.env中讀子網(wǎng)
func ReadSubnetFromSubnetFile(path string) ip.IP4Net {
    var prevSubnet ip.IP4Net
    if _, err := os.Stat(path); !os.IsNotExist(err) {
        prevSubnetVals, err := godotenv.Read(path)
        if err != nil {
            log.Errorf("Couldn't fetch previous subnet from subnet file at %s: %s", path, err)
        } else if prevSubnetString, ok := prevSubnetVals["FLANNEL_SUBNET"]; ok {
            err = prevSubnet.UnmarshalJSON([]byte(prevSubnetString))
            if err != nil {
                log.Errorf("Couldn't parse previous subnet from subnet file at %s: %s", path, err)
            }
        }
    }
    return prevSubnet
}
===> subnet/etcdv2/local_manager.go
func NewLocalManager(config *EtcdConfig, prevSubnet ip.IP4Net) (Manager, error) {
    r, err := newEtcdSubnetRegistry(config, nil)
    if err != nil {
        return nil, err
    }
    return newLocalManager(r, prevSubnet), nil
}

func newLocalManager(r Registry, prevSubnet ip.IP4Net) Manager {
    return &LocalManager{
        registry:       r,
        previousSubnet: prevSubnet,
    }
}

可以看到LocalManager中有兩個(gè)屬性:

registry: 可以訪問(wèn)etcd數(shù)據(jù)的客戶端.
previousSubnet: 該機(jī)器上上一次分配的子網(wǎng)信息.

運(yùn)行

[root@master flannel]# ./flannel --etcd-endpoints="http://172.21.0.16:2379"
I1103 11:22:18.097587   10865 main.go:480] Determining IP address of default interface
I1103 11:22:18.097773   10865 main.go:493] Using interface with name eth0 and address 172.21.0.16
I1103 11:22:18.097783   10865 main.go:510] Defaulting external address to interface address (172.21.0.16)
I1103 11:22:18.097791   10865 main.go:232] ======>IfaceAddr:172.21.0.16, ExtAddr:172.21.0.16, Iface.Name:eth0, Iface.Index:2, Iface.MTU:1500
I1103 11:22:18.097807   10865 main.go:168] ======>prevSubnet:0.0.0.0/0
...

因?yàn)閯倓傞_(kāi)始的時(shí)候/run/flannel/subnet.env文件可能不存在, 所以表明以前該機(jī)器上沒(méi)有分配過(guò)子網(wǎng), 或者該文件被刪除了. 所以顯示了======>prevSubnet:0.0.0.0/0.

4. 獲得backend

4.1 vxlan注冊(cè)信息

在flannel配置的時(shí)候需要告訴使用什么Type, 前文 [docker 網(wǎng)絡(luò)][flannel] 配置安裝測(cè)試 用的vxlan, 比如還有udp等等, 這些Type在flannel這里就是一個(gè)backend, 每個(gè)backend要做的事情是一樣的, 只是實(shí)現(xiàn)方式不一樣而已.

type Backend interface {
    RegisterNetwork(ctx context.Context, config *subnet.Config) (Network, error)
}
type Network interface {
    Lease() *subnet.Lease
    MTU() int
    Run(ctx context.Context)
}
type BackendCtor func(sm subnet.Manager, ei *ExternalInterface) (Backend, error)

在程序啟動(dòng)的時(shí)候每個(gè)backend都會(huì)去backend manager注冊(cè)自己的信息. 以vxlan為例

===> backend/vxlan/vxlan.go
func init() {
    backend.Register("vxlan", New)
}
const (
    defaultVNI = 1
)

type VXLANBackend struct {
    subnetMgr subnet.Manager
    extIface  *backend.ExternalInterface
}
func New(sm subnet.Manager, extIface *backend.ExternalInterface) (backend.Backend, error) {
    backend := &VXLANBackend{
        subnetMgr: sm,
        extIface:  extIface,
    }
    return backend, nil
}
// backend/manager.go
var constructors = make(map[string]BackendCtor)
func Register(name string, ctor BackendCtor) {
    constructors[name] = ctor
}

可以看到vxlan向backend manager中的constructors注冊(cè)了自己的New方法, 告訴backend manager如何創(chuàng)建一個(gè)vxlan backend.

4.2 獲得backend

既然所有的type在程序啟動(dòng)的時(shí)候都會(huì)注冊(cè)自己的信息, 那如何獲取一個(gè)特定的backend呢? 回頭看看main方法.

func main() {
...
// Fetch the network config (i.e. what backend to use etc..).
    config, err := getConfig(ctx, sm)
    if err == errCanceled {
        wg.Wait()
        os.Exit(0)
    }

    // Create a backend manager then use it to create the backend and register the network with it.
    bm := backend.NewManager(ctx, sm, extIface)
    be, err := bm.GetBackend(config.BackendType)
    if err != nil {
        log.Errorf("Error fetching backend: %s", err)
        cancel()
        wg.Wait()
        os.Exit(1)
    }
...
}

1. config, err := getConfig(ctx, sm)就是從etcd獲取如下的配置信息.

[root@master flannel]# etcdctl get /coreos.com/network/config
{"Network": "10.0.0.0/16", "SubnetLen": 24, "SubnetMin": "10.0.1.0","SubnetMax": "10.0.20.0", "Backend": {"Type": "vxlan"}}
[root@master flannel]# 

2. bm := backend.NewManager(ctx, sm, extIface)生成一個(gè)backend manager, 為什么需要傳入sm和extIface, 所以生成某一個(gè)具體的backend這兩個(gè)參數(shù), 這兩個(gè)參數(shù)是給具體的backend來(lái)操作的. 從此定義從可以看到type BackendCtor func(sm subnet.Manager, ei *ExternalInterface) (Backend, error)其用法.

func NewManager(ctx context.Context, sm subnet.Manager, extIface *ExternalInterface) Manager {
    return &manager{
        ctx:      ctx,
        sm:       sm,
        extIface: extIface,
        active:   make(map[string]Backend),
    }
}

3. be, err := bm.GetBackend(config.BackendType)具體實(shí)現(xiàn)如下, 邏輯非常簡(jiǎn)單, 有該類型的backend就直接返回, 沒(méi)有則用注冊(cè)的New創(chuàng)建一個(gè).

func (bm *manager) GetBackend(backendType string) (Backend, error) {
    bm.mux.Lock()
    defer bm.mux.Unlock()
    betype := strings.ToLower(backendType)
    // see if one is already running
    if be, ok := bm.active[betype]; ok {
        return be, nil
    }
    // first request, need to create and run it
    befunc, ok := constructors[betype]
    if !ok {
        return nil, fmt.Errorf("unknown backend type: %v", betype)
    }
    be, err := befunc(bm.sm, bm.extIface)
    if err != nil {
        return nil, err
    }
    bm.active[betype] = be
    bm.wg.Add(1)
    go func() {
        <-bm.ctx.Done()
        bm.mux.Lock()
        delete(bm.active, betype)
        bm.mux.Unlock()

        bm.wg.Done()
    }()
    return be, nil
}

很明顯進(jìn)行完之后可以獲得一個(gè)VXLANBackend實(shí)例.

5. 注冊(cè)網(wǎng)絡(luò)

===> main.go
func main() {
...
bn, err := be.RegisterNetwork(ctx, config)
...
}
===> backend/vxlan/vxlan.go
func (be *VXLANBackend) RegisterNetwork(ctx context.Context, config *subnet.Config) (backend.Network, error) {
    // Parse our configuration
    cfg := struct {
        VNI           int
        Port          int
        GBP           bool
        DirectRouting bool
    }{
        VNI: defaultVNI,
    }

    if len(config.Backend) > 0 {
        if err := json.Unmarshal(config.Backend, &cfg); err != nil {
            return nil, fmt.Errorf("error decoding VXLAN backend config: %v", err)
        }
    }
    log.Infof("VXLAN config: VNI=%d Port=%d GBP=%v DirectRouting=%v", cfg.VNI, cfg.Port, cfg.GBP, cfg.DirectRouting)

    devAttrs := vxlanDeviceAttrs{
        vni:       uint32(cfg.VNI),
        name:      fmt.Sprintf("flannel.%v", cfg.VNI),
        vtepIndex: be.extIface.Iface.Index,
        vtepAddr:  be.extIface.IfaceAddr,
        vtepPort:  cfg.Port,
        gbp:       cfg.GBP,
    }
    // 生成flannel.1 設(shè)備
    dev, err := newVXLANDevice(&devAttrs)
    if err != nil {
        return nil, err
    }
    dev.directRouting = cfg.DirectRouting

    subnetAttrs, err := newSubnetAttrs(be.extIface.ExtAddr, dev.MACAddr())
    if err != nil {
        return nil, err
    }
    // 分配子網(wǎng)的時(shí)候 需要保存出口地址以及flannel.1 mac地址
    lease, err := be.subnetMgr.AcquireLease(ctx, subnetAttrs)
    switch err {
    case nil:
    case context.Canceled, context.DeadlineExceeded:
        return nil, err
    default:
        return nil, fmt.Errorf("failed to acquire lease: %v", err)
    }

    // Ensure that the device has a /32 address so that no broadcast routes are created.
    // This IP is just used as a source address for host to workload traffic (so
    // the return path for the traffic has an address on the flannel network to use as the destination)
    if err := dev.Configure(ip.IP4Net{IP: lease.Subnet.IP, PrefixLen: 32}); err != nil {
        return nil, fmt.Errorf("failed to configure interface %s: %s", dev.link.Attrs().Name, err)
    }

    return newNetwork(be.subnetMgr, be.extIface, dev, ip.IP4Net{}, lease)
}

運(yùn)行:

===> 運(yùn)行前
[root@master flannel]# etcdctl ls /coreos.com/network/subnets
[root@master flannel]# 
===> 運(yùn)行
[root@master flannel]# ./flannel --etcd-endpoints="http://172.21.0.16:2379"
I1103 12:15:50.487232   18595 main.go:480] Determining IP address of default interface
I1103 12:15:50.487439   18595 main.go:493] Using interface with name eth0 and address 172.21.0.16
I1103 12:15:50.487449   18595 main.go:510] Defaulting external address to interface address (172.21.0.16)
I1103 12:15:50.487457   18595 main.go:232] ======>IfaceAddr:172.21.0.16, ExtAddr:172.21.0.16, Iface.Name:eth0, Iface.Index:2, Iface.MTU:1500
I1103 12:15:50.487505   18595 main.go:168] ======>prevSubnet:10.0.13.0/24
I1103 12:15:50.487570   18595 main.go:240] Created subnet manager: Etcd Local Manager with Previous Subnet: 10.0.13.0/24
I1103 12:15:50.487576   18595 main.go:243] Installing signal handlers
I1103 12:15:50.488757   18595 main.go:358] Found network config - Backend type: vxlan
I1103 12:15:50.488800   18595 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
I1103 12:15:50.490867   18595 local_manager.go:201] Found previously leased subnet (10.0.13.0/24), reusing
I1103 12:15:50.491779   18595 local_manager.go:220] Allocated lease (10.0.13.0/24) to current node (172.21.0.16) 
I1103 12:15:50.491908   18595 main.go:305] Wrote subnet file to /run/flannel/subnet.env
I1103 12:15:50.491926   18595 main.go:309] Running backend.
I1103 12:15:50.492172   18595 vxlan_network.go:60] watching for new subnet leases
I1103 12:15:50.493612   18595 main.go:401] Waiting for 22h59m59.997445279s to renew lease
...
===> 運(yùn)行后
[root@master ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.13.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::d457:dcff:febb:a72e  prefixlen 64  scopeid 0x20<link>
        ether d6:57:dc:bb:a7:2e  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0

[root@master ~]# 

可以看到運(yùn)行后多了flannel.1設(shè)備, 然后再看一下etcd里面的內(nèi)容

[root@master ~]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/10.0.13.0-24
[root@master ~]# etcdctl get /coreos.com/network/subnets/10.0.13.0-24
{"PublicIP":"172.21.0.16","BackendType":"vxlan","BackendData":{"VtepMAC":"d6:57:dc:bb:a7:2e"}}

可以看到etcd給172.21.0.16機(jī)器分配了10.0.13.0/24網(wǎng)絡(luò), 并保存了此機(jī)器上的flannel.1的MAC地址. 這個(gè)是當(dāng)別的機(jī)器加入到flannel中時(shí)會(huì)用到, 在add fdb, neighbor中用到.

另外會(huì)把分配得到的子網(wǎng)信息存到各自機(jī)器的/run/flannel/subnet.env中.

[root@master flannel]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.0.0.0/16
FLANNEL_SUBNET=10.0.13.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false
[root@master flannel]# 

另外AcquireLease調(diào)用tryAcquireLease去申請(qǐng)子網(wǎng). 主要分三步

1. 從etcd中尋找是否有該ip的子網(wǎng)網(wǎng)段存在.
2. 從本地/run/flannel/subnet.env中查看該主機(jī)是否有分配過(guò)子網(wǎng).
3. 分配一個(gè)新的子網(wǎng).

6. 監(jiān)控變化

func main() {
...
log.Info("Running backend.")
    wg.Add(1)
    go func() {
        bn.Run(ctx)
        wg.Done()
    }()
...
}

bn, err := be.RegisterNetwork(ctx, config)是5. 注冊(cè)網(wǎng)絡(luò)中返回的一個(gè)Network對(duì)象, 該對(duì)象擁有分配的子網(wǎng)信息以及到期時(shí)間等等.

func (nw *network) Run(ctx context.Context) {
        ...
        subnet.WatchLeases(ctx, nw.subnetMgr, nw.SubnetLease, events)
        ...
    }()
    ...
    for {
        select {
        case evtBatch := <-events:
            nw.handleSubnetEvents(evtBatch)
        ...
    }
}

這里主要關(guān)注兩個(gè)方法:

subnet.WatchLeases(ctx, nw.subnetMgr, nw.SubnetLease, events): 監(jiān)控etcd中子網(wǎng)的變化情況, 一旦有機(jī)器加入/刪除/更新等等, 就會(huì)把信息傳到events中.
handleSubnetEvents: 一旦有變化, 就需要處理與該變化有關(guān)的操作.

    for _, event := range batch {
        ...
        switch event.Type {
        case subnet.EventAdded:
            if directRoutingOK {
                log.V(2).Infof("Adding direct route to subnet: %s PublicIP: %s", sn, attrs.PublicIP)

                if err := netlink.RouteReplace(&directRoute); err != nil {
                    log.Errorf("Error adding route to %v via %v: %v", sn, attrs.PublicIP, err)
                    continue
                }
            } else {
                log.V(2).Infof("adding subnet: %s PublicIP: %s VtepMAC: %s", sn, attrs.PublicIP, net.HardwareAddr(vxlanAttrs.VtepMAC))
                if err := nw.dev.AddARP(neighbor{IP: sn.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                    log.Error("AddARP failed: ", err)
                    continue
                }

                if err := nw.dev.AddFDB(neighbor{IP: attrs.PublicIP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                    log.Error("AddFDB failed: ", err)

                    // Try to clean up the ARP entry then continue
                    if err := nw.dev.DelARP(neighbor{IP: event.Lease.Subnet.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                        log.Error("DelARP failed: ", err)
                    }

                    continue
                }

                // Set the route - the kernel would ARP for the Gw IP address if it hadn't already been set above so make sure
                // this is done last.
                if err := netlink.RouteReplace(&vxlanRoute); err != nil {
                    log.Errorf("failed to add vxlanRoute (%s -> %s): %v", vxlanRoute.Dst, vxlanRoute.Gw, err)

                    // Try to clean up both the ARP and FDB entries then continue
                    if err := nw.dev.DelARP(neighbor{IP: event.Lease.Subnet.IP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                        log.Error("DelARP failed: ", err)
                    }

                    if err := nw.dev.DelFDB(neighbor{IP: event.Lease.Attrs.PublicIP, MAC: net.HardwareAddr(vxlanAttrs.VtepMAC)}); err != nil {
                        log.Error("DelFDB failed: ", err)
                    }

                    continue
                }
            }
        case subnet.EventRemoved:
            ...
        }
    }
}

可以看到有三個(gè)操作:
1. AddARP
2. AddFDB
3. RouteReplace 增加路由
所以說(shuō)只要etcd中子網(wǎng)有變化, 那每臺(tái)機(jī)器上的flannel都需要更新自己的arp, fdb, route表.

運(yùn)行

在另外一臺(tái)機(jī)器上(172.21.0.12)上啟動(dòng)flannel, 可以看到該機(jī)器分配的子網(wǎng)為10.0.10.0/24, 并且與機(jī)器(172.21.0.16)進(jìn)行了flannel.1連通的相關(guān)操作.

// 在另外一臺(tái)機(jī)器上(172.21.0.12)上啟動(dòng)flannel
[root@worker flannel]# ./flannel  --etcd-endpoints="http://172.21.0.16:2379" --ip-masq=true --v=2
...
I1103 12:53:18.445859    6246 local_manager.go:147] Found lease (10.0.10.0/24) for current IP (172.21.0.12)
...
I1103 12:53:18.449268    6246 vxlan_network.go:138] adding subnet: 10.0.13.0/24 PublicIP: 172.21.0.16 VtepMAC: d6:57:dc:bb:a7:2e
I

查看flannel.1 arp, fdb, route表

[root@worker ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.0.10.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::acf0:22ff:fef1:f63d  prefixlen 64  scopeid 0x20<link>
        ether ae:f0:22:f1:f6:3d  txqueuelen 0  (Ethernet)
        RX packets 9  bytes 756 (756.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 672 (672.0 B)
        TX errors 0  dropped 8 overruns 0  carrier 0  collisions 0
[root@worker ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
...
10.0.13.0       10.0.13.0       255.255.255.0   UG    0      0        0 flannel.1
...
[root@worker ~]# 
[root@worker ~]# bridge fdb show
...
10.0.13.0 dev flannel.1 lladdr d6:57:dc:bb:a7:2e PERMANENT
...
[root@worker ~]# ip neighbor show
...
10.0.13.0 dev flannel.1 lladdr d6:57:dc:bb:a7:2e PERMANENT
...
[root@worker ~]# 

類似的可以看到172.21.0.16中flannel的日志中有了

[root@master flannel]# ./flannel --etcd-endpoints="http://172.21.0.16:2379" --ip-masq=true --v=2
...
I1103 12:53:18.446670   24228 vxlan_network.go:138] adding subnet: 10.0.10.0/24 PublicIP: 172.21.0.12 VtepMAC: ae:f0:22:f1:f6:3d

此時(shí)查看一下172.21.0.16中的flannel.1與172.21.0.12是否可以互通.

// flannel.1(172.21.0.16) ===> flannel.1(172.21.0.12)
[root@master flannel]# ping -c 1 10.0.10.0
PING 10.0.10.0 (10.0.10.0) 56(84) bytes of data.
64 bytes from 10.0.10.0: icmp_seq=1 ttl=64 time=0.402 ms

--- 10.0.10.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.402/0.402/0.402/0.000 ms
[root@master flannel]# 

// flannel.1(172.21.0.12) ===> flannel.1(172.21.0.16)
root@worker ~]# ping -c 1 10.0.13.0
PING 10.0.13.0 (10.0.13.0) 56(84) bytes of data.
64 bytes from 10.0.13.0: icmp_seq=1 ttl=64 time=0.399 ms

--- 10.0.13.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.399/0.399/0.399/0.000 ms

可以看到已經(jīng)互相連通.

7. 總結(jié)

conclusion.png

1. 請(qǐng)求子網(wǎng)分配的時(shí)候, 按如下順序, 原則就是盡量不讓每個(gè)機(jī)器所屬的子網(wǎng)發(fā)生變化:

1.1 從etcd中尋找是否有該ip的子網(wǎng)網(wǎng)段存在.
1.2 從本地/run/flannel/subnet.env中查看該主機(jī)是否有分配過(guò)子網(wǎng).
1.3 分配一個(gè)新的子網(wǎng).

2. 當(dāng)有機(jī)器加入或者刪除或更新等等, 都會(huì)觸發(fā)每臺(tái)機(jī)器去更新對(duì)應(yīng)的arp fdb route表. 從而保證不影響通信.