謝絕轉(zhuǎn)載

序言

上篇我們粗略過了一下docker 命令的調(diào)用流程,可知命令的最后運(yùn)行部分是在服務(wù)端的daemon中完成.這一篇主要看一下docker卷服務(wù)的初始化和相關(guān)卷plugin的加載.

這部分代碼我們需要帶著兩個(gè)疑問去看,第一就是plugin是怎么被加載的,第二就是創(chuàng)建卷的時(shí)候怎么與我們自己實(shí)現(xiàn)的plugin關(guān)聯(lián)起來的.

注意事項(xiàng):
1.本文共有四篇,每篇都有編號(hào),編號(hào)類似1.2.1這種,其中1是文章編號(hào),因?yàn)楹竺娴恼{(diào)用關(guān)系需要去前面篇幅中找,所以我標(biāo)注了這個(gè)方便尋找.

2.我是按調(diào)用過程列出代碼,如果當(dāng)前函數(shù)有多個(gè)地方需要講解,比如函數(shù)1.2中有兩個(gè)地方需要講解,那么要展開的地方便是1.2.1,1.2.2這樣排列.

3.鏈接:
第一篇:http://www.itdecent.cn/p/9900ec52f2c1 (命令的調(diào)用流程)
第二篇:http://www.itdecent.cn/p/db08b7d57721 (卷服務(wù)初始化)
第三篇:http://www.itdecent.cn/p/bbc73f5687a2 (plugin的管理)
第四篇:http://www.itdecent.cn/p/a92b1b11c8dd (卷相關(guān)命令的執(zhí)行)

Daemon中關(guān)于volume服務(wù)的初始化

main函數(shù)是萬物之源,我們繼續(xù)從服務(wù)端的main函數(shù)入手,找到相關(guān)的代碼(不知道怎么找的朋友請(qǐng)看上篇).我們要找的代碼在NewDaemon函數(shù)中:

2.1 NewDaemon函數(shù)

NewDaemon是docker進(jìn)程的初始化函數(shù),找到NewDaemon中對(duì)卷服務(wù)的初始化代碼:

func NewDaemon(ctx context.Context, config *config.Config, pluginStore *plugin.Store) (daemon *Daemon, err error) {
    setDefaultMtu(config)

    ...

    # 創(chuàng)建deamon
    d := &Daemon{
        configStore: config,
        PluginStore: pluginStore,
        startupDone: make(chan struct{}),
    }
    ...
    # d.pluginManager的初始化中加載了/var/lib/docker/plugins下的所有plugin
    # 此處為重點(diǎn)
    // Plugin system initialization should happen before restore. Do not change order.
    d.pluginManager, err = plugin.NewManager(plugin.ManagerConfig{
        Root:               filepath.Join(config.Root, "plugins"),
        ExecRoot:           getPluginExecRoot(config.Root),
        Store:              d.PluginStore,
        CreateExecutor:     createPluginExec,
        RegistryService:    registryService,
        LiveRestoreEnabled: config.LiveRestoreEnabled,
        LogPluginEvent:     d.LogPluginEvent, // todo: make private
        AuthzMiddleware:    config.AuthzMiddleware,
    })

    ...
    # d.volumes 具體執(zhí)行對(duì)卷的管理
    # 此處為重點(diǎn)
    d.volumes, err = volumesservice.NewVolumeService(config.Root, d.PluginStore, rootIDs, d)
    if err != nil {
        return nil, err
    }

    ...
}

在上面NewDaemon函數(shù)中標(biāo)注出來兩個(gè)重要的環(huán)節(jié),首先是pluginManager的初始化,其次是d.volumes的初始化.這塊涉及到的接口比較多,我們先根據(jù)調(diào)用流程過一遍,最后再來總結(jié).

2.2 plugin的加載過程

2.2.1 Store 結(jié)構(gòu)體

這里有一個(gè)很重要的參數(shù)pluginStore, 在執(zhí)行NewDaemon的時(shí)候傳入的pluginStore是一個(gè)空的Store結(jié)構(gòu)體,主要作用就是保存plugin:

// Store manages the plugin inventory in memory and on-disk
type Store struct {
    sync.RWMutex
    plugins  map[string]*v2.Plugin
    specOpts map[string][]SpecOpt
    /* handlers are necessary for transition path of legacy plugins
     * to the new model. Legacy plugins use Handle() for registering an
     * activation callback.*/
    handlers map[string][]func(string, *plugins.Client)
}

2.2.2 Manager結(jié)構(gòu)體

先看d.pluginManager的定義,它是一個(gè)結(jié)構(gòu)體,定義如下,用來管理plugin:

// Manager controls the plugin subsystem.
type Manager struct {
    config    ManagerConfig
    mu        sync.RWMutex // protects cMap
    muGC      sync.RWMutex // protects blobstore deletions
    cMap      map[*v2.Plugin]*controller
    blobStore *basicBlobStore
    publisher *pubsub.Publisher
    executor  Executor
}

2.2.3 NewManager函數(shù)

再來看Manager的新建函數(shù):

// NewManager returns a new plugin manager.
func NewManager(config ManagerConfig) (*Manager, error) {
    ...
    manager := &Manager{
        config: config,
    }
    ...
    # 創(chuàng)建manager.executor
    # 此處為重點(diǎn)
    manager.executor, err = config.CreateExecutor(manager)
    if err != nil {
        return nil, err
    }
    ...
    manager.cMap = make(map[*v2.Plugin]*controller)
    # 此處為重點(diǎn)
    if err := manager.reload(); err != nil {
        return nil, errors.Wrap(err, "failed to restore plugins")
    }

    manager.publisher = pubsub.NewPublisher(0, 0)
    return manager, nil
}

首先,傳入的唯一參數(shù)config中綁定了d.PluginStore,對(duì)應(yīng)的是config.Store(這部分代碼代碼賦值在NewDaemon中完成,參考2.1章節(jié)的代碼).這里重點(diǎn)講一下manager.executor和manager.reload().

2.2.4 初始化manager.executor

• 2.2.4.1 Executor接口

在上面的函數(shù)中,manager.executor是實(shí)現(xiàn)了Executor接口的結(jié)構(gòu)體,Executor的主要作用是用來啟動(dòng)和停止plugin(創(chuàng)建對(duì)應(yīng)的容器,此容器不同于正常的docker容器,只能用docker-runc命令查看):

// Executor is the interface that the plugin manager uses to interact with for starting/stopping plugins
type Executor interface {
    Create(id string, spec specs.Spec, stdout, stderr io.WriteCloser) error
    IsRunning(id string) (bool, error)
    Restore(id string, stdout, stderr io.WriteCloser) (alive bool, err error)
    Signal(id string, signal int) error
}

而config.CreateExecutor在創(chuàng)建的時(shí)候傳入值是以下匿名函數(shù):

• 2.2.4.2 createPluginExec對(duì)應(yīng)的匿名函數(shù)

createPluginExec := func(m *plugin.Manager) (plugin.Executor, error) {
        var pluginCli *containerd.Client

        // Windows is not currently using containerd, keep the
        // client as nil
        if config.ContainerdAddr != "" {
            pluginCli, err = containerd.New(config.ContainerdAddr, containerd.WithDefaultNamespace(pluginexec.PluginNamespace), containerd.WithDialOpts(gopts))
            if err != nil {
                return nil, errors.Wrapf(err, "failed to dial %q", config.ContainerdAddr)
            }
        }
        # getPluginExecRoot(config.Root) = "/var/lib/docker/plugin"
        return pluginexec.New(ctx, getPluginExecRoot(config.Root), pluginCli, m)
    }

該函數(shù)調(diào)用pluginexec.New函數(shù),返回一個(gè)Executor結(jié)構(gòu)體,如下:

• 2.2.4.3 New函數(shù)

func New(ctx context.Context, rootDir string, cli *containerd.Client, exitHandler ExitHandler) (*Executor, error) {
    e := &Executor{
        rootDir:     rootDir,
        exitHandler: exitHandler,
    }

    client, err := libcontainerd.NewClient(ctx, cli, rootDir, PluginNamespace, e)
    if err != nil {
        return nil, errors.Wrap(err, "error creating containerd exec client")
    }
    e.client = client
    return e, nil
}

這里創(chuàng)建的結(jié)構(gòu)體Executor實(shí)現(xiàn)了上面的Executor接口(定義位置不同),然后初始化libcontainerd的客戶端,這里就不深入看了,其實(shí)我也不懂太底層的:-).

• 2.2.4.4 Executor結(jié)構(gòu)體

// Executor is the containerd client implementation of a plugin executor
type Executor struct {
    rootDir     string
    client      Client
    exitHandler ExitHandler
}

看完manager.executor,讓我們回到NewManager[2.2.3章節(jié)],繼續(xù)執(zhí)行manager.reload()方法

2.2.5 manager.reload()加載plugin

• 2.2.5.1 reload方法

func (pm *Manager) reload() error { // todo: restore
    # 從/var/lib/docker/plugins 路徑下加載plugin
    dir, err := ioutil.ReadDir(pm.config.Root)
    ...
    plugins := make(map[string]*v2.Plugin)
    for _, v := range dir {
        if validFullID.MatchString(v.Name()) {
            # 從對(duì)應(yīng)的plugin路徑的json文件下加載plugin
            p, err := pm.loadPlugin(v.Name())
            if err != nil {
                handleLoadError(err, v.Name())
                continue
            }
            plugins[p.GetID()] = p
        }
    ...
    }

    pm.config.Store.SetAll(plugins)
    ...
    return nil
}

加載plugin調(diào)用的是pm.loadPlugin方法,這里不仔細(xì)看這個(gè)方法了,但是v2.Plugin這個(gè)結(jié)構(gòu)體比較重要,要看一下:

• 2.2.5.2 Plugin結(jié)構(gòu)體

// Plugin represents an individual plugin.
type Plugin struct {
    mu        sync.RWMutex
    PluginObj types.Plugin `json:"plugin"` // todo: embed struct
    pClient   *plugins.Client
    refCount  int
    Rootfs    string // TODO: make private

    Config   digest.Digest
    Blobsums []digest.Digest

    modifyRuntimeSpec func(*specs.Spec)

    SwarmServiceID string
    timeout        time.Duration
    addr           net.Addr
}

其中plugins.Client定義如下,是plugin的http客戶端:

• 2.2.5.3 Client結(jié)構(gòu)體

// Client represents a plugin client.
type Client struct {
    http           *http.Client // http client to use
    requestFactory transport.RequestFactory
}

加載plugins后我們看到調(diào)用了SetAll方法,看一下SetAll的定義:

• 2.2.5.4 SetAll方法

// SetAll initialized plugins during daemon restore.
func (ps *Store) SetAll(plugins map[string]*v2.Plugin) {
    ps.Lock()
    defer ps.Unlock()

    for _, p := range plugins {
        ps.setSpecOpts(p)
    }
    ps.plugins = plugins
}

上面的方法最后把plugins保存到config中的Store中,也就是之前綁定的d.PluginStore, 也就是說d.PluginStore.plugins = plugins

2.3 卷服務(wù)的創(chuàng)建

回到NewDaemon函數(shù)[2.1章節(jié)],繼續(xù)看卷服務(wù)的創(chuàng)建,volumesservice也就是d.volumes是卷的相關(guān)方法的實(shí)際執(zhí)行者,我們先看下它的初始化過程.

d.volumes, err = volumesservice.NewVolumeService(config.Root, d.PluginStore, rootIDs, d)

2.3.1 VolumesService結(jié)構(gòu)體

先看VolumesService定義:

// VolumesService manages access to volumes
type VolumesService struct {
    vs           *VolumeStore
    ds           ds
    pruneRunning int32
    eventLogger  volumeEventLogger
}

2.3.2 NewVolumeService方法

NewVolumeService方法, 實(shí)參是上面的d.PluginStore, 虛參是一個(gè)PluginGetter接口.注意在NewVolumeService函數(shù)中d.PluginStore變成了pg即plugingetter.PluginGetter.

// NewVolumeService creates a new volume service
func NewVolumeService(root string, pg plugingetter.PluginGetter, rootIDs idtools.Identity, logger volumeEventLogger) (*VolumesService, error) {
    # ds是一個(gè)接口,負(fù)責(zé)driver的存儲(chǔ)
    ds := drivers.NewStore(pg)
    if err := setupDefaultDriver(ds, root, rootIDs); err != nil {
        return nil, err
    }

    # vs 用來存儲(chǔ)卷的信息
    vs, err := NewStore(root, ds)
    if err != nil {
        return nil, err
    }
    return &VolumesService{vs: vs, ds: ds, eventLogger: logger}, nil
}

首先看一下PluginGetter接口

• 2.3.2.1 PluginGetter接口

// PluginGetter is the interface implemented by Store
type PluginGetter interface {
    Get(name, capability string, mode int) (CompatPlugin, error)
    GetAllByCap(capability string) ([]CompatPlugin, error)
    GetAllManagedPluginsByCap(capability string) []CompatPlugin
    Handle(capability string, callback func(string, *plugins.Client))
}

• 2.3.2.2 ds接口

再看ds, ds接口只定義了一個(gè)方法GetDriverList

type ds interface {
    GetDriverList() []string
}

實(shí)現(xiàn)這個(gè)接口的是Store結(jié)構(gòu)體,此Store非彼Store,不同于之前的保存Plugin的那個(gè)Store,這個(gè)store用來保存driver,一定要注意:

• 2.3.2.3 Store結(jié)構(gòu)體

// Store is an in-memory store for volume drivers
type Store struct {
    extensions   map[string]volume.Driver
    mu           sync.Mutex
    driverLock   *locker.Locker
    pluginGetter getter.PluginGetter
}

在drivers.NewStore(pg)執(zhí)行后d.pluginStore被綁定到ds的pluginGetter上,也就是ds.pluginGetter = d.pluginStore.而vs是一個(gè)管理卷的結(jié)構(gòu)體,定義如下:

• 2.3.2.4 VolumeStore結(jié)構(gòu)體

// VolumeStore is a struct that stores the list of volumes available and keeps track of their usage counts
type VolumeStore struct {
    // locks ensures that only one action is being performed on a particular volume at a time without locking the entire store
    // since actions on volumes can be quite slow, this ensures the store is free to handle requests for other volumes.
    locks   *locker.Locker
    drivers *drivers.Store
    // globalLock is used to protect access to mutable structures used by the store object
    globalLock sync.RWMutex
    // names stores the volume name -> volume relationship.
    // This is used for making lookups faster so we don't have to probe all drivers
    names map[string]volume.Volume
    // refs stores the volume name and the list of things referencing it
    refs map[string]map[string]struct{}
    // labels stores volume labels for each volume
    labels map[string]map[string]string
    // options stores volume options for each volume
    options map[string]map[string]string
    db      *bolt.DB
}

vs的drivers值為ds, vs.drivers.pluginGetter = d.pluginStore

總結(jié)

這部分代碼就是知道在Daemon的初始化中如何加載plugin并初始化相關(guān)卷服務(wù),在后續(xù)的代碼中會(huì)調(diào)用這篇講到的內(nèi)容.