謝絕轉(zhuǎn)載

序言

之前研究了一段時間的docker源碼的調(diào)用機(jī)制，主要是想學(xué)習(xí)一下go，并了解一下docker volume plugin的加載機(jī)制，最近有點(diǎn)忘記了，就寫下來加深記憶。

docker 版本：docker-ce(18.09)

本文會列出一些docker源碼中的函數(shù),因為篇幅原因會用...來省略一些內(nèi)容,只留下我認(rèn)為在調(diào)用流程中重要的一些內(nèi)容. 在部分函數(shù)中會用注釋標(biāo)注關(guān)鍵點(diǎn).

注意事項:
1.本文共有四篇,每篇都有編號,編號類似1.2.1這種,其中1是文章編號,因為后面的調(diào)用關(guān)系需要去前面篇幅中找,所以我標(biāo)注了這個方便尋找.

2.我是按調(diào)用過程列出代碼,如果當(dāng)前函數(shù)有多個地方需要講解,比如函數(shù)1.2中有兩個地方需要講解,那么要展開的地方便是1.2.1,1.2.2這樣排列.

3.鏈接:
第一篇:http://www.itdecent.cn/p/9900ec52f2c1 (命令的調(diào)用流程)
第二篇:http://www.itdecent.cn/p/db08b7d57721 (卷服務(wù)初始化)
第三篇:http://www.itdecent.cn/p/bbc73f5687a2 (plugin的管理)
第四篇:http://www.itdecent.cn/p/a92b1b11c8dd (卷相關(guān)命令的執(zhí)行)

以下開始正文

docker命令的調(diào)用流程

以下是docker-ce代碼的主要結(jié)構(gòu),主要關(guān)注前兩個文件夾,即客戶端和服務(wù)端.

markdown-img-paste-2018102921143250.png

下面通過docker命令的執(zhí)行過程,來探究一下docker的代碼調(diào)用流程.

1.1 客戶端部分

客戶端入口是docker.go中的main函數(shù):

markdown-img-paste-20181029212322244.png

func main() {
    // Set terminal emulation based on platform as required.
    stdin, stdout, stderr := term.StdStreams()
    logrus.SetOutput(stderr)

    # 新建dockerCli
    dockerCli := command.NewDockerCli(stdin, stdout, stderr, contentTrustEnabled(), containerizedengine.NewClient)
    # 綁定命令
    cmd := newDockerCommand(dockerCli)

    if err := cmd.Execute(); err != nil {
        ...
    }
}

main函數(shù)先創(chuàng)建了dockerCli客戶端,然后調(diào)用了newDockerCommand函數(shù)來初始化docker命令.

1.1.1 DockerCli結(jié)構(gòu)體

DockerCli是docker命令行客戶端:

// DockerCli is an instance the docker command line client.
// Instances of the client can be returned from NewDockerCli.
type DockerCli struct {
    configFile            *configfile.ConfigFile
    in                    *InStream
    out                   *OutStream
    err                   io.Writer
    client                client.APIClient
    serverInfo            ServerInfo
    clientInfo            ClientInfo
    contentTrust          bool
    newContainerizeClient func(string) (clitypes.ContainerizedClient, error)
}

1.1.2 newDockerCommand函數(shù)

newDockerCommand函數(shù)來初始化docker命令, 用到的是golang的命令行庫Cobra.

func newDockerCommand(dockerCli *command.DockerCli) *cobra.Command {
    ...
    cmd := &cobra.Command{
        Use:              "docker [OPTIONS] COMMAND [ARG...]",
        Short:            "A self-sufficient runtime for containers",
        SilenceUsage:     true,
        SilenceErrors:    true,
        TraverseChildren: true,
        Args:             noArgs,
        ...
    }
    # 設(shè)置默認(rèn)信息
    cli.SetupRootCommand(cmd)
  ...
    # 加載子命令
    commands.AddCommands(cmd, dockerCli)

    ...
}

1.1.3 AddCommands函數(shù)

函數(shù)commands.AddCommands定義如下,可知加載了所有的子命令:

func AddCommands(cmd *cobra.Command, dockerCli *command.DockerCli) {
    cmd.AddCommand(
        ...
        // container
        container.NewContainerCommand(dockerCli),
        container.NewRunCommand(dockerCli),

        // image
        image.NewImageCommand(dockerCli),
        image.NewBuildCommand(dockerCli),

        ...

        // network
        network.NewNetworkCommand(dockerCli),

        ...

        // volume
        volume.NewVolumeCommand(dockerCli),
        ...
    )
}

看到這里我們已經(jīng)知道了客戶端命令是如何綁定的,也可以說是知道了docker命令執(zhí)行的起點(diǎn),再通過具體命令的執(zhí)行過程來看一下客戶端命令是如何和服務(wù)端的代碼對應(yīng)起來的.以docker run命令為例.

1.2.docker run命令執(zhí)行流程

根據(jù)上面的分析,我們找到docker run命令對應(yīng)的函數(shù)NewRunCommand

1.2.1 NewRunCommand函數(shù)

func NewRunCommand(dockerCli command.Cli) *cobra.Command {
  ...
    cmd := &cobra.Command{
        Use:   "run [OPTIONS] IMAGE [COMMAND] [ARG...]",
        Short: "Run a command in a new container",
        Args:  cli.RequiresMinArgs(1),
        RunE: func(cmd *cobra.Command, args []string) error {
            copts.Image = args[0]
            if len(args) > 1 {
                copts.Args = args[1:]
            }
            return runRun(dockerCli, cmd.Flags(), &opts, copts)
        },
    }
    ...
    return cmd
}

命令運(yùn)行則會運(yùn)行同路徑下的runRun函數(shù),runRun函數(shù)繼續(xù)調(diào)用到同路徑下runContainer函數(shù),runContainer函數(shù)最終調(diào)用createContainer函數(shù),可以發(fā)現(xiàn)docker run與docker create命令調(diào)用的是同一個函數(shù):

1.2.2 createContainer函數(shù)

func createContainer(ctx context.Context, dockerCli command.Cli, containerConfig *containerConfig, name string, platform string) (*container.ContainerCreateCreatedBody, error) {
    ...

    //create the container
    response, err := dockerCli.Client().ContainerCreate(ctx, config, hostConfig, networkingConfig, name)
    ...
}

這里可以看到調(diào)用的是dockerCli.Client()的ContainerCreate方法,我們追蹤溯源,看一下dockerCli.Client()是什么. 這里的dockerCli是command.Cli接口:

ps:以下兩個小章節(jié)都是根據(jù)上面的函數(shù)進(jìn)行展開,所以就用這種排版了,以后出現(xiàn)類似的排版同理.

• 1.2.2.1 Cli接口

// Cli represents the docker command line client.
type Cli interface {
    Client() client.APIClient
    ...
}

通過Cli的接口定義可知Client()返回的是client.APIClient, 再看一下DockerCli[1.1章節(jié)]中對接口方法Client()的具體實現(xiàn):

• 1.2.2.2 Client方法

// Client returns the APIClient
func (cli *DockerCli) Client() client.APIClient {
    return cli.client
}

通過接口的實現(xiàn)可知返回值就是cli.client,而cli.client是在newDockerCommand函數(shù)中調(diào)用dockerCli.Initialize初始化的:

1.2.3 Initialize方法

func (cli *DockerCli) Initialize(opts *cliflags.ClientOptions) error {
    ...
    cli.client, err = NewAPIClientFromFlags(opts.Common, cli.configFile)
    ...
}

在NewAPIClientFromFlags中最終調(diào)用到了服務(wù)端的NewClientWithOpts函數(shù),注意,連接客戶端代碼和服務(wù)端代碼的關(guān)鍵來了:

1.2.4 NewClientWithOpts函數(shù)

func NewClientWithOpts(ops ...func(*Client) error) (*Client, error) {
    client, err := defaultHTTPClient(DefaultDockerHost)
    if err != nil {
        return nil, err
    }
    c := &Client{
        host:    DefaultDockerHost,
        version: api.DefaultVersion,
        scheme:  "http",
        client:  client,
        proto:   defaultProto,
        addr:    defaultAddr,
    }

    ...

    return c, nil
}

首先,可知返回的c是一個Client類型的結(jié)構(gòu)體指針,再看成員變量Client.client則是docker服務(wù)端的http客戶端. Client的具體定義如下:

1.2.5 Client結(jié)構(gòu)體

// Client is the API client that performs all operations
// against a docker server.
type Client struct {
    // scheme sets the scheme for the client
    scheme string
    // host holds the server address to connect to
    host string
    // proto holds the client protocol i.e. unix.
    proto string
    // addr holds the client address.
    addr string
    // basePath holds the path to prepend to the requests.
    basePath string
    // client used to send and receive http requests.
    client *http.Client
    // version of the server to talk to.
    version string
    // custom http headers configured by users.
    customHTTPHeaders map[string]string
    // manualOverride is set to true when the version was set by users.
    manualOverride bool
}

在client.go 的相同路徑下找到ContainerCreate方法,可知最后發(fā)出的是一個post請求:

1.2.6 ContainerCreate方法

func (cli *Client) ContainerCreate(ctx context.Context, config *container.Config, hostConfig *container.HostConfig, networkingConfig *network.NetworkingConfig, containerName string) (container.ContainerCreateCreatedBody, error) {
    var response container.ContainerCreateCreatedBody

    ...

    serverResp, err := cli.post(ctx, "/containers/create", query, body, nil)
    if err != nil {
        if serverResp.statusCode == 404 && strings.Contains(err.Error(), "No such image") {
            return response, objectNotFoundError{object: "image", id: config.Image}
        }
        return response, err
    }

    err = json.NewDecoder(serverResp.body).Decode(&response)
    ensureReaderClosed(serverResp)
    return response, err
}

這里我們注意的是url的路徑 "/containers/create".
那另一個疑問來了,該請求最終調(diào)用到什么地方去了?讓我們話分兩頭,再去看看服務(wù)端的進(jìn)程是如何啟動的.

To be continued...

1.3 服務(wù)端程序

同樣,我們找到入口函數(shù),即服務(wù)端代碼的main函數(shù):

func main() {
    ...
    cmd := newDaemonCommand()
    cmd.SetOutput(stdout)
    if err := cmd.Execute(); err != nil {
        fmt.Fprintf(stderr, "%s\n", err)
        os.Exit(1)
    }
}

main函數(shù)中只調(diào)用一個命令,那就是后臺進(jìn)程啟動命令,沒有子命令,newDaemonCommand函數(shù)如下:

1.3.1 newDaemonCommand函數(shù)

func newDaemonCommand() *cobra.Command {
    opts := newDaemonOptions(config.New())

    cmd := &cobra.Command{
        Use:           "dockerd [OPTIONS]",
        Short:         "A self-sufficient runtime for containers.",
        SilenceUsage:  true,
        SilenceErrors: true,
        Args:          cli.NoArgs,
        RunE: func(cmd *cobra.Command, args []string) error {
            opts.flags = cmd.Flags()
            return runDaemon(opts)
        },
        ...
    }
    ...

    return cmd
}

類似客戶端命令,這個命令執(zhí)行后會調(diào)用到runDaemon函數(shù),看一下進(jìn)程啟動都會做些什么:

1.3.2 runDaemon函數(shù)

func runDaemon(opts *daemonOptions) error {
    daemonCli := NewDaemonCli()
    return daemonCli.start(opts)
}

NewDaemonCli()返回值是DaemonCli的結(jié)構(gòu)體指針,先看下DaemonCli 的定義:

• 1.3.2.1 DaemonCli結(jié)構(gòu)體

type DaemonCli struct {
    *config.Config
    configFile *string
    flags      *pflag.FlagSet

    api             *apiserver.Server
    d               *daemon.Daemon
    authzMiddleware *authorization.Middleware // authzMiddleware enables to dynamically reload the authorization plugins
}

再看DaemonCli的start方法,重點(diǎn)來了,這個方法包含整個docker進(jìn)程的啟動參數(shù),我們只找到我們要找的關(guān)鍵部分:

• 1.3.2.2 start方法

func (cli *DaemonCli) start(opts *daemonOptions) (err error) {
    ...
    # 創(chuàng)建apiserver
    cli.api = apiserver.New(serverConfig)

    # 綁定監(jiān)聽端口
    hosts, err := loadListeners(cli, serverConfig)
    if err != nil {
        return fmt.Errorf("Failed to load listeners: %v", err)
    }

    ctx, cancel := context.WithCancel(context.Background())
    if cli.Config.ContainerdAddr == "" && runtime.GOOS != "windows" {
            ...
            # 啟動containerd
            r, err := supervisor.Start(ctx, filepath.Join(cli.Config.Root, "containerd"), filepath.Join(cli.Config.ExecRoot, "containerd"), opts...)
            ...
            cli.Config.ContainerdAddr = r.Address()
            ...
    }
    ...

    # 創(chuàng)建pluginstore,用來保存plugin的相關(guān)信息
    pluginStore := plugin.NewStore()
    ...
    # 創(chuàng)建進(jìn)程,并為守護(hù)進(jìn)程設(shè)置一切服務(wù)
    d, err := daemon.NewDaemon(ctx, cli.Config, pluginStore)
    if err != nil {
        return fmt.Errorf("Error starting daemon: %v", err)
    }

    ...

    cli.d = d

    routerOptions, err := newRouterOptions(cli.Config, d)
    if err != nil {
        return err
    }
    routerOptions.api = cli.api
    routerOptions.cluster = c

    # 初始化路由
    initRouter(routerOptions)

    ...

    return nil
}

我們只關(guān)注調(diào)用流程,所以重點(diǎn)是初始化路由,這個函數(shù)的主要目的是綁定http請求到對應(yīng)方法:

• 1.3.2.3 initRouter函數(shù)

func initRouter(opts routerOptions) {
    decoder := runconfig.ContainerDecoder{}

    routers := []router.Router{
        // we need to add the checkpoint router before the container router or the DELETE gets masked
        checkpointrouter.NewRouter(opts.daemon, decoder),
        container.NewRouter(opts.daemon, decoder),
        image.NewRouter(opts.daemon.ImageService()),
        systemrouter.NewRouter(opts.daemon, opts.cluster, opts.buildCache, opts.buildkit, opts.features),
        volume.NewRouter(opts.daemon.VolumesService()),
        build.NewRouter(opts.buildBackend, opts.daemon, opts.features),
        sessionrouter.NewRouter(opts.sessionManager),
        swarmrouter.NewRouter(opts.cluster),
        pluginrouter.NewRouter(opts.daemon.PluginManager()),
        distributionrouter.NewRouter(opts.daemon.ImageService()),
    }

    ...

    opts.api.InitRouter(routers...)
}

1.4 容器相關(guān)的命令執(zhí)行

我們以容器為例具體看一下container.NewRouter():

1.4.1 NewRouter函數(shù)

func NewRouter(b Backend, decoder httputils.ContainerDecoder) router.Router {
    r := &containerRouter{
        backend: b,
        decoder: decoder,
    }
    r.initRoutes()
    return r
}

具體的綁定過程在initRoutes中執(zhí)行:

1.4.2 NewRouter函數(shù)

func (r *containerRouter) initRoutes() {
    r.routes = []router.Route{
        // HEAD
        router.NewHeadRoute("/containers/{name:.*}/archive", r.headContainersArchive),
        // GET
        router.NewGetRoute("/containers/json", r.getContainersJSON),
        router.NewGetRoute("/containers/{name:.*}/export", r.getContainersExport),
        router.NewGetRoute("/containers/{name:.*}/changes", r.getContainersChanges),
        router.NewGetRoute("/containers/{name:.*}/json", r.getContainersByName),
        router.NewGetRoute("/containers/{name:.*}/top", r.getContainersTop),
        router.NewGetRoute("/containers/{name:.*}/logs", r.getContainersLogs, router.WithCancel),
        router.NewGetRoute("/containers/{name:.*}/stats", r.getContainersStats, router.WithCancel),
        router.NewGetRoute("/containers/{name:.*}/attach/ws", r.wsContainersAttach),
        router.NewGetRoute("/exec/{id:.*}/json", r.getExecByID),
        router.NewGetRoute("/containers/{name:.*}/archive", r.getContainersArchive),
        // POST
        # 找到重點(diǎn)了,"/containers/create"
        router.NewPostRoute("/containers/create", r.postContainersCreate),
        router.NewPostRoute("/containers/{name:.*}/kill", r.postContainersKill),
        router.NewPostRoute("/containers/{name:.*}/pause", r.postContainersPause),
        router.NewPostRoute("/containers/{name:.*}/unpause", r.postContainersUnpause),
        router.NewPostRoute("/containers/{name:.*}/restart", r.postContainersRestart),
        router.NewPostRoute("/containers/{name:.*}/start", r.postContainersStart),
        router.NewPostRoute("/containers/{name:.*}/stop", r.postContainersStop),
        router.NewPostRoute("/containers/{name:.*}/wait", r.postContainersWait, router.WithCancel),
        router.NewPostRoute("/containers/{name:.*}/resize", r.postContainersResize),
        router.NewPostRoute("/containers/{name:.*}/attach", r.postContainersAttach),
        router.NewPostRoute("/containers/{name:.*}/copy", r.postContainersCopy), // Deprecated since 1.8, Errors out since 1.12
        router.NewPostRoute("/containers/{name:.*}/exec", r.postContainerExecCreate),
        router.NewPostRoute("/exec/{name:.*}/start", r.postContainerExecStart),
        router.NewPostRoute("/exec/{name:.*}/resize", r.postContainerExecResize),
        router.NewPostRoute("/containers/{name:.*}/rename", r.postContainerRename),
        router.NewPostRoute("/containers/{name:.*}/update", r.postContainerUpdate),
        router.NewPostRoute("/containers/prune", r.postContainersPrune, router.WithCancel),
        router.NewPostRoute("/commit", r.postCommit),
        // PUT
        router.NewPutRoute("/containers/{name:.*}/archive", r.putContainersArchive),
        // DELETE
        router.NewDeleteRoute("/containers/{name:.*}", r.deleteContainers),
    }
}

通過以上的執(zhí)行過程,我們找到了docker run最終調(diào)用的命令,接上文,繼續(xù)docker run命令的執(zhí)行.

1.4.3 postContainersCreate方法

通過路由綁定的http路徑,我們找到了docker run調(diào)用的方法r.postContainersCreate:

func (s *containerRouter) postContainersCreate(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error {
    ...

    ccr, err := s.backend.ContainerCreate(types.ContainerCreateConfig{
        Name:             name,
        Config:           config,
        HostConfig:       hostConfig,
        NetworkingConfig: networkingConfig,
        AdjustCPUShares:  adjustCPUShares,
    })
    ...
}

s.backend也就是前面DaemonCli的start方法中創(chuàng)建的deamon:

d, err := daemon.NewDaemon(ctx, cli.Config, pluginStore)

我們看一下deamon的定義:

1.4.4 Daemon結(jié)構(gòu)體

// Daemon holds information about the Docker daemon.
type Daemon struct {
    ID                string
    repository        string
    containers        container.Store
    containersReplica container.ViewDB
    execCommands      *exec.Store
    imageService      *images.ImageService
    idIndex           *truncindex.TruncIndex
    configStore       *config.Config
    statsCollector    *stats.Collector
    defaultLogConfig  containertypes.LogConfig
    RegistryService   registry.Service
    EventsService     *events.Events
    netController     libnetwork.NetworkController
    volumes           *volumesservice.VolumesService
    discoveryWatcher  discovery.Reloader
    root              string
    seccompEnabled    bool
    apparmorEnabled   bool
    shutdown          bool
    idMapping         *idtools.IdentityMapping
    // TODO: move graphDrivers field to an InfoService
    graphDrivers map[string]string // By operating system

    PluginStore           *plugin.Store // todo: remove
    pluginManager         *plugin.Manager
    linkIndex             *linkIndex
    containerdCli         *containerd.Client
    containerd            libcontainerd.Client
    defaultIsolation      containertypes.Isolation // Default isolation mode on Windows
    clusterProvider       cluster.Provider
    cluster               Cluster
    genericResources      []swarm.GenericResource
    metricsPluginListener net.Listener

    machineMemory uint64

    seccompProfile     []byte
    seccompProfilePath string

    diskUsageRunning int32
    pruneRunning     int32
    hosts            map[string]bool // hosts stores the addresses the daemon is listening on
    startupDone      chan struct{}

    attachmentStore       network.AttachmentStore
    attachableNetworkLock *locker.Locker
}

在相同路徑下,找到Daemon的ContainerCreate方法:

1.4.5 ContainerCreate方法

// ContainerCreate creates a regular container
func (daemon *Daemon) ContainerCreate(params types.ContainerCreateConfig) (containertypes.ContainerCreateCreatedBody, error) {
    return daemon.containerCreate(params, false)
}

對外可見方法ContainerCreate調(diào)用了私有方法containerCreate:

1.4.6 containerCreate方法

func (daemon *Daemon) containerCreate(params types.ContainerCreateConfig, managed bool) (containertypes.ContainerCreateCreatedBody, error) {
    start := time.Now()
    if params.Config == nil {
        return containertypes.ContainerCreateCreatedBody{}, errdefs.InvalidParameter(errors.New("Config cannot be empty in order to create a container"))
    }

    os := runtime.GOOS
    if params.Config.Image != "" {
        img, err := daemon.imageService.GetImage(params.Config.Image)
        if err == nil {
            os = img.OS
        }
    } else {
        // This mean scratch. On Windows, we can safely assume that this is a linux
        // container. On other platforms, it's the host OS (which it already is)
        if runtime.GOOS == "windows" && system.LCOWSupported() {
            os = "linux"
        }
    }

    warnings, err := daemon.verifyContainerSettings(os, params.HostConfig, params.Config, false)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    err = verifyNetworkingConfig(params.NetworkingConfig)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    if params.HostConfig == nil {
        params.HostConfig = &containertypes.HostConfig{}
    }
    err = daemon.adaptContainerSettings(params.HostConfig, params.AdjustCPUShares)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, errdefs.InvalidParameter(err)
    }

    container, err := daemon.create(params, managed)
    if err != nil {
        return containertypes.ContainerCreateCreatedBody{Warnings: warnings}, err
    }
    containerActions.WithValues("create").UpdateSince(start)

    return containertypes.ContainerCreateCreatedBody{ID: container.ID, Warnings: warnings}, nil
}

由此,我們找到了客戶端docker run命令執(zhí)行后的整個代碼流程.

本篇的主要目的是通過命令的代碼流程, 把客戶端的代碼與服務(wù)端的代碼連接起來,為以后探究其他命令的執(zhí)行過程打下基礎(chǔ),簡而言之,就是找到最終執(zhí)行命令的服務(wù)端代碼.

總結(jié)

1.起點(diǎn)

找到該命令調(diào)用的dockerCli.Client()下的函數(shù)名

2.然后跳轉(zhuǎn)到對應(yīng)的服務(wù)端的客戶端代碼

3.然后跳轉(zhuǎn)到對應(yīng)的api server路由找到調(diào)用的函數(shù)

4.上一步找到的函數(shù)一般在daemon中定義具體執(zhí)行