[k8s源碼分析][controller-manager] kube-controller-manager 啟動

1. 前言

轉(zhuǎn)載請說明原文出處, 尊重他人勞動成果!

源碼位置: https://github.com/nicktming/kubernetes/tree/tming-v1.13/pkg/controller/replicaset
分支: tming-v1.13 (基于v1.13版本)

本文將分析controller-manager的啟動過程.

2. 啟動

在kubernetes/cmd/kube-controller-manager/controller-manager.go中啟動.

// kubernetes/cmd/kube-controller-manager/controller-manager.go
func main() {
    rand.Seed(time.Now().UnixNano())
    command := app.NewControllerManagerCommand()
    ...
    if err := command.Execute(); err != nil {
        fmt.Fprintf(os.Stderr, "%v\n", err)
        os.Exit(1)
    }
}

進(jìn)入到

func NewControllerManagerCommand() *cobra.Command {
    s, err := options.NewKubeControllerManagerOptions()
    ...
    cmd := &cobra.Command{
        Use: "kube-controller-manager",
        ...
        Run: func(cmd *cobra.Command, args []string) {
            ...
            c, err := s.Config(KnownControllers(), ControllersDisabledByDefault.List())
            ...
            if err := Run(c.Complete(), wait.NeverStop); err != nil {
                fmt.Fprintf(os.Stderr, "%v\n", err)
                os.Exit(1)
            }
        },
    }
    ...
    return cmd
}

這里需要注意三個地方:
1. s, err := options.NewKubeControllerManagerOptions()創(chuàng)建一個默認(rèn)的KubeControllerManagerOptions對象s.
2. c, err := s.Config(KnownControllers(), ControllersDisabledByDefault.List())通過傳入的參數(shù)將s變成一個Config對象.
3. Run(c.Complete(), wait.NeverStop)啟動程序.

接下來將總這三個部分來繼續(xù)分析.

2.1 創(chuàng)建KubeControllerManagerOptions對象

// cmd/kube-controller-manager/app/options/options.go
func NewKubeControllerManagerOptions() (*KubeControllerManagerOptions, error) {
    componentConfig, err := NewDefaultComponentConfig(ports.InsecureKubeControllerManagerPort)
    if err != nil {
        return nil, err
    }
...
}

這里主要根據(jù)系統(tǒng)的一些默認(rèn)值生成了一個KubeControllerManagerOptions對象.

2.2 生成Config對象

===> kubernetes/cmd/kube-controller-manager/app/controllermanager.go
func KnownControllers() []string {
    ret := sets.StringKeySet(NewControllerInitializers(IncludeCloudLoops))
    ret.Insert(
        saTokenControllerName,
    )
    return ret.List()
}
func NewControllerInitializers(loopMode ControllerLoopMode) map[string]InitFunc {
    controllers := map[string]InitFunc{}
    controllers["endpoint"] = startEndpointController
    ...
    controllers["deployment"] = startDeploymentController
    ...
    controllers["nodeipam"] = startNodeIpamController
    if loopMode == IncludeCloudLoops {
        controllers["service"] = startServiceController
        controllers["route"] = startRouteController
    }
    ...
    controllers["root-ca-cert-publisher"] = startRootCACertPublisher
    return controllers
}

var ControllersDisabledByDefault = sets.NewString(
    "bootstrapsigner",
    "tokencleaner",
)

可以看到KnownControllers是所有需要啟動的controller的名字, 對應(yīng)的方法就是啟動其對應(yīng)controller的實(shí)體

func (s KubeControllerManagerOptions) Config(allControllers []string, disabledByDefaultControllers []string) (*kubecontrollerconfig.Config, error) {
    if err := s.Validate(allControllers, disabledByDefaultControllers); err != nil {
        return nil, err
    }

    if err := s.SecureServing.MaybeDefaultWithSelfSignedCerts("localhost", nil, []net.IP{net.ParseIP("127.0.0.1")}); err != nil {
        return nil, fmt.Errorf("error creating self-signed certificates: %v", err)
    }
    
    kubeconfig, err := clientcmd.BuildConfigFromFlags(s.Master, s.Kubeconfig)
    if err != nil {
        return nil, err
    }
    kubeconfig.ContentConfig.ContentType = s.Generic.ClientConnection.ContentType
    kubeconfig.QPS = s.Generic.ClientConnection.QPS
    kubeconfig.Burst = int(s.Generic.ClientConnection.Burst)
    
    // 生成一個與api-server打交道的client
    client, err := clientset.NewForConfig(restclient.AddUserAgent(kubeconfig, KubeControllerManagerUserAgent))
    if err != nil {
        return nil, err
    }

    // shallow copy, do not modify the kubeconfig.Timeout.
    config := *kubeconfig
    config.Timeout = s.Generic.LeaderElection.RenewDeadline.Duration
    leaderElectionClient := clientset.NewForConfigOrDie(restclient.AddUserAgent(&config, "leader-election"))

    eventRecorder := createRecorder(client, KubeControllerManagerUserAgent)
    
    c := &kubecontrollerconfig.Config{
        Client:               client,
        Kubeconfig:           kubeconfig,
        EventRecorder:        eventRecorder,
        LeaderElectionClient: leaderElectionClient,
    }
    if err := s.ApplyTo(c); err != nil {
        return nil, err
    }
    return c, nil
}

可以看到Config中生成四個變量:

client 用于與api-server進(jìn)行交流.
kubeconfig 配置
eventRecorder 記錄
LeaderElectionClient 高可用的時(shí)候用到

2.3 Run

===>cmd/kube-controller-manager/app/controllermanager.go
func Run(c *config.CompletedConfig, stopCh <-chan struct{}) error {
    ...
    run := func(ctx context.Context) {
        ...
        if c.ComponentConfig.KubeCloudShared.UseServiceAccountCredentials {
            if len(c.ComponentConfig.SAController.ServiceAccountKeyFile) == 0 {
                // 這個就是在啟動的時(shí)候需要指定--service-account-private-key-file
                klog.Warningf("--use-service-account-credentials was specified without providing a --service-account-private-key-file")
            }
            clientBuilder = controller.SAControllerClientBuilder{
                ClientConfig:         restclient.AnonymousClientConfig(c.Kubeconfig),
                CoreClient:           c.Client.CoreV1(),
                AuthenticationClient: c.Client.AuthenticationV1(),
                Namespace:            "kube-system",
            }
        } else {
            clientBuilder = rootClientBuilder
        }
        controllerContext, err := CreateControllerContext(c, rootClientBuilder, clientBuilder, ctx.Done())
        if err != nil {
            klog.Fatalf("error building controller context: %v", err)
        }
        saTokenControllerInitFunc := serviceAccountTokenControllerStarter{rootClientBuilder: rootClientBuilder}.startServiceAccountTokenController

        // 啟動所有controller
        if err := StartControllers(controllerContext, saTokenControllerInitFunc, NewControllerInitializers(controllerContext.LoopMode), unsecuredMux); err != nil {
            klog.Fatalf("error starting controllers: %v", err)
        }
        // 啟動所有注冊的infromers
        controllerContext.InformerFactory.Start(controllerContext.Stop)
        close(controllerContext.InformersStarted)

        select {}
    }

    if !c.ComponentConfig.Generic.LeaderElection.LeaderElect {
        run(context.TODO())
        panic("unreachable")
    }

    id, err := os.Hostname()
    if err != nil {
        return err
    }

    // add a uniquifier so that two processes on the same host don't accidentally both become active
    id = id + "_" + string(uuid.NewUUID())
    rl, err := resourcelock.New(c.ComponentConfig.Generic.LeaderElection.ResourceLock,
        "kube-system",
        "kube-controller-manager",
        c.LeaderElectionClient.CoreV1(),
        resourcelock.ResourceLockConfig{
            Identity:      id,
            EventRecorder: c.EventRecorder,
        })
    if err != nil {
        klog.Fatalf("error creating lock: %v", err)
    }

    // 高可用
    leaderelection.RunOrDie(context.TODO(), leaderelection.LeaderElectionConfig{
        Lock:          rl,
        LeaseDuration: c.ComponentConfig.Generic.LeaderElection.LeaseDuration.Duration,
        RenewDeadline: c.ComponentConfig.Generic.LeaderElection.RenewDeadline.Duration,
        RetryPeriod:   c.ComponentConfig.Generic.LeaderElection.RetryPeriod.Duration,
        Callbacks: leaderelection.LeaderCallbacks{
            OnStartedLeading: run,
            OnStoppedLeading: func() {
                klog.Fatalf("leaderelection lost")
            },
        },
        WatchDog: electionChecker,
        Name:     "kube-controller-manager",
    })
    panic("unreachable")
}

這里需要注意幾點(diǎn):

1. 這個就是在啟動的時(shí)候需要指定--service-account-private-key-file文件, 在 k8s源碼編譯以及二進(jìn)制安裝(用于源碼開發(fā)調(diào)試版) 中已經(jīng)遇到過了, 該問題會在別的文章分析.
2. controllerContext屬性中有InformerFactory, 并且所有的controller用的都是這個InformerFactory, 因此所有用到同一個informer的controller里面都是一樣的, 比如所有用到podInformer的controller用的都是同一個對象.
3. StartControllers方法中啟動所有的controller, 每個controller都是以goroutine的方式啟動.

===>cmd/kube-controller-manager/app/controllermanager.go
func StartControllers(ctx ControllerContext, startSATokenController InitFunc, controllers map[string]InitFunc, unsecuredMux *mux.PathRecorderMux) error {
    ...
    for controllerName, initFn := range controllers {
        if !ctx.IsControllerEnabled(controllerName) {
            klog.Warningf("%q is disabled", controllerName)
            continue
        }
        ...
        // 都是以goroutine的方式啟動
        debugHandler, started, err := initFn(ctx)
        ...
    }
    return nil
}
  1. 高可用, 使用的其實(shí)也是endpoint資源類型, 這里不多說了, 可以參考 [k8s源碼分析][client-go] k8s選舉leaderelection (分布式資源鎖實(shí)現(xiàn)) 和 [k8s源碼分析][kube-scheduler]scheduler之高可用及原理 .
[root@master kubectl]# ./kubectl get endpoints -n kube-system
NAME                      ENDPOINTS   AGE
kube-controller-manager   <none>      22d
kube-scheduler            <none>      22d
[root@master kubectl]# ./kubectl get endpoints kube-controller-manager -o yaml -n kube-system
apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master_6efbea00-0168-11ea-b452-525400d54f7e","leaseDurationSeconds":15,"acquireTime":"2019-11-07T14:11:21Z","renewTime":"2019-11-07T14:12:01Z","leaderTransitions":15}'
  creationTimestamp: "2019-10-15T14:57:16Z"
  name: kube-controller-manager
  namespace: kube-system
  resourceVersion: "131420"
  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager
  uid: 13f00d33-ef5c-11e9-af01-525400d54f7e
[root@master kubectl]#

3. 總結(jié)

conclusion.png
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請結(jié)合常識與多方信息審慎甄別。
平臺聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡書系信息發(fā)布平臺，僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容