1. 前言

轉(zhuǎn)載請說明原文出處, 尊重他人勞動成果!

源碼位置: https://github.com/nicktming/kubernetes/tree/tming-v1.13/staging/src/k8s.io/code-generator
分支: tming-v1.13 (基于v1.13版本)

本文將在上文 [k8s源碼分析][code-generator] crd代碼生成的基礎(chǔ)上進(jìn)行分析代碼生成是如何工作的. 本文著重分析list, 由于代碼生成這個功能的作用就是為了簡化繁瑣的操作, 即使不使用代碼生成, 自己也可以手寫相關(guān)代碼, 所以說不能一個不能或缺的功能, 只是說錦上添花而已, 因此本文將簡單分析其流程, 不會特別細(xì)致到每個變量的作用等等.

2. 分析

上文生成的client, 下面有三個文件夾, 主要看一下listers文件夾是如何生成的.

[root@master client]# pwd
/root/go/src/github.com/nicktming/k8s-crd-controller/pkg/client
[root@master client]# ls
clientset  informers  listers
[root@master client]# tree listers
listers
└── example.com
    └── v1
        ├── database.go
        └── expansion_generated.go
2 directories, 2 files
[root@master client]# 

上文關(guān)于lister的生成命令是/root/go/bin/lister-gen --input-dirs github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1 --output-package github.com/nicktming/k8s-crd-controller/pkg/client/listers

接下來將以這條主線進(jìn)行分析:

2.1 main方法

// kubernetes/staging/src/k8s.io/code-generator/cmd/lister-gen/main.go
    // Run it.
    if err := genericArgs.Execute(
        generators.NameSystems(),
        generators.DefaultNameSystem(),
        generators.Packages,
    );

genericArgs是一個GeneratoArgs對象, 在k8s.io/gengo/args/args.go中, 可以不用管, 只需要知道此處會執(zhí)行什么就可以了.

//k8s.io/gengo/args/args.go
func (g *GeneratorArgs) Execute(nameSystems namer.NameSystems, defaultSystem string, pkgs func(*generator.Context, *GeneratorArgs) generator.Packages) error {
    ...
    b, err := g.NewBuilder()
    if err != nil {
        return fmt.Errorf("Failed making a parser: %v", err)
    }

    c, err := generator.NewContext(b, nameSystems, defaultSystem)
    if err != nil {
        return fmt.Errorf("Failed making a context: %v", err)
    }

    c.Verify = g.VerifyOnly
    packages := pkgs(c, g)
    if err := c.ExecutePackages(g.OutputBase, packages); err != nil {
        return fmt.Errorf("Failed executing generator: %v", err)
    }

    return nil
}

可以看到傳進(jìn)來三個參數(shù)nameSystems, defaultSystem和pkgs, 這里的作用是利用pkgs方法得到需要生成的packages, 然后調(diào)用ExecutePackages去生成真正的文件.
這里需要注意的是pkgs中第一個參數(shù)c已經(jīng)解析了傳進(jìn)來的--input-dirs對應(yīng)的文件的內(nèi)容, 在后面會看到.

2.2 Packages

這個方法就是生成最終需要生成的Packages, 然后GeneratorArgs調(diào)用ExecutePackages方法根據(jù)Packages生成最終的文件. 定義如下:

// k8s.io/gengo/generator/generator.go
type Packages []Package
type Package interface {
    // Name returns the package short name.
    Name() string
    // Path returns the package import path.
    Path() string
    // 判斷是否需要過濾此Type
    Filter(*Context, *types.Type) bool
    // 頭部信息
    Header(filename string) []byte
    // 如何生成
    Generators(*Context) []Generator
}

由于該方法是核心方法, 所以就拆分一段一段進(jìn)行分析:

生成頭部信息

boilerplate, err := arguments.LoadGoBoilerplate()

此段代碼是文件頭部信息, 內(nèi)容如下:

/*
Copyright The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

// Code generated by lister-gen. DO NOT EDIT.

分析每個InputDirs

p := context.Universe.Package(inputDir)

對應(yīng)內(nèi)容如下, 上文已經(jīng)說過context中已經(jīng)對inputDir解析過了, 這里u[packagePath]就會直接返回該packagePath下解析處理的Package.

func (u Universe) Package(packagePath string) *Package {
    if p, ok := u[packagePath]; ok {
        return p
    }
    p := &Package{
        Path:      packagePath,
        Types:     map[string]*Type{},
        Functions: map[string]*Type{},
        Variables: map[string]*Type{},
        Imports:   map[string]*Package{},
    }
    u[packagePath] = p
    return p
}

Package對象如下: 該結(jié)構(gòu)體描述了整個packagePath的所有信息.

type Package struct {
    // Canonical name of this package-- its path.
    Path string
    // The location this package was loaded from
    SourcePath string
    // Short name of this package; the name that appears in the
    // 'package x' line. 
    Name string
    // The comment right above the package declaration in doc.go, if any.
    DocComments []string
    // All comments from doc.go, if any.
    Comments []string
    // 所有的Type(結(jié)構(gòu)體和接口)
    Types map[string]*Type
    // 所有的方法
    Functions map[string]*Type
    // 所有的變量
    Variables map[string]*Type
    // 所有的import
    Imports map[string]*Package
}

對應(yīng)的Package信息如下:
inputDir: github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1
Path:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1
SourcePath:/root/go/src/github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1
Name:v1
DocComments:[+groupName=nicktming.example.com]
Comments:[+k8s:deepcopy-gen=package,register +groupName=nicktming.example.com]
Types:map[Database:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.Database DatabaseList:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.DatabaseList DatabaseSpec:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.DatabaseSpec]
Functions:map[Kind:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.Kind Resource:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.Resource addKnownTypes:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.addKnownTypes]
Variables:map[AddToScheme:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.AddToScheme SchemeBuilder:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.SchemeBuilder SchemeGroupVersion:github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.SchemeGroupVersion]
Imports:map[k8s.io/apimachinery/pkg/apis/meta/v1:0xc005497d00 k8s.io/apimachinery/pkg/runtime:0xc005497a80 k8s.io/apimachinery/pkg/runtime/schema:0xc005497c00]

生成objectMeta和group version

        objectMeta, internal, err := objectMetaForPackage(p)
        if err != nil {
            klog.Fatal(err)
        }
        if objectMeta == nil {
            // no types in this package had genclient
            continue
        }

        fmt.Printf("objectMeta:%v, internal:%v\n", objectMeta.Name, internal)

        var gv clientgentypes.GroupVersion
        var internalGVPkg string

        if internal {
            lastSlash := strings.LastIndex(p.Path, "/")
            if lastSlash == -1 {
                klog.Fatalf("error constructing internal group version for package %q", p.Path)
            }
            gv.Group = clientgentypes.Group(p.Path[lastSlash+1:])
            internalGVPkg = p.Path
        } else {
            // 根據(jù)Path來區(qū)別group verion
            parts := strings.Split(p.Path, "/")
            gv.Group = clientgentypes.Group(parts[len(parts)-2])
            gv.Version = clientgentypes.Version(parts[len(parts)-1])

            internalGVPkg = strings.Join(parts[0:len(parts)-1], "/")
        }
        groupPackageName := strings.ToLower(gv.Group.NonEmpty())

        // If there's a comment of the form "http:// +groupName=somegroup" or
        // "http:// +groupName=somegroup.foo.bar.io", use the first field (somegroup) as the name of the
        // group when generating.
        if override := types.ExtractCommentTags("+", p.Comments)["groupName"]; override != nil {
            gv.Group = clientgentypes.Group(strings.SplitN(override[0], ".", 2)[0])
        }

        fmt.Printf("Group:%v, verion:%v\n", gv.Group, gv.Version)

尋找第一個帶有genclient和ObjectMeat的Type, 并且將其返回.

// objectMetaForPackage returns the type of ObjectMeta used by package p.
func objectMetaForPackage(p *types.Package) (*types.Type, bool, error) {
    generatingForPackage := false
    for _, t := range p.Types {
        // filter out types which dont have genclient.
        // 如果該Type上面沒有g(shù)enclient注釋 直接跳過
        if !util.MustParseClientGenTags(append(t.SecondClosestCommentLines, t.CommentLines...)).GenerateClient {
            continue
        }
        generatingForPackage = true
        for _, member := range t.Members {
            // 返回?fù)碛蠴bjectMeta屬性的Type 在例子是Database
            if member.Name == "ObjectMeta" {
                return member.Type, isInternal(member), nil
            }
        }
    }
    if generatingForPackage {
        return nil, false, fmt.Errorf("unable to find ObjectMeta for any types in package %s", p.Path)
    }
    return nil, false, nil
}

// tag中不擁有json 表明是internal
// isInternal returns true if the tags for a member do not contain a json tag
func isInternal(m types.Member) bool {
    return !strings.Contains(m.Tags, "json")
}

輸出結(jié)果如下:

objectMeta:k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta, internal:false
Group:nicktming, verion:v1

這里需要說明一下, group名字選的是+groupName中按.劃分的第一個, 這里只是為了生成方法, 但是真正的GroupVersion是在register.go中的變量

// k8s-crd-controller/pkg/apis/example.com/v1/register.go
var SchemeGroupVersion = schema.GroupVersion{Group: "nicktming.example.com", Version: "v1"}

// k8s-crd-controller/pkg/client/clientset/versioned/typed/example.com/v1/example.com_client.go
func setConfigDefaults(config *rest.Config) error {
    gv := v1.SchemeGroupVersion
    config.GroupVersion = &gv
    config.APIPath = "/apis"
    config.NegotiatedSerializer = serializer.DirectCodecFactory{CodecFactory: scheme.Codecs}

    if config.UserAgent == "" {
        config.UserAgent = rest.DefaultKubernetesUserAgent()
    }

    return nil
}

在setConfigDefaults中使用了SchemeGroupVersion然后在rest中生成的request URL會用到. 比如:curl http://localhost:8080/apis/nicktming.example.com/v1/namespaces/default/databases/my-database.

生成typesToGenerate

var typesToGenerate []*types.Type
        for _, t := range p.Types {
            tags := util.MustParseClientGenTags(append(t.SecondClosestCommentLines, t.CommentLines...))
            if !tags.GenerateClient || !tags.HasVerb("list") || !tags.HasVerb("get") {
                continue
            }
            typesToGenerate = append(typesToGenerate, t)
        }
        if len(typesToGenerate) == 0 {
            continue
        }
        orderer := namer.Orderer{Namer: namer.NewPrivateNamer(0)}
        typesToGenerate = orderer.OrderTypes(typesToGenerate)
        packagePath := filepath.Join(arguments.OutputPackagePath, groupPackageName, strings.ToLower(gv.Version.NonEmpty()))
        fmt.Printf("typesToGenerate:%v, packagePath:%v\n", typesToGenerate, packagePath)

這段代碼的意思是確定哪些Type需要生成lister. (哪些含有+genclient注解的或者tag中含有list或get的), 很顯然只有Database符合.
輸出結(jié)果如下:

typesToGenerate:[github.com/nicktming/k8s-crd-controller/pkg/apis/example.com/v1.Database], 
packagePath:github.com/nicktming/k8s-crd-controller/pkg/client/listers/example.com/v1

生成Package并加入到packageList中

主要看一下GeneratorFunc

GeneratorFunc: func(c *generator.Context) (generators []generator.Generator) {
                generators = append(generators, &expansionGenerator{
                    DefaultGen: generator.DefaultGen{
                        OptionalName: "expansion_generated",
                    },
                    packagePath: filepath.Join(arguments.OutputBase, packagePath),
                    types:       typesToGenerate,
                })

                for _, t := range typesToGenerate {
                    generators = append(generators, &listerGenerator{
                        DefaultGen: generator.DefaultGen{
                            OptionalName: strings.ToLower(t.Name.Name),
                        },
                        outputPackage:  arguments.OutputPackagePath,
                        groupVersion:   gv,
                        internalGVPkg:  internalGVPkg,
                        typeToGenerate: t,
                        imports:        generator.NewImportTracker(),
                        objectMeta:     objectMeta,
                    })
                }
                return generators
            },

可以看到第一個方法是生成expansion_generated.go文件. 對應(yīng)著expansionGenerator類型.
第二個方法是生成database.go文件. 對應(yīng)著listerGenerator類型.

expansionGenerator

func (g *expansionGenerator) GenerateType(c *generator.Context, t *types.Type, w io.Writer) error {
    sw := generator.NewSnippetWriter(w, c, "$", "$")
    for _, t := range g.types {
        tags := util.MustParseClientGenTags(append(t.SecondClosestCommentLines, t.CommentLines...))
        if _, err := os.Stat(filepath.Join(g.packagePath, strings.ToLower(t.Name.Name+"_expansion.go"))); os.IsNotExist(err) {
            sw.Do(expansionInterfaceTemplate, t)
            if !tags.NonNamespaced {
                sw.Do(namespacedExpansionInterfaceTemplate, t)
            }
        }
    }
    return sw.Error()
}

var expansionInterfaceTemplate = `
// $.|public$ListerExpansion allows custom methods to be added to
// $.|public$Lister.
type $.|public$ListerExpansion interface {}
`

var namespacedExpansionInterfaceTemplate = `
// $.|public$NamespaceListerExpansion allows custom methods to be added to
// $.|public$NamespaceLister.
type $.|public$NamespaceListerExpansion interface {}
`

就是替換template中的某些字段成該type的某些字段就可以了.

listerGenerator

func (g *listerGenerator) GenerateType(c *generator.Context, t *types.Type, w io.Writer) error {
    ...
    sw.Do(typeListerStruct, m)
    sw.Do(typeListerConstructor, m)
    sw.Do(typeLister_List, m)
    ...
    return sw.Error()
}

var typeListerInterface = `
// $.type|public$Lister helps list $.type|publicPlural$.
type $.type|public$Lister interface {
    // List lists all $.type|publicPlural$ in the indexer.
    List(selector labels.Selector) (ret []*$.type|raw$, err error)
    // $.type|publicPlural$ returns an object that can list and get $.type|publicPlural$.
    $.type|publicPlural$(namespace string) $.type|public$NamespaceLister
    $.type|public$ListerExpansion
}
`
...

作用一樣, 替換字段即可.

4. 總結(jié)

1. 遍歷每一個inputDir.
2. 解析該inputDir下面所有文件并整理成一個package結(jié)構(gòu)體, 里面存有關(guān)于該inputDir下面所有類型(Type), 方法(funcs), 變量(variables)等等.
3. 找到需要生成lister的類型. 指定的是那些帶有+genclient或tag中帶有list或get的類型(Type). (tag指的是Encoding string `json:"encoding,omitempty"后面用`那段包括的內(nèi)容)
4. 根據(jù)已經(jīng)定義好的模板創(chuàng)建DefaultPackage, 里面包括了如何生成等等.
5. 調(diào)用ExecutePackages將返回的Packages生成對應(yīng)文件.