概覽

• kata container簡介
• cri-containerd plugin + containerd方式
• cri-o方式

kata container簡介

kata containers是由OpenStack基金會管理，但獨立于OpenStack項目之外的容器項目。它是一個可 以使用容器鏡像以超輕量級虛機的形式創(chuàng)建容器的運行時工具。 kata containers整合了Intel的 Clear Containers 和 Hyper.sh 的 runV，能夠支持不同平臺的硬件 （x86-64，arm等），并符合OCI(Open Container Initiative)規(guī)范，同時還可以兼容k8s的 CRI（Container Runtime Interface）接口規(guī)范。目前項目包含幾個配套組件，即Runtime，Agent， Proxy，Shim，Kernel等。目前Kata Containers的運行時還沒有整合，即Clear containers 和 runV 還在獨立的組織內(nèi)。

• runtime：符合OCI規(guī)范的容器運行時命令工具，主要用來創(chuàng)建輕量級虛機，并通過agent控制虛 擬機內(nèi)容器的生命周期。目前kata containers還沒有一個統(tǒng)一的運行時工具，用戶可以選擇 clear containers和runV中的其中之一。

• agent：運行在虛機中的一個運行時代理組件，主要用來執(zhí)行runtime傳給他的指令，在虛機內(nèi) 管理容器的生命周期。

• shim：以對接docker為例，這里的shim相當于是containerd-shim的一個適配，用來處理容器 進程的stdio和signals。shim可以將containerd-shim發(fā)來的數(shù)據(jù)流（如stdin）傳給proxy，然 后轉交給agent，也可以將agent經(jīng)由proxy發(fā)過來的數(shù)據(jù)流（stdout/stderr）傳給containerdshim，同時也可以傳輸signal。

• kernel：kernel其實比較好理解，就是提供一個輕量化虛機的linux內(nèi)核，根據(jù)不同的需要，提供 幾個內(nèi)核選擇，最小的內(nèi)核僅有4M多

kata和傳統(tǒng)容器對比

kata and docker

kata in kubernetes

OCI部分即為kata,也可以是符合oci標準的運行時容器。

以下環(huán)境安裝默認都是overlayfs ,centos 建議版本7.4，7.5版本默認f_type=0,不支持D_TYPE參數(shù)，會導致containerd和crio啟動失敗

kata + cri-containerd plugin + containerd

環(huán)境版本

kubeadm      v1.10.3
kubenetes    v1.10.3
crictl       v1.0.0-beta.0
containerd   v1.1.0
kata-runtime 1.0.0

kubeadm安裝

http://www.itdecent.cn/p/3ec8945a864f

此處只需安裝一個主節(jié)點即可，kata容器所在虛擬機均為從節(jié)點

鏡像版本

k8s.gcr.io/kube-proxy-amd64               v1.10.3 
k8s.gcr.io/kube-apiserver-amd64           v1.10.3 
k8s.gcr.io/kube-scheduler-amd64           v1.10.3 
k8s.gcr.io/kube-controller-manager-amd64  v1.10.3 
k8s.gcr.io/etcd-amd64                     3.1.12  
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64    1.14.8  
k8s.gcr.io/k8s-dns-sidecar-amd64          1.14.8  
k8s.gcr.io/k8s-dns-kube-dns-amd64         1.14.8  
k8s.gcr.io/pause-amd64                    3.1     
quay.io/coreos/flannel                    v0.10.0-amd64

kata 安裝

sudo yum -y install yum-utils

source /etc/os-release

sudo -E VERSION_ID=$VERSION_ID yum-config-manager --add-repo \
"http://download.opensuse.org/repositories/home:/katacontainers:/release/CentOS_${VERSION_ID}/home:katacontainers:release.repo"

sudo -E yum -y install kata-runtime kata-proxy kata-shim

安裝完成后測試

[root@localhost tmp]# kata-runtime -v
kata-runtime  : 1.0.0
   commit   : a3d11b73ac4f1d5c5ae6451d83fc4b1fca385098
   OCI specs: 1.0.1

cri-containerd 安裝

wget https://github.com/containerd/containerd/releases/download/v1.1.0/containerd-1.1.0.linux-amd64.tar.gz

sudo tar -C / -xzf  containerd-1.1.0.linux-amd64.tar.gz

sudo systemctl start containerd

sudo systemctl status containerd

containerd config default #生成默認配置，位置/etc/containerd/config.toml

crictl 安裝

確保有go環(huán)境

go get github.com/kubernetes-incubator/cri-tools/cmd/crictl

export PATH=$PATH:$GOPATH/bin

配置修改

crictl配置

[root@localhost bin]# cat /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: true

containerd配置

[root@localhost bin]# cat /etc/containerd/config.toml 
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = ""
    stream_server_port = "10010"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause-amd64:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      ## default_runtime為默認runtime
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      ## untrusted_workload_runtime當pod參數(shù)帶有
      ## io.kubernetes.cri.untrusted-workload: "true"時生效
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = "/usr/bin/kata-runtime"
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://9c56nwew.mirror.aliyuncs.com", "https://registry-1.docker.io"]
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

各參數(shù)配置含義：https://github.com/containerd/cri/blob/release/1.0/docs/config.md

kubelet配置

[root@localhost bin]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

## 注意該行配置，修改kubelet啟動使用containerd
Environment="KUBELET_EXTRA_ARGS=--runtime-cgroups=/system.slice/containerd.service --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

關于鏡像下載的問題

由于默認鏡像都是 k8s.gcr.io上的無法訪問，所以最好有docker環(huán)境給鏡像打tag，然后使用

ctr cri load xxx.tar 

cri-o + kata

kata安裝

同上

cri-o安裝

yum install -y \
  btrfs-progs-devel \
  device-mapper-devel \
  git \
  glib2-devel \
  glibc-devel \
  glibc-static \
  go \
  golang-github-cpuguy83-go-md2man \
  gpgme-devel \
  libassuan-devel \
  libgpg-error-devel \
  libseccomp-devel \
  libselinux-devel \
  ostree-devel \
  pkgconfig \
  runc \
  skopeo-containers

git clone https://github.com/kubernetes-incubator/cri-o # or your fork
cd cri-o

make install.tools
make
sudo make install

##第一次安裝時生成初始化配置文件
make install.config

make install.tools時可能有些package下載不下來，建議直接在git clone到go path下

配置

修改crio配置

vim /etc/crio/crio.conf 

主要修改crio.runtime的配置

[crio.runtime]
runtime = "/usr/bin/runc"
runtime_untrusted_workload = "/usr/bin/kata-runtime"
default_workload_trust = "trusted"

kubelet配置

[root@localhost crio]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

## 注意該行配置，修改kubelet啟動使用crio
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///var/run/crio/crio.sock"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

關于鏡像

cri-o缺少像containerd那樣的鏡像管理工具，個人采用的方式時修改主節(jié)點kube-proxy和kube-flannel-ds這兩個DaemonSet的鏡像為本地個人上傳的阿里云的鏡像，才能啟動成功。

[root@kata-node1 tmp]# kubectl get ds -n kube-system
NAME              DESIRED   CURRENT   READY UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-flannel-ds   3            3        3       3             3    beta.kubernetes.io/arch=amd64   4h
kube-proxy        3            3        3       3             3       <none>                    5h              

kubectl edit kube-flannel-ds -n kube-system

加入集群

環(huán)境安裝完成后各節(jié)點使用

kubeadm join 172.20.95.208:6443 --token ss60t6.fb1hymd6qxf5tbx7 --discovery-token-ca-cert-hash sha256:ea3a8f578071f73e7a5aab2e783f9d1cce22d4cf70faad24b8eb9090a4ef6b3b --ignore-preflight-errors all

具體命令使用主節(jié)點初始化完成后的kubeadm join命令 ，只需加上--ignore-preflight-errors all即可

測試

[root@kata-node1 tmp]# cat nginx-untrusted-1.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted-1
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx-1
    image: nginx


kubecrl create -f  nginx-untrusted-1.yaml