容器與宿主機(jī)共用內(nèi)核，修改內(nèi)核參數(shù)可能會(huì)影響到宿主機(jī)和其他容器。非特權(quán)容器大部分內(nèi)核參數(shù)無(wú)法修改，可以修改的內(nèi)核參數(shù)主要涉及到 IPC Namespace和Net Namespace,可修改內(nèi)核參數(shù)需要滿足3個(gè)條件

• docker白名單
• 可命名空間化
• 容器中可見(jiàn)

1、docker白名單

• IPC Namespace
  kernel.msgmax
  kernel.msgmnb
  kernel.msgmni
  kernel.sem
  kernel.shmall
  kernel.shmmax
  kernel.shmmni
  kernel.shm_rmid_forced
  fs.mqueue.msg_default
  fs.mqueue.msg_max
  fs.mqueue.msgsize_default
  fs.mqueue.msgsize_max
  fs.mqueue.queues_max
• Net Namespace
  net.*

2、可命名空間化的

使用特權(quán)的容器可以修改所有容器可見(jiàn)的參數(shù)，其中不影響宿主機(jī)的為可命名空間化的，與白名單大致一致。

# sysctl -a | grep fs.mqueue.msg_max
fs.mqueue.msg_max = 10
# docker run   --privileged --rm centos /bin/bash -c "sysctl -w fs.mqueue.msg_max=5"
fs.mqueue.msg_max = 5
# sysctl -a | grep fs.mqueue.msg_max                            
fs.mqueue.msg_max = 10

3、符合白名單但是容器中不可見(jiàn)的參數(shù)

符合白名單且不可見(jiàn)的參數(shù)有：
net.bridge.bridge-nf-call-arptables
net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-filter-pppoe-tagged
net.bridge.bridge-nf-filter-vlan-tagged
net.bridge.bridge-nf-pass-vlan-input-dev
net.core.bpf_jit_enable
net.core.bpf_jit_harden
net.core.bpf_jit_kallsyms
net.core.bpf_jit_limit
net.core.busy_poll
net.core.busy_read
net.core.default_qdisc
net.core.dev_weight
net.core.dev_weight_rx_bias
net.core.dev_weight_tx_bias
net.core.fb_tunnels_only_for_init_net
net.core.flow_limit_cpu_bitmap
net.core.flow_limit_table_len
net.core.max_skb_frags
net.core.message_burst
net.core.message_cost
net.core.netdev_budget
net.core.netdev_budget_usecs
net.core.netdev_max_backlog
net.core.netdev_rss_key
net.core.netdev_tstamp_prequeue
net.core.optmem_max
net.core.rmem_default
net.core.rmem_max
net.core.rps_sock_flow_entries
net.core.tstamp_allow_data
net.core.warnings
net.core.wmem_default
net.core.wmem_max
net.ipv4.icmp_msgs_burst
net.ipv4.icmp_msgs_per_sec
net.ipv4.inet_peer_maxttl
net.ipv4.inet_peer_minttl
net.ipv4.inet_peer_threshold
net.ipv4.ipfrag_secret_interval
net.ipv4.route.error_burst
net.ipv4.route.error_cost
net.ipv4.route.gc_elasticity
net.ipv4.route.gc_interval
net.ipv4.route.gc_min_interval
net.ipv4.route.gc_min_interval_ms
net.ipv4.route.gc_thresh
net.ipv4.route.gc_timeout
net.ipv4.route.max_size
net.ipv4.route.min_adv_mss
net.ipv4.route.min_pmtu
net.ipv4.route.mtu_expires
net.ipv4.route.redirect_load
net.ipv4.route.redirect_number
net.ipv4.route.redirect_silence
net.ipv4.tcp_allowed_congestion_control
net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_ulp
net.ipv4.tcp_low_latency
net.ipv4.tcp_max_orphans
net.ipv4.tcp_mem
net.ipv4.udp_mem
net.netfilter.nf_log_all_netns
net.nf_conntrack_max
net.unix.max_dgram_qlen
net.netfilter.nf_conntrack_tcp_timeout_close_wait
net.netfilter.nf_conntrack_tcp_timeout_established
net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.all.disable_ipv6
net.netfilter.nf_conntrack_count

4、常見(jiàn)可修改的內(nèi)核參數(shù)

net.core.somaxconn 
net.ipv4.conf.xxx.proxy_arp_pvlan 
net.ipv4.ip_default_ttl 
net.ipv4.ip_forward 
net.ipv4.tcp_base_mss 
net.ipv4.tcp_sack 
net.ipv4.tcp_syncookies 
net.ipv4.tcp_timestamps 
net.ipv4.tcp_tw_reuse
net.ipv4.tcp_window_scaling 
net.ipv4.tcp_wmem
net.ipv4.udp_rmem_min 
net.ipv4.udp_wmem_min 
...

5、容器缺省會(huì)修改的內(nèi)核參數(shù)

docker中有部分內(nèi)核參數(shù)缺省會(huì)修改，需要注意

net.unix.max_dgram_qlen = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.netfilter.nf_conntrack_count = 0

6、docker中修改內(nèi)核參數(shù)

如果ipc和net配置為host，則無(wú)法修改。
擁有特權(quán)的容器可以修改所有可見(jiàn)參數(shù)，可能會(huì)影響宿主機(jī)和其他容器。
非特權(quán)容器內(nèi)內(nèi)核參數(shù)為只讀文件系統(tǒng)，任何內(nèi)核參數(shù)在容器內(nèi)部修改會(huì)報(bào)錯(cuò)。

# sysctl -w net.ipv6.icmp.ratelimit=500
sysctl: setting key "net.ipv6.icmp.ratelimit": Read-only file system

在docker run中修改不在白名單中的內(nèi)核參數(shù)，則會(huì)報(bào)錯(cuò)不在白名單

#  docker run -it --sysctl vm.swappiness=10 centos /bin/bash    invalid argument "vm.swappiness=10" for "--sysctl" flag: sysctl 'vm.swappiness=10' is not whitelisted
See 'docker run --help'.

在docker run中修改不可見(jiàn)內(nèi)核參數(shù)，容器會(huì)報(bào)文件不存在

#  docker run -it --sysctl net.core.busy_poll=1 centos /bin/bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"write sysctl key net.core.busy_poll: open /proc/sys/net/core/busy_poll: no such file or directory\"": unknown.
ERRO[0000] error waiting for container: context canceled

在docker run中修改可修改的內(nèi)核參數(shù)net.ipv4.ip_default_ttl=32

# sysctl -a | grep net.ipv4.ip_default_ttl
net.ipv4.ip_default_ttl = 64
# docker run --rm centos /bin/bash -c "ping -c 1 172.17.0.2"
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=64 time=0.279 ms

--- 172.17.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.279/0.279/0.279/0.000 ms
# docker run --sysctl net.ipv4.ip_default_ttl=32 --rm centos /bin/bash -c "ping -c 1 172.17.0.2"
PING 172.17.0.2 (172.17.0.2) 56(84) bytes of data.
64 bytes from 172.17.0.2: icmp_seq=1 ttl=32 time=0.033 ms

--- 172.17.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.033/0.033/0.033/0.000 ms

調(diào)整內(nèi)核參數(shù)啟動(dòng)ipv6

# docker run --rm  centos ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
148: eth0@if149: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
# docker run --sysctl net.ipv6.conf.default.disable_ipv6=0 --sysctl net.ipv6.conf.all.disable_ipv6=0 --rm  centos ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
150: eth0@if151: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link tentative
       valid_lft forever preferred_lft forever

6、K8S中修改內(nèi)核參數(shù)

查看相關(guān)參數(shù)

# kubectl explain pod.spec.securityContext.sysctls
KIND:     Pod
VERSION:  v1
RESOURCE: sysctls <[]Object>
DESCRIPTION:
     Sysctls hold a list of namespaced sysctls used for the pod. Pods with
     unsupported sysctls (by the container runtime) might fail to launch.
     Sysctl defines a kernel parameter to be set
FIELDS:
   name <string> -required-
   Name of a property to set
   value        <string> -required-
   Value of a property to set

不是docker中可以修改的內(nèi)核參數(shù)在k8s中就可以修改，缺省k8s只認(rèn)為下面三個(gè)參數(shù)是安全的：
kernel.shm_rmid_forced
net.ipv4.ip_local_port_range
net.ipv4.tcp_syncookies
要使用其他內(nèi)核參數(shù)，需要在kubelet中啟用參數(shù)

# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS= "--allowed-unsafe-sysctls=net.ipv6.conf.*"

在pod中啟用ipv6的內(nèi)核參數(shù)

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: test1
  name: test1
  namespace: default
spec:
  selector:
    matchLabels:
      run: test1
  template:
    metadata:
      labels:
        run: test1
    spec:
      containers:
      - args:
        - "/bin/sh"
        - "-c"
        - "sleep 120"
        image: centos:latest
        name: test1
      securityContext:
        sysctls:
          - name: net.ipv6.conf.default.disable_ipv6
            value: '0'
          - name: net.ipv6.conf.all.disable_ipv6
            value: '0'

查看ipv6地址

# kubectl exec test1-66ccc967c5-7xrpc  ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if154: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
    link/ether 66:cf:30:b8:22:71 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.16.137.96/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::64cf:30ff:feb8:2271/64 scope link
       valid_lft forever preferred_lft forever