背景

考慮到k8s需要一個可共享到各個node的存儲，準(zhǔn)備搭建一個ceph，以rbd的方式提供

操作系統(tǒng)準(zhǔn)備

centos 7.2
centos-base.repo epel.repo
ntp同步配置(chrony.conf)
內(nèi)網(wǎng)服務(wù)器，暫時關(guān)閉selinux與防火墻
可選：dns配置，如忽略采用/etc/hosts方式

集群規(guī)劃

此處名字應(yīng)和hostname -s 輸出相一致

安裝過程

1、創(chuàng)建ceph用戶（所有主機）

useradd ceph
echo 'ceph ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
sed -i 's/Defaults requiretty/#Defaults requiretty/' /etc/sudoers

2、所有主機配置ssh免密登錄（ceph用戶）

ssh-keygen
cat ~/.ssh/id_rsa.pub #將三個服務(wù)器的三行文本copy到一個文本文件中
touch ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
vim ~/.ssh/authorized_keys #將上面的三行文本復(fù)制到此文件
ssh cloud4ourself-c1
ssh cloud4ourself-c2
ssh cloud4ourself-c3 #都是免登錄

3、配置集群（cloud4ourself-c1）

mkdir k8s && cd k8s
sudo yum install http://download.ceph.com/rpm-hammer/el7/noarch/ceph-release-1-1.el7.noarch.rpm -y
sudo yum install ceph-deploy -y
$ceph-deploy new cloud4ourself-c1

[ceph@cloud4ourself-c1 ~]$ ceph-deploy new cloud4ourself-c1

[ceph@cloud4ourself-c1 k8s]$ ls
ceph.conf  ceph.log  ceph.mon.keyring

echo "osd pool default size = 2" >> ceph.conf

4、安裝

ceph-deploy install cloud4ourself-c1 cloud4ourself-c2 cloud4ourself-c3
上面是安裝最新版本的ceph，如果安裝指定版本
ceph-deploy install cloud4ourself-c1 --repo-url=http://mirrors.aliyun.com/ceph/rpm-hammer/el7/
如有如下錯誤

[c2][WARNIN] Error: Package: 1:ceph-selinux-10.2.6-0.el7.x86_64 (Ceph)
[c2][WARNIN]            Requires: selinux-policy-base >= 3.13.1-102.el7_3.13
[c2][WARNIN]            Installed: selinux-policy-targeted-3.13.1-60.el7.noarch (@anaconda)
[c2][WARNIN]                selinux-policy-base = 3.13.1-60.el7
[c2][WARNIN]            Available: selinux-policy-minimum-3.13.1-102.el7.noarch (base)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7
[c2][WARNIN]            Available: selinux-policy-minimum-3.13.1-102.el7_3.4.noarch (updates)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7_3.4
[c2][WARNIN]            Available: selinux-policy-minimum-3.13.1-102.el7_3.7.noarch (updates)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7_3.7
[c2][WARNIN]            Available: selinux-policy-mls-3.13.1-102.el7.noarch (base)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7
[c2][WARNIN]            Available: selinux-policy-mls-3.13.1-102.el7_3.4.noarch (updates)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7_3.4
[c2][WARNIN]            Available: selinux-policy-mls-3.13.1-102.el7_3.7.noarch (updates)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7_3.7
[c2][WARNIN]            Available: selinux-policy-targeted-3.13.1-102.el7.noarch (base)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7
[c2][WARNIN]            Available: selinux-policy-targeted-3.13.1-102.el7_3.4.noarch (updates)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7_3.4
[c2][WARNIN]            Available: selinux-policy-targeted-3.13.1-102.el7_3.7.noarch (updates)
[c2][WARNIN]                selinux-policy-base = 3.13.1-102.el7_3.7
[c2][DEBUG ]  You could try running: rpm -Va --nofiles --nodigest
[c2][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y install ceph ceph-radosgw

可能是相關(guān)repo沒有更新。

如有如下錯誤

[c1][DEBUG ] Retrieving https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm
[c1][WARNIN]    file /etc/yum.repos.d/ceph.repo from install of ceph-release-1-1.el7.noarch conflicts with file from package ceph-release-1-1.el7.noarch
[c1][DEBUG ] Preparing...                          ########################################
[c1][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: rpm -Uvh --replacepkgs https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm

可執(zhí)行sudo rpm -e ceph-release
然后再試。

monitor初始化

[ceph@cloud4ourself-c1 k8s]$ ceph-deploy mon create-initial
[ceph@cloud4ourself-c1 k8s]$ ls 
ceph.bootstrap-mds.keyring  ceph.bootstrap-osd.keyring  ceph.bootstrap-rgw.keyring  ceph.client.admin.keyring  ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring

cloud4ourself-c2\cloud4ourself-c3主機

sudo mkdir -p /var/local/cephfs
sudo chown ceph:ceph  /var/local/cephfs

cloud4ourself-c1主機
（由于使用了虛擬機，使用xfs文件系統(tǒng)替代磁盤）

ceph-deploy osd prepare  cloud4ourself-c2:/var/local/cephfs cloud4ourself-c3:/var/local/cephfs
ceph-deploy osd activate cloud4ourself-c2:/var/local/cephfs cloud4ourself-c3:/var/local/cephfs
sudo chmod +r /etc/ceph/ceph.client.admin.keyring
ceph -s

后續(xù)

增加兩個mon
修改ceph.conf

public_network=10.9.5.0/24

ceph-deploy --overwrite-conf mon add cloud4ourself-c2
ceph-deploy --overwrite-conf mon add cloud4ourself-c3

在測試k8s過程中，關(guān)閉一個node后，發(fā)現(xiàn)存在rbd lock的情況

Mar  8 17:53:18 cloud4ourself-mytest2 kubelet: E0308 17:53:18.391278    2162 nestedpendingoperations.go:262] Operation for "\"kubernetes.io/rbd/a539e906-03d7-11e7-9826-fa163eec323b-pvc-a55f4a04-0317-11e7-9826-fa163eec323b\" (\"a539e906-03d7-11e7-9826-fa163eec323b\")" failed. No retries permitted until 2017-03-08 17:55:18.391251562 +0800 CST (durationBeforeRetry 2m0s). Error: MountVolume.SetUp failed for volume "kubernetes.io/rbd/a539e906-03d7-11e7-9826-fa163eec323b-pvc-a55f4a04-0317-11e7-9826-fa163eec323b" (spec.Name: "pvc-a55f4a04-0317-11e7-9826-fa163eec323b") pod "a539e906-03d7-11e7-9826-fa163eec323b" (UID: "a539e906-03d7-11e7-9826-fa163eec323b") with: rbd: image kubernetes-dynamic-pvc-52f919af-0321-11e7-b778-fa163eec323b is locked by other nodes

解決方式是收到解鎖
rbd lock remove

參考鏈接：
http://docs.ceph.org.cn/start/
http://www.cnblogs.com/clouding/p/6115447.html
http://tonybai.com/2017/02/17/temp-fix-for-pod-unable-mount-cephrbd-volume/
http://tonybai.com/2016/11/07/integrate-kubernetes-with-ceph-rbd/