ceph官方文檔：https://docs.ceph.com/en/latest/start/intro/一. ceph組件：Monitors，Managers，OSDs，MDSs
Monitors：Ceph Monitor?(?ceph-mon) 維護(hù)集群狀態(tài)的映射，包括監(jiān)視器映射、管理器映射、OSD 映射、MDS 映射和 CRUSH 映射。這些映射是 Ceph 守護(hù)進(jìn)程相互協(xié)調(diào)所需的關(guān)鍵集群狀態(tài)。監(jiān)視器還負(fù)責(zé)管理守護(hù)進(jìn)程和客戶端之間的身份驗(yàn)證。通常至少需要三個(gè)監(jiān)視器才能實(shí)現(xiàn)冗余和高可用性。
Managers：Ceph 管理器守護(hù)進(jìn)程 (?ceph-mgr) 負(fù)責(zé)跟蹤運(yùn)行時(shí)指標(biāo)和 Ceph 集群的當(dāng)前狀態(tài)，包括存儲(chǔ)利用率、當(dāng)前性能指標(biāo)和系統(tǒng)負(fù)載。Ceph 管理器守護(hù)進(jìn)程還托管基于 Python 的模塊來管理和公開 Ceph 集群信息，包括基于 Web 的Ceph 儀表板和?REST API。高可用性通常至少需要兩個(gè)管理器。
Ceph OSD：Ceph OSD（對(duì)象存儲(chǔ)守護(hù)進(jìn)程?ceph-osd）存儲(chǔ)數(shù)據(jù)，處理數(shù)據(jù)復(fù)制、恢復(fù)、重新平衡，并通過檢查其他 Ceph OSD 守護(hù)進(jìn)程的心跳來向 Ceph 監(jiān)視器和管理器提供一些監(jiān)控信息。通常至少需要 3 個(gè) Ceph OSD 來實(shí)現(xiàn)冗余和高可用性。
MDS：Ceph 元數(shù)據(jù)服務(wù)器（MDS，ceph-mds）代表Ceph 文件系統(tǒng)存儲(chǔ)元數(shù)據(jù)（即 Ceph 塊設(shè)備和 Ceph 對(duì)象存儲(chǔ)不使用 MDS）。Ceph的元數(shù)據(jù)服務(wù)器允許POSIX文件系統(tǒng)的用戶來執(zhí)行基本的命令（如?ls，find沒有放置在一個(gè)Ceph存儲(chǔ)集群的巨大負(fù)擔(dān)，等等）。
二.可以支持的文件系統(tǒng)： 文件存儲(chǔ)，塊存儲(chǔ)，對(duì)象存儲(chǔ)，這三種存儲(chǔ)方式?jīng)]有優(yōu)劣之分，取決于應(yīng)用場(chǎng)景。
三.ceph數(shù)據(jù)的寫流程：分為兩步，第一步，根據(jù)Hash算法來計(jì)算對(duì)象名稱，將其映射到對(duì)應(yīng)的PG；第二部，根據(jù)Crush算法將PG映射到對(duì)應(yīng)的OSD。
四.ceph的安裝：
環(huán)境部署
windows10操作系統(tǒng) + VMware15 + Ubuntu18.04 安裝 + Ceph Pacific
在windows10上面安裝了VMware15，ceph部署在ubuntu18上，使用10臺(tái)服務(wù)器
ceph組件：
網(wǎng)絡(luò)：在VMWare上設(shè)置的，集群網(wǎng)絡(luò)用的僅主機(jī)模式，節(jié)點(diǎn)網(wǎng)絡(luò)使用的nat模式，因此出公網(wǎng)用的節(jié)點(diǎn)網(wǎng)絡(luò)，配置的時(shí)候。默認(rèn)網(wǎng)關(guān)配置在節(jié)點(diǎn)網(wǎng)絡(luò)
集群網(wǎng)絡(luò)：? 192.168.133.0/24
節(jié)點(diǎn)網(wǎng)絡(luò)：? 192.168.241.0/24
部署節(jié)點(diǎn)：
192.168.241.11 ceph-deploy
monitor節(jié)點(diǎn)：
192.168.241.12 ceph-mon1
192.168.241.13 ceph-mon2
192.168.241.14 ceph-mon3
manger節(jié)點(diǎn)：
192.168.241.15 ceph-mgr1
192.168.241.16 ceph-mgr2
node節(jié)點(diǎn)4個(gè)
192.168.241.17 ceph-node1
192.168.241.18 ceph-node2
192.168.241.19 ceph-node3
192.168.241.20 ceph-node4
1.配置主機(jī)名和網(wǎng)卡信息，按照上面的ip和主機(jī)名對(duì)應(yīng)，vi /etc/hostname，并重啟服務(wù)器
采用主機(jī)模式和nat模式,ens33是主機(jī)模式，ens38是nat模式，要想在
huahualin@ceph-deploy:~$ cat /etc/netplan/01-netcfg.yaml
# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
? version: 2
? renderer: networkd
? ethernets:
? ? ens33:
? ? ? dhcp4: no
? ? ? dhcp6: no
? ? ? addresses: [192.168.133.11/24]
? ? ? #? ? ? gateway4: 192.168.1.1
? ? ? #nameservers:
? ? ? #? ? ? ? addresses: [114.114.114.114]
? ? ens38:
? ? ? dhcp4: no
? ? ? dhcp6: no
? ? ? addresses: [192.168.241.11/24]
? ? ? gateway4: 192.168.241.2
? ? ? nameservers:
? ? ? ? ? ? ? addresses: [114.114.114.114]
#重啟網(wǎng)卡服務(wù)
huahualin@ceph-deploy:~$ sudo netplan apply? ? ? ? ? ? ?
@ceph-deploy:~$ ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>? mtu 1500
? ? ? ? inet 192.168.133.11? netmask 255.255.255.0? broadcast 192.168.133.255
? ? ? ? inet6 fe80::20c:29ff:fe9f:ab01? prefixlen 64? scopeid 0x20<link>
? ? ? ? ether 00:0c:29:9f:ab:01? txqueuelen 1000? (Ethernet)
? ? ? ? RX packets 150? bytes 24996 (24.9 KB)
? ? ? ? RX errors 0? dropped 0? overruns 0? frame 0
? ? ? ? TX packets 15? bytes 1146 (1.1 KB)
? ? ? ? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0
ens38: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>? mtu 1500
? ? ? ? inet 192.168.241.11? netmask 255.255.255.0? broadcast 192.168.241.255
? ? ? ? inet6 fe80::20c:29ff:fe9f:ab0b? prefixlen 64? scopeid 0x20<link>
? ? ? ? ether 00:0c:29:9f:ab:0b? txqueuelen 1000? (Ethernet)
? ? ? ? RX packets 3957? bytes 432025 (432.0 KB)
? ? ? ? RX errors 0? dropped 0? overruns 0? frame 0
? ? ? ? TX packets 3215? bytes 1097834 (1.0 MB)
? ? ? ? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING>? mtu 65536
? ? ? ? inet 127.0.0.1? netmask 255.0.0.0
? ? ? ? inet6 ::1? prefixlen 128? scopeid 0x10<host>=
? ? ? ? loop? txqueuelen 1000? (Local Loopback)
? ? ? ? RX packets 8? bytes 836 (836.0 B)
? ? ? ? RX errors 0? dropped 0? overruns 0? frame 0
? ? ? ? TX packets 8? bytes 836 (836.0 B)
? ? ? ? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0
2.所有服務(wù)器上執(zhí)行提升普通用戶權(quán)限，免密執(zhí)行命令，指定完畢后退出到huahualin普通用戶環(huán)境
~$ sudo -i
~# echo 'huahualin ALL=(ALL:ALL) NOPASSWD:ALL'? >> /etc/sudoers
~# exit
3.在ceph-deploy節(jié)點(diǎn)安裝并部署ansible
#安裝ansible
huahualin@ceph-deploy:~$ sudo apt install ansible -y
#配置免密登錄,一路回車
huahualin@ceph-deploy:~$ ssh-keygen
#配置deploy節(jié)點(diǎn)到各服務(wù)器的互信，huahualin到huahualin的互信，將公鑰拷貝到相應(yīng)節(jié)點(diǎn)
huahualin@ceph-deploy:~$for i in `cat 1` ;do echo huahualin@1234| ssh-copy-id -i /home/huahualin/.ssh/id_rsa.pub $i;done
cat /etc/ansible/hosts
[mon]
192.168.241.12? ansible_ssh_user=huahualin
192.168.241.13? ansible_ssh_user=huahualin
192.168.241.14? ansible_ssh_user=huahualin
[mgr]
192.168.241.15? ansible_ssh_user=huahualin
192.168.241.16? ansible_ssh_user=huahualin
[node]
192.168.241.17? ansible_ssh_user=huahualin
192.168.241.18? ansible_ssh_user=huahualin
192.168.241.19? ansible_ssh_user=huahualin
192.168.241.20? ansible_ssh_user=huahualin
#配置主機(jī)名解析
huahualin@ceph-deploy:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 192.localdomain 192
# The following lines are desirable for IPv6 capable hosts
::1? ? localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.241.11 ceph-deploy
192.168.241.12 ceph-mon1
192.168.241.13 ceph-mon2
192.168.241.14 ceph-mon3
192.168.241.15 ceph-mgr1
192.168.241.16 ceph-mgr2
192.168.241.17 ceph-node1
192.168.241.18 ceph-node2
192.168.241.19 ceph-node3
192.168.241.20 ceph-node4
#將/etc/hosts文件主機(jī)名解析拷貝到各服務(wù)器，-f是并發(fā)的線程數(shù)，-m指定copy模塊，-a指定參數(shù)
huahualin@ceph-deploy:~$ ansible all -f 9 -m copy -a "src=/etc/hosts dest=/etc"
#配置ceph的源，找到清華鏡像源，按照網(wǎng)站的提示替換相應(yīng)的源就是了
先導(dǎo)入key
https://mirrors.tuna.tsinghua.edu.cn/help/ceph/
只需要把文檔中出現(xiàn)的 download.ceph.com 替換為 mirrors.tuna.tsinghua.edu.cn/ceph 即可。
即：wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -?
替換為： 導(dǎo)入密鑰需要在每個(gè)服務(wù)器都要導(dǎo)入
wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | sudo apt-key add -
而ceph的版本這是使用pacific ，系統(tǒng)ubuntu18也要替換成bonic
sudo apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-octopus/ buster main'
替換成：
sudo apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main'
配置好鏡像倉庫將配置部署到所有服務(wù)器：普通用戶只能拷貝到tmp目錄下，這塊我也沒找到直接拷貝過去的辦法，所以先拷貝到tmp然后再拷貝到相應(yīng)/etc/apt目錄下
huahualin@ceph-deploy:~$ansible all -f 9 -m copy -a "src=/etc/apt/sources.list dest=/tmp"
huahualin@ceph-deploy:~$ ansible all -f 9 -m shell? -a "sudo cp? /tmp/sources.list /etc/apt/"
#然后所有節(jié)點(diǎn)更新源，執(zhí)行
huahualin@ceph-deploy:~$? sudo apt update
#創(chuàng)建一個(gè)普通賬戶ceph來部署ceph
sudo groupadd ceph -g 2021 && sudo useradd -m -s /bin/bash? -u 2021 -g 2021? ceph && sudo echo ceph:ceph@1234| sudo chpasswd
#所有節(jié)點(diǎn)執(zhí)行，讓ceph以非交互式的方式執(zhí)行命令,官方安裝部署需要ceph用戶執(zhí)行
huahualin@ceph-deploy:~$ sudo -i
root@ceph-deploy:~# sudo echo 'ceph ALL=(ALL:ALL) NOPASSWD:ALL' >> /etc/sudoers
root@ceph-deploy:~# exit
#配置deploy節(jié)點(diǎn)的ceph用戶到各服務(wù)器ceph用戶的互信，先切換到ceph用戶
huahualin@ceph-deploy:~$ su - ceph
Password:
ceph@ceph-deploy:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ceph/.ssh/id_rsa):
Created directory '/home/ceph/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ceph/.ssh/id_rsa.
Your public key has been saved in /home/ceph/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:41lBo965Bsmv+a3fhDZN4NCoD0FXm3rSB2qC0t3nXIw ceph@ceph-deploy
The key's randomart image is:
+---[RSA 2048]----+
|? ? ? ? . +..? ? |
|? ? ? . + + o? |
|? ? ? ? o + *? ? |
|? ? . = * O =? |
|? ? . o S X E =? |
|? ? . . @ B *? |
|? ? ? ? o = * o? |
|? ? ? ? + o +? |
|? ? ? ? o.ooo .? |
+----[SHA256]-----+
ceph@ceph-deploy:~$ for i in {12..20};do ssh-copy-id -i /home/ceph/.ssh/id_rsa.pub 192.168.241.$i;done
#到這里為止，可以可以給每個(gè)服務(wù)器，包括deploy創(chuàng)建快照，以便于后來重新部署恢復(fù)
#安裝ceph-deploy部署工具，這里使用ceph，但是此知名用戶名被人熟知不太安全，ceph名被保留給了Ceph守護(hù)進(jìn)程，升級(jí)前必須先刪掉這個(gè)用戶，也可以使用別的普通用戶名替代，
ceph@ceph-deploy:~$ sudo apt-cache madison ceph-deploy
ceph-deploy |? ? ? 2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages
ceph-deploy |? ? ? 2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main i386 Packages
ceph-deploy | 1.5.38-0ubuntu1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages
ceph-deploy | 1.5.38-0ubuntu1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages
ceph@ceph-deploy:~$ sudo apt install ceph-deploy -y
#創(chuàng)建生成集群
mkdir ceph-cluster
cd ceph-clusteceph-deploy new? --cluster-network 192.168.133.0/24 --public-network 192.168.241.0/24 ceph-mon1
會(huì)生成：只是生成配置文件，還沒有在mon服務(wù)器上安裝任何包
ceph.conf? ceph-deploy-ceph.log? ceph.mon.keyring
ceph@ceph-deploy:~/ceph-cluster$ cat ceph.conf
[global]
fsid = 30e28db1-cfea-4e9d-9be6-148d4fa39545? #集群id
public_network = 192.168.241.0/24
cluster_network = 192.168.133.0/24
mon_initial_members = ceph-mon1
mon_host = 192.168.241.12
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
#在所有mon節(jié)點(diǎn)安裝ceph-mon,用于初始化mon，一路默認(rèn)就好了
sudo apt install? ceph-mon -y
#到ceph-deploy節(jié)點(diǎn)上初始化mon節(jié)點(diǎn)，會(huì)讀取ceph.conf配置文件讀取配置去到相應(yīng)的節(jié)點(diǎn)初始化，并在mon節(jié)點(diǎn)上重建ceph用戶的家目錄，uid和gid不變，裝包的時(shí)候會(huì)新建一個(gè)用戶，key就會(huì)變了，到后面可以重新導(dǎo)入一次key。并在deploy節(jié)點(diǎn)上生成各組件的key
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy mon create-initial
到mon節(jié)點(diǎn)查看，
huahualin@ceph-mon1:~$ cat /etc/passwd
ceph:x:2021:2021:Ceph storage service:/var/lib/ceph:/bin/bash
進(jìn)程也起來了：
huahualin@ceph-mon1:~$ ps axu|grep mon
ceph? ? ? 12110? 0.1? 2.0 480392 41648 ?? ? ? ? Ssl? 23:39? 0:00 /usr/bin/ceph-mon -f --cluster ceph --id ceph-mon1 --setuser ceph --setgroup ceph
#到deploy節(jié)點(diǎn)安裝ceph-comman查看集群狀態(tài),這里要退出ceph登錄，也不要經(jīng)過ceph的登錄終端，直接點(diǎn)從別的用戶登錄安裝，不然會(huì)和ceph的家目錄沖突，要么使用ceph之外的用戶安裝，要么重新登錄終端安裝
ceph@ceph-deploy:~/ceph-cluster$ exit
logout
huahualin@ceph-deploy:~/ceph-cluster$ sudo -i
root@ceph-deploy:~# apt install -y ceph-common
#能使用ceph這個(gè)命令說明就已經(jīng)好了，因?yàn)榕渲眠€沒有弄好，所以匯報(bào)這個(gè)錯(cuò)
huahualin@ceph-deploy:~/ceph-cluster$ ceph -s
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)',)
#ceph-deploy上把key拷貝到新生成的家目錄，
sudo - ceph
cp /home/ceph/ceph-cluster? /var/lib/ceph/ -r
#然后到ceph-cluster目錄下執(zhí)行ceph -s，會(huì)看到很多錯(cuò)誤，需要使用admin去推送這些key才能管理ceph
ceph@ceph-deploy:/home/ceph/ceph-cluster$ cd /var/lib/ceph/ceph-cluster/
ceph@ceph-deploy:~/ceph-cluster$ ceph -s
#推送key到指定服務(wù)器，就會(huì)在相應(yīng)機(jī)器的/etc/ceph目錄下得到ceph.conf和key文件
到node1上看
ceph-deploy? admin ceph-node1 ceph-deploy? #推送給自己
ceph-deploy? admin ceph-node1 ceph-node2 ceph-node3 ceph-node4? #推送給別的節(jié)點(diǎn)
#到剛剛推送的幾個(gè)節(jié)點(diǎn)上查看
root@ceph-node1:~# ls /etc/ceph/
ceph.client.admin.keyring? ceph.conf? rbdmap? tmpMLt9SR
#授權(quán)，到推送的個(gè)節(jié)點(diǎn)上修改key文件權(quán)限，默認(rèn)是root的，普通用戶不能使用
-u 把用戶改為ceph，下面到推送了key的節(jié)點(diǎn)進(jìn)行授權(quán)
ceph@ceph-deploy:~/ceph-cluster$ sudo setfacl -m u:ceph:rw /etc/ceph/ceph.client.admin.keyring
如果沒有setfacl這個(gè)命令，需要安裝acl包
ceph@ceph-deploy:~/ceph-cluster$ sudo apt install acl
#可以查看有一個(gè)mon節(jié)點(diǎn)
ceph@ceph-deploy:~/ceph-cluster$ ceph -s
? cluster:
? ? id:? ? 30e28db1-cfea-4e9d-9be6-148d4fa39545
? ? health: HEALTH_WARN
? ? ? ? ? ? mon is allowing insecure global_id reclaim
? services:
? ? mon: 1 daemons, quorum ceph-mon1 (age 80m)
? ? mgr: no daemons active
? ? osd: 0 osds: 0 up, 0 in
? data:
? ? pools:? 0 pools, 0 pgs
? ? objects: 0 objects, 0 B
? ? usage:? 0 B used, 0 B / 0 B avail
? ? pgs:? ?
#到mgr節(jié)點(diǎn)安裝mgr
sudo apt install? ceph-mgr
#創(chuàng)建一個(gè)mrg節(jié)點(diǎn)，到deploy上指執(zhí)行
ceph@ceph-deploy:~/ceph-cluster$ ceph-deploy? mgr create ceph-mgr1
可以查看，有一個(gè)mgr節(jié)點(diǎn)了
ceph@ceph-deploy:~/ceph-cluster$ ceph -s
? cluster:
? ? id:? ? 30e28db1-cfea-4e9d-9be6-148d4fa39545
? ? health: HEALTH_WARN
? ? ? ? ? ? mon is allowing insecure global_id reclaim
? services:
? ? mon: 1 daemons, quorum ceph-mon1 (age 82m)
? ? mgr: ceph-mgr1(active, since 4s)
? ? osd: 0 osds: 0 up, 0 in
? data:
? ? pools:? 0 pools, 0 pgs
? ? objects: 0 objects, 0 B
? ? usage:? 0 B used, 0 B / 0 B avail
? ? pgs:? ?
#deploy添加osd
添加osd之前安裝node節(jié)點(diǎn)的基本環(huán)境，初始化node環(huán)境：
ceph@ceph-deploy:~$ ceph-deploy install --no-adjust-repos? --nogpgcheck? ceph-node1
ceph@ceph-deploy:~$ ceph-deploy install --no-adjust-repos? --nogpgcheck? ceph-node2
ceph@ceph-deploy:~$ ceph-deploy install --no-adjust-repos? --nogpgcheck? ceph-node3
ceph@ceph-deploy:~$ ceph-deploy install --no-adjust-repos? --nogpgcheck? ceph-node4
解決ceph -s 告警? mon is allowing insecure global_id reclaim，需要禁用非安全模式，執(zhí)行如下命令
ceph config set mon auth_allow_insecure_global_id_reclaim false
#擦除磁盤之前安裝基本運(yùn)行環(huán)境
ceph@ceph-deploy:~$ ceph-deploy install --release pacific ceph-node1
ceph@ceph-deploy:~$ ceph-deploy install --release pacific ceph-nod32
ceph@ceph-deploy:~$ ceph-deploy install --release pacific ceph-node4
ceph@ceph-deploy:~$ ceph-deploy install --release pacific ceph-node4
#新加磁盤不重啟系統(tǒng)
root@ceph-node2:~# ls /sys/class/scsi_host/
host0? host11? host14? host17? host2? host22? host25? host28? host30? host4? host7
host1? host12? host15? host18? host20? host23? host26? host29? host31? host5? host8
host10? host13? host16? host19? host21? host24? host27? host3? host32? host6? host9
root@ceph-node2:~# for i in {0..32};do echo "- - -" > /sys/class/scsi_host/host$i/scan;done
? #擦除磁盤,只能擦除b到d盤，因?yàn)閟da是系統(tǒng)盤不能擦除
? ceph@ceph-deploy:~$ ceph-deploy disk zap ceph-node1 /dev/sdb
? for i in {b..d};do ceph-deploy disk zap ceph-node1 /dev/sd$i ;done
? for i in {b..d};do ceph-deploy disk zap ceph-node2 /dev/sd$i ;done
? for i in {b..d};do ceph-deploy disk zap ceph-node3 /dev/sd$i ;done
? echo ceph@1234|for i in {b..d};do ceph-deploy disk zap ceph-node4 /dev/sd$i ;done
? #添加主機(jī)磁盤osd,id從0開始
? 數(shù)據(jù)：對(duì)象數(shù)據(jù)，元數(shù)據(jù)，wal日志即預(yù)寫日志
? for i in {b..d};do ceph-deploy osd create ceph-node1 --data /dev/sd$i;done
? for i in {b..d};do ceph-deploy osd create ceph-node2 --data /dev/sd$i;done
? for i in {b..d};do ceph-deploy osd create ceph-node3 --data /dev/sd$i;done
? for i in {b..d};do ceph-deploy osd create ceph-node4 --data /dev/sd$i;done
? #查看集群狀態(tài)，正常
? ceph@ceph-deploy:~$ ceph -s
? cluster:
? ? id:? ? 30e28db1-cfea-4e9d-9be6-148d4fa39545
? ? health: HEALTH_OK
? services:
? ? mon: 1 daemons, quorum ceph-mon1 (age 4h)
? ? mgr: ceph-mgr1(active, since 5h)
? ? osd: 12 osds: 12 up (since 14s), 12 in (since 22s)
? data:
? ? pools:? 1 pools, 1 pgs
? ? objects: 0 objects, 0 B
? ? usage:? 74 MiB used, 240 GiB / 240 GiB avail
? ? pgs:? ? 1 active+clean
#創(chuàng)建存儲(chǔ)池
ceph@ceph-deploy:~$ ceph osd pool create? --help
osd pool create <pool> [<pg_num:int>] [<pgp_num:? ? ? create pool
int>] [replicated|erasure] [<erasure_code_profile>]?
[<rule>] [<expected_num_objects:int>] [<size:int>]?
[<pg_num_min:int>] [on|off|warn] [<target_size_? ? ?
bytes:int>] [<target_size_ratio:float>]
ceph osd pool create mypool 32 32
ceph osd pool ls? #查看有多少存儲(chǔ)池 rados lspools
ceph pg ls-by-pool mypool
ceph@ceph-deploy:~$ ceph pg ls-by-pool mypool | awk '{print $1,$2,$15}'
pg一定是跨主機(jī)的
ceph@ceph-deploy:~$ ceph osd tree
ID? CLASS? WEIGHT? TYPE NAME? ? ? ? ? ? STATUS? REWEIGHT? PRI-AFF
-1? ? ? ? 0.23383? root default? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
-3? ? ? ? 0.05846? ? ? host ceph-node1? ? ? ? ? ? ? ? ? ? ? ? ?
0? ? hdd? 0.01949? ? ? ? ? osd.0? ? ? ? ? ? up? 1.00000? 1.00000
1? ? hdd? 0.01949? ? ? ? ? osd.1? ? ? ? ? ? up? 1.00000? 1.00000
2? ? hdd? 0.01949? ? ? ? ? osd.2? ? ? ? ? ? up? 1.00000? 1.00000
-5? ? ? ? 0.05846? ? ? host ceph-node2? ? ? ? ? ? ? ? ? ? ? ? ?
3? ? hdd? 0.01949? ? ? ? ? osd.3? ? ? ? ? ? up? 1.00000? 1.00000
5? ? hdd? 0.01949? ? ? ? ? osd.5? ? ? ? ? ? up? 1.00000? 1.00000
7? ? hdd? 0.01949? ? ? ? ? osd.7? ? ? ? ? ? up? 1.00000? 1.00000
-7? ? ? ? 0.05846? ? ? host ceph-node3? ? ? ? ? ? ? ? ? ? ? ? ?
4? ? hdd? 0.01949? ? ? ? ? osd.4? ? ? ? ? ? up? 1.00000? 1.00000
6? ? hdd? 0.01949? ? ? ? ? osd.6? ? ? ? ? ? up? 1.00000? 1.00000
8? ? hdd? 0.01949? ? ? ? ? osd.8? ? ? ? ? ? up? 1.00000? 1.00000
-9? ? ? ? 0.05846? ? ? host ceph-node4? ? ? ? ? ? ? ? ? ? ? ? ?
9? ? hdd? 0.01949? ? ? ? ? osd.9? ? ? ? ? ? up? 1.00000? 1.00000
10? ? hdd? 0.01949? ? ? ? ? osd.10? ? ? ? ? up? 1.00000? 1.00000
11? ? hdd? 0.01949? ? ? ? ? osd.11? ? ? ? ? up? 1.00000? 1.00000
#ceph 的 rados 命令可以實(shí)現(xiàn)訪問 ceph 對(duì)象存儲(chǔ)的功能
ceph@ceph-deploy:~$ sudo rados put msg1 /var/log/syslog --pool=mypool
--pool=mypool #把 messages 文件上傳到 mypool 并指定對(duì)象 id 為 msg1
ceph@ceph-deploy:~$ sudo rados ls --pool=mypool
msg1
#mon的高可用，現(xiàn)在mon節(jié)點(diǎn)安裝mon的組件包c(diǎn)eph-mon
? sudo apt -y install ceph-mon
#deploy節(jié)點(diǎn)上添加mon節(jié)點(diǎn)，此前已經(jīng)有mon1了
ceph@ceph-deploy:~$ ceph-deploy mon add ceph-mon2
ceph@ceph-deploy:~$ ceph-deploy mon add ceph-mon3
#查看mon節(jié)點(diǎn)狀態(tài)：
ceph@ceph-deploy:~$ ceph quorum_status -f json-pretty
#mgr節(jié)點(diǎn)擴(kuò)展，mgr不能進(jìn)行選舉，沒有高可用功能，一般兩個(gè)就夠了，也有M/S這種高可用，一個(gè)節(jié)點(diǎn)掛了，另外一個(gè)也會(huì)主動(dòng)接管服務(wù)，此前已經(jīng)有一個(gè)mgr1了，只需要?jiǎng)?chuàng)建mgr2
ceph@ceph-deploy:~$ ceph-deploy mgr create ceph-mgr2
#創(chuàng)建塊存儲(chǔ)
$ ceph osd pool create myrbd1 64 64 #創(chuàng)建存儲(chǔ)池,指定 pg 和 pgp 的數(shù)量，pgp 是對(duì)存在
于 pg 的數(shù)據(jù)進(jìn)行組合存儲(chǔ)，pgp 通常等于 pg 的值
ceph osd pool application enable myrbd1 rbd #對(duì)存儲(chǔ)池啟用 RBD 功能
rbd pool init -p myrbd1 #初始化、
還不能直接用，需要先掛上去
#需要
集群網(wǎng)絡(luò)給自己管理使用，外部網(wǎng)絡(luò)是給客戶端使用的
#mon節(jié)點(diǎn)
#部署客戶端
如果還要部署客戶端，客戶端也要導(dǎo)入key
