前面我們手工部署了redis-cluster集群,可以看到步驟煩多,現(xiàn)在使用docker的方式部署,非常簡(jiǎn)單,共分為二步。
參考鏡像https://github.com/publicisworldwide/docker-stacks/tree/master/oracle-linux/environments/storage/redis-cluster
Docker方式部署redis-cluster步驟
1、redis容器初始化
2、redis容器集群配置
一、redis容器初始化
容器初始化,使用docker-compose方式,先創(chuàng)建一個(gè)docker-compose.yml文件,內(nèi)容如下:
version: '3'
services:
redis1:
image: publicisworldwide/redis-cluster
network_mode: host
restart: always
volumes:
- /data/redis/8001/data:/data
environment:
- REDIS_PORT=8001
redis2:
image: publicisworldwide/redis-cluster
network_mode: host
restart: always
volumes:
- /data/redis/8002/data:/data
environment:
- REDIS_PORT=8002
redis3:
image: publicisworldwide/redis-cluster
network_mode: host
restart: always
volumes:
- /data/redis/8003/data:/data
environment:
- REDIS_PORT=8003
redis4:
image: publicisworldwide/redis-cluster
network_mode: host
restart: always
volumes:
- /data/redis/8004/data:/data
environment:
- REDIS_PORT=8004
redis5:
image: publicisworldwide/redis-cluster
network_mode: host
restart: always
volumes:
- /data/redis/8005/data:/data
environment:
- REDIS_PORT=8005
redis6:
image: publicisworldwide/redis-cluster
network_mode: host
restart: always
volumes:
- /data/redis/8006/data:/data
environment:
- REDIS_PORT=8006
這里引用了別人的一個(gè)鏡像publicisworldwide/redis-cluster,方便快捷。
這里使用host(主機(jī))網(wǎng)絡(luò)模式,把redis數(shù)據(jù)掛載到本機(jī)目錄/data/redis/800*下。
若不想使用host模式,也可以把network_mode去掉,但就要加ports映射。
redis-cluster的節(jié)點(diǎn)端口共分為2種,一種是節(jié)點(diǎn)提供服務(wù)的端口,如6379;一種是節(jié)點(diǎn)間通信的端口,固定格式為:10000+6379。
version: '3'
services:
redis1:
image: publicisworldwide/redis-cluster
restart: always
volumes:
- /data/redis/8001/data:/data
environment:
- REDIS_PORT=8001
ports:
- '8001:8001' #服務(wù)端口
- '18001:18001' #集群端口
redis2:
image: publicisworldwide/redis-cluster
restart: always
volumes:
- /data/redis/8002/data:/data
environment:
- REDIS_PORT=8002
ports:
- '8002:8002'
- '18002:18002'
redis3:
image: publicisworldwide/redis-cluster
restart: always
volumes:
- /data/redis/8003/data:/data
environment:
- REDIS_PORT=8003
ports:
- '8003:8003'
- '18003:18003'
redis4:
image: publicisworldwide/redis-cluster
restart: always
volumes:
- /data/redis/8004/data:/data
environment:
- REDIS_PORT=8004
ports:
- '8004:8004'
- '18004:18004'
redis5:
image: publicisworldwide/redis-cluster
restart: always
volumes:
- /data/redis/8005/data:/data
environment:
- REDIS_PORT=8005
ports:
- '8005:8005'
- '18005:18005'
redis6:
image: publicisworldwide/redis-cluster
restart: always
volumes:
- /data/redis/8006/data:/data
environment:
- REDIS_PORT=8006
ports:
- '8006:8006'
- '18006:18006'
創(chuàng)建文件后,直接啟動(dòng)服務(wù)
窗口模式
docker-compose up
后臺(tái)進(jìn)程
docker-compose up -d
查看啟動(dòng)的進(jìn)程
[root@localhost redis-cluster]# docker-compose ps
Name Command State Ports
----------------------------------------------------------------------
rediscluster_redis1_1 /usr/local/bin/entrypoint. ... Up
rediscluster_redis2_1 /usr/local/bin/entrypoint. ... Up
rediscluster_redis3_1 /usr/local/bin/entrypoint. ... Up
rediscluster_redis4_1 /usr/local/bin/entrypoint. ... Up
rediscluster_redis5_1 /usr/local/bin/entrypoint. ... Up
rediscluster_redis6_1 /usr/local/bin/entrypoint. ... Up
狀態(tài)為Up,說(shuō)明服務(wù)均已啟動(dòng),鏡像無(wú)問(wèn)題。
注意:以上鏡像不能設(shè)置永久密碼,其實(shí)redis一般是內(nèi)網(wǎng)訪問(wèn),可以不需密碼。
二、redis容器集群配置
上面只是啟動(dòng)了6個(gè)redis容器,并沒(méi)有設(shè)置集群,通過(guò)下面的命令可以設(shè)置集群。
docker run --rm -it inem0o/redis-trib create --replicas 1 172.19.165.222:8001 172.19.165.222:8002 172.19.165.222:8003 172.19.165.222:8004 172.19.165.222:8005 172.19.165.222:8006
這里同樣使用了另一個(gè)鏡像inem0o/redis-trib,執(zhí)行時(shí)會(huì)自動(dòng)下載。
日志如下:
[root@localhost disconf]# docker run --rm -it inem0o/redis-trib create --replicas 1 172.19.165.222:8001 172.19.165.222:8002 172.19.165.222:8003 172.19.165.222:8004 172.19.165.222:8005 172.19.165.222:8006
Unable to find image 'inem0o/redis-trib:latest' locally
latest: Pulling from inem0o/redis-trib
a2b2998a36ab: Pull complete
a3ed95caeb02: Pull complete
46ab6b64c08e: Pull complete
3d82c3ac2025: Pull complete
Digest: sha256:0b89d25b387f70ef1c54605bdf061dd86e0833dbc0e2149390570b8b372278f8
Status: Downloaded newer image for inem0o/redis-trib:latest
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.19.165.222:8001
172.19.165.222:8002
172.19.165.222:8003
Adding replica 172.19.165.222:8004 to 172.19.165.222:8001
Adding replica 172.19.165.222:8005 to 172.19.165.222:8002
Adding replica 172.19.165.222:8006 to 172.19.165.222:8003
M: 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950 172.19.165.222:8001
slots:0-5460 (5461 slots) master
M: 206626063f31dcd7e69010ce13c258e786197f1e 172.19.165.222:8002
slots:5461-10922 (5462 slots) master
M: e9924018d95772b8ff535f6bc0605a6630837069 172.19.165.222:8003
slots:10923-16383 (5461 slots) master
S: 548f4e65fbab8dcde8aac187849d50983d68599d 172.19.165.222:8004
replicates 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950
S: 0a5c799c1f8fed083c50902639fc354e4c25aa8c 172.19.165.222:8005
replicates 206626063f31dcd7e69010ce13c258e786197f1e
S: 94e2530ddd05b0e9eb3e71a9616342bd6647a5e6 172.19.165.222:8006
replicates e9924018d95772b8ff535f6bc0605a6630837069
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 172.19.165.222:8001)
M: 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950 172.19.165.222:8001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 94e2530ddd05b0e9eb3e71a9616342bd6647a5e6 172.19.165.222:8006@18006
slots: (0 slots) slave
replicates e9924018d95772b8ff535f6bc0605a6630837069
S: 0a5c799c1f8fed083c50902639fc354e4c25aa8c 172.19.165.222:8005@18005
slots: (0 slots) slave
replicates 206626063f31dcd7e69010ce13c258e786197f1e
M: e9924018d95772b8ff535f6bc0605a6630837069 172.19.165.222:8003@18003
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 206626063f31dcd7e69010ce13c258e786197f1e 172.19.165.222:8002@18002
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: 548f4e65fbab8dcde8aac187849d50983d68599d 172.19.165.222:8004@18004
slots: (0 slots) slave
replicates 67d9a6bb6875f3a0f9a53e5bb05ddeca8e656950
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
可以看到設(shè)置了3個(gè)Master,3個(gè)Slave,若想測(cè)試下集群,可以參考【Redis-Cluster集群】
只需二步就完成了集群的部署,但上面是使用了別人的鏡像,若是要自己創(chuàng)建鏡像該如何呢?
手動(dòng)創(chuàng)建鏡像
編寫(xiě)Dockerfile
#基礎(chǔ)鏡像
FROM redis
#修復(fù)時(shí)區(qū)
RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' >/etc/timezone
#環(huán)境變量
ENV REDIS_PORT 8000
#ENV REDIS_PORT_NODE 18000
#暴露變量
EXPOSE $REDIS_PORT
#EXPOSE $REDIS_PORT_NODE
#復(fù)制
COPY entrypoint.sh /usr/local/bin/
COPY redis.conf /usr/local/etc/
#for config rewrite
RUN chmod 777 /usr/local/etc/redis.conf
RUN chmod +x /usr/local/bin/entrypoint.sh
#入口
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
#命令
CMD ["redis-server", "/usr/local/etc/redis.conf"]
編寫(xiě)shell文件entrypoint.sh
#!/bin/sh
#只作用于當(dāng)前進(jìn)程,不作用于其創(chuàng)建的子進(jìn)程
set -e
#$0--Shell本身的文件名 $1--第一個(gè)參數(shù) $@--所有參數(shù)列表
# allow the container to be started with `--user`
if [ "$1" = 'redis-server' -a "$(id -u)" = '0' ]; then
sed -i 's/REDIS_PORT/'$REDIS_PORT'/g' /usr/local/etc/redis.conf
chown -R redis . #改變當(dāng)前文件所有者
exec gosu redis "$0" "$@" #gosu是sudo輕量級(jí)”替代品”
fi
exec "$@"
編寫(xiě)redis.conf
#端口
port REDIS_PORT
#開(kāi)啟集群
cluster-enabled yes
#配置文件
cluster-config-file nodes.conf
cluster-node-timeout 5000
#更新操作后進(jìn)行日志記錄
appendonly yes
#設(shè)置主服務(wù)的連接密碼
# masterauth
#設(shè)置從服務(wù)的連接密碼
# requirepass
注意:
- requirepass和masterauth不能啟用,否則redis-trib創(chuàng)建集群失敗。
- protected-mode 保護(hù)模式是禁止公網(wǎng)訪問(wèn),但是不能設(shè)置密碼和bind ip。
以上編寫(xiě)了3個(gè)文件,其實(shí)可以創(chuàng)建鏡像了,不過(guò)也可以在docker-compose中自動(dòng)創(chuàng)建。
編寫(xiě)docker-compose.yml文件
version: '3'
services:
redis1:
build: ./
restart: always
volumes:
- /data/redis/8001/data:/data
environment:
- REDIS_PORT=8001
ports:
- '8001:8001' #服務(wù)端口
- '18001:18001' #集群端口
redis2:
build: ./
restart: always
volumes:
- /data/redis/8002/data:/data
environment:
- REDIS_PORT=8002
ports:
- '8002:8002'
- '18002:18002'
redis3:
build: ./
restart: always
volumes:
- /data/redis/8003/data:/data
environment:
- REDIS_PORT=8003
ports:
- '8003:8003'
- '18003:18003'
redis4:
build: ./
restart: always
volumes:
- /data/redis/8004/data:/data
environment:
- REDIS_PORT=8004
ports:
- '8004:8004'
- '18004:18004'
redis5:
build: ./
restart: always
volumes:
- /data/redis/8005/data:/data
environment:
- REDIS_PORT=8005
ports:
- '8005:8005'
- '18005:18005'
redis6:
build: ./
restart: always
volumes:
- /data/redis/8006/data:/data
environment:
- REDIS_PORT=8006
ports:
- '8006:8006'
- '18006:18006'
創(chuàng)建鏡像
窗口模式
docker-compose up
后臺(tái)進(jìn)程
docker-compose up -d
重新創(chuàng)建
docker-compose up --build
若創(chuàng)建成功,可以使用命令登錄并查看集群信息。
[root@localhost src]# ./redis-cli -c -p 8003
127.0.0.1:8003> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
從上面的日志里看到,當(dāng)前集群狀態(tài)失敗。
與上面相同,重新執(zhí)行上面的第二步,再次查看結(jié)果如下:
127.0.0.1:8003> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:3
cluster_stats_messages_ping_sent:39
cluster_stats_messages_pong_sent:37
cluster_stats_messages_meet_sent:4
cluster_stats_messages_sent:80
cluster_stats_messages_ping_received:34
cluster_stats_messages_pong_received:43
cluster_stats_messages_meet_received:3
cluster_stats_messages_received:80
以上信息顯示集群狀態(tài)ok。
設(shè)置集群密碼
上面提供了publicisworldwide/redis-cluster的鏡像不能設(shè)置永久密碼,是因?yàn)樗鼪](méi)有給redis.conf寫(xiě)入權(quán)限,在我們重創(chuàng)建的鏡像里是有寫(xiě)入權(quán)限的,所以可以創(chuàng)建永久密碼。
集群構(gòu)建完畢后再通過(guò)config set + config rewrite命令逐個(gè)機(jī)器設(shè)置密碼
在之前的鏡像上設(shè)置密碼會(huì)有權(quán)限問(wèn)題,如下:
127.0.0.1:8001> config set masterauth 1111
OK
127.0.0.1:8001> config set requirepass 1111
OK
127.0.0.1:8001> auth
(error) ERR wrong number of arguments for 'auth' command
127.0.0.1:8001> auth 1111
OK
127.0.0.1:8001> config rewrite
(error) ERR Rewriting config file: Permission denied
在新的鏡像上添加密碼,如下:
[root@localhost src]# ./redis-cli -c -p 8001
127.0.0.1:8001> config set masterauth 1111
OK
127.0.0.1:8001> config set requirepass 1111
OK
127.0.0.1:8001> config rewrite
(error) NOAUTH Authentication required.
127.0.0.1:8001> auth 1111
OK
127.0.0.1:8001> config rewrite
OK
當(dāng)重登錄8001機(jī)器并get數(shù)據(jù)時(shí)就會(huì)提示需要密碼了。其它機(jī)器也要一并設(shè)置密碼才行。
查看密碼文件
是否將密碼寫(xiě)入到配置文件redis.conf,可以登錄到容器查看即可。
docker exec -it 容器ID /bin/bash
cat /usr/local/etc/redis.conf
可以看到masterauth和requirepass被追加到文件的最后,即使重啟密碼也還生效。
在單機(jī)上部署意義不大,可以把slave結(jié)點(diǎn)放到不同的服務(wù)器上,再通過(guò)docker的redis-trib容器來(lái)連接,十分靈活和方便。
異常處理
在第三步執(zhí)行可能會(huì)現(xiàn)以下異常,如下:
[root@localhost redis-cluster]# docker run --rm -it inem0o/redis-trib create --replicas 1 172.19.165.222:8001 172.19.165.222:8002 172.19.165.222:8003 172.19.165.222:8004 172.19.165.222:8005 172.19.165.222:8006
>>> Creating cluster
[ERR] Node 172.19.165.222:8001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
這個(gè)異常是因?yàn)樵谀夸浵?code>/data/redis/800*已經(jīng)存在redis的配置文件,要先清除掉。
[root@localhost redis-cluster]# rm /data/redis/800* -rf