實驗背景
A文件服務(wù)器的/opt/dataA 和 B文件服務(wù)器的/opt/dataB目錄實時同步
A正常時，A為主，B為備，B去實時同步A
A掛了時，B為主，待A起來后，A為備，去實時同步B，并且能補(bǔ)上掛掉期間B增加的數(shù)據(jù)，反之亦然
keepalived + rsync +inotify 實現(xiàn)真正的高效數(shù)據(jù)實時同步
https://fandenggui.com/post/keepalived-rsync-inotify.html
真正的inotify+rsync實時同步 徹底告別同步慢
http://www.ttlsa.com/web/let-infotify-rsync-fast
一、實驗環(huán)境
操作系統(tǒng)：CentOS7.2
serverA： 192.168.1.102
serverB： 192.168.1.103
vip： ? ? ? ? ?192.168.1.120
二、軟件安裝
在 serverA和serverB上
# yum ?-y install ?epel-release
# yum ?-y install ?keepalived ?inotity-tools? rsync ?httpd
# systemctl enable ? httpd ?keepalived
rsync-inotify.service?會由后面的Keepalived程序根據(jù)主備角色變化情況，自動的運(yùn)行和關(guān)閉
設(shè)置selinux為寬松模式
# setenforce 0
# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' ? /etc/selinux/config
關(guān)閉防火墻
# systemctl stop ? ? firewalld
# systemctl disable firewalld
主機(jī)互信
我們同步采用rsync的ssh模式，需要讓serverA和?serverB 相互互信
三、主機(jī)配置
serverA?
# vim ?/opt/data_realtime_rsync.sh
#################################################
#!/bin/bash
# File name:data_realtime_sync.sh
# File path: /opt/data_realtime_sync.sh
src_dir="/opt/dataA"
dest_dir="/opt/dataB"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
. /etc/init.d/functions
cd ${src_dir}
inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |
while read file;do
? ? INO_EVENT=$(echo $file | awk '{print $1}') ? ? ?
? ? INO_FILE=$(echo $file ?| awk '{print $2}') ? ? ?
? ? if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then
? ? ? ? for ip in ${dest_ip};do
? ? ? ? ? ? rsync -avzcR -e ssh ?$(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
? ? ? ? done
? ? fi
? ? if [[ $INO_EVENT =~ 'ATTRIB' ]];then
? ? ? ? if [ ! -d "$INO_FILE" ];then
? ? ? ? ? ? for ip in ${dest_ip};do
? ? ? ? ? ? ? ? rsync -avzcR ?-e ssh ?$(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
? ? ? ? ? ? done
? ? ? ? fi
? ? fi
? ? if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then
? ? ? ? for ip in ${dest_ip};do
? ? ? ? ? ? rsync -avzcR --delete ?-e ssh ?$(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
? ? ? ? done
? ? fi
done
##############################################
# vim ?/etc/keepalived/keepalived.conf
###########################################
! Configuration File for keepalived
global_defs {
? ?router_id LVS_DEVEL
}
vrrp_script check_httpd {?
? ? script "/etc/keepalived/check_httpd.sh"?
? ? interval 5 ? ??
} ??
vrrp_instance VI_1 {
? ? state BACKUP
? ? interface eno16777736
? ? virtual_router_id 100
? ? priority 100
? ? advert_int 1
? ? nopreempt
? ? authentication {
? ? ? ? auth_type PASS
? ? ? ? auth_pass 1111
? ? }
? ? track_script { ??
? ? ? check_httpd
? ? } ??
? ? virtual_ipaddress {
? ? ? ? 192.168.1.120
? ? }
notify_master "/etc/keepalived/rsync-inotify.sh start"
notify_backup "/etc/keepalived/rsync-inotify.sh stop"
notify_stop ? "/etc/keepalived/rsync-inotify.sh send"
}
##########################################
# vim ?/etc/keepalived/check_httpd.sh
##############################
#!/bin/sh
#
httpd_status1=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status1}" = "0" ]; then
? systemctl start httpd.service
? sleep 5
? httpd_status2=$(ps -C httpd --no-heading|wc -l)
? if [ "${httpd_status2}" = "0" ]; then
? ? systemctl stop keepalived.service
? fi
fi
#############################
# vim ?/etc/keepalived/rsync-inotify.sh
########################
#!/bin/bash
case "$1" in
? start )
? ? systemctl start rsync-inotity.service
? ;;
? stop )
? ? systemctl stop rsync-inotity.service
? ;;
? restart )
? ? systemctl restart rsync-inotity.service
? ;;
? send )
? ? echo "Keepalived stoppd in 192.168.1.102" | mail -s "Alarm Mail" ?example@qq.com
? ;;
? * )
? ? echo "Usage:$0 start|stop|restart|send"
? ;;
esac
########################
# vim ?/etc/systemd/system/rsync-inotity.service
################################
[Unit]
Description=File instant rsync script
[Service]
ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh ?&
[Install]
WantedBy=multi-user.target
################################
# vim ?/etc/full-rsync.conf
#################################
src_dir="/opt/dataA"
dest_dir="/opt/dataB"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
##################################
# vim /etc/systemd/system/full-rsync.service
####################################
[Unit]
Description=Full rsync script
After=network.target sshd.service
Wants=sshd.service
[Service]
EnvironmentFile=/etc/full-rsync.conf
ExecStart=/usr/bin/rsync -avz --delete ?-e ssh ?root@${dest_ip}:${dest_dir}/ ?${src_dir}
Type=oneshot
[Install]
WantedBy=multi-user.target
#####################################
serverB
# vim ?/opt/data_realtime_rsync.sh
#############################
#!/bin/bash
# File name:data_realtime_sync.sh
# File path: /opt/data_realtime_sync.sh
src_dir="/opt/dataB"
dest_dir="/opt/dataA"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
. /etc/init.d/functions
cd ${src_dir}
inotifywait -mrq -e modify,attrib,close_write,move,create,delete --format '%e %w%f' ./ |
while read file;do
? ? INO_EVENT=$(echo $file | awk '{print $1}') ? ? ?
? ? INO_FILE=$(echo $file ?| awk '{print $2}') ? ? ?
? ? if [[ $INO_EVENT =~ 'CREATE' ]] || [[ $INO_EVENT =~ 'MODIFY' ]] || [[ $INO_EVENT =~ 'CLOSE_WRITE' ]] || [[ $INO_EVENT =~ 'MOVED_TO' ]];then
? ? ? ? for ip in ${dest_ip};do
? ? ? ? ? ? rsync -avzcR ?-e ?ssh ?$(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
? ? ? ? done
? ? fi
? ? if [[ $INO_EVENT =~ 'ATTRIB' ]];then
? ? ? ? if [ ! -d "$INO_FILE" ];then
? ? ? ? ? ? for ip in ${dest_ip};do
? ? ? ? ? ? ? ? rsync -avzcR -e ?ssh ?$(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
? ? ? ? ? ? done
? ? ? ? fi
? ? fi
? ? if [[ $INO_EVENT =~ 'DELETE' ]] || [[ $INO_EVENT =~ 'MOVED_FROM' ]];then
? ? ? ? for ip in ${dest_ip};do
? ? ? ? ? ? rsync -avzcR --delete ?-e ssh ?$(dirname ${INO_FILE}) root@${dest_ip}:${dest_dir}
? ? ? ? done
? ? fi
done
###################################
# vim ?/etc/keepalived/keepalived.conf
###########################################
! Configuration File for keepalived
global_defs {
? ?router_id LVS_DEVEL
}
vrrp_script check_httpd {?
? ? script "/etc/keepalived/check_httpd.sh"?
? ? interval 5 ? ??
} ??
vrrp_instance VI_1 {
? ? state BACKUP
? ? interface eno16777736
? ? virtual_router_id 90
? ? priority 100
? ? advert_int 1
? ? nopreempt
? ? authentication {
? ? ? ? auth_type PASS
? ? ? ? auth_pass 1111
? ? }
? ? track_script { ??
? ? ? check_httpd
? ? } ??
? ? virtual_ipaddress {
? ? ? ? 192.168.1.120
? ? }
notify_master "/etc/keepalived/rsync-inotify.sh start"
notify_backup "/etc/keepalived/rsync-inotify.sh stop"
notify_stop ? "/etc/keepalived/rsync-inotify.sh send"
}
##########################################
# vim ?/etc/keepalived/check_httpd.sh
##############################
#!/bin/sh
#
httpd_status1=$(ps -C httpd --no-heading|wc -l)
if [ "${httpd_status1}" = "0" ]; then
? systemctl start httpd.service
? sleep 5
? httpd_status2=$(ps -C httpd --no-heading|wc -l)
? if [ "${httpd_status2}" = "0" ]; then
? ? systemctl stop keepalived.service
? fi
fi
#############################
# vim ?/etc/keepalived/rsync-inotify.sh
########################
#!/bin/bash
case "$1" in
? start )
? ? systemctl start rsync-inotity.service
? ;;
? stop )
? ? systemctl stop rsync-inotity.service
? ;;
? restart )
? ? systemctl restart rsync-inotity.service
? ;;
? send )
? ? echo "Keepalived stoppd in 192.168.1.102" | mail -s "Alarm Mail" ??example@qq.com
? ;;
? * )
? ? echo "Usage:$0 start|stop|restart|send"
? ;;
esac
########################
# vim ?/etc/systemd/system/rsync-inotity.service
################################
[Unit]
Description=File instant rsync script
[Service]
ExecStart=/usr/bin/nohup /opt/data_realtime_rsync.sh ?&
[Install]
WantedBy=multi-user.target
################################
# vim ?/etc/full-rsync.conf
#################################
src_dir="/opt/dataB"
dest_dir="/opt/dataA"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
##################################
# vim /etc/systemd/system/full-rsync.service
####################################
[Unit]
Description=Full rsync script
After=network.target sshd.service
Wants=sshd.service
[Service]
EnvironmentFile=/etc/full-rsync.conf
ExecStart=/usr/bin/rsync -avz --delete ?-e ssh ?root@${dest_ip}:${dest_dir}/ ?${src_dir}
Type=oneshot
[Install]
WantedBy=multi-user.target
#####################################
四、測試
# systemctl ?status httpd
# systemctl ?status keepalived
# systemctl ?status ?rsync-inotity
五、全量同步
手動方式
A 全量同步B
# ?vim ?full_rsync_from_B_to_A.sh
######################################################
#!/bin/bash
src_dir="/opt/dataA"
dest_dir="/opt/dataB"
src_ip="192.168.1.102"
dest_ip="192.168.1.103"
rsync -avz?--delete ?-e ssh ?root@${dest_ip}:${dest_dir}/ ?${src_dir}/
#######################################################
B 全量同步A
# vim full_rsync_from_A_to_B.sh
#######################################################
#!/bin/bash
src_dir="/opt/dataB"
dest_dir="/opt/dataA"
src_ip="192.168.1.103"
dest_ip="192.168.1.102"
rsync -avz?--delete ?-e ssh ?root@${dest_ip}:${dest_dir}/ ?${src_dir}/
#######################################################
系統(tǒng)服務(wù)
#?systemctl ?daemon-reload
# systemctl ?start? full-rsync
# ?systemctl ?status ?full-rsync
通過/etc/rc.d/rc.local 開機(jī)執(zhí)行
# vim?/etc/rc.d/rc.local
##################
# Full rsync?
src_dir="/xxx"
dest_dir="/xxx"
src_ip="xx.xx.xx.xx"
dest_ip="xx.xx.xx.xx"
rsync -avz --delete ?-e ssh ?root@${dest_ip}:${dest_dir}/ ?${src_dir}
##################
# chmod +x ?/etc/rc.d/rc.local
六、優(yōu)化inotity
如果不進(jìn)行優(yōu)化，遇到瓶頸會有如下錯誤提示：
Failed to watch /your/path; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.
主要優(yōu)化?max_user_watches?及?max_queued_events?
具體大小可以等于當(dāng)前數(shù)據(jù)盤的 inode最大值
這里以1T數(shù)據(jù)盤為例：
# echo "67108864" > /proc/sys/fs/inotify/max_user_watches
# echo "67108864" > /proc/sys/fs/inotify/max_queued_events
將下面參數(shù)寫入 /etc/sysctl.conf，永久生效
# vim ?
/etc/sysctl.conf
fs.inotify.max_user_watches = 67108864
fs.inotify.max_queued_events = 67108864
七、參考
keepalived + rsync +inotify 實現(xiàn)真正的高效數(shù)據(jù)實時同步
https://fandenggui.com/post/keepalived-rsync-inotify.html
真正的inotify+rsync實時同步 徹底告別同步慢
http://www.ttlsa.com/web/let-infotify-rsync-fast
【Keepalived 】keepalived check and notify scripts
大家知道keepalived會有四種狀態(tài)的變化，每種狀態(tài)變化時，都可以調(diào)用一個腳本
當(dāng)進(jìn)入Master狀態(tài)時會呼叫notify_master
當(dāng)進(jìn)入Backup狀態(tài)時會呼叫notify_backup
當(dāng)發(fā)現(xiàn)異常情況時進(jìn)入Fault狀態(tài)呼叫notify_fault
當(dāng)Keepalived程序終止時則呼叫notify_stop
進(jìn)入Master和Backup這兩種狀態(tài)很容易理解了，就是分別變?yōu)橹骱蛷?/p>
進(jìn)入Fault，簡單的說就是keepalived發(fā)現(xiàn)自己有問題了，不能能再去參與Master競選了，你得修好他才行。
進(jìn)入這種狀態(tài)一般是keepalived自身出問題了，或者keepalived檢測的網(wǎng)卡出問題不通了，再或者就是我們自己寫的檢測業(yè)務(wù)的腳本返回錯誤
進(jìn)入Stop 這個就容易理解了，執(zhí)行service keepalived stop 或者 systemctl stop keepalived ?就會進(jìn)入這個狀態(tài)。
設(shè)置selinux為寬松模式
# setenforce 0
# sed -i 's/^SELINUX=.*/SELINUX=permissive/g' ? /etc/selinux/config
CentOS防火墻默認(rèn)是不允許keepalived使用?vrrp的組播。
如果不開啟組播ip,keepalived雙機(jī)不能實現(xiàn)熱備的效果,只能實現(xiàn)負(fù)載的效果,即虛擬ip不能實現(xiàn)漂移?。
Check that the multicast IP and protocol for VRRP are allowed in the firewall on both servers.?
For firewalld:
添加規(guī)則
# firewall-cmd --direct --permanent--add-rule?ipv4 filterINPUT ? ??0 ??? --in-interface eth0? ?--destination?224.0.0.18?--protocol vrrp -j ACCEPT
# firewall-cmd --direct --permanent--add-ruleipv4 filterOUTPUT?0 ? ?--out-interface eth0? --destination?224.0.0.18?--protocol vrrp -j ACCEPT
# firewall-cmd --reload
刪除規(guī)則
# firewall-cmd --direct --permanent--remove-ruleipv4 filter INPUT ? ??0 ??? --in-interface eth0? ?--destination?224.0.0.18?--protocol vrrp -j ACCEPT
# firewall-cmd --direct --permanent--remove-rule?ipv4 filter ?OUTPUT?0 ? ?--out-interface eth0? --destination?224.0.0.18?--protocol vrrp -j ACCEPT
# firewall-cmd --reload
For iptables:
添加規(guī)則
# iptables?-A?INPUT?-p?vrrp?-j?ACCEPT
# iptables?-A?OUTPUT?-p?vrrp?-j?ACCEPT
# service iptables save
刪除規(guī)則
# iptables ?-D ?INPUT?-p?vrrp?-j?ACCEPT
# iptables ?-D?OUTPUT?-p?vrrp?-j?ACCEPT
# service iptables save
keepalived基本應(yīng)用解析
http://blog.51cto.com/pangge/1301878
keepalived官方文檔
http://www.keepalived.org/doc/introduction.html
使用keepalived實現(xiàn)redis的故障切換
http://peiqiang.net/2014/11/21/keepalived-and-redis.html
Kamailio High Availability Done Right with Keepalived
http://blog.unicsolution.com/2015/01/kamailio-high-availability-with.html
Keepalived Check and Notify Scripts
