一、高可用集群
（一）提升系統(tǒng)高可用性的解決方案：冗余(redundant)
-
工作模式
- active/passive：主備
- active/active：雙主
-
以心跳方式通告
- active --> HEARTBEAT --> passive
- active <--> HEARTBEAT <--> active
-
故障處理
- failover：故障切換，即某資源的主節(jié)點(diǎn)故障時(shí)，將資源轉(zhuǎn)移至其它節(jié)點(diǎn)的操作
- failback：故障移回，即某資源的主節(jié)點(diǎn)故障后重新修改上線后，將之前已轉(zhuǎn)移至其它節(jié)點(diǎn)的資源重新切回的過(guò)程
（二）HA Cluster實(shí)現(xiàn)方案
ais：應(yīng)用接口規(guī)范完備復(fù)雜的HA集群
RHCS：Red Hat Cluster Suite紅帽集群套件
heartbeat
corosyncvrrp協(xié)議實(shí)現(xiàn)：虛擬路由冗余協(xié)議
keepalived
二、KeepAlived基本介紹
（一）VRRP(Virtual Router Redundancy Protocol)協(xié)議術(shù)語(yǔ)
-
虛擬路由器：Virtual Router，多個(gè)物理路由器對(duì)外以一個(gè)IP地址提供服務(wù)，仿佛一臺(tái)路由器
- 虛擬路由器標(biāo)識(shí)：VRID(0-255)，唯一標(biāo)識(shí)虛擬路由器
- VIP：Virtual IP，虛擬IP
- VMAC：Virutal MAC (00-00-5e-00-01-VRID)，虛擬MAC
物理路由器
master：主設(shè)備
backup：備用設(shè)備
priority：優(yōu)先級(jí)
（二）KeepAlived的工作特性
通告：心跳，優(yōu)先級(jí)等；周期性
工作方式：搶占式，非搶占式
-
安全認(rèn)證：
- 無(wú)認(rèn)證
- 簡(jiǎn)單字符認(rèn)證：預(yù)共享密鑰
- MD5
-
工作模式：
- 主/備：?jiǎn)翁摂M路由器
- 主/主：主/備（虛擬路由器1），備/主（虛擬路由器2）
（三）KeepAlived的功能
vrrp協(xié)議完成地址流動(dòng)
為vip地址所在的節(jié)點(diǎn)生成ipvs規(guī)則(在配置文件中預(yù)先定義)
為ipvs集群的各RS做健康狀態(tài)檢測(cè)
基于腳本調(diào)用接口通過(guò)執(zhí)行腳本完成腳本中定義的功能，進(jìn)而影響集群事務(wù)，以此支持nginx, haproxy等服務(wù)
三、KeepAlived的配置
（一）HA Cluster配置準(zhǔn)備：
-
各節(jié)點(diǎn)時(shí)間必須同步：ntp服務(wù)（CentOS 6）, chrony（CentOS 7）
// 由于ntp/chrony服務(wù)不能同步差距過(guò)大的時(shí)間，需要先使用ntpdate命令同步一次，再開(kāi)啟服務(wù) ntpdate ntp_server_ip // 開(kāi)啟chronyd服務(wù)（CentOS 7） vim /etc/chrony.conf server 172.18.0.1 iburst systemctl enable chronyd systemctl start chronyd // 開(kāi)啟ntp服務(wù)（CentOS 6） vim /etc/ntp.conf server 172.18.0.1 iburst chkconfig ntpd on service ntpd start 確保iptables及selinux不會(huì)成為阻礙
各節(jié)點(diǎn)之間可通過(guò)主機(jī)名互相通信（對(duì)KA并非必須），建議使用/etc/hosts文件實(shí)現(xiàn)
-
各節(jié)點(diǎn)之間的root用戶可以基于密鑰認(rèn)證的ssh服務(wù)完成互相通信（對(duì)KA并非必須）
ssh-keygen ssh-copy-id destination_ip
（二）KeepAlived的程序環(huán)境
主配置文件：/etc/keepalived/keepalived.conf
主程序文件：/usr/sbin/keepalived
Unit File：/usr/lib/systemd/system/keepalived.service
Unit File的環(huán)境配置文件：/etc/sysconfig/keepalived
（三）KeepAlived的配置文件結(jié)構(gòu)
GLOBAL CONFIGURATION：全局設(shè)置
Global definitions
Static routes/addressesVRRPD CONFIGURATION：VRRP設(shè)置
VRRP synchronization group(s)：vrrp同步組
VRRP instance(s)：即一個(gè)vrrp虛擬路由器LVS CONFIGURATION：LVS設(shè)置
Virtual server group(s)
Virtual server(s)：ipvs集群的vs和rs
（四）配置虛擬路由器
-
語(yǔ)法：
vrrp_instance <STRING> { .... } -
專用參數(shù)：
- state MASTER | BACKUP
當(dāng)前節(jié)點(diǎn)在此虛擬路由器上的初始狀態(tài)；只能有一個(gè)是MASTER，余下的都應(yīng)該為BACKUP - interface IFACE_NAME
綁定為當(dāng)前虛擬路由器使用的物理接口 - virtual_router_id VRID
當(dāng)前虛擬路由器惟一標(biāo)識(shí)，范圍是0-255 - priority 100
當(dāng)前物理節(jié)點(diǎn)在此虛擬路由器中的優(yōu)先級(jí)；范圍1-254 - advert_int 1
vrrp通告的時(shí)間間隔，默認(rèn)1s - authentication：認(rèn)證機(jī)制
authentication { auth_type AH|PASS auth_pass <PASSWORD> 僅前8位有效 }- virtual_ipaddress：虛擬IP
virtual_ipaddress { <IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL> }- track_interface：配置監(jiān)控網(wǎng)絡(luò)接口，一旦出現(xiàn)故障，則轉(zhuǎn)為FAULT狀態(tài)實(shí)現(xiàn)地址轉(zhuǎn)移
track_interface { eth0 eth1 … }- nopreempt：定義工作模式為非搶占模式
- preempt_delay 300：搶占式模式，節(jié)點(diǎn)上線后觸發(fā)新選舉操作的延遲時(shí)長(zhǎng)，默認(rèn)模式
- 定義通知腳本：
notify_master <STRING> | <QUOTED-STRING>：
當(dāng)前節(jié)點(diǎn)成為主節(jié)點(diǎn)時(shí)觸發(fā)的腳本
notify_backup <STRING> | <QUOTED-STRING>：
當(dāng)前節(jié)點(diǎn)轉(zhuǎn)為備節(jié)點(diǎn)時(shí)觸發(fā)的腳本
notify_fault <STRING> | <QUOTED-STRING>：
當(dāng)前節(jié)點(diǎn)轉(zhuǎn)為“失敗”狀態(tài)時(shí)觸發(fā)的腳本
notify <STRING> | <QUOTED-STRING>：
通用格式的通知觸發(fā)機(jī)制，一個(gè)腳本可完成以上三種狀態(tài)的轉(zhuǎn)換時(shí)的通知
- state MASTER | BACKUP
-
實(shí)驗(yàn)1：實(shí)現(xiàn)主/備虛擬路由器
- 實(shí)驗(yàn)環(huán)境：
物理路由器1：ip: 192.168.136.230, 主機(jī)名: node1, MASTER
物理路由器2：ip: 192.168.136.130, 主機(jī)名: node2, BACKUP
VIP：192.168.136.100
// 配置物理路由器1 vim /etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost } notification_email_from node1@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 // vrrp中的路由器主機(jī)名 vrrp_mcast_group4 224.0.0.58 // 設(shè)置組播ip地址 } vrrp_instance VI_1 { state MASTER interface ens37 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 // openssl rand -hex 4 生成8位16進(jìn)制密碼 } virtual_ipaddress { 192.168.136.100/24 } } systemctl start keepalived // 配置物理路由器2 vim /etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost } notification_email_from node2@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node2 vrrp_mcast_group4 224.0.0.58 } vrrp_instance VI_1 { state BACKUP interface ens37 virtual_router_id 51 priority 90 //作為BACKUP優(yōu)先級(jí)比MASTER要低 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 // 密碼與node1相同 } virtual_ipaddress { 192.168.136.100/24 } } systemctl start keepalived- 測(cè)試
node1的ip地址已經(jīng)出現(xiàn)VIP
監(jiān)聽(tīng)組播地址的tcp連接
tcpdump -i ens37 -nn host 224.0.0.58，此時(shí)關(guān)閉node1的keepalived服務(wù)systemctl stop keepalived，自動(dòng)由node2接管并開(kāi)始聲明自身?yè)碛刑摂M路由器的IPVIP此時(shí)已經(jīng)被node2接管
- 實(shí)驗(yàn)環(huán)境：
- 實(shí)驗(yàn)2：實(shí)現(xiàn)keepalived日志
vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 3" // -D:詳細(xì)日志，-S 3: 設(shè)置日志facility為local3
vim /etc/rsyslog.conf
local3.* /var/log/keepalived.log // 設(shè)置日志存儲(chǔ)路徑
systemctl restart rsyslog
systemctl restart keepalived
tail -f /var/log/keepalived.log
-
實(shí)驗(yàn)3：實(shí)現(xiàn)主/主虛擬路由器，并且當(dāng)節(jié)點(diǎn)發(fā)生變化時(shí)主動(dòng)發(fā)送郵件
- 實(shí)驗(yàn)環(huán)境
物理路由器1：ip: 192.168.136.230, 主機(jī)名: node1
物理路由器2：ip: 192.168.136.130, 主機(jī)名: node2
虛擬路由器1：MASTER: node1, BACKUP: node2, VIP: 192.168.136.100
虛擬路由器2：MASTER: node2, BACKUP: node1, VIP: 192.168.136.200
// 配置物理路由器1（虛擬路由器1的MASTER，虛擬路由器2的BACKUP） vim /etc/keepalived/keepalived.conf smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.0.0.58 } // 虛擬路由器1的設(shè)置 vrrp_instance VI_1 { state MASTER interface ens37 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 } virtual_ipaddress { 192.168.136.100/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } // 虛擬路由器2的設(shè)置 vrrp_instance VI_2 { state BACKUP interface ens37 virtual_router_id 61 priority 80 advert_int 1 authentication { auth_type PASS auth_pass a56c19be } virtual_ipaddress { 192.168.136.200/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } systemctl restart keepalived // 配置物理路由器2（虛擬路由器1的BACKUP，虛擬路由器2的MASTER） vim /etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost } notification_email_from node2@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node2 vrrp_mcast_group4 224.0.0.58 } // 虛擬路由器1的設(shè)置 vrrp_instance VI_1 { state BACKUP interface ens37 virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 } virtual_ipaddress { 192.168.136.100/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } // 虛擬路由器2的設(shè)置 vrrp_instance VI_2 { state MASTER interface ens37 virtual_router_id 61 priority 100 advert_int 1 authentication { auth_type PASS auth_pass a56c19be } virtual_ipaddress { 192.168.136.200/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } // 在物理路由器1,2上添加腳本文件 vim /etc/keepalived/notify.sh #! /bin/bash contact='root@localhost' notify() { mailsubject="$(hostname) to be $1, vip floating" mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1" echo "$mailbody" | mail -s "$mailsubject" $contact } case $1 in master) notify master ;; backup) notify backup ;; fault) notify fault ;; *) echo "Usage: $(basename $0) {master|backup|fault}" exit 1 ;; esac chmod +x /etc/keepalived/notify.sh- 測(cè)試
監(jiān)聽(tīng)組播地址的tcp連接tcpdump -i ens37 -nn host 224.0.0.58，可以看到node1, node2分別聲明擁有虛擬路由器1(vrid 51)、虛擬路由器2(vrid61)的IP地址
分別查看node1和node2的網(wǎng)卡IP地址，進(jìn)一步確認(rèn)上述結(jié)果
此時(shí)，斷開(kāi)node1的網(wǎng)絡(luò)連接
虛擬路由器1的VIP立即由node2的網(wǎng)卡接管恢復(fù)node1的網(wǎng)絡(luò)連接，在node1和node2上都可以看到相應(yīng)的郵件通知：
node1上通知出錯(cuò)，很快通知自身被切換為BACKUP，恢復(fù)網(wǎng)絡(luò)連接后通知自身重新變?yōu)镸ASTER；node2上通知自身切換為MASTER，恢復(fù)網(wǎng)絡(luò)連接后通知自身切換為BACKUP
- 實(shí)驗(yàn)環(huán)境
（五）Keepalived支持IPVS
- 語(yǔ)法：
virtual_server {IP port | fwmark int}
{
...
real_server{
...
}
...
}
-
virtual_server常用參數(shù)
- delay_loop <INT>
檢查后端服務(wù)器的時(shí)間間隔 - lb_algo rr|wrr|lc|wlc|lblc|sh|dh
定義調(diào)度方法 - lb_kind NAT|DR|TUN
集群的類型 - persistence_timeout <INT>
持久連接時(shí)長(zhǎng) - protocol TCP
服務(wù)協(xié)議，僅支持TCP - sorry_server<IPADDR> <PORT>
所有RS故障時(shí)，備用服務(wù)器地址
- delay_loop <INT>
-
real_server <IPADDR> <PORT>常用參數(shù)
- weight <INT>
RS權(quán)重 - notify_up <STRING>|<QUOTED-STRING>
RS上線通知腳本 - notify_down <STRING>|<QUOTED-STRING>
RS下線通知腳本 - HTTP_GET|SSL_GET|TCP_CHECK|SMTP_CHECK|MISC_CHECK { ... }
定義當(dāng)前主機(jī)的健康狀態(tài)檢測(cè)方法
- weight <INT>
-
HTTP_GET|SSL_GET：應(yīng)用層健康狀態(tài)檢測(cè)
HTTP_GET|SSL_GET { url { path <URL_PATH> // 定義要監(jiān)控的URL status_code <INT> // 判斷上述檢測(cè)機(jī)制為健康狀態(tài)的響應(yīng)碼 digest <STRING> // 判斷為健康狀態(tài)的響應(yīng)的內(nèi)容的校驗(yàn)碼 } connect_timeout <INTEGER> // 連接請(qǐng)求的超時(shí)時(shí)長(zhǎng) nb_get_retry <INT> // 重試次數(shù) delay_before_retry <INT> // 重試之前的延遲時(shí)長(zhǎng) connect_ip <IP ADDRESS> // 向當(dāng)前RS哪個(gè)IP地址發(fā)起健康狀態(tài)檢測(cè)請(qǐng)求 connect_port <PORT> // 向當(dāng)前RS的哪個(gè)PORT發(fā)起健康狀態(tài)檢測(cè)請(qǐng)求 bindto <IP ADDRESS> // 發(fā)出健康狀態(tài)檢測(cè)請(qǐng)求時(shí)使用的源地址 bind_port <PORT> // 發(fā)出健康狀態(tài)檢測(cè)請(qǐng)求時(shí)使用的源端口 } -
TCP_CHECK參數(shù)
- connect_ip <IP ADDRESS>
向當(dāng)前RS的哪個(gè)IP地址發(fā)起健康狀態(tài)檢測(cè)請(qǐng)求 - connect_port <PORT>
向當(dāng)前RS的哪個(gè)PORT發(fā)起健康狀態(tài)檢測(cè)請(qǐng)求 - bindto <IP ADDRESS>
發(fā)出健康狀態(tài)檢測(cè)請(qǐng)求時(shí)使用的源地址 - bind_port <PORT>
發(fā)出健康狀態(tài)檢測(cè)請(qǐng)求時(shí)使用的源端口 - connect_timeout <INTEGER>
連接請(qǐng)求的超時(shí)時(shí)長(zhǎng)
- connect_ip <IP ADDRESS>
-
實(shí)驗(yàn)4：實(shí)現(xiàn)主/備模型的IPVS集群
- 實(shí)驗(yàn)環(huán)境：
LB1(master)/VS：IP: 192.168.136.230
LB2(backup)/VS：IP: 192.168.136.130
VIP：192.168.136.100
RS1：IP: 192.168.136.229
RS2：IP: 192.168.136.129
// 配置LB1的keepalived設(shè)置 vim /etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost } notification_email_from node1@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 vrrp_mcast_group4 224.0.0.58 } vrrp_instance VI_1 { state MASTER interface ens37 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 } virtual_ipaddress { 192.168.136.100/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } virtual_server 192.168.136.100 80{ delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.136.229 80{ weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } real_server 192.168.136.129 80{ weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } // 配置LB2的keepalived設(shè)置 vim /etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost } notification_email_from node2@localhost smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node2 vrrp_mcast_group4 224.0.0.58 } vrrp_instance VI_1 { state BACKUP interface ens37 virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 } virtual_ipaddress { 192.168.136.100/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } virtual_server 192.168.136.100 80{ delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.136.229 80{ weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } real_server 192.168.136.129 80{ weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } // 配置LB1, LB2的sorry server服務(wù) echo sorry on LB1 > /var/www/html/index.html // LB1上操作 echo sorry on LB2 > /var/www/html/index.html // LB2上操作 systemctl start httpd // 配置RS1, RS2的Web服務(wù) echo RS1 homepage > /var/www/html/index.html // RS1上操作 echo RS2 homepage > /var/www/html/index.html // RS2上操作 systemctl start httpd // 編輯腳本實(shí)現(xiàn)：禁止RS響應(yīng)ARP請(qǐng)求，并將網(wǎng)卡綁定VIP vim lvs_dr_rs.sh #! /bin/bash vip='192.168.136.100' mask='255.255.255.255' dev=lo:1 rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null service httpd start &> /dev/null && echo "The httpd Server is Ready!" case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $dev $vip netmask $mask broadcast $vip up echo "The RS Server is Ready!" ;; stop) ifconfig $dev down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce echo "The RS Server is Canceled!" ;; *) echo "Usage: $(basename $0) start|stop" exit 1 ;; esac chmod +x lvs_dr_rs.sh bash lvs_dr_rs.sh start // LB1, LB2啟動(dòng)KeepAlived服務(wù)，進(jìn)行測(cè)試 systemctl start keepalived訪問(wèn)VIP(192.168.136.100)的Web服務(wù)，正常工作
停止RS2的Web服務(wù)，自動(dòng)進(jìn)行健康檢查，全部調(diào)度至RS1
停止RS1的Web服務(wù)，自動(dòng)進(jìn)行健康檢查，調(diào)度至LB1的sorry server
停止LB1的KeepAlived服務(wù)，自動(dòng)切換至LB2
- 實(shí)驗(yàn)環(huán)境：
-
實(shí)驗(yàn)5：實(shí)現(xiàn)主/主模型的IPVS集群
- 實(shí)驗(yàn)環(huán)境：
LB1/VS1：IP: 192.168.136.230，后端RS: RS1, RS2
LB2/VS2：IP: 192.168.136.130，后端RS: RS3, RS4
LB1 VIP：192.168.136.100
LB2 VIP：192.168.136.200
RS1：IP: 192.168.136.229
RS2：IP: 192.168.136.129
RS3：IP: 192.168.136.240
RS4：IP: 192.168.136.250
LB之間互為MASTER與BACKUP的關(guān)系
MASTER：LB1，BACKUP：LB2
MASTER：LB2，BACKUP：LB1
// 配置LB1, LB2的keepalived設(shè)置 global_defs { notification_email { root@localhost } notification_email_from node1@localhost // LB1上操作 notification_email_from node1@localhost // LB2上操作 smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 // LB1上操作 router_id node2 // LB2上操作 vrrp_mcast_group4 224.0.0.58 } vrrp_instance VI_1 { state MASTER // LB1上操作 state BACKUP // LB2上操作 interface ens37 virtual_router_id 51 priority 100 // LB1上操作 priority 90 // LB2上操作 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 } virtual_ipaddress { 192.168.136.100/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } vrrp_instance VI_2 { state BACKUP // LB1上操作 state MASTER // LB2上操作 interface ens37 virtual_router_id 61 priority 80 // LB1上操作 priority 100 // LB2上操作 advert_int 1 authentication { auth_type PASS auth_pass a56c19be } virtual_ipaddress { 192.168.136.200/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" } virtual_server 192.168.136.100 80{ delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.136.229 80{ weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } real_server 192.168.136.129 80{ weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } virtual_server 192.168.136.200 80{ delay_loop 3 lb_algo wrr lb_kind DR protocol TCP sorry_server 127.0.0.1 80 real_server 192.168.136.240 80{ weight 2 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } real_server 192.168.136.250 80{ weight 1 HTTP_GET { url { path / status_code 200 } connect_timeout 1 nb_get_retry 3 delay_before_retry 1 } } } // 配置LB1, LB2的sorry server服務(wù) echo sorry on LB1 > /var/www/html/index.html // LB1上操作 echo sorry on LB2 > /var/www/html/index.html // LB2上操作 systemctl start httpd // 配置RS1, RS2, RS3, RS4的Web服務(wù) echo RS1 homepage > /var/www/html/index.html // RS1上操作 echo RS2 homepage > /var/www/html/index.html // RS2上操作 echo RS3 homepage > /var/www/html/index.html // RS3上操作 echo RS4 homepage > /var/www/html/index.html // RS4上操作 systemctl start httpd // 編輯腳本實(shí)現(xiàn)：禁止RS響應(yīng)ARP請(qǐng)求，并將網(wǎng)卡綁定VIP vim lvs_dr_rs.sh #! /bin/bash vip='192.168.136.100' // RS1, RS2上操作 vip='192.168.136.200' // RS3, RS4上操作 mask='255.255.255.255' dev=lo:1 rpm -q httpd &> /dev/null || yum -y install httpd &>/dev/null service httpd start &> /dev/null && echo "The httpd Server is Ready!" case $1 in start) echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $dev $vip netmask $mask broadcast $vip up echo "The RS Server is Ready!" ;; stop) ifconfig $dev down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce echo "The RS Server is Canceled!" ;; *) echo "Usage: $(basename $0) start|stop" exit 1 ;; esac chmod +x lvs_dr_rs.sh bash lvs_dr_rs.sh start // LB1, LB2啟動(dòng)KeepAlived服務(wù)，進(jìn)行測(cè)試 systemctl start keepalived - 實(shí)驗(yàn)環(huán)境：
使用ipvsadm -Ln命令查看ipvs調(diào)度策略，與KeepAlived的配置吻合
訪問(wèn)VIP1, VIP2(192.168.136.100, 192.168.136.200)的Web服務(wù)，正常工作
停止RS1的Web服務(wù)，自動(dòng)進(jìn)行健康檢查，全部調(diào)度至RS2
停止RS2的Web服務(wù)，自動(dòng)進(jìn)行健康檢查，調(diào)度至LB1的sorry server
停止LB1的KeepAlived服務(wù)，自動(dòng)切換至LB2
停止RS3的Web服務(wù)，自動(dòng)進(jìn)行健康檢查，全部調(diào)度至RS4
停止RS4的Web服務(wù)，自動(dòng)進(jìn)行健康檢查，調(diào)度至LB2的sorry server
（六）Keepalived調(diào)用腳本進(jìn)行資源監(jiān)控
keepalived調(diào)用外部的輔助腳本進(jìn)行資源監(jiān)控，并根據(jù)監(jiān)控的結(jié)果狀態(tài)實(shí)現(xiàn)優(yōu)先動(dòng)態(tài)調(diào)整
vrrp_script：自定義資源監(jiān)控腳本，vrrp實(shí)例根據(jù)腳本返回值，公共定義，可被多個(gè)實(shí)例調(diào)用，定義在vrrp實(shí)例之外
-
track_script：調(diào)用vrrp_script定義的腳本去監(jiān)控資源，定義在實(shí)例之內(nèi)，調(diào)用事先定義的vrrp_script
- 分兩步：(1) 先定義一個(gè)腳本；(2) 調(diào)用此腳本
格式：
// 定義腳本，定義在實(shí)例外 vrrp_script <SCRIPT_NAME> { script "" // 引號(hào)內(nèi)為腳本命令 interval INT weight -INT } // 調(diào)用腳本，定義在實(shí)例內(nèi) track_script { SCRIPT_NAME_1 SCRIPT_NAME_2 } - 分兩步：(1) 先定義一個(gè)腳本；(2) 調(diào)用此腳本
-
實(shí)驗(yàn)6：實(shí)現(xiàn)主/主模型的高可用Nginx反向代理
- 實(shí)驗(yàn)環(huán)境：
LB1/VS1：IP: 192.168.136.230，后端RS: RS1, RS2
LB2/VS2：IP: 192.168.136.130，后端RS: RS3, RS4
LB1 VIP：192.168.136.100
LB2 VIP：192.168.136.200
RS1：IP: 192.168.136.229
RS2：IP: 192.168.136.129
RS3：IP: 192.168.136.240
RS4：IP: 192.168.136.250
LB之間互為MASTER與BACKUP的關(guān)系
MASTER：LB1，BACKUP：LB2
MASTER：LB2，BACKUP：LB1
// 配置LB1, LB2的KeepAlived設(shè)置 vim /etc/keepalived/keepalived.conf global_defs { notification_email { root@localhost } notification_email_from node1@localhost // LB1上操作 notification_email_from node2@localhost // LB2上操作 smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id node1 // LB1上操作 router_id node2 // LB2上操作 vrrp_mcast_group4 224.0.0.58 } vrrp_script chk_nginx { script "killall -0 nginx && exit 0 || exit 1;" interval 1 weight -20 fall 3 rise 3 } vrrp_instance VI_1 { state MASTER // LB1上操作 state BACKUP // LB2上操作 interface ens37 virtual_router_id 51 priority 100 // LB1上操作 priority 90 // LB2上操作 advert_int 1 authentication { auth_type PASS auth_pass dd73f9d6 } virtual_ipaddress { 192.168.136.100/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" // 下面的腳本引用僅在LB1的配置文件出現(xiàn) track_script { chk_nginx } } vrrp_instance VI_2 { state BACKUP // LB1上操作 state MASTER // LB2上操作 interface ens37 virtual_router_id 61 priority 90 // LB1上操作 priority 100 // LB2上操作 advert_int 1 authentication { auth_type PASS auth_pass a56c19be } virtual_ipaddress { 192.168.136.200/24 } notify_master "/etc/keepalived/notify.sh master" notify_backup "/etc/keepalived/notify.sh backup" notify_fault "/etc/keepalived/notify.sh fault" // 下面的腳本引用僅在LB2的配置文件出現(xiàn) track_script { chk_nginx } } // 配置LB1,LB2的nginx反向代理 vim /etc/nginx/nginx.conf http { upstream websrvs1 { server 192.168.136.229:80 weight=2; server 192.168.136.129:80 weight=1; } upstream websrvs2 { server 192.168.136.240:80 weight=2; server 192.168.136.250:80 weight=1; } server { listen 192.168.136.100:80; location / { proxy_pass http://websrvs1; } } server { listen 192.168.136.200:80; location / { proxy_pass http://websrvs2; } } } nginx -t systemctl start nginx // 配置RS1, RS2, RS3, RS4的Web服務(wù) echo RS1 homepage > /var/www/html/index.html // RS1上操作 echo RS2 homepage > /var/www/html/index.html // RS2上操作 echo RS3 homepage > /var/www/html/index.html // RS3上操作 echo RS4 homepage > /var/www/html/index.html // RS4上操作 systemctl start httpd // LB1, LB2啟動(dòng)KeepAlived服務(wù)，進(jìn)行測(cè)試 systemctl start keepalived登錄192.168.136.100和192.168.136.200的web服務(wù)，確實(shí)按照設(shè)置要求調(diào)度
停止RS2的httpd服務(wù)，全部調(diào)度至RS1
停止RS3的httpd服務(wù)，全部調(diào)度至RS4
關(guān)閉LB2的nginx反向代理服務(wù)，通過(guò)
tcpdump -i ens37 -nn host 224.0.0.58查看組播情況。三個(gè)紅框依次表達(dá)：
（1）未關(guān)閉nginx前的組播狀態(tài)
（2）關(guān)閉nginx后，LB2的vrid 61權(quán)重減去20變作80，而LB1vrid 61的權(quán)重為90
（3）由于LB1的權(quán)重高，VIP2的所有權(quán)被LB1接管關(guān)閉LB1的nginx反向代理服務(wù)，通過(guò)
tcpdump -i ens37 -nn host 224.0.0.58查看組播情況。三個(gè)紅框依次表達(dá)：
（1）未關(guān)閉nginx前的組播狀態(tài)
（2）關(guān)閉nginx后，LB1的vrid 51權(quán)重減去20變作80，而LB2的vrid 51權(quán)重為90
（3）由于LB2的權(quán)重高，VIP1的所有權(quán)被LB2接管由于此時(shí)兩個(gè)nginx反向代理均關(guān)閉，故訪問(wèn)192.168.136.100和192.168.136.200的web服務(wù)全部失敗
打開(kāi)LB2的nginx反向代理服務(wù)，通過(guò)
tcpdump -i ens37 -nn host 224.0.0.58查看組播情況。三個(gè)紅框依次表達(dá)：
（1）未打開(kāi)nginx前的組播狀態(tài)
（2）打開(kāi)nginx后，LB2的vrid 61權(quán)重增加20變作100，而LB1的vrid 61權(quán)重為90
（3）由于LB2的權(quán)重高，VIP2的所有權(quán)被LB2接管此時(shí)VIP1和VIP2均由LB2上的nginx服務(wù)器進(jìn)行反向代理，192.168.136.100和192.168.136.200的web服務(wù)全部恢復(fù)
- 實(shí)驗(yàn)環(huán)境：
（七）Keepalived同步組
LVS NAT模型VIP和DIP需要同步，需要同步組
-
格式：
vrrp_sync_group VG_1 { group { VI_1 # name of vrrp_instance(below) VI_2 # One for each moveable IP. } } vrrp_instance VI_1 { eth0 vip } vrrp_instance VI_2 { eth1 dip }
