keepalived
- 通过在两台主、备调度器上实现Keepalived高可用解决调度器单点失败问题
- 主、备调度器上配置LVS，正常情况下主调度器工作
- 主调度器异常时，Keepalived启用备用调度器，保证了调度器的高可用
- 调度器保证了后台服务的高可用
一、keepalived+lvs-DR模式配置高可用负载均衡
拓扑如下：
二、服务器配置信息如下：
1、LVS_MASTER
[root@master ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:5C:9B:35 inet addr:192.168.1.107 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe5c:9b35/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1167 errors:0 dropped:0 overruns:0 frame:0 TX packets:417 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:99957 (97.6 KiB) TX bytes:46517 (45.4 KiB)2、LVS_BACKUP
[root@backup ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:8B:95:97 inet addr:192.168.1.108 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe8b:9597/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:968 errors:0 dropped:0 overruns:0 frame:0 TX packets:382 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:84935 (82.9 KiB) TX bytes:50663 (49.4 KiB)3、web1主机
root@realserver1 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:EB:5A:68 inet addr:192.168.1.109 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:feeb:5a68/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:19546 errors:0 dropped:0 overruns:0 frame:0 TX packets:11171 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2870466 (2.7 MiB) TX bytes:1742194 (1.6 MiB)4、web2主机
[root@realserver2 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:A9:D1:DC inet addr:192.168.1.110 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fea9:d1dc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:20086 errors:0 dropped:0 overruns:0 frame:0 TX packets:12236 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3012984 (2.8 MiB) TX bytes:1924957 (1.8 MiB)三、在master和backup分别查看是否有ipvsadm软件
1、检查ipvsadm是否安装成功，可以直接输入
[root@master ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn[root@backup ~]# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn2、检查模板是否加入内核
[root@master ~]# lsmod | grep ip_vs
ip_vs 126534 0 libcrc32c 1246 1 ip_vsipv6 335525 270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6[root@backup ~]# lsmod | grep ip_vs
ip_vs 126534 0 libcrc32c 1246 1 ip_vsipv6 335525 270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6四、分别在web1和web2上面安装apache服务
开放端口80
[root@realserver1 ~]# netstat -ntpl | grep :80
tcp 0 0 :::80 :::* LISTEN 2841/httpd[root@realserver2 ~]# netstat -ntpl | grep :80
tcp 0 0 :::80 :::* LISTEN 1602/httpd五、在master主机（192.168.1.107）和backup主机（192.168.1.108）上安装keepalived
1、解压
[root@master ~]# tar xf keepalived-1.2.7.tar.gz
2、配置
[root@master ~]# cd keepalived-1.2.7
[root@master keepalived-1.2.7]# ./configure --prefix=/home/keepalived/ --with-kernel-dir=/usr/src/kernels/2.6.32-573.7.1.el6.x86_64报错：
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
[root@master keepalived-1.2.7]# yum -y install openssl-devel
报错：
configure: error: Popt libraries is required
[root@master keepalived-1.2.7]# yum -y install popt-devel
再次执行
[root@master keepalived-1.2.7]# ./configure --prefix=/home/keepalived/ --with-kernel-dir=/usr/src/kernels/2.6.32-573.7.1.el6.x86_64
成功安装：
3、make
[root@master keepalived-1.2.7]# make
4、make install
[root@master keepalived-1.2.7]#make install
5、配置文件的路径
[root@master keepalived-1.2.7]# cp /home/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[root@master keepalived-1.2.7]# cp /home/keepalived/etc/sysconfig/keepalived /etc/sysconfig/[root@master keepalived-1.2.7]# mkdir /etc/keepalived[root@master keepalived-1.2.7]#cp /home/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@master keepalived-1.2.7]# cp /home/keepalived/sbin/keepalived /sbin/6、配置keepalived.conf文件（master）
[root@master ~]# vim /etc/keepalived/keepalived.conf
----------------------------------------------------------------------------------------------------------------
#global define
global_defs {
router_id LVS #全局配置，设置本机id
}
vrr_sync_group b1_group {
group {
b1_one
}
}
vrrp_instance b1_one {
state MASTER #状态为master，从服务器为backup。必须大写
interface eth0 #指定端口以提供虚拟ip的绑定
virtual_router_id 38 #vrrp的组名，主从必须相同且不能与其他组相同
priority 100 #指定优先级，值越大越优先
advert_int 3 #组播信息发送间隔
authentication { #验证信息，两节点相同
auth_type PASS
auth_pass 1111
}
virtual_ipaddress { #执行虚拟ip地址
192.168.1.113/24 dev eth0
}
}
virtual_server 192.168.1.113 80 { #虚拟服务器的相应端口
delay_loop 3 #延时时间
lb_algo rr #lvs的算法
lb_kind DR #lvs的模式
persistence_timeout 1 #超时时间，在此时间内，同一用户发送多次请求，被转发到同一服务器上
protocol TCP #使用的协议
real_server 192.168.1.109 80 { #真实主机的ip以及端口，另一个服务器上ip为110，这个为109
weight 1 #权重，根据此进行请求的分发比例
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.1.110 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
----------------------------------------------------------------------------------------------------------------
7、配置keepalived.conf文件（BACKUP）
[root@backup ~]# vim /etc/keepalived/keepalived.conf
----------------------------------------------------------------------------------------------------------------
#global define
global_defs {
router_id LVS
}
vrr_sync_group b1_group {
group {
b1_one
}
}
vrrp_instance b1_one {
state BACKUP
interface eth0
virtual_router_id 38
priority 99
advert_int 3
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.113/24 dev eth0
}
}
virtual_server 192.168.1.113 80 {
delay_loop 3
lb_algo rr
lb_kind DR
persistence_timeout 50
protocol TCP
real_server 192.168.1.109 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.1.110 80 {
weight 1
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
----------------------------------------------------------------------------------------------------------------
8、分别在主备上启动keepalived
[root@master ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ][root@master ~]# chkconfig --add keepalived
[root@master ~]# chkconfig keepalived on[root@master ~]# chkconfig keepalived --list keepalived 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@master ~]# tail -f /var/log/messagesOct 19 16:50:47 master Keepalived_vrrp[1601]: Netlink reflector reports IP 192.168.1.107 addedOct 19 16:50:47 master Keepalived_vrrp[1601]: Netlink reflector reports IP fe80::20c:29ff:fe5c:9b35 addedOct 19 16:50:47 master Keepalived_vrrp[1601]: Registering Kernel netlink reflectorOct 19 16:50:47 master Keepalived_vrrp[1601]: Registering Kernel netlink command channelOct 19 16:50:47 master Keepalived_vrrp[1601]: Registering gratuitous ARP shared channelOct 19 16:50:47 master Keepalived_healthcheckers[1600]: Interface queue is emptyOct 19 16:50:47 master Keepalived_healthcheckers[1600]: Netlink reflector reports IP 192.168.1.107 addedOct 19 16:50:47 master Keepalived_healthcheckers[1600]: Netlink reflector reports IP fe80::20c:29ff:fe5c:9b35 addedOct 19 16:50:47 master Keepalived_healthcheckers[1600]: Registering Kernel netlink reflectorOct 19 16:50:47 master Keepalived_healthcheckers[1600]: Registering Kernel netlink command channelOct 19 16:51:08 master Keepalived_vrrp[1601]: Opening file '/etc/keepalived/keepalived.conf'.Oct 19 16:51:08 master Keepalived_vrrp[1601]: Configuration is using : 61515 BytesOct 19 16:51:08 master Keepalived_vrrp[1601]: Using LinkWatch kernel netlink reflector...Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Opening file '/etc/keepalived/keepalived.conf'.Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Configuration is using : 12492 BytesOct 19 16:51:08 master Keepalived_healthcheckers[1600]: Using LinkWatch kernel netlink reflector...Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Activating healthchecker for service [192.168.1.109]:80 监控两台主机Oct 19 16:51:08 master Keepalived_healthcheckers[1600]: Activating healthchecker for service [192.168.1.110]:80Oct 19 16:51:08 master Keepalived_vrrp[1601]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]Oct 19 16:51:11 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Transition to MASTER STATEOct 19 16:51:14 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Entering MASTER STATEOct 19 16:51:14 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) setting protocol VIPs.Oct 19 16:51:14 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113Oct 19 16:51:14 master Keepalived_healthcheckers[1600]: Netlink reflector reports IP 192.168.1.113 addedOct 19 16:51:19 master Keepalived_vrrp[1601]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113[root@backup ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ][root@backup ~]# tail -f /var/log/messagesOct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Netlink reflector reports IP 192.168.1.108 addedOct 19 16:54:08 backup Keepalived_vrrp[1618]: Interface queue is emptyOct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Netlink reflector reports IP fe80::20c:29ff:fe8b:9597 addedOct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Registering Kernel netlink reflectorOct 19 16:54:08 backup Keepalived_healthcheckers[1617]: Registering Kernel netlink command channelOct 19 16:54:08 backup Keepalived_vrrp[1618]: Netlink reflector reports IP 192.168.1.108 addedOct 19 16:54:08 backup Keepalived_vrrp[1618]: Netlink reflector reports IP fe80::20c:29ff:fe8b:9597 addedOct 19 16:54:08 backup Keepalived_vrrp[1618]: Registering Kernel netlink reflectorOct 19 16:54:08 backup Keepalived_vrrp[1618]: Registering Kernel netlink command channelOct 19 16:54:08 backup Keepalived_vrrp[1618]: Registering gratuitous ARP shared channelOct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Opening file '/etc/keepalived/keepalived.conf'.Oct 19 16:54:28 backup Keepalived_vrrp[1618]: Opening file '/etc/keepalived/keepalived.conf'.Oct 19 16:54:28 backup Keepalived_vrrp[1618]: Configuration is using : 61515 BytesOct 19 16:54:28 backup Keepalived_vrrp[1618]: Using LinkWatch kernel netlink reflector...Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Configuration is using : 12492 BytesOct 19 16:54:28 backup Keepalived_vrrp[1618]: VRRP_Instance(b1_one) Entering BACKUP STATEOct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Using LinkWatch kernel netlink reflector...Oct 19 16:54:28 backup Keepalived_vrrp[1618]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Activating healthchecker for service [192.168.1.109]:80Oct 19 16:54:28 backup Keepalived_healthcheckers[1617]: Activating healthchecker for service [192.168.1.110]:80# PS:这里说的LVS.并不是用ipvsadm软件配置.而是直接用keepalived的virtual_server配置项控制的.
# 安装ipvsadm只是可以看到负载状况.其实只需要keepalived也可以实现负载均衡集群.
分别在主备服务器上查看VIP：
[root@master ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:5c:9b:35 brd ff:ff:ff:ff:ff:ff inet 192.168.1.107/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.113/24 scope global secondary eth0 inet6 fe80::20c:29ff:fe5c:9b35/64 scope link valid_lft forever preferred_lft forever[root@backup ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:8b:95:97 brd ff:ff:ff:ff:ff:ff inet 192.168.1.108/24 brd 192.168.1.255 scope global eth0 inet6 fe80::20c:29ff:fe8b:9597/64 scope link valid_lft forever preferred_lft foreverVIP目前在master上
六、在web1和web2主机上配置LVS-DR的相关配置
1、[root@realserver1 ~]# vim /etc/init.d/lvsDR_node
---------------------------------------------------------------------------------------------------
#! /bin/sh
# descript : start real server DR
VIP=192.168.1.113
. /etc/rc.d/init.d/functions
case "$1" in
start)
echo "start LVS of RealServer DR"
/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
;;
stop)
/sbin/ifconfig lo:0 down
echo "close LVS of RealServer DR"
echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce
;;
*)
echo "Usage : $0 {start|stop}"
exit 1
esac
---------------------------------------------------------------------------------------------------
[root@realserver1 ~]# chmod +x /etc/init.d/lvsDR_node
将启动脚本加入到 /etc/rc.local 开机启动
/etc/init.d/lvsDR_node start
七、web1和web2主机上开启httpd服务
[root@realserver1 ~]# netstat -ntpl | grep httpd
tcp 0 0 :::80 :::* LISTEN 1228/httpd[root@realserver2 ~]# netstat -ntpl | grep httpd
tcp 0 0 :::80 :::* LISTEN 1238/httpd八、测试是否可以负载均衡
编写测试脚本
#! /bin/sh
for((i=1;i<=100;i++));docurl http://192.168.1.113:80 >> /tmp/q;done结果：
[root@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.1.113:80 rr -> 192.168.1.109:80 Route 1 0 100 -> 192.168.1.110:80 Route 1 0 100九、测试断开一个web服务或加入一个web服务
现在关闭web2主机（192.168.1.110）的web服务
查看日志可以看到：
Oct 19 20:39:59 master Keepalived_healthcheckers[1796]: TCP connection to [192.168.1.110]:80 failed !!!
Oct 19 20:39:59 master Keepalived_healthcheckers[1796]: Removing service [192.168.1.110]:80 from VS [192.168.1.113]:80
[root@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.1.113:80 rr -> 192.168.1.109:80 Route 1 0 0 ---只有1.109现在再把web2主机（192.168.1.110）的web服务加进来：
查看日志可以看到：
Oct 19 20:42:51 master Keepalived_healthcheckers[1796]: TCP connection to [192.168.1.110]:80 success.
Oct 19 20:42:51 master Keepalived_healthcheckers[1796]: Adding service [192.168.1.110]:80 to VS [192.168.1.113]:80
[root@master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 192.168.1.113:80 rr -> 192.168.1.109:80 Route 1 0 0 -> 192.168.1.110:80 Route 1 0 0十、测试HA
现在模拟主（master）的keepalived主机死掉
备机的日志文件
Oct 19 20:45:33 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Transition to MASTER STATE
Oct 19 20:45:36 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Entering MASTER STATE
Oct 19 20:45:36 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) setting protocol VIPs.
Oct 19 20:45:36 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113
Oct 19 20:45:36 backup Keepalived_healthcheckers[1795]: Netlink reflector reports IP 192.168.1.113 added
Oct 19 20:45:41 backup Keepalived_vrrp[1796]: VRRP_Instance(b1_one) Sending gratuitous ARPs on eth0 for 192.168.1.113
可以看到备机的keepalived已经开始工作.......
整个部署就完成了