mysql 高可用架构 mha 之二 master_ip_failover详解数据库

os:centos 7.4
mysql: 5.7
mha: 0.58

上一篇blog介绍了 mha 的安装,这一篇blog介绍下 master_ip_failover_script 的设置。

ip 规划如下:

192.168.56.101 node1 (mha manager)

192.168.56.102 node2 (mysql master)
192.168.56.103 node3 (mysql master candicate)
192.168.56.104 node4 (mysql slave)

追加 mysql master 的 vip 192.168.56.100/24

master_ip_failover参数

mha manager 节点上

# vi /etc/masterha/app1.cnf  
 
#自动failover时候的切换脚本 
master_ip_failover_script= /usr/local/bin/master_ip_failover 

master_ip_failover_script 指的是mysql master 失败时执行的切换脚本。
没有使用 keepalived ,通过脚本的方式管理vip。

# cp /usr/local/bin/master_ip_failover /usr/local/bin/master_ip_failover.bak 
# vi /usr/local/bin/master_ip_failover 
 
#!/usr/bin/env perl 
use strict; 
use warnings FATAL => 'all'; 
 
use Getopt::Long; 
 
my ( 
    $command,          $ssh_user,        $orig_master_host, $orig_master_ip, 
    $orig_master_port, $new_master_host, $new_master_ip,    $new_master_port 
); 
 
my $vip = '192.168.56.100'; 
my $brdc = '192.168.56.255'; 
my $ifdev = 'enp0s8'; 
my $key = '1'; 
my $ssh_start_vip = "/usr/sbin/ip addr add $vip/24 brd $brdc dev $ifdev label $ifdev:$key;/usr/sbin/arping -q -A -c 1 -I $ifdev $vip;iptables -F;"; 
my $ssh_stop_vip = "/usr/sbin/ip addr del $vip/24 dev $ifdev label $ifdev:$key"; 
 
GetOptions( 
    'command=s'          => /$command, 
    'ssh_user=s'         => /$ssh_user, 
    'orig_master_host=s' => /$orig_master_host, 
    'orig_master_ip=s'   => /$orig_master_ip, 
    'orig_master_port=i' => /$orig_master_port, 
    'new_master_host=s'  => /$new_master_host, 
    'new_master_ip=s'    => /$new_master_ip, 
    'new_master_port=i'  => /$new_master_port, 
); 
 
exit &main(); 
 
sub main { 
 
    print "/n/nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===/n/n"; 
 
    if ( $command eq "stop" || $command eq "stopssh" ) { 
 
        my $exit_code = 1; 
        eval { 
            print "Disabling the VIP on old master: $orig_master_host /n"; 
            &stop_vip(); 
            $exit_code = 0; 
        }; 
        if ($@) { 
            warn "Got Error: $@/n"; 
            exit $exit_code; 
        } 
        exit $exit_code; 
    } 
    elsif ( $command eq "start" ) { 
 
        my $exit_code = 10; 
        eval { 
            print "Enabling the VIP - $vip on the new master - $new_master_host /n"; 
            &start_vip(); 
            $exit_code = 0; 
        }; 
        if ($@) { 
            warn $@; 
            exit $exit_code; 
        } 
        exit $exit_code; 
    } 
    elsif ( $command eq "status" ) { 
        print "Checking the Status of the script.. OK /n"; 
        exit 0; 
    } 
    else { 
        &usage(); 
        exit 1; 
    } 
} 
sub start_vip() { 
    `ssh $ssh_user/@$new_master_host /" $ssh_start_vip /"`; 
} 
# A simple system call that disable the VIP on the old_master 
sub stop_vip() { 
    `ssh $ssh_user/@$orig_master_host /" $ssh_stop_vip /"`; 
} 
 
sub usage { 
    print 
    "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port/n"; 
}

更换ip后,一定要执行下 arping

检查复制环境ssh

# masterha_check_ssh --conf=/etc/masterha/app1.cnf 

检查整个复制环境

# masterha_check_repl --conf=/etc/masterha/app1.cnf 
 
Tue Aug  7 03:44:13 2018 - [info] Checking master_ip_failover_script status: 
Tue Aug  7 03:44:13 2018 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.56.102 --orig_master_ip=192.168.56.102 --orig_master_port=3306  
 
 
IN SCRIPT TEST====/usr/sbin/ip addr del 192.168.56.100/24 dev enp0s8 label enp0s8:1==/usr/sbin/ip addr add 192.168.56.100/24 brd 192.168.56.255 dev enp0s8 label enp0s8:1;/usr/sbin/arping -q -A -c 1 -I enp0s8 192.168.56.100;=== 
 
Checking the Status of the script.. OK  
Tue Aug  7 03:44:13 2018 - [info]  OK. 

启动 mha manager

启动 manager

# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1-manager.log 2>&1 &  

查看 manager status

# masterha_check_status --conf=/etc/masterha/app1.cnf

查看 manager log

# tail -n 1000 -f /var/log/masterha/app1-manager.log

验证 failover

node2上 kill 掉 mysqld 进程

# ps -ef|grep -i mysql 
mysql     3114     1  0 Aug06 ?        00:00:51 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid 
root     15551 10466  0 Aug06 pts/1    00:00:00 mysql 
root     25521 21213  0 03:52 pts/2    00:00:00 grep --color=auto -i mysql 
# kill -9 3114

观察 mha manager 之前打开的日志输出

# tail -n 1000 -f /var/log/masterha/app1-manager.log 
 
Tue Aug  7 03:53:08 2018 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query) 
Tue Aug  7 03:53:08 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin 
Tue Aug  7 03:53:08 2018 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s node3 -s node2  --user=root  --master_host=192.168.56.102  --master_ip=192.168.56.102  --master_port=3306 --master_user=mha_mon --master_password=2wsx3edc --ping_type=SELECT 
Monitoring server node3 is reachable, Master is not reachable from node3. OK. 
Tue Aug  7 03:53:09 2018 - [info] HealthCheck: SSH to 192.168.56.102 is reachable. 
Monitoring server node2 is reachable, Master is not reachable from node2. OK. 
Tue Aug  7 03:53:09 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start. 
Tue Aug  7 03:53:13 2018 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

居然把node2上的mysqld 重新拉起来了。

node2 虚拟机掉电

虚拟机掉电验证的更直接。

# tail -n 1000 -f /var/log/masterha/app1-manager.log 
 
Tue Aug  7 03:58:29 2018 - [warning] Got timeout on MySQL Ping(SELECT) child process and killed it! at /usr/local/share/perl5/MHA/HealthCheck.pm line 432. 
Tue Aug  7 03:58:29 2018 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s node3 -s node2  --user=root  --master_host=192.168.56.102  --master_ip=192.168.56.102  --master_port=3306 --master_user=mha_mon --master_password=2wsx3edc --ping_type=SELECT 
Tue Aug  7 03:58:29 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin 
Tue Aug  7 03:58:34 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.102' (110)) 
Tue Aug  7 03:58:34 2018 - [warning] Connection failed 2 time(s).. 
Tue Aug  7 03:58:34 2018 - [warning] HealthCheck: Got timeout on checking SSH connection to 192.168.56.102! at /usr/local/share/perl5/MHA/HealthCheck.pm line 343. 
Monitoring server node3 is reachable, Master is not reachable from node3. OK. 
Tue Aug  7 03:58:39 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.102' (110)) 
Tue Aug  7 03:58:39 2018 - [warning] Connection failed 3 time(s).. 
ssh: connect to host node2 port 22: Connection timed out 
Monitoring server node2 is NOT reachable! 
Tue Aug  7 03:58:39 2018 - [warning] At least one of monitoring servers is not reachable from this script. This is likely a network problem. Failover should not happen.

居然输出 This is likely a network problem. Failover should not happen 。本来是测试 failover的,结果 mha 认为 Failover should not happen。

难道node2宕机,node3、node4存活,mha还认为是网络问题,不进行切换?是哪里设置出了问题吗?

再把 node2 加电起来,观察 mha manager 日志

# tail -n 1000 -f /var/log/masterha/app1-manager.log 
 
Tue Aug  7 04:06:44 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.102' (111)) 
Tue Aug  7 04:06:44 2018 - [warning] Connection failed 1 time(s).. 
Tue Aug  7 04:06:44 2018 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s node3 -s node2  --user=root  --master_host=192.168.56.102  --master_ip=192.168.56.102  --master_port=3306 --master_user=mha_mon --master_password=2wsx3edc --ping_type=SELECT 
Tue Aug  7 04:06:44 2018 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin 
Monitoring server node3 is reachable, Master is not reachable from node3. OK. 
Monitoring server node2 is reachable, Master is not reachable from node2. OK. 
Tue Aug  7 04:06:47 2018 - [info] Master is not reachable from all other monitoring servers. Failover should start. 
Tue Aug  7 04:06:47 2018 - [info] HealthCheck: SSH to 192.168.56.102 is reachable. 
··· 
··· 
··· 
Tue Aug  7 04:07:02 2018 - [info] New master is 192.168.56.103(192.168.56.103:3306) 
Tue Aug  7 04:07:02 2018 - [info] Starting master failover.. 
Tue Aug  7 04:07:02 2018 - [info]  
From: 
192.168.56.102(192.168.56.102:3306) (current master) 
 +--192.168.56.103(192.168.56.103:3306) 
 +--192.168.56.104(192.168.56.104:3306) 
 
To: 
192.168.56.103(192.168.56.103:3306) (new master) 
 +--192.168.56.104(192.168.56.104:3306) 
··· 
··· 
··· 
----- Failover Report ----- 
 
app1: MySQL Master failover 192.168.56.102(192.168.56.102:3306) to 192.168.56.103(192.168.56.103:3306) succeeded 
 
Master 192.168.56.102(192.168.56.102:3306) is down! 
 
Check MHA Manager logs at node1:/var/log/masterha/app1-manager.log for details. 
 
Started automated(non-interactive) failover. 
Invalidated master IP address on 192.168.56.102(192.168.56.102:3306) 
The latest slave 192.168.56.103(192.168.56.103:3306) has all relay logs for recovery. 
Selected 192.168.56.103(192.168.56.103:3306) as a new master. 
192.168.56.103(192.168.56.103:3306): OK: Applying all logs succeeded. 
192.168.56.103(192.168.56.103:3306): OK: Activated master IP address. 
192.168.56.104(192.168.56.104:3306): This host has the latest relay log events. 
Generating relay diff files from the latest slave succeeded. 
192.168.56.104(192.168.56.104:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.56.103(192.168.56.103:3306) 
192.168.56.103(192.168.56.103:3306): Resetting slave info succeeded. 
Master failover to 192.168.56.103(192.168.56.103:3306) completed successfully. 

把node1 起来之后,提示出现 Failover should start

查看 node3 的ip

# ifconfig  enp0s8:1 
enp0s8:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500 
        inet 192.168.56.100  netmask 255.255.255.0  broadcast 192.168.56.255 
        ether 08:00:27:e8:24:31  txqueuelen 1000  (Ethernet) 

查看 node4 的 mysql slave status,已经发生了变化。

mysql> show slave status/G 
*************************** 1. row *************************** 
               Slave_IO_State: Waiting for master to send event 
                  Master_Host: 192.168.56.103 
                  Master_User: replicator 
                  Master_Port: 3306 
                Connect_Retry: 60 
              Master_Log_File: mysql-bin.000006 
          Read_Master_Log_Pos: 947 
               Relay_Log_File: node4-relay-bin.000002 
                Relay_Log_Pos: 320 
        Relay_Master_Log_File: mysql-bin.000006 
             Slave_IO_Running: Yes 
            Slave_SQL_Running: Yes 

关闭 mha manager

关闭 manager

# masterha_stop --conf=/etc/masterha/app1.cnf 
Stopped app1 successfully.

查看 manager log

# tail -n 1000 -f /var/log/masterha/app1-manager.log

vip 的一些操作

# ip addr add 192.168.56.100/24 brd 192.168.56.255 dev enp0s8 label enp0s8:1 
# /usr/sbin/arping -q -A -c 1 -I enp0s8 192.168.56.100 
# ip addr del 192.168.56.100/24 dev enp0s8 label enp0s8:1

slave 的设置

mysql> change master to 
  master_host='192.168.56.102', 
  master_user='replicator', 
  master_password='2wsx3edc', 
  master_port=3306, 
  master_log_file='mysql-bin.000007', 
  master_log_pos=154; 
mysql> start slave; 

参考:
IT虾米网
IT虾米网

IT虾米网
IT虾米网

IT虾米网
IT虾米网

failover的过程,基本为以下步骤:

1.配置文件检查阶段,这个阶段会检查整个集群配置文件配置
2.宕机的master处理,这个阶段包括虚拟ip摘除操作,主机关机操作
3.复制dead maste和最新slave相差的relay log,并保存到MHA Manger具体的目录下
4.识别含有最新更新的slave
5.应用从master保存的二进制日志事件(binlog events)
6.提升一个slave为新的master进行复制
7.使其他的slave连接新的master进行复制

下面是完整日志输出

Tue Aug  7 04:06:59 2018 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.56.102' (111)) 
Tue Aug  7 04:06:59 2018 - [warning] Connection failed 4 time(s).. 
Tue Aug  7 04:06:59 2018 - [warning] Master is not reachable from health checker! 
Tue Aug  7 04:06:59 2018 - [warning] Master 192.168.56.102(192.168.56.102:3306) is not reachable! 
Tue Aug  7 04:06:59 2018 - [warning] SSH is reachable. 
Tue Aug  7 04:06:59 2018 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status.. 
Tue Aug  7 04:06:59 2018 - [info] Reading default configuration from /etc/masterha_default.cnf.. 
Tue Aug  7 04:06:59 2018 - [info] Reading application default configuration from /etc/masterha/app1.cnf.. 
Tue Aug  7 04:06:59 2018 - [info] Reading server configuration from /etc/masterha/app1.cnf.. 
Tue Aug  7 04:07:00 2018 - [info] GTID failover mode = 0 
Tue Aug  7 04:07:00 2018 - [info] Dead Servers: 
Tue Aug  7 04:07:00 2018 - [info]   192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:00 2018 - [info] Alive Servers: 
Tue Aug  7 04:07:00 2018 - [info]   192.168.56.103(192.168.56.103:3306) 
Tue Aug  7 04:07:00 2018 - [info]   192.168.56.104(192.168.56.104:3306) 
Tue Aug  7 04:07:00 2018 - [info] Alive Slaves: 
Tue Aug  7 04:07:00 2018 - [info]   192.168.56.103(192.168.56.103:3306)  Version=5.7.22-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:00 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:00 2018 - [info]   192.168.56.104(192.168.56.104:3306)  Version=5.7.23-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:00 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:00 2018 - [info] Checking slave configurations.. 
Tue Aug  7 04:07:00 2018 - [info] Checking replication filtering settings.. 
Tue Aug  7 04:07:00 2018 - [info]  Replication filtering check ok. 
Tue Aug  7 04:07:00 2018 - [info] Master is down! 
Tue Aug  7 04:07:00 2018 - [info] Terminating monitoring script. 
Tue Aug  7 04:07:00 2018 - [info] Got exit code 20 (Master dead). 
Tue Aug  7 04:07:00 2018 - [info] MHA::MasterFailover version 0.58. 
Tue Aug  7 04:07:00 2018 - [info] Starting master failover. 
Tue Aug  7 04:07:00 2018 - [info]  
Tue Aug  7 04:07:00 2018 - [info] * Phase 1: Configuration Check Phase.. 
Tue Aug  7 04:07:00 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] GTID failover mode = 0 
Tue Aug  7 04:07:01 2018 - [info] Dead Servers: 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info] Checking master reachability via MySQL(double check)... 
Tue Aug  7 04:07:01 2018 - [info]  ok. 
Tue Aug  7 04:07:01 2018 - [info] Alive Servers: 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.103(192.168.56.103:3306) 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.104(192.168.56.104:3306) 
Tue Aug  7 04:07:01 2018 - [info] Alive Slaves: 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.103(192.168.56.103:3306)  Version=5.7.22-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:01 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.104(192.168.56.104:3306)  Version=5.7.23-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:01 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info] Starting Non-GTID based failover. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] ** Phase 1: Configuration Check Phase completed. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] * Phase 2: Dead Master Shutdown Phase.. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] Forcing shutdown so that applications never connect to the current master.. 
Tue Aug  7 04:07:01 2018 - [info] Executing master IP deactivation script: 
Tue Aug  7 04:07:01 2018 - [info]   /usr/local/bin/master_ip_failover --orig_master_host=192.168.56.102 --orig_master_ip=192.168.56.102 --orig_master_port=3306 --command=stopssh --ssh_user=root   
 
 
IN SCRIPT TEST====/usr/sbin/ip addr del 192.168.56.100/24 dev enp0s8 label enp0s8:1==/usr/sbin/ip addr add 192.168.56.100/24 brd 192.168.56.255 dev enp0s8 label enp0s8:1;/usr/sbin/arping -q -A -c 1 -I enp0s8 192.168.56.100;=== 
 
Disabling the VIP on old master: 192.168.56.102  
RTNETLINK answers: Cannot assign requested address 
Tue Aug  7 04:07:01 2018 - [info]  done. 
Tue Aug  7 04:07:01 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master. 
Tue Aug  7 04:07:01 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] * Phase 3: Master Recovery Phase.. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase.. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] The latest binary log file/position on all slaves is mysql-bin.000006:154 
Tue Aug  7 04:07:01 2018 - [info] Latest slaves (Slaves that received relay log files to the latest): 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.103(192.168.56.103:3306)  Version=5.7.22-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:01 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.104(192.168.56.104:3306)  Version=5.7.23-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:01 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info] The oldest binary log file/position on all slaves is mysql-bin.000006:154 
Tue Aug  7 04:07:01 2018 - [info] Oldest slaves: 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.103(192.168.56.103:3306)  Version=5.7.22-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:01 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info]   192.168.56.104(192.168.56.104:3306)  Version=5.7.23-log (oldest major version between slaves) log-bin:enabled 
Tue Aug  7 04:07:01 2018 - [info]     Replicating from 192.168.56.102(192.168.56.102:3306) 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:01 2018 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase.. 
Tue Aug  7 04:07:01 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] Fetching dead master's binary logs.. 
Tue Aug  7 04:07:02 2018 - [info] Executing command on the dead master 192.168.56.102(192.168.56.102:3306): save_binary_logs --command=save --start_file=mysql-bin.000006  --start_pos=154 --binlog_dir=/var/lib/mysql --output_file=/tmp/saved_master_binlog_from_192.168.56.102_3306_20180807040700.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58 
  Creating /tmp if not exists..    ok. 
 Concat binary/relay logs from mysql-bin.000006 pos 154 to mysql-bin.000006 EOF into /tmp/saved_master_binlog_from_192.168.56.102_3306_20180807040700.binlog .. 
 Binlog Checksum enabled 
  Dumping binlog format description event, from position 0 to 154.. ok. 
  No need to dump effective binlog data from /var/lib/mysql/mysql-bin.000006 (pos starts 154, filesize 154). Skipping. 
 Binlog Checksum enabled 
 /tmp/saved_master_binlog_from_192.168.56.102_3306_20180807040700.binlog has no effective data events. 
Event not exists. 
Tue Aug  7 04:07:02 2018 - [info] Additional events were not found from the orig master. No need to save. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] * Phase 3.3: Determining New Master Phase.. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves.. 
Tue Aug  7 04:07:02 2018 - [info] All slaves received relay logs to the same position. No need to resync each other. 
Tue Aug  7 04:07:02 2018 - [info] Searching new master from slaves.. 
Tue Aug  7 04:07:02 2018 - [info]  Candidate masters from the configuration file: 
Tue Aug  7 04:07:02 2018 - [info]  Non-candidate masters: 
Tue Aug  7 04:07:02 2018 - [info] New master is 192.168.56.103(192.168.56.103:3306) 
Tue Aug  7 04:07:02 2018 - [info] Starting master failover.. 
Tue Aug  7 04:07:02 2018 - [info]  
From: 
192.168.56.102(192.168.56.102:3306) (current master) 
 +--192.168.56.103(192.168.56.103:3306) 
 +--192.168.56.104(192.168.56.104:3306) 
 
To: 
192.168.56.103(192.168.56.103:3306) (new master) 
 +--192.168.56.104(192.168.56.104:3306) 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] * Phase 3.4: New Master Diff Log Generation Phase.. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info]  This server has all relay logs. No need to generate diff files from the latest slave. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] * Phase 3.5: Master Log Apply Phase.. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed. 
Tue Aug  7 04:07:02 2018 - [info] Starting recovery on 192.168.56.103(192.168.56.103:3306).. 
Tue Aug  7 04:07:02 2018 - [info]  This server has all relay logs. Waiting all logs to be applied..  
Tue Aug  7 04:07:02 2018 - [info]   done. 
Tue Aug  7 04:07:02 2018 - [info]  All relay logs were successfully applied. 
Tue Aug  7 04:07:02 2018 - [info] Getting new master's binlog name and position.. 
Tue Aug  7 04:07:02 2018 - [info]  mysql-bin.000006:947 
Tue Aug  7 04:07:02 2018 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.56.103', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000006', MASTER_LOG_POS=947, MASTER_USER='replicator', MASTER_PASSWORD='xxx'; 
Tue Aug  7 04:07:02 2018 - [info] Executing master IP activate script: 
Tue Aug  7 04:07:02 2018 - [info]   /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.56.102 --orig_master_ip=192.168.56.102 --orig_master_port=3306 --new_master_host=192.168.56.103 --new_master_ip=192.168.56.103 --new_master_port=3306 --new_master_user='mha_mon'   --new_master_password=xxx 
Unknown option: new_master_user 
Unknown option: new_master_password 
 
 
IN SCRIPT TEST====/usr/sbin/ip addr del 192.168.56.100/24 dev enp0s8 label enp0s8:1==/usr/sbin/ip addr add 192.168.56.100/24 brd 192.168.56.255 dev enp0s8 label enp0s8:1;/usr/sbin/arping -q -A -c 1 -I enp0s8 192.168.56.100;=== 
 
Enabling the VIP - 192.168.56.100 on the new master - 192.168.56.103  
Tue Aug  7 04:07:02 2018 - [info]  OK. 
Tue Aug  7 04:07:02 2018 - [info] Setting read_only=0 on 192.168.56.103(192.168.56.103:3306).. 
Tue Aug  7 04:07:02 2018 - [info]  ok. 
Tue Aug  7 04:07:02 2018 - [info] ** Finished master recovery successfully. 
Tue Aug  7 04:07:02 2018 - [info] * Phase 3: Master Recovery Phase completed. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] * Phase 4: Slaves Recovery Phase.. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. 
Tue Aug  7 04:07:02 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info] -- Slave diff file generation on host 192.168.56.104(192.168.56.104:3306) started, pid: 24628. Check tmp log /etc/masterha/app1/192.168.56.104_3306_20180807040700.log if it takes time.. 
Tue Aug  7 04:07:03 2018 - [info]  
Tue Aug  7 04:07:03 2018 - [info] Log messages from 192.168.56.104 ... 
Tue Aug  7 04:07:03 2018 - [info]  
Tue Aug  7 04:07:02 2018 - [info]  This server has all relay logs. No need to generate diff files from the latest slave. 
Tue Aug  7 04:07:03 2018 - [info] End of log messages from 192.168.56.104. 
Tue Aug  7 04:07:03 2018 - [info] -- 192.168.56.104(192.168.56.104:3306) has the latest relay log events. 
Tue Aug  7 04:07:03 2018 - [info] Generating relay diff files from the latest slave succeeded. 
Tue Aug  7 04:07:03 2018 - [info]  
Tue Aug  7 04:07:03 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. 
Tue Aug  7 04:07:03 2018 - [info]  
Tue Aug  7 04:07:03 2018 - [info] -- Slave recovery on host 192.168.56.104(192.168.56.104:3306) started, pid: 24630. Check tmp log /etc/masterha/app1/192.168.56.104_3306_20180807040700.log if it takes time.. 
Tue Aug  7 04:07:04 2018 - [info]  
Tue Aug  7 04:07:04 2018 - [info] Log messages from 192.168.56.104 ... 
Tue Aug  7 04:07:04 2018 - [info]  
Tue Aug  7 04:07:03 2018 - [info] Starting recovery on 192.168.56.104(192.168.56.104:3306).. 
Tue Aug  7 04:07:03 2018 - [info]  This server has all relay logs. Waiting all logs to be applied..  
Tue Aug  7 04:07:03 2018 - [info]   done. 
Tue Aug  7 04:07:03 2018 - [info]  All relay logs were successfully applied. 
Tue Aug  7 04:07:03 2018 - [info]  Resetting slave 192.168.56.104(192.168.56.104:3306) and starting replication from the new master 192.168.56.103(192.168.56.103:3306).. 
Tue Aug  7 04:07:03 2018 - [info]  Executed CHANGE MASTER. 
Tue Aug  7 04:07:03 2018 - [info]  Slave started. 
Tue Aug  7 04:07:04 2018 - [info] End of log messages from 192.168.56.104. 
Tue Aug  7 04:07:04 2018 - [info] -- Slave recovery on host 192.168.56.104(192.168.56.104:3306) succeeded. 
Tue Aug  7 04:07:04 2018 - [info] All new slave servers recovered successfully. 
Tue Aug  7 04:07:04 2018 - [info]  
Tue Aug  7 04:07:04 2018 - [info] * Phase 5: New master cleanup phase.. 
Tue Aug  7 04:07:04 2018 - [info]  
Tue Aug  7 04:07:04 2018 - [info] Resetting slave info on the new master.. 
Tue Aug  7 04:07:04 2018 - [info]  192.168.56.103: Resetting slave info succeeded. 
Tue Aug  7 04:07:04 2018 - [info] Master failover to 192.168.56.103(192.168.56.103:3306) completed successfully. 
Tue Aug  7 04:07:04 2018 - [info] Deleted server1 entry from /etc/masterha/app1.cnf . 
Tue Aug  7 04:07:04 2018 - [info]  
 
----- Failover Report ----- 
 
app1: MySQL Master failover 192.168.56.102(192.168.56.102:3306) to 192.168.56.103(192.168.56.103:3306) succeeded 
 
Master 192.168.56.102(192.168.56.102:3306) is down! 
 
Check MHA Manager logs at node1:/var/log/masterha/app1-manager.log for details. 
 
Started automated(non-interactive) failover. 
Invalidated master IP address on 192.168.56.102(192.168.56.102:3306) 
The latest slave 192.168.56.103(192.168.56.103:3306) has all relay logs for recovery. 
Selected 192.168.56.103(192.168.56.103:3306) as a new master. 
192.168.56.103(192.168.56.103:3306): OK: Applying all logs succeeded. 
192.168.56.103(192.168.56.103:3306): OK: Activated master IP address. 
192.168.56.104(192.168.56.104:3306): This host has the latest relay log events. 
Generating relay diff files from the latest slave succeeded. 
192.168.56.104(192.168.56.104:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.56.103(192.168.56.103:3306) 
192.168.56.103(192.168.56.103:3306): Resetting slave info succeeded. 
Master failover to 192.168.56.103(192.168.56.103:3306) completed successfully. 
 

原创文章,作者:Maggie-Hunter,如若转载,请注明出处:https://blog.ytso.com/3885.html

(0)
上一篇 2021年7月16日
下一篇 2021年7月16日

相关推荐

发表回复

登录后才能评论