MYSQL主从复制部署MHA自动故障转移

发布时间:2020-08-07 07:07:47 作者:lixiaxin200319
来源:ITPUB博客 阅读:313

. 环境描述

操作系统:RHEL 6.4

mysql    5.6.27

 

角色

IP 地址

主机名

Server ID

类型

主库

192.168.222.156

oradb

10

写入

备库、MHA管理服务器

192.168.222.157

mysqldb2

11

读、监控集群组

 

 

二、部署MHA

 

1.主库安装 MHA NODE

1.1 安装需要的PERL模块

 

yum -y install *zlib-dev*

 

 install.sh

#!/bin/bash

wget http://xrl.us/cpanm --no-check-certificate

mv cpanm /usr/bin

chmod 755 /usr/bin/cpanm

cat >/root/list<<EOF

install DBD::mysql

EOF

for package in `cat /root/list`

do

cpanm $package

done

 

 

 




1.2 安装 MHA NODE

https://github.com/yoshinorim/mha4mysql-node 下载最新版本的 mha4mysql-node 源代码(目前是 0.57版)

unzip  mha4mysql-node-master.zip

cd  mha4mysql-node-master

perl  Makefile.PL

make

make  install

 

 

 

 

2. 备库安装 MHA Manager

2.1 安装需要的perl模块

yum -y install *zlib-dev*

 

 install.sh

#!/bin/bash

wget http://xrl.us/cpanm --no-check-certificate

mv cpanm /usr/bin

chmod 755 /usr/bin/cpanm

cat >/root/list<<EOF

install DBD::mysql

install Config::Tiny

install Log::Dispatch

install Parallel::ForkManager

install Time::HiRes

EOF

for package in `cat /root/list`

do

cpanm $package

done

 

 

 

2.2 安装MHA NODE

https://github.com/yoshinorim/mha4mysql-node 下载最新版本的 mha4mysql-node 源代码(目前是 0.57版)

unzip  mha4mysql-node-master.zip

cd  mha4mysql-node-master

perl  Makefile.PL

make

make  install

 

 

 

2.3 安装 MHA Manager

 

https://github.com/yoshinorim/mha4mysql-manager下载最新版本的 mha4mysql-manager 源代码(目前是 0.57版)

 

unzip  mha4mysql-manager-master.zip

cd   mha4mysql-manager-master

perl  Makefile.PL

make

make  install

 

 

 

3. 配置 SSH 登录无密码验证

3.1 主库配置到 MHA manager 节点(备库节点)SSH登录无密码验证

vi  /etc/hosts

192.168.222.157  mysqldb2

192.168.222.156  oradb

 

 

ssh-keygen   -t  rsa

ssh-coyp-id  -i  /root/.ssh/id_rsa.pub  root@192.168.222.157

ssh-coyp-id  -i  /root/.ssh/id_rsa.pub  root@192.168.222.156

 

ssh  root@192.168.222.156

ssh  root@oradb

ssh   root@192.168.222.157

ssh   root@mysqldb2

 

 

 

 

3.2 MHA manager 节点(备库节点)配置到主库配置 SSH登录无密码验证

vi  /etc/hosts

192.168.222.157  mysqldb2

192.168.222.156  oradb

 

 

ssh-keygen   -t  rsa

ssh-coyp-id  -i  /root/.ssh/id_rsa.pub  root@192.168.222.157

ssh-coyp-id  -i  /root/.ssh/id_rsa.pub  root@192.168.222.156

 

ssh  root@192.168.222.156

ssh  root@oradb

ssh   root@192.168.222.157

ssh   root@mysqldb2

 

 

 

4. 搭建主从复制

4.1  修改主库my.cnf 配置服务器ID10

vi  /app/oracle/oradata2/mysql-5.6.27/my.cnf

[mysqld]

server_id = 10

 

 

4.1  修改备库my.cnf 配置

vi  /usr/local/mysql/my.cnf

[mysqld]

server_id = 10

 

# slave config

log_bin=/usr/local/mysql/data/mysql-bin

relay_log=/usr/local/mysql/data/relay-bin

binlog_format=row

relay_log_purge=0

read_only=1

 

 

 

 

4.2  在主库上备份

mysqldump  --master-data=2  --single-transaction  -R  -A > all.sql

 

 

 

其中 --master-data=2代表备份时刻记录 master Binlog 位置和 Position--single-transaction 意思是获取一致性快照, -R 意思是备份相关的存储过程

-A代表备份所有的 schema

 

4.3  在主库上创建复制用户和 MHA监控用户

mysql>  grant  replication  slave  on *.*  to 'repl'@'192.168.222.%'  identified  by  '123456';

mysql>  grant   all  on *.*   to   'root'@'192.168.222.%'    identified  by  '123456';  /*MHA 监控用户*/

 

4.4  查看主库上备份时刻 Binlog 的名称和位置,MASTER_LOG MASTER_LOG_POS

head -n 30 all.sql|grep -i "CHANGE MASTER TO"

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000068', MASTER_LOG_POS=325;

 

 

4.5  将备份复制到备库

scp  all.sql  root@192.168.222.157:/root/olm

 

 

4.6  在备库导入数据

mysql>  source  /root/olm/all.sql

mysql>  flush privileges;

mysql> 

change master to master_host='192.168.222.156',master_user='repl',master_password='123456',

master_port=3306,master_log_file='mysql-bin.000068',master_log_pos=325;

start   slave;

--查看复制状态

show  slave  status  \G

 

 

 

 

5. 配置MHA

5.1  创建 MHA 工作目录,并且创建相关配置文件

mkdir   -p   /etc/masterha

vi   /etc/masterha/app1.cnf

[server default]

manager_log=/masterha/app1/app1.log

manager_workdir=/masterha/app1

master_binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data

master_ip_failover_script=/usr/local/bin/master_ip_failover

master_ip_online_change_script=/usr/local/bin/master_ip_online_change

password=123456

ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repl

secondary_check_script=/usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb --user=root --master_host=oradb --master_ip='192.168.222.156' --master_port=3306

shutdown_script=""

ssh_user=root

user=root

 

[server1]

hostname=192.168.222.156

port=3306

 

[server2]

candidate_master=1

check_repl_delay=0

hostname=192.168.222.157

port=3306

 

 

 

5.2  设置定期清理中继日志的脚本

crontab  -e

0 4 * * * /usr/local/bin/purge_relay_logs  --user=root  --password=123456  --disable_replay_log_purge   --port=3306  --workdir=/usr/local/mysql/data   --disable_relay_log_purge >> /usr/loca/masterha/log/purge_relay_logs.log  2>&1

 

 

5.2  编辑 /etc/bashrc 文件,在文件末尾添加以下内容,避免MHA执行故障切换时无法找到 mysqlbinlog

PATH="$PATH:/usr/local/mysql/bin"

export  PATH

 

 

5.3  检查 SSH 的配置

masterha_check_ssh   --conf=/etc/masterha/app1.cnf

 

5.4  检查MYSQL复制

masterha_check_repl   --conf=/etc/masterha/app1.cnf

 




5.5  启动 MHA Manger 监控

tail -f /masterha/app1/app1.log

 

--在另一个SSH 窗口启动 MHA Manger 监控。--remove_dead_master_conf 该参数代表当发生主从切换后,老的主库的IP将会从配置文件中移除。

--ignore_last_failover  在缺省情况下,如果MHA检测到连续发生宕机,而且两次宕机时间不足8小时的话,则不会进行 Failover,之所以这样限制

--是为了避免 ping-pong效应。该参数代表忽悠上次 MHA 触发切换产生的文件,默认情况下,MHA 发生切换后将会在 /masterha/paa1下产生 app1.

--failover.complete 文件,下次再次切换的时候如果发现目录存在该文件将不允许触发切换,除非在第一次切换后收到 rm  -rf  /masterha/paa1/app1. --failover.complete,出于方便考虑,我们每次启动MHA时会添加 --ignore_last_failover 参数。

 

nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/masterha/app1/manager.log 2>&1 &

 

--查看 MHA Manager 监控是否正常

masterha_check_status  --conf=/etc/masterha/app1.cnf

app1 (pid:4758) is running(0:PING_OK), master:192.168.222.156

--在默认情况下,10秒内状态为10INITIALIZING_MONITOR,当状态转变为0:PING_OK后表明已经开启了到 MASTER 端的监控。

 

 

关闭MHA的命令如下:

masterha_stop   --conf=/etc/masterha/app1.cnf

 


6. 配置 VIP

6.1  在主库下载集群心跳软件 keepalived,并进行安装

访问 http://www.keepalived.org/software/  下载 keepalived-1.2.18.tar.gz

tar  vzxf  keepalived-1.2.18.tar.gz

cd  keepalived-1.2.18

./configure --prefix=/usr/local/keepalived

make

make  install

cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

cd /etc/keepalived/

 

 

vi  keepalived.conf

--下面是添加的配置

[root@oradb ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

 

global_defs {

   notification_email {

     acassen@firewall.loc

     failover@firewall.loc

     sysadmin@firewall.loc

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id MySQL-ha

}

 

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 150

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.222.158/24

    }

}

 

 

service keepalived start

 

--是用 ip  add 命令可以看到 eth0上已经绑定了虚拟IP 192.168.222.158 ,注意使用 ifconfig  是看不到虚拟IP

[root@oradb ~]# ip add

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:53:7e:f0 brd ff:ff:ff:ff:ff:ff

    inet 192.168.222.156/24 brd 192.168.222.255 scope global eth0

    inet 192.168.222.158/24 scope global secondary eth0

    inet6 fe80::20c:29ff:fe53:7ef0/64 scope link

       valid_lft forever preferred_lft forever

3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:53:7e:fa brd ff:ff:ff:ff:ff:ff

    inet 192.168.1.11/24 brd 192.168.1.255 scope global eth2

    inet6 fe80::20c:29ff:fe53:7efa/64 scope link

       valid_lft forever preferred_lft forever

 

 

6.2  在备库下载集群心跳软件 keepalived,并进行安装

访问 http://www.keepalived.org/software/  下载 keepalived-1.2.18.tar.gz

tar  vzxf  keepalived-1.2.18.tar.gz

cd  keepalived-1.2.18

./configure --prefix=/usr/local/keepalived

make

make  install

cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/rc.d/init.d/

cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

cd /etc/keepalived/

 

 

vi  keepalived.conf

--下面是添加的配置

! Configuration File for keepalived

 

global_defs {

   notification_email {

     acassen@firewall.loc

     failover@firewall.loc

     sysadmin@firewall.loc

   }

   notification_email_from Alexandre.Cassen@firewall.loc

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id MySQL-ha

}

 

vrrp_instance VI_1 {

    state BACKUP

    interface eth0

    virtual_router_id 51

    priority 120

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.222.158/24

    }

}

 

service keepalived start

 

 

 

6.2  MHA引入 keepalived

 

MHA监控节点(备库)编辑 master_ip_failover 脚本

vi  /usr/local/bin/master_ip_failover

#!/usr/bin/env perl

 

#  Copyright (C) 2011 DeNA Co.,Ltd.

#

#  This program is free software; you can redistribute it and/or modify

#  it under the terms of the GNU General Public License as published by

#  the Free Software Foundation; either version 2 of the License, or

#  (at your option) any later version.

#

#  This program is distributed in the hope that it will be useful,

#  but WITHOUT ANY WARRANTY; without even the implied warranty of

#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

#  GNU General Public License for more details.

#

#  You should have received a copy of the GNU General Public License

#   along with this program; if not, write to the Free Software

#  Foundation, Inc.,

#  51 Franklin Street, Fifth Floor, Boston, MA  02110-1301  USA

 

## Note: This is a sample script and is not complete. Modify the script based on your environment.

 

use strict;

use warnings FATAL => 'all';

 

use Getopt::Long;

use MHA::DBHelper;

 

my (

  $command,        $ssh_user,         $orig_master_host,

  $orig_master_ip, $orig_master_port, $new_master_host,

  $new_master_ip,  $new_master_port,  $new_master_user,

  $new_master_password,  $cmd

);

 

GetOptions(

  'command=s'             => \$command,

  'ssh_user=s'            => \$ssh_user,

  'orig_master_host=s'    => \$orig_master_host,

  'orig_master_ip=s'      => \$orig_master_ip,

  'orig_master_port=i'    => \$orig_master_port,

  'new_master_host=s'     => \$new_master_host,

  'new_master_ip=s'       => \$new_master_ip,

  'new_master_port=i'     => \$new_master_port,

  'new_master_user=s'     => \$new_master_user,

  'new_master_password=s' => \$new_master_password,

);

 

exit &main();

 

sub main {

  if ( $command eq "stop" || $command eq "stopssh" ) {

 

    # $orig_master_host, $orig_master_ip, $orig_master_port are passed.

    # If you manage master ip address at global catalog database,

    # invalidate orig_master_ip here.

    my $exit_code = 1;

    eval {

 

      # updating global catalog, etc

      $exit_code = 0;

    };

    if ($@) {

      warn "Got Error: $@\n";

      exit $exit_code;

    }

    exit $exit_code;

  }

  elsif ( $command eq "start" ) {

 

    # all arguments are passed.

    # If you manage master ip address at global catalog database,

    # activate new_master_ip here.

    # You can also grant write access (create user, set read_only=0, etc) here.

    my $exit_code = 10;

    eval {

      my $new_master_handler = new MHA::DBHelper();

 

      # args: hostname, port, user, password, raise_error_or_not

      $new_master_handler->connect( $new_master_ip, $new_master_port,

       $new_master_user, $new_master_password, 1 );

 

      ## Set read_only=0 on the new master

      $new_master_handler->disable_log_bin_local();

      print "Set read_only=0 on the new master.\n";

      $new_master_handler->disable_read_only();

 

      ## Creating an app user on the new master

      print "Creating app user on the new master..\n";

      #FIXME_xxx_create_user( $new_master_handler->{dbh} );

      #$new_master_handler->enable_log_bin_local();

      #$new_master_handler->disconnect();

 

      ## Update master ip on the catalog database, etc

      #FIXME_xxx;

      #$cmd = 'ssh root@192.168.222.156 \./lvs-admin stop';

      #$cmd = 'ssh'.$ssh_user.'@'.$orig_master_ip.'\'./lvs-admin stop\'';

      $cmd = 'ssh '.$ssh_user.'@'.$orig_master_ip.'   \./lvs-admin stop';

      system($cmd);

 

      $exit_code = 0;

    };

    if ($@) {

      warn $@;

 

      # If you want to continue failover, exit 10.

      exit $exit_code;

    }

    exit $exit_code;

  }

  elsif ( $command eq "status" ) {

 

    # do nothing

    exit 0;

  }

  else {

    &usage();

    exit 1;

  }

}

 

sub usage {

  print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";

}

 

在脚本中绿色背景的代码是新增的,红色背景的代码是注释的。


在主库添加关闭 keepalived 的脚本

 

vi /root/lvs-admin

 

case "$1" in

      "stop")

      echo;

      echo "stop keepalived ......";

      service keepalived stop

      ;;

      "start")

      echo;

      echo "start keepalive ......";

      service keepalived start

      ;;

      *)

      echo "please input you select!"

      ;;

esac

 

chmod  755  /root/lvs-admin

 

 

 

7. 测试主库故障MHA自动把备库提升为主库并把虚拟IP漂移到新的主库(原来的备库)

 

7.1 在备库停止复制默认有二进制日志没传到备库

 

mysql>   stop  slave;

 

7.2  sysbench 生成测试数据,在主库上的 sbtest 库下创建 sbtest 表,共100万条数据

mysql>   create  database  sbtest;

 

./sysbench --num-threads=4 --test=oltp --oltp-table-size=1000000 --oltp-read-only=off --init-rng=on  --max-requests=0 \

--oltp-dist-type=uniform --max-time=1800 --mysql-user=root --db-driver=test --mysql-table-engine=innodb \

--oltp-test-mode=complex prepare

 

7.3  关闭主库 MHA会自动把备份库提升为新的主库,并把虚拟IP漂移到新的主库(原来的备库)

--关闭主库

mysqladmin  -u root  shutdown

 

 

--MHA监控节点(备库)实时查看MHA日志

tail   -f  /masterha/app1/app1.log

 

Thu Feb  2 23:53:06 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb  2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb  2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb  2 23:53:07 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb  2 23:53:07 2017 - [info] Got exit code 1 (Not master dead).

Thu Feb  2 23:54:15 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb  2 23:54:16 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb  2 23:54:16 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb  2 23:54:16 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb  2 23:54:16 2017 - [info] Got exit code 1 (Not master dead).

Thu Feb  2 23:58:13 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb  2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb  2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb  2 23:58:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb  2 23:58:14 2017 - [info] Got exit code 1 (Not master dead).

Thu Feb  2 23:59:48 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb  2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Thu Feb  2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Thu Feb  2 23:59:49 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb  2 23:59:49 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb  3 00:04:10 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb  3 00:04:11 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb  3 00:04:11 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb  3 00:04:11 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb  3 00:04:11 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb  3 00:09:04 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb  3 00:09:05 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb  3 00:09:05 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb  3 00:09:05 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb  3 00:09:05 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb  3 00:13:13 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb  3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb  3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb  3 00:13:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb  3 00:13:14 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb  3 00:20:21 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb  3 00:20:23 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln671] Master 192.168.222.156:3306 from which slave 192.168.222.157(192.168.222.157:330

6) replicates is not defined in the configuration file!

Fri Feb  3 00:20:23 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/Master

Monitor.pm line 329.

Fri Feb  3 00:20:23 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Fri Feb  3 00:20:23 2017 - [info] Got exit code 1 (Not master dead).

Fri Feb  3 00:21:54 2017 - [info] MHA::MasterMonitor version 0.57.

Fri Feb  3 00:21:55 2017 - [info] GTID failover mode = 0

Fri Feb  3 00:21:55 2017 - [info] Dead Servers:

Fri Feb  3 00:21:55 2017 - [info] Alive Servers:

Fri Feb  3 00:21:55 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:21:55 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Fri Feb  3 00:21:55 2017 - [info] Alive Slaves:

Fri Feb  3 00:21:55 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb  3 00:21:55 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:21:55 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Fri Feb  3 00:21:55 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:21:55 2017 - [info] Checking slave configurations..

Fri Feb  3 00:21:55 2017 - [info] Checking replication filtering settings..

Fri Feb  3 00:21:55 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Fri Feb  3 00:21:55 2017 - [info]  Replication filtering check ok.

Fri Feb  3 00:21:55 2017 - [info] GTID (with auto-pos) is not supported

Fri Feb  3 00:21:55 2017 - [info] Starting SSH connection tests..

Fri Feb  3 00:21:58 2017 - [info] All SSH connection tests passed successfully.

Fri Feb  3 00:21:58 2017 - [info] Checking MHA Node version..

Fri Feb  3 00:21:58 2017 - [info]  Version check ok.

Fri Feb  3 00:21:58 2017 - [info] Checking SSH publickey authentication settings on the current master..

Fri Feb  3 00:21:59 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Fri Feb  3 00:21:59 2017 - [info] Master MHA Node version is 0.57.

Fri Feb  3 00:21:59 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Fri Feb  3 00:21:59 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=

/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000068

Fri Feb  3 00:21:59 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

  Creating /tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000068

Fri Feb  3 00:21:59 2017 - [info] Binlog setting check done.

Fri Feb  3 00:21:59 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Fri Feb  3 00:21:59 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.15

7 --slave_port=3306 --workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info  --relay_dir=/usr/local/mys

ql/data/  --slave_pass=xxx

Fri Feb  3 00:21:59 2017 - [info]   Connecting to root@192.168.222.157(192.168.222.157:22)..

  Checking slave recovery environment settings..

    Opening /usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at /usr/local/mysql/data, up to relay-bin.000002

    Temporary relay log file is /usr/local/mysql/data/relay-bin.000002

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Fri Feb  3 00:22:00 2017 - [info] Slaves settings check done.

Fri Feb  3 00:22:00 2017 - [info]

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

 

Fri Feb  3 00:22:00 2017 - [info] Checking master_ip_failover_script status:

Fri Feb  3 00:22:00 2017 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.

156 --orig_master_port=3306

Fri Feb  3 00:22:00 2017 - [info]  OK.

Fri Feb  3 00:22:00 2017 - [warning] shutdown_script is not defined.

Fri Feb  3 00:22:00 2017 - [info] Set master ping interval 1 seconds.

Fri Feb  3 00:22:00 2017 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb --user=root --master_host=oradb --master_ip=

'192.168.222.156' --master_port=3306

Fri Feb  3 00:22:00 2017 - [info] Starting ping health check on 192.168.222.156(192.168.222.156:3306)..

Fri Feb  3 00:22:00 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

Fri Feb  3 00:25:06 2017 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)

Fri Feb  3 00:25:06 2017 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s mysqldb2 -s oradb --user=root --master_host=orad

b --master_ip='192.168.222.156' --master_port=3306  --user=root  --master_host=192.168.222.156  --master_ip=192.168.222.156  --master_port=3306 --master_user=root --ma

ster_password=123456 --ping_type=SELECT

Fri Feb  3 00:25:06 2017 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --outpu

t_file=/tmp/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin

Fri Feb  3 00:25:07 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Fri Feb  3 00:25:07 2017 - [warning] Connection failed 2 time(s)..

Monitoring server mysqldb2 is reachable, Master is not reachable from mysqldb2. OK.

Fri Feb  3 00:25:08 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Fri Feb  3 00:25:08 2017 - [warning] Connection failed 3 time(s)..

Monitoring server oradb is reachable, Master is not reachable from oradb. OK.

Fri Feb  3 00:25:08 2017 - [info] Master is not reachable from all other monitoring servers. Failover should start.

Fri Feb  3 00:25:08 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Fri Feb  3 00:25:09 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Fri Feb  3 00:25:09 2017 - [warning] Connection failed 4 time(s)..

Fri Feb  3 00:25:09 2017 - [warning] Master is not reachable from health checker!

Fri Feb  3 00:25:09 2017 - [warning] Master 192.168.222.156(192.168.222.156:3306) is not reachable!

Fri Feb  3 00:25:09 2017 - [warning] SSH is reachable.

Fri Feb  3 00:25:09 2017 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and tryi

ng to connect to all servers to check server status..

Fri Feb  3 00:25:09 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Fri Feb  3 00:25:09 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Fri Feb  3 00:25:09 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Fri Feb  3 00:25:10 2017 - [warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

Fri Feb  3 00:25:10 2017 - [info] GTID failover mode = 0

Fri Feb  3 00:25:10 2017 - [info] Dead Servers:

Fri Feb  3 00:25:10 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:10 2017 - [info] Alive Servers:

Fri Feb  3 00:25:10 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Fri Feb  3 00:25:10 2017 - [info] Alive Slaves:

Fri Feb  3 00:25:10 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb  3 00:25:10 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:10 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Fri Feb  3 00:25:10 2017 - [info] Checking slave configurations..

Fri Feb  3 00:25:10 2017 - [info] Checking replication filtering settings..

Fri Feb  3 00:25:10 2017 - [info]  Replication filtering check ok.

Fri Feb  3 00:25:10 2017 - [info] Master is down!

Fri Feb  3 00:25:10 2017 - [info] Terminating monitoring script.

Fri Feb  3 00:25:10 2017 - [info] Got exit code 20 (Master dead).

Fri Feb  3 00:25:10 2017 - [info] MHA::MasterFailover version 0.57.

Fri Feb  3 00:25:10 2017 - [info] Starting master failover.

Fri Feb  3 00:25:10 2017 - [info]

Fri Feb  3 00:25:10 2017 - [info] * Phase 1: Configuration Check Phase..

Fri Feb  3 00:25:10 2017 - [info]

Fri Feb  3 00:25:11 2017 - [warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

Fri Feb  3 00:25:11 2017 - [info] GTID failover mode = 0

Fri Feb  3 00:25:11 2017 - [info] Dead Servers:

Fri Feb  3 00:25:11 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:11 2017 - [info] Checking master reachability via MySQL(double check)...

Fri Feb  3 00:25:11 2017 - [info]  ok.

Fri Feb  3 00:25:11 2017 - [info] Alive Servers:

Fri Feb  3 00:25:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Fri Feb  3 00:25:11 2017 - [info] Alive Slaves:

Fri Feb  3 00:25:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb  3 00:25:11 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:11 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Fri Feb  3 00:25:11 2017 - [info]  Starting SQL thread on 192.168.222.157(192.168.222.157:3306) ..

Fri Feb  3 00:25:11 2017 - [info]   done.

Fri Feb  3 00:25:11 2017 - [info] Starting Non-GTID based failover.

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] ** Phase 1: Configuration Check Phase completed.

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] * Phase 2: Dead Master Shutdown Phase..

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] Forcing shutdown so that applications never connect to the current master..

Fri Feb  3 00:25:11 2017 - [info] Executing master IP deactivation script:

Fri Feb  3 00:25:11 2017 - [info]   /usr/local/bin/master_ip_failover --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306 --com

mand=stopssh --ssh_user=root 

Fri Feb  3 00:25:11 2017 - [info]  done.

Fri Feb  3 00:25:11 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Fri Feb  3 00:25:11 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] * Phase 3: Master Recovery Phase..

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] The latest binary log file/position on all slaves is mysql-bin.000068:530

Fri Feb  3 00:25:11 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):

Fri Feb  3 00:25:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb  3 00:25:11 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:11 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Fri Feb  3 00:25:11 2017 - [info] The oldest binary log file/position on all slaves is mysql-bin.000068:530

Fri Feb  3 00:25:11 2017 - [info] Oldest slaves:

Fri Feb  3 00:25:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb  3 00:25:11 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:11 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:11 2017 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..

Fri Feb  3 00:25:11 2017 - [info]

Fri Feb  3 00:25:12 2017 - [info] Fetching dead master's binary logs..

Fri Feb  3 00:25:12 2017 - [info] Executing command on the dead master 192.168.222.156(192.168.222.156:3306): save_binary_logs --command=save --start_file=mysql-bin.00

0068  --start_pos=530 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog --hand

le_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57

  Creating /tmp if not exists..    ok.

 Concat binary/relay logs from mysql-bin.000068 pos 530 to mysql-bin.000068 EOF into /tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog ..

 Binlog Checksum enabled

  Dumping binlog format description event, from position 0 to 120.. ok.

  Dumping effective binlog data from /app/oracle/oradata2/mysql-5.5.37/data/mysql-bin.000068 position 530 to tail(61261921).. ok.

 Binlog Checksum enabled

 Concat succeeded.

Fri Feb  3 00:25:16 2017 - [info] scp from root@192.168.222.156:/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog to local:/masterha/app1/saved_

master_binlog_from_192.168.222.156_3306_20170203002510.binlog succeeded.

Fri Feb  3 00:25:17 2017 - [info] HealthCheck: SSH to 192.168.222.157 is reachable.

Fri Feb  3 00:25:17 2017 - [info]

Fri Feb  3 00:25:17 2017 - [info] * Phase 3.3: Determining New Master Phase..

Fri Feb  3 00:25:17 2017 - [info]

Fri Feb  3 00:25:17 2017 - [info] Finding the latest slave that has all relay logs for recovering other slaves..

Fri Feb  3 00:25:17 2017 - [info] All slaves received relay logs to the same position. No need to resync each other.

Fri Feb  3 00:25:17 2017 - [info] Searching new master from slaves..

Fri Feb  3 00:25:17 2017 - [info]  Candidate masters from the configuration file:

Fri Feb  3 00:25:17 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Fri Feb  3 00:25:17 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Fri Feb  3 00:25:17 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Fri Feb  3 00:25:17 2017 - [info]  Non-candidate masters:

Fri Feb  3 00:25:17 2017 - [info]  Searching from candidate_master slaves which have received the latest relay log events..

Fri Feb  3 00:25:17 2017 - [info] New master is 192.168.222.157(192.168.222.157:3306)

Fri Feb  3 00:25:17 2017 - [info] Starting master failover..

Fri Feb  3 00:25:17 2017 - [info]

From:

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

 

To:

192.168.222.157(192.168.222.157:3306) (new master)

Fri Feb  3 00:25:17 2017 - [info]

Fri Feb  3 00:25:17 2017 - [info] * Phase 3.3: New Master Diff Log Generation Phase..

Fri Feb  3 00:25:17 2017 - [info]

Fri Feb  3 00:25:17 2017 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.

Fri Feb  3 00:25:17 2017 - [info] Sending binlog..

Fri Feb  3 00:25:20 2017 - [info] scp from local:/masterha/app1/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog to root@192.168.222.157:/tmp/saved_

master_binlog_from_192.168.222.156_3306_20170203002510.binlog succeeded.

Fri Feb  3 00:25:20 2017 - [info]

Fri Feb  3 00:25:20 2017 - [info] * Phase 3.4: Master Log Apply Phase..

Fri Feb  3 00:25:20 2017 - [info]

Fri Feb  3 00:25:20 2017 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.

Fri Feb  3 00:25:20 2017 - [info] Starting recovery on 192.168.222.157(192.168.222.157:3306)..

Fri Feb  3 00:25:20 2017 - [info]  Generating diffs succeeded.

Fri Feb  3 00:25:20 2017 - [info] Waiting until all relay logs are applied.

Fri Feb  3 00:25:20 2017 - [info]  done.

Fri Feb  3 00:25:20 2017 - [info] Getting slave status..

Fri Feb  3 00:25:20 2017 - [info] This slave(192.168.222.157)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000068:530). No need to recover from Exec_M

aster_Log_Pos.

Fri Feb  3 00:25:20 2017 - [info] Connecting to the target slave host 192.168.222.157, running recover script..

Fri Feb  3 00:25:20 2017 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157

 --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog --workdir=/tmp --target_version=5.6.27-log --timestamp=201702

03002510 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx

Fri Feb  3 00:26:48 2017 - [info]

MySQL client version is 5.6.27. Using --binary-mode.

Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.222.156_3306_20170203002510.binlog on 192.168.222.157:3306. This may take long time.

..

Applying log files succeeded.

Fri Feb  3 00:26:48 2017 - [info]  All relay logs were successfully applied.

Fri Feb  3 00:26:48 2017 - [info] Getting new master's binlog name and position..

Fri Feb  3 00:26:48 2017 - [info]  mysql-bin.000002:196188901

Fri Feb  3 00:26:48 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.222.157', MASTER_POR

T=3306, MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=196188901, MASTER_USER='repl', MASTER_PASSWORD='xxx';

Fri Feb  3 00:26:48 2017 - [info] Executing master IP activate script:

Fri Feb  3 00:26:48 2017 - [info]   /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.1

56 --orig_master_port=3306 --new_master_host=192.168.222.157 --new_master_ip=192.168.222.157 --new_master_port=3306 --new_master_user='root'   --new_master_password=xx

x

Set read_only=0 on the new master.

Creating app user on the new master..

sh: sshroot@192.168.222.156./lvs-admin stop: No such file or directory

Fri Feb  3 00:26:49 2017 - [info]  OK.

Fri Feb  3 00:26:49 2017 - [info] ** Finished master recovery successfully.

Fri Feb  3 00:26:49 2017 - [info] * Phase 3: Master Recovery Phase completed.

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] * Phase 4: Slaves Recovery Phase..

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] Generating relay diff files from the latest slave succeeded.

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] All new slave servers recovered successfully.

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] * Phase 5: New master cleanup phase..

Fri Feb  3 00:26:49 2017 - [info]

Fri Feb  3 00:26:49 2017 - [info] Resetting slave info on the new master..

Fri Feb  3 00:26:49 2017 - [info]  192.168.222.157: Resetting slave info succeeded.

Fri Feb  3 00:26:49 2017 - [info] Master failover to 192.168.222.157(192.168.222.157:3306) completed successfully.

Fri Feb  3 00:26:49 2017 - [info] Deleted server1 entry from /etc/masterha/app1.cnf .

Fri Feb  3 00:26:49 2017 - [info]

 

----- Failover Report -----

 

app1: MySQL Master failover 192.168.222.156(192.168.222.156:3306) to 192.168.222.157(192.168.222.157:3306) succeeded

 

Master 192.168.222.156(192.168.222.156:3306) is down!

 

Check MHA Manager logs at mysqldb2:/masterha/app1/app1.log for details.

 

Started automated(non-interactive) failover.

Invalidated master IP address on 192.168.222.156(192.168.222.156:3306)

The latest slave 192.168.222.157(192.168.222.157:3306) has all relay logs for recovery.

Selected 192.168.222.157(192.168.222.157:3306) as a new master.

192.168.222.157(192.168.222.157:3306): OK: Applying all logs succeeded.

192.168.222.157(192.168.222.157:3306): OK: Activated master IP address.

Generating relay diff files from the latest slave succeeded.

192.168.222.157(192.168.222.157:3306): Resetting slave info succeeded.

Master failover to 192.168.222.157(192.168.222.157:3306) completed successfully.

 

 

三、部署 MHA遇到的问题

问题:

[root@localhost mha]# ./install.sh

--> Working on install

Fetching http://www.cpan.org/authors/id/D/DA/DAGOLDEN/install-0.01.tar.gz ... OK

Configuring install-0.01 ... OK

Building and testing install-0.01 ... FAIL

! Installing install failed. See /root/.cpanm/work/1485934105.3288/build.log for details. Retry with --force to force install it.

--> Working on DBD::mysql

Fetching http://www.cpan.org/authors/id/M/MI/MICHIELB/DBD-mysql-4.041.tar.gz ... OK

==> Found dependencies: DBI

--> Working on DBI

Fetching http://www.cpan.org/authors/id/T/TI/TIMB/DBI-1.636.tar.gz ... OK

Configuring DBI-1.636 ... N/A

! Configure failed for DBI-1.636. See /root/.cpanm/work/1485934125.3353/build.log for details.

! Installing the dependencies failed: Module 'DBI' is not installed

! Bailing out the installation for DBD-mysql-4.041.

 

 

问题:

/usr/bin/ld: cannot find -lz

 

usr/bin/ld: cannot find -lz

collect2: ld returned 1 exit status

make: *** [blib/arch/auto/DBD/mysql/mysql.so] Error 1

-> FAIL Installing DBD::mysql failed. See /root/.cpanm/work/1485938925.8246/build.log for details. Retry with --force to force install it.

2 distributions installed

 

 

处理:

yum -y install *zlib-dev*

 

[root@localhost mha]# ./install.sh

install is up to date. (0.01)

--> Working on DBD::mysql

Fetching http://www.cpan.org/authors/id/M/MI/MICHIELB/DBD-mysql-4.041.tar.gz ... OK

Configuring DBD-mysql-4.041 ... OK

Building and testing DBD-mysql-4.041 ... OK

Successfully installed DBD-mysql-4.041 (upgraded from 4.013)

1 distribution installed

 

 

 

 

 

 

 

 

[root@localhost mha4mysql-node-0.53]# perl Makefile.PL

*** Module::AutoInstall version 1.03

*** Checking for Perl dependencies...

[Core Features]

- DBI        ...loaded. (1.609)

- DBD::mysql ...loaded. (4.041)

*** Module::AutoInstall configuration finished.

Checking if your kit is complete...

Looks good

Generating a Unix-style Makefile

Writing Makefile for mha4mysql::node

Writing MYMETA.yml and MYMETA.json

[root@localhost mha4mysql-node-0.53]# make

cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm

cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm

cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm

cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm

cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm

cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm

cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm

cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm

cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm

cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs

cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog

cp bin/purge_relay_logs blib/script/purge_relay_logs

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs

cp bin/save_binary_logs blib/script/save_binary_logs

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs

Manifying 4 pod documents

[root@localhost mha4mysql-node-0.53]# make install

Manifying 4 pod documents

Installing /usr/local/share/perl5/MHA/NodeUtil.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFindManager.pm

Installing /usr/local/share/perl5/MHA/NodeConst.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFinder.pm

Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFinderXid.pm

Installing /usr/local/share/perl5/MHA/BinlogPosFinderElp.pm

Installing /usr/local/share/perl5/MHA/BinlogManager.pm

Installing /usr/local/share/perl5/MHA/SlaveUtil.pm

Installing /usr/local/share/man/man1/purge_relay_logs.1

Installing /usr/local/share/man/man1/filter_mysqlbinlog.1

Installing /usr/local/share/man/man1/save_binary_logs.1

Installing /usr/local/share/man/man1/apply_diff_relay_logs.1

Installing /usr/local/bin/apply_diff_relay_logs

Installing /usr/local/bin/save_binary_logs

Installing /usr/local/bin/filter_mysqlbinlog

Installing /usr/local/bin/purge_relay_logs

Appending installation info to /usr/lib64/perl5/perllocal.pod

 

 

 

 

 

问题10:从 https://github.com/yoshinorim/mha4mysql-node 下载 mha4mysql-node 安装

[root@mysqldb2 mha4mysql-node-master]# perl Makefile.PL

Can't locate inc/Module/Install.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Makefile.PL line 1.

BEGIN failed--compilation aborted at Makefile.PL line 1.

 

 

处理:

--查看 Makefile.PL 第一行的内容

[root@mysqldb2 mha4mysql-node-master]# vi Makefile.PL

 

use inc::Module::Install;

 

--安装模块

cpanm inc::Module::Install

 

 

[root@mysqldb2 mha4mysql-node-master]# perl Makefile.PL

include /root/olm/mha/mha4mysql-node-master/inc/Module/Install.pm

include inc/Module/Install/Metadata.pm

include inc/Module/Install/Base.pm

include inc/Module/Install/Makefile.pm

include inc/Module/Install/Scripts.pm

include inc/Module/Install/AutoInstall.pm

include inc/Module/Install/Include.pm

include inc/Module/AutoInstall.pm

*** Module::AutoInstall version 1.17

*** Checking for Perl dependencies...

[Core Features]

- DBI        ...loaded. (1.609)

- DBD::mysql ...loaded. (4.041)

*** Module::AutoInstall configuration finished.

include inc/Module/Install/WriteAll.pm

include inc/Module/Install/Win32.pm

include inc/Module/Install/Can.pm

include inc/Module/Install/Fetch.pm

Checking if your kit is complete...

Warning: the following files are missing in your kit:

        META.yml

Please inform the author.

Generating a Unix-style Makefile

Writing Makefile for mha4mysql::node

Writing MYMETA.yml and MYMETA.json

Writing META.yml

[root@mysqldb2 mha4mysql-node-master]# make

cp lib/MHA/BinlogManager.pm blib/lib/MHA/BinlogManager.pm

cp lib/MHA/BinlogPosFindManager.pm blib/lib/MHA/BinlogPosFindManager.pm

cp lib/MHA/BinlogPosFinderXid.pm blib/lib/MHA/BinlogPosFinderXid.pm

cp lib/MHA/BinlogHeaderParser.pm blib/lib/MHA/BinlogHeaderParser.pm

cp lib/MHA/BinlogPosFinder.pm blib/lib/MHA/BinlogPosFinder.pm

cp lib/MHA/BinlogPosFinderElp.pm blib/lib/MHA/BinlogPosFinderElp.pm

cp lib/MHA/NodeUtil.pm blib/lib/MHA/NodeUtil.pm

cp lib/MHA/SlaveUtil.pm blib/lib/MHA/SlaveUtil.pm

cp lib/MHA/NodeConst.pm blib/lib/MHA/NodeConst.pm

cp bin/apply_diff_relay_logs blib/script/apply_diff_relay_logs

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/apply_diff_relay_logs

cp bin/filter_mysqlbinlog blib/script/filter_mysqlbinlog

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/filter_mysqlbinlog

cp bin/purge_relay_logs blib/script/purge_relay_logs

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/purge_relay_logs

cp bin/save_binary_logs blib/script/save_binary_logs

"/usr/bin/perl" "-Iinc" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/save_binary_logs

Manifying 4 pod documents

[root@mysqldb2 mha4mysql-node-master]# make install

Manifying 4 pod documents

Installing /usr/local/share/perl5/MHA/NodeUtil.pm

Installing /usr/local/share/perl5/MHA/NodeConst.pm

Installing /usr/local/share/perl5/MHA/BinlogHeaderParser.pm

Installing /usr/local/share/perl5/MHA/BinlogManager.pm

Installing /usr/local/share/perl5/MHA/SlaveUtil.pm

Installing /usr/local/share/man/man1/purge_relay_logs.1

Installing /usr/local/share/man/man1/filter_mysqlbinlog.1

Installing /usr/local/share/man/man1/save_binary_logs.1

Installing /usr/local/share/man/man1/apply_diff_relay_logs.1

Installing /usr/local/bin/apply_diff_relay_logs

Installing /usr/local/bin/save_binary_logs

Installing /usr/local/bin/purge_relay_logs

Appending installation info to /usr/lib64/perl5/perllocal.pod

 

 

 

 

问题11:处理完问题10

[root@mysqldb2 debian]# /usr/local/bin/masterha_check_ssh --conf=/etc/masterha/app1.cnf

"NI_NUMERICHOST" is not exported by the Socket module

 "getaddrinfo" is not exported by the Socket module

 "getnameinfo" is not exported by the Socket module

Can't continue after import errors at /usr/local/share/perl5/MHA/NodeUtil.pm line 29.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/NodeUtil.pm line 29.

Compilation failed in require at /usr/local/share/perl5/MHA/SlaveUtil.pm line 27.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/SlaveUtil.pm line 27.

Compilation failed in require at /usr/local/share/perl5/MHA/DBHelper.pm line 26.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/DBHelper.pm line 26.

Compilation failed in require at /usr/local/share/perl5/MHA/HealthCheck.pm line 30.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/HealthCheck.pm line 30.

Compilation failed in require at /usr/local/share/perl5/MHA/Server.pm line 28.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/Server.pm line 28.

Compilation failed in require at /usr/local/share/perl5/MHA/Config.pm line 29.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/Config.pm line 29.

Compilation failed in require at /usr/local/share/perl5/MHA/SSHCheck.pm line 32.

BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/SSHCheck.pm line 32.

Compilation failed in require at /usr/local/bin/masterha_check_ssh line 25.

BEGIN failed--compilation aborted at /usr/local/bin/masterha_check_ssh line 25.

 

处理:

 

 

[root@mysqldb2 mha4mysql-node-master]# cpanm Socket

--> Working on Socket

Fetching http://www.cpan.org/authors/id/P/PE/PEVANS/Socket-2.024.tar.gz ... OK

==> Found dependencies: ExtUtils::Constant

--> Working on ExtUtils::Constant

Fetching http://www.cpan.org/authors/id/N/NW/NWCLARK/ExtUtils-Constant-0.23.tar.gz ... OK

Configuring ExtUtils-Constant-0.16 ... OK

Building and testing ExtUtils-Constant-0.23 ... OK

Successfully installed ExtUtils-Constant-0.23 (upgraded from 0.22)

Configuring Socket-2.024 ... OK

Building and testing Socket-2.024 ... OK

Successfully installed Socket-2.024 (upgraded from 1.82)

2 distributions installed

 

 

[root@mysqldb2 mha4mysql-manager-master]# /usr/local/bin/masterha_check_ssh --conf=/etc/masterha/app1.cnf

Wed Feb  1 19:41:34 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 19:41:34 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 19:41:34 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 19:41:34 2017 - [info] Starting SSH connection tests..

Wed Feb  1 19:41:51 2017 - [debug]

Wed Feb  1 19:41:35 2017 - [debug]  Connecting via SSH from root@192.168.222.157(192.168.222.157:22) to root@192.168.222.156(192.168.222.156:22)..

Wed Feb  1 19:41:50 2017 - [debug]   ok.

Wed Feb  1 19:41:55 2017 - [debug]

Wed Feb  1 19:41:34 2017 - [debug]  Connecting via SSH from root@192.168.222.156(192.168.222.156:22) to root@192.168.222.157(192.168.222.157:22)..

Wed Feb  1 19:41:55 2017 - [debug]   ok.

Wed Feb  1 19:41:55 2017 - [info] All SSH connection tests passed successfully.

 

 

 

问题12

[root@mysqldb2 mha]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Wed Feb  1 20:04:52 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 20:04:52 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:04:52 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:04:52 2017 - [info] MHA::MasterMonitor version 0.57.

Creating directory /masterha/app1.. done.

Wed Feb  1 20:04:57 2017 - [info] GTID failover mode = 0

Wed Feb  1 20:04:57 2017 - [info] Dead Servers:

Wed Feb  1 20:04:57 2017 - [info] Alive Servers:

Wed Feb  1 20:04:57 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:04:57 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Wed Feb  1 20:04:57 2017 - [info] Alive Slaves:

Wed Feb  1 20:04:57 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27 (oldest major version between slaves) log-bin:disabled

Wed Feb  1 20:04:57 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:04:57 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Wed Feb  1 20:04:57 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:04:57 2017 - [info] Checking slave configurations..

Wed Feb  1 20:04:57 2017 - [warning]  log-bin is not set on slave 192.168.222.157(192.168.222.157:3306). This host cannot be a master.

Wed Feb  1 20:04:57 2017 - [info] Checking replication filtering settings..

Wed Feb  1 20:04:57 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Wed Feb  1 20:04:57 2017 - [info]  Replication filtering check ok.

Wed Feb  1 20:04:57 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln364] None of slaves can be master. Check failover configuration file or log-bin settings in my.cnf

Wed Feb  1 20:04:57 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.

Wed Feb  1 20:04:57 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:04:57 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

 

 

处理:

vi  /usr/local/mysql/my.cnf

 

log_bin=/usr/local/mysql/data/mysql-bin

#relay_log=/usr/local/mysql/data/relay-bin

skip_slave_start

 

 

/usr/local/mysql/bin/mysqld_safe --defaults-file=/usr/local/mysql/my.cnf --user=mysql &

mysql> start slave;

 

 

[root@mysqldb2 mysql]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Wed Feb  1 20:16:43 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 20:16:43 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:16:43 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:16:43 2017 - [info] MHA::MasterMonitor version 0.57.

Wed Feb  1 20:16:44 2017 - [info] GTID failover mode = 0

Wed Feb  1 20:16:44 2017 - [info] Dead Servers:

Wed Feb  1 20:16:44 2017 - [info] Alive Servers:

Wed Feb  1 20:16:44 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:16:44 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Wed Feb  1 20:16:44 2017 - [info] Alive Slaves:

Wed Feb  1 20:16:44 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Wed Feb  1 20:16:44 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:16:44 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Wed Feb  1 20:16:44 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:16:44 2017 - [info] Checking slave configurations..

Wed Feb  1 20:16:44 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:16:44 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:16:44 2017 - [info] Checking replication filtering settings..

Wed Feb  1 20:16:44 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Wed Feb  1 20:16:44 2017 - [info]  Replication filtering check ok.

Wed Feb  1 20:16:44 2017 - [info] GTID (with auto-pos) is not supported

Wed Feb  1 20:16:44 2017 - [info] Starting SSH connection tests..

Wed Feb  1 20:16:50 2017 - [info] All SSH connection tests passed successfully.

Wed Feb  1 20:16:50 2017 - [info] Checking MHA Node version..

Wed Feb  1 20:16:51 2017 - [info]  Version check ok.

Wed Feb  1 20:16:51 2017 - [info] Checking SSH publickey authentication settings on the current master..

Wed Feb  1 20:16:51 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Wed Feb  1 20:16:51 2017 - [info] Master MHA Node version is 0.57.

Wed Feb  1 20:16:51 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Wed Feb  1 20:16:51 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/binlog --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000058

Wed Feb  1 20:16:51 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

Failed to save binary log: Binlog not found from /home/binlog! If you got this error at MHA Manager, please set "master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the MHA Manager's configuration file and try again.

 at /usr/local/bin/save_binary_logs line 123.

        eval {...} called at /usr/local/bin/save_binary_logs line 70

        main::main() called at /usr/local/bin/save_binary_logs line 66

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln161] Binlog setting check failed!

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln408] Master configuration failed.

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:16:51 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

 

问题12已经处理好,但出现了新的问题,命名为问题13

 

 

 

问题13

 

 

[root@mysqldb2 mysql]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Wed Feb  1 20:16:43 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 20:16:43 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:16:43 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:16:43 2017 - [info] MHA::MasterMonitor version 0.57.

Wed Feb  1 20:16:44 2017 - [info] GTID failover mode = 0

Wed Feb  1 20:16:44 2017 - [info] Dead Servers:

Wed Feb  1 20:16:44 2017 - [info] Alive Servers:

Wed Feb  1 20:16:44 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:16:44 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Wed Feb  1 20:16:44 2017 - [info] Alive Slaves:

Wed Feb  1 20:16:44 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Wed Feb  1 20:16:44 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:16:44 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Wed Feb  1 20:16:44 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:16:44 2017 - [info] Checking slave configurations..

Wed Feb  1 20:16:44 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:16:44 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:16:44 2017 - [info] Checking replication filtering settings..

Wed Feb  1 20:16:44 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Wed Feb  1 20:16:44 2017 - [info]  Replication filtering check ok.

Wed Feb  1 20:16:44 2017 - [info] GTID (with auto-pos) is not supported

Wed Feb  1 20:16:44 2017 - [info] Starting SSH connection tests..

Wed Feb  1 20:16:50 2017 - [info] All SSH connection tests passed successfully.

Wed Feb  1 20:16:50 2017 - [info] Checking MHA Node version..

Wed Feb  1 20:16:51 2017 - [info]  Version check ok.

Wed Feb  1 20:16:51 2017 - [info] Checking SSH publickey authentication settings on the current master..

Wed Feb  1 20:16:51 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Wed Feb  1 20:16:51 2017 - [info] Master MHA Node version is 0.57.

Wed Feb  1 20:16:51 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Wed Feb  1 20:16:51 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/binlog --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000058

Wed Feb  1 20:16:51 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

Failed to save binary log: Binlog not found from /home/binlog! If you got this error at MHA Manager, please set "master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the MHA Manager's configuration file and try again.

 at /usr/local/bin/save_binary_logs line 123.

        eval {...} called at /usr/local/bin/save_binary_logs line 70

        main::main() called at /usr/local/bin/save_binary_logs line 66

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln161] Binlog setting check failed!

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln408] Master configuration failed.

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.

Wed Feb  1 20:16:51 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:16:51 2017 - [info] Got exit code 1 (Not master dead).

 

处理:

修改 MHA配置文件中主库二进制日志的路径

 

[server default]

manager_log=/masterha/app1/app1.log

manager_workdir=/masterha/app1

#master_binlog_dir=/home/binlog 修改前的配置

master_binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data

master_ip_failover_script=/usr/local/bin/master_ip_failover

master_ip_online_change_script=/usr/local/bin/master_ip_online_change

password=123456

ping_interval=1

remote_workdir=/tmp

repl_password=123456

repl_user=repl

report_script=/usr/local/bin/send_report

secondary_check_script=/usr/local/bin/masterha_secondary_check -s '192.168.222.157' -s '192.168.222.156' --user=root --master_ip='192.168.222.156' --master_port=3306

shutdown_script=""

ssh_user=root

user=root

[server1]

hostname=192.168.222.156

port=3306

 

[server2]

hostname=192.168.222.157

port=3306

candidate_master=1

check_repl_delay=0

 

 

问题14Can't exec "mysqlbinlog": No such file or directory at /usr/local/share/perl5/MHA/BinlogManager.pm line 106.

 

[root@mysqldb2 mysql]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Wed Feb  1 20:25:05 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 20:25:05 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:25:05 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:25:05 2017 - [info] MHA::MasterMonitor version 0.57.

Wed Feb  1 20:25:06 2017 - [info] GTID failover mode = 0

Wed Feb  1 20:25:06 2017 - [info] Dead Servers:

Wed Feb  1 20:25:06 2017 - [info] Alive Servers:

Wed Feb  1 20:25:06 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:25:06 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Wed Feb  1 20:25:06 2017 - [info] Alive Slaves:

Wed Feb  1 20:25:06 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Wed Feb  1 20:25:06 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:25:06 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Wed Feb  1 20:25:06 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:25:06 2017 - [info] Checking slave configurations..

Wed Feb  1 20:25:06 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:25:06 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:25:06 2017 - [info] Checking replication filtering settings..

Wed Feb  1 20:25:06 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Wed Feb  1 20:25:06 2017 - [info]  Replication filtering check ok.

Wed Feb  1 20:25:06 2017 - [info] GTID (with auto-pos) is not supported

Wed Feb  1 20:25:06 2017 - [info] Starting SSH connection tests..

Wed Feb  1 20:25:13 2017 - [info] All SSH connection tests passed successfully.

Wed Feb  1 20:25:13 2017 - [info] Checking MHA Node version..

Wed Feb  1 20:25:13 2017 - [info]  Version check ok.

Wed Feb  1 20:25:13 2017 - [info] Checking SSH publickey authentication settings on the current master..

Wed Feb  1 20:25:13 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Wed Feb  1 20:25:14 2017 - [info] Master MHA Node version is 0.57.

Wed Feb  1 20:25:14 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Wed Feb  1 20:25:14 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000058

Wed Feb  1 20:25:14 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

  Creating /tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000058

Wed Feb  1 20:25:14 2017 - [info] Binlog setting check done.

Wed Feb  1 20:25:14 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Wed Feb  1 20:25:14 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306 --workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info  --relay_dir=/usr/local/mysql/data/  --slave_pass=xxx

Wed Feb  1 20:25:14 2017 - [info]   Connecting to root@192.168.222.157(192.168.222.157:22)..

Can't exec "mysqlbinlog": No such file or directory at /usr/local/share/perl5/MHA/BinlogManager.pm line 106.

mysqlbinlog version command failed with rc 1:0, please verify PATH, LD_LIBRARY_PATH, and client options

 at /usr/local/bin/apply_diff_relay_logs line 493.

Wed Feb  1 20:25:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln208] Slaves settings check failed!

Wed Feb  1 20:25:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln416] Slave configuration failed.

Wed Feb  1 20:25:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.

Wed Feb  1 20:25:14 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:25:14 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

 

 

处理:

[root@mysqldb2 mysql]# vi /etc/bashrc

#在文件的末尾添加

PATH="$PATH:/usr/local/mysql/bin"

export PATH

 

 

 

问题15Wed Feb  1 20:31:04 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. Can't exec "/usr/local/bin/master_ip_failover": No such file or directory at /usr/local/share/perl5/MHA/ManagerUtil.pm line 68.

 

[root@mysqldb2 mysql]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Wed Feb  1 20:30:56 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 20:30:56 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:30:56 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:30:56 2017 - [info] MHA::MasterMonitor version 0.57.

Wed Feb  1 20:30:57 2017 - [info] GTID failover mode = 0

Wed Feb  1 20:30:57 2017 - [info] Dead Servers:

Wed Feb  1 20:30:57 2017 - [info] Alive Servers:

Wed Feb  1 20:30:57 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:30:57 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Wed Feb  1 20:30:57 2017 - [info] Alive Slaves:

Wed Feb  1 20:30:57 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Wed Feb  1 20:30:57 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:30:57 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Wed Feb  1 20:30:57 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:30:57 2017 - [info] Checking slave configurations..

Wed Feb  1 20:30:57 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:30:57 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:30:57 2017 - [info] Checking replication filtering settings..

Wed Feb  1 20:30:57 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Wed Feb  1 20:30:57 2017 - [info]  Replication filtering check ok.

Wed Feb  1 20:30:57 2017 - [info] GTID (with auto-pos) is not supported

Wed Feb  1 20:30:57 2017 - [info] Starting SSH connection tests..

Wed Feb  1 20:31:03 2017 - [info] All SSH connection tests passed successfully.

Wed Feb  1 20:31:03 2017 - [info] Checking MHA Node version..

Wed Feb  1 20:31:03 2017 - [info]  Version check ok.

Wed Feb  1 20:31:03 2017 - [info] Checking SSH publickey authentication settings on the current master..

Wed Feb  1 20:31:04 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Wed Feb  1 20:31:04 2017 - [info] Master MHA Node version is 0.57.

Wed Feb  1 20:31:04 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Wed Feb  1 20:31:04 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000058

Wed Feb  1 20:31:04 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

  Creating /tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000058

Wed Feb  1 20:31:04 2017 - [info] Binlog setting check done.

Wed Feb  1 20:31:04 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Wed Feb  1 20:31:04 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306 --workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info  --relay_dir=/usr/local/mysql/data/  --slave_pass=xxx

Wed Feb  1 20:31:04 2017 - [info]   Connecting to root@192.168.222.157(192.168.222.157:22)..

  Checking slave recovery environment settings..

    Opening /usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at /usr/local/mysql/data, up to mysqldb2-relay-bin.000005

    Temporary relay log file is /usr/local/mysql/data/mysqldb2-relay-bin.000005

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Wed Feb  1 20:31:04 2017 - [info] Slaves settings check done.

Wed Feb  1 20:31:04 2017 - [info]

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

 

Wed Feb  1 20:31:04 2017 - [info] Checking replication health on 192.168.222.157..

Wed Feb  1 20:31:04 2017 - [info]  ok.

Wed Feb  1 20:31:04 2017 - [info] Checking master_ip_failover_script status:

Wed Feb  1 20:31:04 2017 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306

Wed Feb  1 20:31:04 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. Can't exec "/usr/local/bin/master_ip_failover": No such file or directory at /usr/local/share/perl5/MHA/ManagerUtil.pm line 68.

Wed Feb  1 20:31:04 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:31:04 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

Wed Feb  1 20:31:04 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln229]  Failed to get master_ip_failover_script status with return code 1:0.

Wed Feb  1 20:31:04 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.

Wed Feb  1 20:31:04 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:31:04 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

 

 

 

处理:

 

[root@mysqldb2 scripts]# cp master_ip_failover /usr/local/bin

[root@mysqldb2 scripts]# pwd

/root/olm/mha/mha4mysql-manager-master/samples/scripts

 

 

 

问题16

 

[root@mysqldb2 scripts]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Wed Feb  1 20:40:41 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Feb  1 20:40:41 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:40:41 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Wed Feb  1 20:40:41 2017 - [info] MHA::MasterMonitor version 0.57.

Wed Feb  1 20:40:43 2017 - [info] GTID failover mode = 0

Wed Feb  1 20:40:43 2017 - [info] Dead Servers:

Wed Feb  1 20:40:43 2017 - [info] Alive Servers:

Wed Feb  1 20:40:43 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:40:43 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Wed Feb  1 20:40:43 2017 - [info] Alive Slaves:

Wed Feb  1 20:40:43 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Wed Feb  1 20:40:43 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:40:43 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Wed Feb  1 20:40:43 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Wed Feb  1 20:40:43 2017 - [info] Checking slave configurations..

Wed Feb  1 20:40:43 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:40:43 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Wed Feb  1 20:40:43 2017 - [info] Checking replication filtering settings..

Wed Feb  1 20:40:43 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Wed Feb  1 20:40:43 2017 - [info]  Replication filtering check ok.

Wed Feb  1 20:40:43 2017 - [info] GTID (with auto-pos) is not supported

Wed Feb  1 20:40:43 2017 - [info] Starting SSH connection tests..

Wed Feb  1 20:40:59 2017 - [info] All SSH connection tests passed successfully.

Wed Feb  1 20:40:59 2017 - [info] Checking MHA Node version..

Wed Feb  1 20:40:59 2017 - [info]  Version check ok.

Wed Feb  1 20:40:59 2017 - [info] Checking SSH publickey authentication settings on the current master..

Wed Feb  1 20:40:59 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Wed Feb  1 20:40:59 2017 - [info] Master MHA Node version is 0.57.

Wed Feb  1 20:40:59 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Wed Feb  1 20:40:59 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000058

Wed Feb  1 20:40:59 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

  Creating /tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000058

Wed Feb  1 20:41:00 2017 - [info] Binlog setting check done.

Wed Feb  1 20:41:00 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Wed Feb  1 20:41:00 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306 --workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info  --relay_dir=/usr/local/mysql/data/  --slave_pass=xxx

Wed Feb  1 20:41:00 2017 - [info]   Connecting to root@192.168.222.157(192.168.222.157:22)..

  Checking slave recovery environment settings..

    Opening /usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at /usr/local/mysql/data, up to mysqldb2-relay-bin.000005

    Temporary relay log file is /usr/local/mysql/data/mysqldb2-relay-bin.000005

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Wed Feb  1 20:41:00 2017 - [info] Slaves settings check done.

Wed Feb  1 20:41:00 2017 - [info]

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

 

Wed Feb  1 20:41:00 2017 - [info] Checking replication health on 192.168.222.157..

Wed Feb  1 20:41:00 2017 - [info]  ok.

Wed Feb  1 20:41:00 2017 - [info] Checking master_ip_failover_script status:

Wed Feb  1 20:41:00 2017 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306

Bareword "FIXME_xxx" not allowed while "strict subs" in use at /usr/local/bin/master_ip_failover line 93.

Execution of /usr/local/bin/master_ip_failover aborted due to compilation errors.

Wed Feb  1 20:41:00 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln229]  Failed to get master_ip_failover_script status with return code 255:0.

Wed Feb  1 20:41:00 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/bin/masterha_check_repl line 48.

Wed Feb  1 20:41:00 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Wed Feb  1 20:41:00 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

 

 

 

处理:

暂时注释 master_ip_failover 脚本

 

 

 

############################################################################################################################

Keepalived

############################################################################################################################

 

问题17

 

./configure --prefix=/usr/local/keepalived

 

configure: error:

  !!! OpenSSL is not properly installed on your system. !!!

  !!! Can not include OpenSSL headers files.            !!!

 

 

处理:

yum -y install openssl-devel  

 

 

 

问题18

[root@mysqldb2 keepalived-1.3.2]# make

Making all in lib

make[1]: Entering directory `/root/olm/mha/keepalived-1.3.2/lib'

make  all-am

make[2]: Entering directory `/root/olm/mha/keepalived-1.3.2/lib'

  CC       memory.o

  CC       utils.o

  CC       notify.o

notify.c: 在函数‘find_path’中:

notify.c:226: 警告:此函数中的‘sgid_num’在使用前可能未初始化

  CC       timer.o

  CC       scheduler.o

  CC       vector.o

  CC       list.o

  CC       html.o

  CC       parser.o

  CC       signals.o

  CC       logger.o

  CC       rttables.o

  AR       liblib.a

make[2]: Leaving directory `/root/olm/mha/keepalived-1.3.2/lib'

make[1]: Leaving directory `/root/olm/mha/keepalived-1.3.2/lib'

Making all in keepalived

make[1]: Entering directory `/root/olm/mha/keepalived-1.3.2/keepalived'

Making all in core

make[2]: Entering directory `/root/olm/mha/keepalived-1.3.2/keepalived/core'

  CC       main.o

  CC       daemon.o

  CC       pidfile.o

  CC       layer4.o

  CC       smtp.o

  CC       global_data.o

  CC       global_parser.o

  CC       process.o

  CC       namespaces.o

namespaces.c: 在函数‘setns’中:

namespaces.c:187: 错误:‘SYS_setns’未声明(在此函数内第一次使用)

namespaces.c:187: 错误:(即使在一个函数内多次出现,每个未声明的标识符在其

namespaces.c:187: 错误:所在的函数内也只报告一次。)

make[2]: *** [namespaces.o] 错误 1

make[2]: Leaving directory `/root/olm/mha/keepalived-1.3.2/keepalived/core'

make[1]: *** [all-recursive] 错误 1

make[1]: Leaving directory `/root/olm/mha/keepalived-1.3.2/keepalived'

make: *** [all-recursive] 错误 1

 

 

处理:

--安装 glibc 库仍无法解决问题

yum -y install glibc*

 

--使用低版本keepalived keepalived-1.2.18

 

 

 

问题19

[root@oradb sysbench]# ./sysbench --test=olm --oltp-table-size=1000000 --oltp-read-only=off --init-rng=on --num-threads=16 --max-requests=0 --oltp-dist-type=uniform --max-time=1800 --mysql-user=root --db-driver=mysql --mysql-table-engine=innodb --oltp-test-mode=complexprepare

./sysbench: /usr/lib64/libmysqlclient.so.18: no version information available (required by ./sysbench)

 

处理:

ln -s /app/oracle/oradata2/mysql-5.6.27/lib/libmysqlclient.so /usr/lib64/libmysqlclient.so.18

 

 

 

 

问题:20

 

[root@mysqldb2 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf

Thu Feb  2 19:13:17 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Thu Feb  2 19:13:17 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Thu Feb  2 19:13:17 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Thu Feb  2 19:13:17 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb  2 19:13:17 2017 - [error][/usr/local/share/perl5/MHA/Server.pm, ln180] Got MySQL error when connecting 192.168.222.157(192.168.222.157:3306) :1045:Access denied for user 'root'@'mysqldb2' (using password: YES), but this is not a MySQL crash. Check MySQL server settings.

Thu Feb  2 19:13:17 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln301]  at /usr/local/share/perl5/MHA/ServerManager.pm line 297.

Thu Feb  2 19:13:18 2017 - [error][/usr/local/share/perl5/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations

Thu Feb  2 19:13:18 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/local/share/perl5/MHA/MasterMonitor.pm line 329.

Thu Feb  2 19:13:18 2017 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.

Thu Feb  2 19:13:18 2017 - [info] Got exit code 1 (Not master dead).

 

MySQL Replication Health is NOT OK!

 

原因:从库没有MHA监控用户

处理:

1.再主库创建监控用户

2.然后导出DUMP文件

3.导入备库创建 主从复制

4.执行权限刷新  flush privileges;

 

 

 

问题21

 

Thu Feb  2 19:56:56 2017 - [info] Connecting to the target slave host 192.168.222.157, running recover script..

Thu Feb  2 19:56:56 2017 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157  --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.222.156_3306_20170202195650.binlog --workdir=/tmp --target_version=5.6.27-log --timestamp=20170202195650 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx

Thu Feb  2 19:56:57 2017 - [info]

MySQL client version is 5.6.27. Using --binary-mode.

Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.222.156_3306_20170202195650.binlog on 192.168.222.157:3306. This may take long time.

 

.....

 

Thu Feb  2 19:56:57 2017 - [info]

MySQL client version is 5.6.27. Using --binary-mode.

Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.222.156_3306_20170202195650.binlog on 192.168.222.157:3306. This may take long time...

FATAL: applying log files failed with rc 1:0!

Error logs from mysqldb2:/tmp/relay_log_apply_for_192.168.222.157_3306_20170202195650_err.log (the last 200 lines)..

 

.....

 

at line 197: Cannot execute statement: impossible to write to binary log since statement is in row format and BINLOG_FORMAT = STATEMENT

 

....

 

--------------

 

Bye

 at /usr/local/bin/apply_diff_relay_logs line 515.

        eval {...} called at /usr/local/bin/apply_diff_relay_logs line 475

        main::main() called at /usr/local/bin/apply_diff_relay_logs line 120

Thu Feb  2 19:56:57 2017 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln1398]  Applying diffs failed with return code 22:0.

Thu Feb  2 19:56:57 2017 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln1561] Recovering master server failed.

Thu Feb  2 19:56:57 2017 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR:  at /usr/local/bin/masterha_manager line 65.

Thu Feb  2 19:56:57 2017 - [info]

 

----- Failover Report -----

 

app1: MySQL Master failover 192.168.222.156(192.168.222.156:3306)

 

Master 192.168.222.156(192.168.222.156:3306) is down!

 

Check MHA Manager logs at mysqldb2:/masterha/app1/app1.log for details.

 

Started automated(non-interactive) failover.

Invalidated master IP address on 192.168.222.156(192.168.222.156:3306)

The latest slave 192.168.222.157(192.168.222.157:3306) has all relay logs for recovery.

Selected 192.168.222.157(192.168.222.157:3306) as a new master.

Recovering master server failed.

Got Error so couldn't continue failover from here.

 

 

原因:二进制日志格式错误,需要修改为行格式

处理:

1.修改主库和备库的二进制日志格式

vi my.cnf

#添加

binlog_format=row

 

重启主备库MYSQL

 

2. 启动mha监控

 

nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/masterha/app1/manager.log 2>&1 &

 

Thu Feb  2 20:24:51 2017 - [info] MHA::MasterMonitor version 0.57.

Thu Feb  2 20:24:52 2017 - [info] GTID failover mode = 0

Thu Feb  2 20:24:52 2017 - [info] Dead Servers:

Thu Feb  2 20:24:52 2017 - [info] Alive Servers:

Thu Feb  2 20:24:52 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:24:52 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Thu Feb  2 20:24:52 2017 - [info] Alive Slaves:

Thu Feb  2 20:24:52 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Thu Feb  2 20:24:52 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:24:52 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu Feb  2 20:24:52 2017 - [info] Current Alive Master: 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:24:52 2017 - [info] Checking slave configurations..

Thu Feb  2 20:24:52 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Thu Feb  2 20:24:52 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Thu Feb  2 20:24:52 2017 - [info] Checking replication filtering settings..

Thu Feb  2 20:24:52 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Thu Feb  2 20:24:52 2017 - [info]  Replication filtering check ok.

Thu Feb  2 20:24:52 2017 - [info] GTID (with auto-pos) is not supported

Thu Feb  2 20:24:52 2017 - [info] Starting SSH connection tests..

Thu Feb  2 20:24:54 2017 - [info] All SSH connection tests passed successfully.

Thu Feb  2 20:24:54 2017 - [info] Checking MHA Node version..

Thu Feb  2 20:24:54 2017 - [info]  Version check ok.

Thu Feb  2 20:24:54 2017 - [info] Checking SSH publickey authentication settings on the current master..

Thu Feb  2 20:24:54 2017 - [info] HealthCheck: SSH to 192.168.222.156 is reachable.

Thu Feb  2 20:24:55 2017 - [info] Master MHA Node version is 0.57.

Thu Feb  2 20:24:55 2017 - [info] Checking recovery script configurations on 192.168.222.156(192.168.222.156:3306)..

Thu Feb  2 20:24:55 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --start_file=mysql-bin.000064

Thu Feb  2 20:24:55 2017 - [info]   Connecting to root@192.168.222.156(192.168.222.156:22)..

  Creating /tmp if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /app/oracle/oradata2/mysql-5.5.37/data, up to mysql-bin.000064

Thu Feb  2 20:24:55 2017 - [info] Binlog setting check done.

Thu Feb  2 20:24:55 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Thu Feb  2 20:24:55 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.222.157 --slave_ip=192.168.222.157 --slave_port=3306 --workdir=/tmp --target_version=5.6.27-log --manager_version=0.57 --relay_log_info=/usr/local/mysql/data/relay-log.info  --relay_dir=/usr/local/mysql/data/  --slave_pass=xxx

Thu Feb  2 20:24:55 2017 - [info]   Connecting to root@192.168.222.157(192.168.222.157:22)..

  Checking slave recovery environment settings..

    Opening /usr/local/mysql/data/relay-log.info ... ok.

    Relay log found at /usr/local/mysql/data, up to relay-bin.000005

    Temporary relay log file is /usr/local/mysql/data/relay-bin.000005

    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Thu Feb  2 20:24:56 2017 - [info] Slaves settings check done.

Thu Feb  2 20:24:56 2017 - [info]

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

 

Thu Feb  2 20:24:56 2017 - [info] Checking master_ip_failover_script status:

Thu Feb  2 20:24:56 2017 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306

Thu Feb  2 20:24:56 2017 - [info]  OK.

Thu Feb  2 20:24:56 2017 - [warning] shutdown_script is not defined.

Thu Feb  2 20:24:56 2017 - [info] Set master ping interval 1 seconds.

Thu Feb  2 20:24:56 2017 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check  -s oradb --user=root --master_host=oradb --master_ip='192.168.222.156' --master_port=3306

Thu Feb  2 20:24:56 2017 - [info] Starting ping health check on 192.168.222.156(192.168.222.156:3306)..

Thu Feb  2 20:24:56 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

 

 

 

3. 关闭主库

mysqladmin -u root shutdown

 

4. 启动MHA监控

 

nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover< /dev/null >/masterha/app1/manager.log 2>&1 &

 

Thu Feb  2 20:37:56 2017 - [warning] Got error on MySQL select ping: 2013 (Lost connection to MySQL server during query)

Thu Feb  2 20:37:56 2017 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check  -s oradb --user=root --master_host=oradb --master_ip='192.168.222.156' --master_port=3306  --user=root  --master_host=192.168.222.156  --master_ip=192.168.222.156  --master_port=3306 --master_user=root --master_password=123456 --ping_type=SELECT

Thu Feb  2 20:37:56 2017 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/oracle/oradata2/mysql-5.5.37/data --output_file=/tmp/save_binary_logs_test --manager_version=0.57 --binlog_prefix=mysql-bin

Thu Feb  2 20:37:57 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Thu Feb  2 20:37:57 2017 - [warning] Connection failed 2 time(s)..

Thu Feb  2 20:37:58 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Thu Feb  2 20:37:58 2017 - [warning] Connection failed 3 time(s)..

Thu Feb  2 20:37:59 2017 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.222.156' (111))

Thu Feb  2 20:37:59 2017 - [warning] Connection failed 4 time(s)..

Thu Feb  2 20:38:01 2017 - [warning] HealthCheck: Got timeout on checking SSH connection to 192.168.222.156! at /usr/local/share/perl5/MHA/HealthCheck.pm line 342.

Monitoring server oradb is reachable, Master is not reachable from oradb. OK.

Thu Feb  2 20:38:08 2017 - [info] Master is not reachable from all other monitoring servers. Failover should start.

Thu Feb  2 20:38:08 2017 - [warning] Master is not reachable from health checker!

Thu Feb  2 20:38:08 2017 - [warning] Master 192.168.222.156(192.168.222.156:3306) is not reachable!

Thu Feb  2 20:38:08 2017 - [warning] SSH is NOT reachable.

Thu Feb  2 20:38:08 2017 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..

Thu Feb  2 20:38:08 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Thu Feb  2 20:38:08 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..

Thu Feb  2 20:38:08 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..

Thu Feb  2 20:38:09 2017 - [warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

Thu Feb  2 20:38:09 2017 - [info] GTID failover mode = 0

Thu Feb  2 20:38:09 2017 - [info] Dead Servers:

Thu Feb  2 20:38:09 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:09 2017 - [info] Alive Servers:

Thu Feb  2 20:38:09 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Thu Feb  2 20:38:09 2017 - [info] Alive Slaves:

Thu Feb  2 20:38:09 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Thu Feb  2 20:38:09 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:09 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu Feb  2 20:38:09 2017 - [info] Checking slave configurations..

Thu Feb  2 20:38:09 2017 - [info]  read_only=1 is not set on slave 192.168.222.157(192.168.222.157:3306).

Thu Feb  2 20:38:09 2017 - [warning]  relay_log_purge=0 is not set on slave 192.168.222.157(192.168.222.157:3306).

Thu Feb  2 20:38:09 2017 - [info] Checking replication filtering settings..

Thu Feb  2 20:38:09 2017 - [info]  Replication filtering check ok.

Thu Feb  2 20:38:09 2017 - [info] Master is down!

Thu Feb  2 20:38:09 2017 - [info] Terminating monitoring script.

Thu Feb  2 20:38:09 2017 - [info] Got exit code 20 (Master dead).

Thu Feb  2 20:38:09 2017 - [info] MHA::MasterFailover version 0.57.

Thu Feb  2 20:38:09 2017 - [info] Starting master failover.

Thu Feb  2 20:38:09 2017 - [info]

Thu Feb  2 20:38:09 2017 - [info] * Phase 1: Configuration Check Phase..

Thu Feb  2 20:38:09 2017 - [info]

Thu Feb  2 20:38:10 2017 - [warning] SQL Thread is stopped(no error) on 192.168.222.157(192.168.222.157:3306)

Thu Feb  2 20:38:10 2017 - [info] GTID failover mode = 0

Thu Feb  2 20:38:10 2017 - [info] Dead Servers:

Thu Feb  2 20:38:10 2017 - [info]   192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:10 2017 - [info] Checking master reachability via MySQL(double check)...

Thu Feb  2 20:38:10 2017 - [info]  ok.

Thu Feb  2 20:38:10 2017 - [info] Alive Servers:

Thu Feb  2 20:38:10 2017 - [info]   192.168.222.157(192.168.222.157:3306)

Thu Feb  2 20:38:10 2017 - [info] Alive Slaves:

Thu Feb  2 20:38:10 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Thu Feb  2 20:38:10 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:10 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu Feb  2 20:38:10 2017 - [info]  Starting SQL thread on 192.168.222.157(192.168.222.157:3306) ..

Thu Feb  2 20:38:10 2017 - [info]   done.

Thu Feb  2 20:38:10 2017 - [info] Starting Non-GTID based failover.

Thu Feb  2 20:38:10 2017 - [info]

Thu Feb  2 20:38:10 2017 - [info] ** Phase 1: Configuration Check Phase completed.

Thu Feb  2 20:38:10 2017 - [info]

Thu Feb  2 20:38:10 2017 - [info] * Phase 2: Dead Master Shutdown Phase..

Thu Feb  2 20:38:10 2017 - [info]

Thu Feb  2 20:38:10 2017 - [info] Forcing shutdown so that applications never connect to the current master..

Thu Feb  2 20:38:10 2017 - [info] Executing master IP deactivation script:

Thu Feb  2 20:38:10 2017 - [info]   /usr/local/bin/master_ip_failover --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306 --command=stop

Thu Feb  2 20:38:10 2017 - [info]  done.

Thu Feb  2 20:38:10 2017 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.

Thu Feb  2 20:38:10 2017 - [info] * Phase 2: Dead Master Shutdown Phase completed.

Thu Feb  2 20:38:10 2017 - [info]

Thu Feb  2 20:38:10 2017 - [info] * Phase 3: Master Recovery Phase..

Thu Feb  2 20:38:10 2017 - [info]

Thu Feb  2 20:38:10 2017 - [info] * Phase 3.1: Getting Latest Slaves Phase..

Thu Feb  2 20:38:10 2017 - [info]

Thu Feb  2 20:38:10 2017 - [info] The latest binary log file/position on all slaves is mysql-bin.000064:120

Thu Feb  2 20:38:10 2017 - [info] Latest slaves (Slaves that received relay log files to the latest):

Thu Feb  2 20:38:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Thu Feb  2 20:38:11 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:11 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu Feb  2 20:38:11 2017 - [info] The oldest binary log file/position on all slaves is mysql-bin.000064:120

Thu Feb  2 20:38:11 2017 - [info] Oldest slaves:

Thu Feb  2 20:38:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Thu Feb  2 20:38:11 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:11 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [warning] Dead Master is not SSH reachable. Could not save it's binlogs. Transactions that were not sent to the latest slave (Read_Master_Log_Pos to the tail of the dead master's binlog) were lost.

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 3.3: Determining New Master Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] Finding the latest slave that has all relay logs for recovering other slaves..

Thu Feb  2 20:38:11 2017 - [info] All slaves received relay logs to the same position. No need to resync each other.

Thu Feb  2 20:38:11 2017 - [info] Searching new master from slaves..

Thu Feb  2 20:38:11 2017 - [info]  Candidate masters from the configuration file:

Thu Feb  2 20:38:11 2017 - [info]   192.168.222.157(192.168.222.157:3306)  Version=5.6.27-log (oldest major version between slaves) log-bin:enabled

Thu Feb  2 20:38:11 2017 - [info]     Replicating from 192.168.222.156(192.168.222.156:3306)

Thu Feb  2 20:38:11 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu Feb  2 20:38:11 2017 - [info]  Non-candidate masters:

Thu Feb  2 20:38:11 2017 - [info]  Searching from candidate_master slaves which have received the latest relay log events..

Thu Feb  2 20:38:11 2017 - [info] New master is 192.168.222.157(192.168.222.157:3306)

Thu Feb  2 20:38:11 2017 - [info] Starting master failover..

Thu Feb  2 20:38:11 2017 - [info]

From:

192.168.222.156(192.168.222.156:3306) (current master)

 +--192.168.222.157(192.168.222.157:3306)

 

To:

192.168.222.157(192.168.222.157:3306) (new master)

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 3.3: New Master Diff Log Generation Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 3.4: Master Log Apply Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.

Thu Feb  2 20:38:11 2017 - [info] Starting recovery on 192.168.222.157(192.168.222.157:3306)..

Thu Feb  2 20:38:11 2017 - [info]  This server has all relay logs. Waiting all logs to be applied..

Thu Feb  2 20:38:11 2017 - [info]   done.

Thu Feb  2 20:38:11 2017 - [info]  All relay logs were successfully applied.

Thu Feb  2 20:38:11 2017 - [info] Getting new master's binlog name and position..

Thu Feb  2 20:38:11 2017 - [info]  mysql-bin.000003:120

Thu Feb  2 20:38:11 2017 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.222.157', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=120, MASTER_USER='repl', MASTER_PASSWORD='xxx';

Thu Feb  2 20:38:11 2017 - [info] Executing master IP activate script:

Thu Feb  2 20:38:11 2017 - [info]   /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.222.156 --orig_master_ip=192.168.222.156 --orig_master_port=3306 --new_master_host=192.168.222.157 --new_master_ip=192.168.222.157 --new_master_port=3306 --new_master_user='root'   --new_master_password=xxx

Undefined subroutine &main::FIXME_xxx_create_user called at /usr/local/bin/master_ip_failover line 89.

Set read_only=0 on the new master.

Creating app user on the new master..

Thu Feb  2 20:38:11 2017 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln1589]  Failed to activate master IP address for 192.168.222.157(192.168.222.157:3306) with return code 10:0

Thu Feb  2 20:38:11 2017 - [warning] Proceeding.

Thu Feb  2 20:38:11 2017 - [info] ** Finished master recovery successfully.

Thu Feb  2 20:38:11 2017 - [info] * Phase 3: Master Recovery Phase completed.

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 4: Slaves Recovery Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] Generating relay diff files from the latest slave succeeded.

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] All new slave servers recovered successfully.

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] * Phase 5: New master cleanup phase..

Thu Feb  2 20:38:11 2017 - [info]

Thu Feb  2 20:38:11 2017 - [info] Resetting slave info on the new master..

Thu Feb  2 20:38:11 2017 - [info]  192.168.222.157: Resetting slave info succeeded.

Thu Feb  2 20:38:11 2017 - [info] Master failover to 192.168.222.157(192.168.222.157:3306) completed successfully.

Thu Feb  2 20:38:11 2017 - [info] Deleted server1 entry from /etc/masterha/app1.cnf .

Thu Feb  2 20:38:11 2017 - [info]

 

----- Failover Report -----

 

app1: MySQL Master failover 192.168.222.156(192.168.222.156:3306) to 192.168.222.157(192.168.222.157:3306) succeeded

 

Master 192.168.222.156(192.168.222.156:3306) is down!

 

Check MHA Manager logs at mysqldb2:/masterha/app1/app1.log for details.

 

Started automated(non-interactive) failover.

Invalidated master IP address on 192.168.222.156(192.168.222.156:3306)

The latest slave 192.168.222.157(192.168.222.157:3306) has all relay logs for recovery.

Selected 192.168.222.157(192.168.222.157:3306) as a new master.

192.168.222.157(192.168.222.157:3306): OK: Applying all logs succeeded.

Failed to activate master IP address for 192.168.222.157(192.168.222.157:3306) with return code 10:0

Generating relay diff files from the latest slave succeeded.

192.168.222.157(192.168.222.157:3306): Resetting slave info succeeded.

Master failover to 192.168.222.157(192.168.222.157:3306) completed successfully.


推荐阅读:
  1. windows server dhcp 配置故障转移
  2. server 2016部署故障转移群集cluster

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

mha mysql 故障

上一篇:mysqldump --master-data=2参数的含义

下一篇:社群系统ThinkSNS+ V2.2.3更新播报

相关阅读

您好,登录后才能下订单哦!

密码登录
登录注册
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》