private 网络不稳定引起的Evicting instance 2 from cluster

环境：双节点RAC, oracle 11.2.3

客户电话RAC实例２异常，现场查看日志：

实例２：

Fri Aug 25 09:45:16 2017

Received an instance abort message from instance 1Received an instance abort message from instance 1

Please check instance 1 alert and LMON trace files for detail.Please check instance 1 alert and LMON trace files for detail.

LMS0 (ospid: 24510820): terminating the instance due to error 481

Fri Aug 25 09:45:16 2017

System state dump requested by (instance=2, osid=24510820 (LMS0)), summary=[abnormal instance termination].

System State dumped to trace file /oracle/11.2.0/diag/rdbms/ins/ins2/trace/ins2_diag_21561818.trc

Instance terminated by LMS0, pid = 24510820

实例１

Fri Aug 25 09:44:25 2017

IPC Send timeout detected. Sender: ospid 35783054 [oracle@db1 (LMS1)]

Receiver: inst 2 binc 2073329022 ospid 24183072

IPC Send timeout to 2.2 inc 28 for msg type 65518 from opid 14

Fri Aug 25 09:44:27 2017

Communications reconfiguration: instance_number 2

Fri Aug 25 09:45:16 2017

Detected an inconsistent instance membership by instance 1

Evicting instance 2 from cluster

Waiting for instances to leave: 2 

Fri Aug 25 09:45:16 2017

Dumping diagnostic data in directory=[cdmp_20170825094516], requested by (instance=2, osid=24510820 (LMS0)), summary=[abnormal instance termination].

Reconfiguration started (old inc 28, new inc 32)

List of instances:

 1 (myinst: 1) 

查看/oracle/11.2.0/diag/rdbms/gjj/ins2/trace/ins2_diag_21561818.trc

*** 2017-08-25 14:24:35.900

I'm the voting node

Group reconfiguration cleanup

confirm->incar_num 22, rcfgctx->prop_incar 0

Send my bitmap to master 0

kjzgmappropose : incar 0, newmap - 

3000000000000000000000000000000000000000000000000000000000000000

kjzgmappropose : rc from psnd : 30

kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

kjzdattdlm: Can not attach to DLM (LMON up=[TRUE], DB mounted=[FALSE]).

怀疑心跳网络存在问题（这套RAC之前就出现过几次实例被驱逐的问题，但实例自动都启动了，这次实例被驱逐后实例2不能启动，针对之前实例被驱逐的问题进行过参数修改，通过这次的情况来看，实该不是参数设置的问题）。

测试心跳网络，连通性和传输速率都没有问题，后续打算通过haip进一步提升心跳网络可用性，在添加haip过程中发现当服和服务器和交换机新添加网络后出来数据包丢失的情况，丢包率50%，判断心跳网络稳定性存在问题，基于此撤掉新添加的心跳线，更换原来的心跳线，重启被驱逐的实例2，实例正常。

最后判断是原心跳线RJ45头存在某两芯短路的问题造成此次故障。