[rdo-users] Problems with Controller and Instance HA

Hi everyone, My cluster is deployed with both Controller and Instance HA. The deployment completed without errors, but I noticed something strange from the 'pcs status' output from the controllers: Clone Set: compute-unfence-trigger-clone [compute-unfence-trigger] Started: [ overcloud-novacompute-0 ] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 overcloud-novacompute-1 ] nova-evacuate (ocf::openstack:NovaEvacuate): Started overcloud-controller-0 stonith-fence_ipmilan-002590a2d2c7 (stonith:fence_ipmilan): Started overcloud-controller-1 stonith-fence_ipmilan-002590a1c641 (stonith:fence_ipmilan): Started overcloud-controller-2 stonith-fence_ipmilan-002590f25822 (stonith:fence_ipmilan): Started overcloud-controller-0 stonith-fence_ipmilan-002590f3977a (stonith:fence_ipmilan): Started overcloud-controller-2 stonith-fence_ipmilan-002590f2631a (stonith:fence_ipmilan): Started overcloud-controller-1 Notice the stonith-fence_ipmilan lines showed incorrect hosts for the last two devices. The MAC addresses are for the overcloud-novacompute-0 and overcloud-novacompute-1, but it got started on the controller nodes. Is this right? There are also some failed actions from the status output: Failed Actions: * overcloud-novacompute-1_start_0 on overcloud-controller-2 'unknown error' (1): call=3, status=Timed Out, exitreason='', last-rc-change='Wed Oct 3 03:48:55 2018', queued=0ms, exec=0ms * overcloud-novacompute-1_start_0 on overcloud-controller-0 'unknown error' (1): call=23, status=Timed Out, exitreason='', last-rc-change='Wed Oct 3 14:50:25 2018', queued=0ms, exec=0ms * overcloud-novacompute-1_start_0 on overcloud-controller-1 'unknown error' (1): call=3, status=Timed Out, exitreason='', last-rc-change='Wed Oct 3 03:47:51 2018', queued=0ms, exec=0ms I can spin up VMs, but cannot do failover. If I manually trigger a crash on one of the compute nodes, the affected VMs will remain at ERROR state and the affected compute node will be unable to rejoin the cluster afterward. After a manual reboot on the affected compute node, it cannot start the pcs cluster service. Its container 'nova_compute' also remains unhealthy after reboot, with the lastest 'docker logs' message as: ++ cat /run_command + CMD='/var/lib/nova/instanceha/check-run-nova-compute ' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start ++ [[ ! -d /var/log/kolla/nova ]] +++ stat -c %a /var/log/kolla/nova ++ [[ 2755 != \7\5\5 ]] ++ chmod 755 /var/log/kolla/nova ++ . /usr/local/bin/kolla_nova_extend_start +++ [[ ! -d /var/lib/nova/instances ]] + echo 'Running command: '\''/var/lib/nova/instanceha/check-run-nova-compute '\''' Running command: '/var/lib/nova/instanceha/check-run-nova-compute ' + exec /var/lib/nova/instanceha/check-run-nova-compute Waiting for fence-down flag to be cleared Waiting for fence-down flag to be cleared Waiting for fence-down flag to be cleared Waiting for fence-down flag to be cleared Waiting for fence-down flag to be cleared Waiting for fence-down flag to be cleared ... So I guess something may be wrong with fencing, but I have no idea what caused it and how to fix it. Any helps/suggestions/opinions would be greatly appreciated. Thank you very much. Regards. Cody