<div dir="ltr">Hi Yatin,<div><br></div><div>I figured out the cause of the issue, I had the hostnames configured wrong, I was using FQDNs instead of short hostnames and therefore tripleo was appending the domain name to them incorrectly. I'm not sure why this broke pacemaker but it works correctly now. </div><div><br></div><div>Thanks for your help!</div><div>-James H</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 28 Dec 2020 at 16:15, YATIN KAREL <<a href="mailto:yatinkarel@gmail.com">yatinkarel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi James,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 28, 2020 at 6:31 PM James Hirst <<a href="mailto:jdhirst12@gmail.com" target="_blank">jdhirst12@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Yatin,</div><div><br></div><div>I just deleted the overcloud and re-ran the deployment and it got stuck at the same place, when applying puppet host configuration to the controller. </div><div><br></div><div>I have compared my pcsd.log file to the one you linked to and it seems that mine has far less activity; the only requests I'm seeing received by it are auth requests like this:</div><div><font face="monospace">200 POST /remote/auth (10.27.0.4) 45.73ms</font></div><div><br></div></div></blockquote><div>I see pcsd[74686]: WARNING:pcs.daemon:Caught signal: 15, shutting down, which looks suspicious.<br></div><div>Also seems you shared logs from previous runs, can you share the latest sos report for controller, overcloud-deploy.log, or it might be due to you using deployed servers, can you clean those as well before retry.</div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>I have attached the logs here: <a href="https://drive.google.com/file/d/1eN15ZJzn_4GesrqTT2OcYpCigAK8rvnE/view?usp=sharing" target="_blank">ansible.log</a> <a href="https://drive.google.com/file/d/1cTC3lPRf3wjYB5SW1dNqlp--_038GInj/view?usp=sharing" target="_blank">controller log bundle</a> (note: ansible.log does just end without an error during the deployment as I didn't wait for it to retry 1100 times so I CTRL+C'd it.)</div><div><br></div><div>Are there any other logs I should gather?</div><div><br></div><div>Thanks,</div><div>James H</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 28 Dec 2020 at 13:29, YATIN KAREL <<a href="mailto:yatinkarel@gmail.com" target="_blank">yatinkarel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div dir="ltr">Hi James,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 28, 2020 at 4:28 PM James Hirst <<a href="mailto:jdhirst12@gmail.com" target="_blank">jdhirst12@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi Yatin,<div><br></div><div>Thank you for the confirmation! I re-enabled the pacemaker and haproxy roles and I have been since digging into why HA has been failing and I am seeing the following:</div><div><br></div><div>1. pacemaker.service won't start due to Corosync not running.</div><div>2. Corosync seems to be failing to start due to not having the /etc/corosync/corosync.conf file as it does not exist. </div><div>3. The pcsd log file shows the following errors:</div><div>---</div><div>Config files sync started<br>Config files sync skipped, this host does not seem to be in a cluster of at least 2 nodes<br></div><div>---<br></div><div>This is what originally led me to believe that it wouldn't work without a proper HA environment with 3 nodes.</div><div><br></div></div></div></blockquote><div>That is not fatal, i see same in job logs:- <a href="https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-victoria/0bccbf6/logs/overcloud-controller-0/var/log/pcsd/pcsd.log.txt.gz" target="_blank">https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-victoria/0bccbf6/logs/overcloud-controller-0/var/log/pcsd/pcsd.log.txt.gz</a></div><div><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div></div><div>The overcloud deployment itself simply times out at "Wait for puppet host configuration to finish". I saw that step_1 seems to be where things are failing (due to pacemaker), and when running it manually, I am seeing the following messages:</div><div><br></div><div>---</div><div>Debug: Executing: '/bin/systemctl is-enabled -- corosync'<br>Debug: Executing: '/bin/systemctl is-enabled -- pacemaker'<br>Debug: Executing: '/bin/systemctl is-active -- pcsd'<br>Debug: Executing: '/bin/systemctl is-enabled -- pcsd'<br>Debug: Exec[check-for-local-authentication](provider=posix): Executing check '/sbin/pcs status pcsd controller 2>&1 | grep 'Unable to authenticate''<br>Debug: Executing: '/sbin/pcs status pcsd controller 2>&1 | grep 'Unable to authenticate''<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[check-for-local-authentication]: '/bin/echo 'local pcsd auth failed, triggering a reauthentication'' won't be executed because of failed check 'onlyif'<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[reauthenticate-across-all-nodes]: '/sbin/pcs host auth <a href="http://controller.cloud.hirstgroup.net" target="_blank">controller.cloud.hirstgroup.net</a> -u hacluster -p oaJOCgGDxRfJ1dLK' won't be executed because of failed check 'refreshonly'<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]: '/sbin/pcs host auth <a href="http://controller.cloud.hirstgroup.net" target="_blank">controller.cloud.hirstgroup.net</a> -u hacluster -p oaJOCgGDxRfJ1dLK' won't be executed because of failed check 'refreshonly'<br>Debug: Exec[wait-for-settle](provider=posix): Executing check '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1'<br>Debug: Executing: '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1'<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: Error: error running crm_mon, is pacemaker running?<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: Could not connect to the CIB: Transport endpoint is not connected<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: crm_mon: Error: cluster is not available on this node<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: Exec try 1/360<br>Debug: Exec[wait-for-settle](provider=posix): Executing '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1'<br>Debug: Executing: '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1'<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: Sleeping for 10.0 seconds between tries<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: Exec try 2/360<br>Debug: Exec[wait-for-settle](provider=posix): Executing '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1'<br>Debug: Executing: '/sbin/pcs status | grep -q 'partition with quorum' > /dev/null 2>&1'<br>Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: Sleeping for 10.0 seconds between tries<br></div><div>---</div><div><br></div><div>How does the corosync.conf file get created? Is it related to the pcsd error saying that config sync can't proceed due to the cluster not having a minimum of two members?</div><div><br></div></div></div></blockquote><div>No that's not related as per pcsd.log shared above. AFAIK corosync.conf is created by pcs daemon itself by default when pcsd is used.</div><div>You tried that on an already deployed overcloud? If that's just a test setup try with overcloud delete and fresh install as i am not sure how well a re deployment with HA enable/disable works. Also share full logs with this as that will give some hint and also share what docs/steps you are using to see if some customization is being done. Also on your current failure /var/log/pcsd/pcsd.log.txt.gz on controller node should also have some details wrt failure.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div></div><div>Thanks,<br>James H</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 28 Dec 2020 at 11:26, YATIN KAREL <<a href="mailto:yatinkarel@gmail.com" target="_blank">yatinkarel@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi James,</div></div><div><br></div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Dec 27, 2020 at 4:04 PM James Hirst <<a href="mailto:jdhirst12@gmail.com" target="_blank">jdhirst12@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">HI All,<div><br></div><div>I am attempting to set up a single controller overcloud with tripleo Victoria. I keep running into issues where pcsd is attempting to be started in puppet step 1 on the controller and it fails. I attempted to solve this by simply removing the pacemaker service from my roles_data.yaml file, but then I ran into other errors requiring that the pacemaker service be enabled.</div><div><br></div></div></blockquote><div><div>HA Deployment is enabled by default since Ussuri release[1] with [2]. So pacemaker will be deployed by default whether you set up 1 or more controller nodes since Ussuri. Without pacemaker deployment is possible but would need more changes(apart from removing pacemaker from roles_data.yaml file), like adjusting resource_registry to use non pacemaker resources. HA with 1 Controller works fine as we have green jobs[3][4] running with both 1 controller/3 controllers, so would recommend to look why pcsd is failing for you and proceed with HA. But if you still want to go without pacemaker then can try adjusting resource-registry to enable/disable pacemaker resources<br></div><div> </div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I have ControllerCount set to 1, which according to the docs is all I need to do to tell tripleo that I'm not using HA.</div><div><br></div></div></blockquote><div>Docs might be outdated if it specifies just setting ControllerCount to 1 is enough to deploy without a pacemaker, you can report a bug or send a patch to fix that with the docs link you using.<br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div>Thanks,</div><div>James H</div></div>
_______________________________________________<br>
users mailing list<br>
<a href="mailto:users@lists.rdoproject.org" target="_blank">users@lists.rdoproject.org</a><br>
<a href="http://lists.rdoproject.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.rdoproject.org/mailman/listinfo/users</a><br>
<br>
To unsubscribe: <a href="mailto:users-unsubscribe@lists.rdoproject.org" target="_blank">users-unsubscribe@lists.rdoproject.org</a><br>
</blockquote></div></div><div><br></div><div><div><br></div><div>[1] <a href="https://docs.openstack.org/releasenotes/tripleo-heat-templates/ussuri.html#relnotes-12-3-0-stable-ussuri-other-notes" target="_blank">https://docs.openstack.org/releasenotes/tripleo-heat-templates/ussuri.html#relnotes-12-3-0-stable-ussuri-other-notes</a></div><div>[2] <a href="https://review.opendev.org/c/openstack/tripleo-heat-templates/+/359060" target="_blank">https://review.opendev.org/c/openstack/tripleo-heat-templates/+/359060</a></div><div>[3] <a href="https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-victoria/0bccbf6/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz" target="_blank">https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-victoria/0bccbf6/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz</a></div><div>[4] <a href="https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria/a5dd4bc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz" target="_blank">https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria/a5dd4bc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz</a></div><div><br></div><br></div><div>Thanks and regards<br></div><div><div dir="ltr"><div dir="ltr">Yatin Karel</div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><br></div>Thanks and Regards<br><div><div dir="ltr"><div dir="ltr">Yatin Karel</div></div></div></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Yatin Karel</div></div></div>
</blockquote></div>