<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi folks,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Another issue I have noticed is with the nova_placement container using 3 controllers in HA mode with the Queens release.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">More than half the time, users were unable to launch instances after a fresh deployment. Attempts to launch instances would either failed or stuck at BUILD status. Other times the first one or two launches went successfully, but subsequent launches would fail. This phenomenon would last until the nova_placement container on one of the controllers became unhealthy and was given a restart. <br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">My current "solution" is to preemptively restart every nova_placement container on every controller node after a fresh deployment before launching any instances. This doesn't fix the problem every time as I can recall, but it does significantly reduce the likelihood of initial launch failures. <br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I have collected some logs from the nova_placement container and linked them below. To my untrained eye, I am unable to find out clues. Hopefully they are useful to help you debug.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">/var/log/nova/nova-schedular.log : <a href="http://paste.openstack.org/show/731930/">http://paste.openstack.org/show/731930/</a></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">/var/log/nova/nova-placement-api.log : <a href="http://paste.openstack.org/show/731931/">http://paste.openstack.org/show/731931/</a></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">/var/log/nova-conductor.log : <a href="http://paste.openstack.org/show/731932/">http://paste.openstack.org/show/731932/</a></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">/var/log/httpd/placement_wsgi_access.log (with repetitive entries/patterns shortened): <a href="http://paste.openstack.org/show/731933/">http://paste.openstack.org/show/731933/</a></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">/var/log/httpd/placement_wsgi_error.log : <a href="http://paste.openstack.org/show/731934/">http://paste.openstack.org/show/731934/</a><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Thank you very much.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Best regards,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Cody<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div></div></div></div></div></div></div></div>