<div dir="ltr">Hi,<div><br></div><div>I've deleted the nova and neutron services but the issue persists, so I guess it's not related.</div><div><br></div><div>Filing the sosreport.</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 4, 2016 at 4:32 PM, Ilja Maslov <span dir="ltr"><<a href="mailto:imaslov@dispersivegroup.com" target="_blank">imaslov@dispersivegroup.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Not on this fresh install, but what I saw few weeks back was that when controller nodes restart, I see services created with FQDN names that were up and I was able to safely
clean the original services with short host names. But I haven’t re-tested controller restarts afterwards.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">With my fresh install, rabbitmq is not coming up upon reboot (‘unknown error’ (1)), so I need to fix this first before I’m able to proceed with testing. I’ll let you know
how it goes.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Ilja<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Pedro Sousa [mailto:<a href="mailto:pgsousa@gmail.com" target="_blank">pgsousa@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, August 04, 2016 11:23 AM<br>
<b>To:</b> Ilja Maslov <<a href="mailto:imaslov@dispersivegroup.com" target="_blank">imaslov@dispersivegroup.com</a>><br>
<b>Cc:</b> Raoul Scarazzini <<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>>; rdo-list <<a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a>></span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [rdo-list] Overcloud pacemaker services restart behavior causes downtime<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Ilja,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I noticed that too. Did you try to delete the services that are marked down and retest?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Aug 4, 2016 at 4:12 PM, Ilja Maslov <<a href="mailto:imaslov@dispersivegroup.com" target="_blank">imaslov@dispersivegroup.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Hi,<br>
<br>
I've noticed similar behavior on Mitaka installed from trunk/mitaka/passed-ci. Appreciate if you could put me in CC.<br>
<br>
Additional detail is that during initial deployment, nova services, neutron agents and heat engines are registered with the short hostnames and upon controller node restart, these will all show with state=down. Probably because hosts files are re-written after
the services had been started with FQDN as a first entry. I do not know to what extent pacemaker resources are monitored, but it could be related to the problem you are reporting.<br>
<br>
Cheers,<br>
Ilja<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
-----Original Message-----<br>
From: <a href="mailto:rdo-list-bounces@redhat.com" target="_blank">rdo-list-bounces@redhat.com</a> [mailto:<a href="mailto:rdo-list-bounces@redhat.com" target="_blank">rdo-list-bounces@redhat.com</a>] On Behalf Of Raoul Scarazzini<br>
Sent: Thursday, August 04, 2016 9:31 AM<br>
To: Pedro Sousa <<a href="mailto:pgsousa@gmail.com" target="_blank">pgsousa@gmail.com</a>><br>
Cc: rdo-list <<a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a>><br>
Subject: Re: [rdo-list] Overcloud pacemaker services restart behavior causes downtime<br>
<br>
That will be great, thank you, put me in CC so I can follow this.<br>
<br>
Thanks,<br>
<br>
--<br>
Raoul Scarazzini<br>
<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a><br>
<br>
On 04/08/2016 15:29, Pedro Sousa wrote:<br>
> Hi Raoul,<br>
><br>
> this only happens when the node comes back online after booting. When I<br>
> stop the node with "pcs cluster stop", everything works fine, even if<br>
> VIP is active on that node.<br>
><br>
> Anyway I will file a bugzilla.<br>
><br>
> Thanks<br>
><br>
><br>
><br>
><br>
> On Thu, Aug 4, 2016 at 1:51 PM, Raoul Scarazzini <<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a><br>
> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>>> wrote:<br>
><br>
> Ok, so we are on mitaka. Here we have VIPs that are a (Optional)<br>
> dependency for haproxy, which is a (Mandatory) dependency for<br>
> openstack-core from which all the others (nova, neutron, cinder and so<br>
> on) depends.<br>
> This means that if you are rebooting a controller in which a VIP is<br>
> active you will NOT have a restart of openstack-core since haproxy will<br>
> not be restarted, because of the OPTIONAL constraint.<br>
> So the behavior you're describing is quite strange.<br>
> Maybe other components are in the game here. Can you open a bugzilla<br>
> with the exact steps you're using to reproduce the problem and share the<br>
> sosreports of your systems?<br>
><br>
> Thanks,<br>
><br>
> --<br>
> Raoul Scarazzini<br>
> <a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>><br>
><br>
> On 04/08/2016 12:34, Pedro Sousa wrote:<br>
> > Hi,<br>
> ><br>
> > I use mitaka from centos sig repos:<br>
> ><br>
> > Centos 7.2<br>
> > centos-release-openstack-mitaka-1-3.el7.noarch<br>
> > pacemaker-cli-1.1.13-10.el7_2.2.x86_64<br>
> > pacemaker-1.1.13-10.el7_2.2.x86_64<br>
> > pacemaker-remote-1.1.13-10.el7_2.2.x86_64<br>
> > pacemaker-cluster-libs-1.1.13-10.el7_2.2.x86_64<br>
> > pacemaker-libs-1.1.13-10.el7_2.2.x86_64<br>
> > corosynclib-2.3.4-7.el7_2.3.x86_64<br>
> > corosync-2.3.4-7.el7_2.3.x86_64<br>
> > resource-agents-3.9.5-54.el7_2.10.x86_64<br>
> ><br>
> > Let me know if you need more info.<br>
> ><br>
> > Thanks<br>
> ><br>
> ><br>
> ><br>
> > On Thu, Aug 4, 2016 at 11:21 AM, Raoul Scarazzini <<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>><br>
> > <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>>>> wrote:<br>
> ><br>
> > Hi,<br>
> > can you please give us more information about the environment you are<br>
> > using? Release, package versions and so on.<br>
> ><br>
> > --<br>
> > Raoul Scarazzini<br>
> > <a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>><br>
> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a> <mailto:<a href="mailto:rasca@redhat.com" target="_blank">rasca@redhat.com</a>>><br>
> ><br>
> > On 04/08/2016 11:34, Pedro Sousa wrote:<br>
> > > Hi all,<br>
> > ><br>
> > > I have an overcloud with 3 controller nodes, everything is<br>
> working fine,<br>
> > > the problem is when I reboot one of the controllers. When<br>
> the node comes<br>
> > > online, all the services (nova-api, neutron-server) on the<br>
> other nodes<br>
> > > are also restarted, causing a couple of minutes of downtime<br>
> until<br>
> > > everything is recovered.<br>
> > ><br>
> > > In the example below I restarted controller2 and I see these<br>
> messages on<br>
> > > controller0. My question is if this is the expected<br>
> behavior, because in<br>
> > > my opinion it shouldn't happen.<br>
> > ><br>
> > > *Authorization Failed: Service Unavailable (HTTP 503)*<br>
> > > *== Glance images ==*<br>
> > > *Service Unavailable (HTTP 503)*<br>
> > > *== Nova managed services ==*<br>
> > > *No handlers could be found for logger<br>
> > "keystoneauth.identity.generic.base"*<br>
> > > *ERROR (ServiceUnavailable): Service Unavailable (HTTP 503)*<br>
> > > *== Nova networks ==*<br>
> > > *No handlers could be found for logger<br>
> > "keystoneauth.identity.generic.base"*<br>
> > > *ERROR (ServiceUnavailable): Service Unavailable (HTTP 503)*<br>
> > > *== Nova instance flavors ==*<br>
> > > *No handlers could be found for logger<br>
> > "keystoneauth.identity.generic.base"*<br>
> > > *ERROR (ServiceUnavailable): Service Unavailable (HTTP 503)*<br>
> > > *== Nova instances ==*<br>
> > > *No handlers could be found for logger<br>
> > "keystoneauth.identity.generic.base"*<br>
> > > *ERROR (ServiceUnavailable): Service Unavailable (HTTP 503)*<br>
> > > *[root@overcloud-controller-0 ~]# openstack-status *<br>
> > > *Broadcast message from<br>
> > > <a href="mailto:systemd-journald@overcloud-controller-0.localdomain" target="_blank">
systemd-journald@overcloud-controller-0.localdomain</a> (Thu<br>
> 2016-08-04<br>
> > > 09:22:31 UTC):*<br>
> > > *<br>
> > > *<br>
> > > *haproxy[2816]: proxy neutron has no server available!*<br>
> > ><br>
> > > Thanks,<br>
> > > Pedro Sousa<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > rdo-list mailing list<br>
> > > <a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a> <mailto:<a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a>><br>
> <mailto:<a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a> <mailto:<a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a>>><br>
> > > <a href="https://www.redhat.com/mailman/listinfo/rdo-list" target="_blank">
https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
> > ><br>
> > > To unsubscribe: <a href="mailto:rdo-list-unsubscribe@redhat.com" target="_blank">rdo-list-unsubscribe@redhat.com</a> <mailto:<a href="mailto:rdo-list-unsubscribe@redhat.com" target="_blank">rdo-list-unsubscribe@redhat.com</a>><br>
> > <mailto:<a href="mailto:rdo-list-unsubscribe@redhat.com" target="_blank">rdo-list-unsubscribe@redhat.com</a><br>
> <mailto:<a href="mailto:rdo-list-unsubscribe@redhat.com" target="_blank">rdo-list-unsubscribe@redhat.com</a>>><br>
> > ><br>
> ><br>
> ><br>
><br>
><br>
<br>
_______________________________________________<br>
rdo-list mailing list<br>
<a href="mailto:rdo-list@redhat.com" target="_blank">rdo-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/rdo-list" target="_blank">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
<br>
To unsubscribe: <a href="mailto:rdo-list-unsubscribe@redhat.com" target="_blank">rdo-list-unsubscribe@redhat.com</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>