10.03.2016 20:54, Tristan Cacqueray wrote: Thanks Tristan and Cedric! Dogfooding is important so we can improve what we ship. Do you do have any logs that we could give to Fabio and his team to analyze what happened with those crazy wabbits, so we might get it fixed and avoided in the future? Thanks, Alan

On 03/11/2016 09:07 AM, Fabio M. Di Nitto wrote: Unfortunately there are no obvious steps to reproduce beside using nodepool (which is quite good at loading openstack service...). However here are some more technical insights: * the first tell was the lack of amqp queue consumer: # rabbitmqctl list_queues messages consumers name | tail 168 0 reply_478513620c5f4be4a60e294b02dabe54 444 0 scheduler In the logs, this translate to: conductor.log: Timed out waiting for a reply to message ID scheduler.log: The exchange Exchange reply_XXX(direct) to send to reply_XXX doesn't exist yet, retrying... * the second tell was that one of the rabbitmq service was not available, admin port was working fine and cluster_status didn't reported errors, however direct connections to port 5672 was unreachable: ERROR oslo_messaging._drivers.impl_rabbit [req-XXX - - - - -] AMQP server on controler_ip:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. * then after rabbitmq service are restored, controller needed to be powercycled to restore operation. I couldn't find any coredump on controllers (using find / -xdev -iname "*core*" -type f), and I don't know the plateform settings enough to use sosreports now. TL;DR; imo the first real issue is rabbit cluster didn't checked service availability and was unable to report connection error to the service. Then the second issue seems to be the shuffle strategy of oslo_messaging, in this case, even though only one controller had a rabbitmq issue, services seemed to always choose the same controller. This really needs manual testing to better understand why services doesn't recover easily of such failure and how to fix it. Regards, -Tristan