[Rdo-list] rdo-manager failures: instack-install-undercloud failing for non-obvious reasons

[James Slagle]

On Tue, Apr 21, 2015 at 04:37:26PM -0400, Lars Kellogg-Stedman wrote:
> Running "instack-install-undercloud" is failing for me:
> 
>   + echo 'puppet apply exited with exit code 6'
>   puppet apply exited with exit code 6
>   + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
>   + exit 6
>   [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure
>   phase. [Command '['dib-run-parts',
>   '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit
>   status 6]
> 
> Unfortunately, the failure doesn't provide much in the way of useful
> information.  If I scroll up several pages, I find:
> 
>   Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff'
>   Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired!
>   Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired!
>   Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired!
>   Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired!
>   Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired!
> 
> But again, that doesn't really tell me what is failing either (a
> command is still failing? Which command?).

Unfortunately we're pretty much at the mercy of puppet and all of the external
puppet modules here in terms of its helpful output, and the point at which it
chooses to stop applying after an error is encountered. Perhaps some people
more familiar with puppet might chime in here on how to improve this.

> 
> It looks like rabbitmq is having some problems:
> 
>   [stack at localhost ~]$ sudo rabbitmqctl status
>   Status of node rabbit at localhost ...
>   Error: unable to connect to node rabbit at localhost: nodedown
> 
>   DIAGNOSTICS
>   ===========
> 
>   attempted to contact: [rabbit at localhost]
> 
>   rabbit at localhost:
>   * connected to epmd (port 4369) on localhost
>   * epmd reports node 'rabbit' running on port 25672
>   * TCP connection succeeded but Erlang distribution failed
>   * suggestion: hostname mismatch?
>   * suggestion: is the cookie set correctly?
> 
>   current node details:
>   - node name: rabbitmqctl20640 at stack
>   - home dir: /var/lib/rabbitmq
>   - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ==
> 
> If I manually stop and then start rabbitmq:
> 
>     sudo systemctl stop rabbitmq-server
>     sudo systemctl start rabbitmq-server
> 
> It seems to work:
> 
>   # rabbitmqctl status
>   Status of node rabbit at stack ...
>   [{pid,20946},
>    {running_applications,
>        [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
>   ...
> 
> After manually starting rabbit and re-running
> instack-install-undercloud, the process is able to successfully create
> the rabbitmq_user resources and completes successfully.

Are you on RHEL 7.1 or CentOS 7? I'll try to reproduce locally and see if I can
get to the bottom of it.

> 
> -- 
> Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {freenode,twitter,github}
> Cloud Engineering / OpenStack          | http://blog.oddbit.com/



> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
> 
> To unsubscribe: rdo-list-unsubscribe at redhat.com

--
-- James Slagle
--