[Rdo-list] rdo-manager failures: instack-install-undercloud failing for non-obvious reasons

[Lars Kellogg-Stedman]

Running "instack-install-undercloud" is failing for me:

  + echo 'puppet apply exited with exit code 6'
  puppet apply exited with exit code 6
  + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
  + exit 6
  [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure
  phase. [Command '['dib-run-parts',
  '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit
  status 6]

Unfortunately, the failure doesn't provide much in the way of useful
information.  If I scroll up several pages, I find:

  Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff'
  Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired!
  Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired!
  Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired!
  Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired!
  Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired!

But again, that doesn't really tell me what is failing either (a
command is still failing? Which command?).

It looks like rabbitmq is having some problems:

  [stack at localhost ~]$ sudo rabbitmqctl status
  Status of node rabbit at localhost ...
  Error: unable to connect to node rabbit at localhost: nodedown

  DIAGNOSTICS
  ===========

  attempted to contact: [rabbit at localhost]

  rabbit at localhost:
  * connected to epmd (port 4369) on localhost
  * epmd reports node 'rabbit' running on port 25672
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: hostname mismatch?
  * suggestion: is the cookie set correctly?

  current node details:
  - node name: rabbitmqctl20640 at stack
  - home dir: /var/lib/rabbitmq
  - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ==

If I manually stop and then start rabbitmq:

    sudo systemctl stop rabbitmq-server
    sudo systemctl start rabbitmq-server

It seems to work:

  # rabbitmqctl status
  Status of node rabbit at stack ...
  [{pid,20946},
   {running_applications,
       [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
  ...

After manually starting rabbit and re-running
instack-install-undercloud, the process is able to successfully create
the rabbitmq_user resources and completes successfully.

-- 
Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {freenode,twitter,github}
Cloud Engineering / OpenStack          | http://blog.oddbit.com/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20150421/21e8c911/attachment.sig>