Re: [Rdo-list] rdo-manager failures: instack-install-undercloud failing for non-obvious reasons

On 21.4.2015 23:00, James Slagle wrote: > On Tue, Apr 21, 2015 at 04:37:26PM -0400, Lars Kellogg-Stedman wrote: >> Running "instack-install-undercloud" is failing for me: >> >> + echo 'puppet apply exited with exit code 6' >> puppet apply exited with exit code 6 >> + '[' 6 '!=' 2 -a 6 '!=' 0 ']' >> + exit 6 >> [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure >> phase. [Command '['dib-run-parts', >> '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit >> status 6] >> >> Unfortunately, the failure doesn't provide much in the way of useful >> information. If I scroll up several pages, I find: >> >> Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff' >> Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired! >> Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired! >> Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired! >> Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired! >> Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired! >> >> But again, that doesn't really tell me what is failing either (a >> command is still failing? Which command?). > > Unfortunately we're pretty much at the mercy of puppet and all of the external > puppet modules here in terms of its helpful output, and the point at which it > chooses to stop applying after an error is encountered. Perhaps some people > more familiar with puppet might chime in here on how to improve this. Changing "puppet apply" to "puppet apply -d" here [1] should give you more output including the commands which are being run at each step. (Hoping i've found the right spot, i'm not very familiar with instack-undercloud.) Perhaps an env variable could be added to switch on the debug output? (It shouldn't be on by default i guess because Puppet prints a lot of stuff then, including potentially sensitive info.) Regarding the problem as a whole, the hostname/domain issue you outlined in another e-mail might be a good clue. It certainly wouldn't be the first time i've seen problems with Puppet caused by FQDN settings. In general it's a good idea to verify that `facter fqdn` prints the same thing as `hostname -f` before running Puppet.

Cheers J. [1] https://github.com/rdo-management/instack-undercloud/blob/6f75c8dc3c37d48... > >> >> It looks like rabbitmq is having some problems: >> >> [stack@localhost ~]$ sudo rabbitmqctl status >> Status of node rabbit@localhost ... >> Error: unable to connect to node rabbit@localhost: nodedown >> >> DIAGNOSTICS >> =========== >> >> attempted to contact: [rabbit@localhost] >> >> rabbit@localhost: >> * connected to epmd (port 4369) on localhost >> * epmd reports node 'rabbit' running on port 25672 >> * TCP connection succeeded but Erlang distribution failed >> * suggestion: hostname mismatch? >> * suggestion: is the cookie set correctly? >> >> current node details: >> - node name: rabbitmqctl20640@stack >> - home dir: /var/lib/rabbitmq >> - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ== >> >> If I manually stop and then start rabbitmq: >> >> sudo systemctl stop rabbitmq-server >> sudo systemctl start rabbitmq-server >> >> It seems to work: >> >> # rabbitmqctl status >> Status of node rabbit@stack ... >> [{pid,20946}, >> {running_applications, >> [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"}, >> ... >> >> After manually starting rabbit and re-running >> instack-install-undercloud, the process is able to successfully create >> the rabbitmq_user resources and completes successfully. > > Are you on RHEL 7.1 or CentOS 7? I'll try to reproduce locally and see if I can > get to the bottom of it. > >> >> -- >> Lars Kellogg-Stedman <lars(a)redhat.com&gt; | larsks @ {freenode,twitter,github} >> Cloud Engineering / OpenStack | http://blog.oddbit.com/ > > > >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > -- > -- James Slagle > -- > > _______________________________________________ > Rdo-list mailing list > Rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com