Running "instack-install-undercloud" is failing for me: + echo 'puppet apply exited with exit code 6' puppet apply exited with exit code 6 + '[' 6 '!=' 2 -a 6 '!=' 0 ']' + exit 6 [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6] Unfortunately, the failure doesn't provide much in the way of useful information. If I scroll up several pages, I find: Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff' Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired! Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired! Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired! Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired! Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired! But again, that doesn't really tell me what is failing either (a command is still failing? Which command?).

It looks like rabbitmq is having some problems: [stack@localhost ~]$ sudo rabbitmqctl status Status of node rabbit@localhost ... Error: unable to connect to node rabbit@localhost: nodedown DIAGNOSTICS =========== attempted to contact: [rabbit@localhost] rabbit@localhost: * connected to epmd (port 4369) on localhost * epmd reports node 'rabbit' running on port 25672 * TCP connection succeeded but Erlang distribution failed * suggestion: hostname mismatch? * suggestion: is the cookie set correctly? current node details: - node name: rabbitmqctl20640@stack - home dir: /var/lib/rabbitmq - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ== If I manually stop and then start rabbitmq: sudo systemctl stop rabbitmq-server sudo systemctl start rabbitmq-server It seems to work: # rabbitmqctl status Status of node rabbit@stack ... [{pid,20946}, {running_applications, [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"}, ... After manually starting rabbit and re-running instack-install-undercloud, the process is able to successfully create the rabbitmq_user resources and completes successfully.

-- Lars Kellogg-Stedman <lars(a)redhat.com&gt; | larsks @ {freenode,twitter,github} Cloud Engineering / OpenStack | http://blog.oddbit.com/

_______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On Tue, Apr 21, 2015 at 04:37:26PM -0400, Lars Kellogg-Stedman wrote: > Running "instack-install-undercloud" is failing for me: > > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure > phase. [Command '['dib-run-parts', > '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit > status 6] > > Unfortunately, the failure doesn't provide much in the way of useful > information. If I scroll up several pages, I find: > > Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff' > Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired! > Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired! > Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired! > Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired! > Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired! > > But again, that doesn't really tell me what is failing either (a > command is still failing? Which command?). Unfortunately we're pretty much at the mercy of puppet and all of the external puppet modules here in terms of its helpful output, and the point at which it chooses to stop applying after an error is encountered. Perhaps some people more familiar with puppet might chime in here on how to improve this.

> > It looks like rabbitmq is having some problems: > > [stack@localhost ~]$ sudo rabbitmqctl status > Status of node rabbit@localhost ... > Error: unable to connect to node rabbit@localhost: nodedown > > DIAGNOSTICS > =========== > > attempted to contact: [rabbit@localhost] > > rabbit@localhost: > * connected to epmd (port 4369) on localhost > * epmd reports node 'rabbit' running on port 25672 > * TCP connection succeeded but Erlang distribution failed > * suggestion: hostname mismatch? > * suggestion: is the cookie set correctly? > > current node details: > - node name: rabbitmqctl20640@stack > - home dir: /var/lib/rabbitmq > - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ== > > If I manually stop and then start rabbitmq: > > sudo systemctl stop rabbitmq-server > sudo systemctl start rabbitmq-server > > It seems to work: > > # rabbitmqctl status > Status of node rabbit@stack ... > [{pid,20946}, > {running_applications, > [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"}, > ... > > After manually starting rabbit and re-running > instack-install-undercloud, the process is able to successfully create > the rabbitmq_user resources and completes successfully. Are you on RHEL 7.1 or CentOS 7? I'll try to reproduce locally and see if I can get to the bottom of it. > > -- > Lars Kellogg-Stedman <lars(a)redhat.com&gt; | larsks @ {freenode,twitter,github} > Cloud Engineering / OpenStack | http://blog.oddbit.com/ > _______________________________________________ > Rdo-list mailing list > Rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com -- -- James Slagle -- _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On 22.4.2015 17:05, Jiří Stránský wrote: >On 21.4.2015 23:00, James Slagle wrote: >>On Tue, Apr 21, 2015 at 04:37:26PM -0400, Lars Kellogg-Stedman wrote: >>>Running "instack-install-undercloud" is failing for me: >>> >>> + echo 'puppet apply exited with exit code 6' >>> puppet apply exited with exit code 6 >>> + '[' 6 '!=' 2 -a 6 '!=' 0 ']' >>> + exit 6 >>> [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure >>> phase. [Command '['dib-run-parts', >>> '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit >>> status 6] >>> >>>Unfortunately, the failure doesn't provide much in the way of useful >>>information. If I scroll up several pages, I find: >>> >>> Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff' >>> Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired! >>> Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired! >>> Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired! >>> Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired! >>> Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired! >>> >>>But again, that doesn't really tell me what is failing either (a >>>command is still failing? Which command?). >> >>Unfortunately we're pretty much at the mercy of puppet and all of the external >>puppet modules here in terms of its helpful output, and the point at which it >>chooses to stop applying after an error is encountered. Perhaps some people >>more familiar with puppet might chime in here on how to improve this. > >Changing "puppet apply" to "puppet apply -d" here [1] should give you >more output including the commands which are being run at each step. >(Hoping i've found the right spot, i'm not very familiar with >instack-undercloud.) Perhaps an env variable could be added to switch on >the debug output? (It shouldn't be on by default i guess because Puppet >prints a lot of stuff then, including potentially sensitive info.) > >Regarding the problem as a whole, the hostname/domain issue you outlined >in another e-mail might be a good clue. It certainly wouldn't be the >first time i've seen problems with Puppet caused by FQDN settings. In >general it's a good idea to verify that `facter fqdn` prints the same >thing as `hostname -f` before running Puppet. Actually, now i recall that staypuft-installer has a check for this ^^ built in, and if the two values don't match, it refuses to run. Maybe we should do the same with instack-undercloud.

J. > >Cheers > >J. > >[1] >https://github.com/rdo-management/instack-undercloud/blob/6f75c8dc3c37d489763b7310a7b57d00e1e70da2/elements/puppet-stack-config/os-refresh-config/configure.d/50-puppet-stack-config#L7 > >> >>> >>>It looks like rabbitmq is having some problems: >>> >>> [stack@localhost ~]$ sudo rabbitmqctl status >>> Status of node rabbit@localhost ... >>> Error: unable to connect to node rabbit@localhost: nodedown >>> >>> DIAGNOSTICS >>> =========== >>> >>> attempted to contact: [rabbit@localhost] >>> >>> rabbit@localhost: >>> * connected to epmd (port 4369) on localhost >>> * epmd reports node 'rabbit' running on port 25672 >>> * TCP connection succeeded but Erlang distribution failed >>> * suggestion: hostname mismatch? >>> * suggestion: is the cookie set correctly? >>> >>> current node details: >>> - node name: rabbitmqctl20640@stack >>> - home dir: /var/lib/rabbitmq >>> - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ== >>> >>>If I manually stop and then start rabbitmq: >>> >>> sudo systemctl stop rabbitmq-server >>> sudo systemctl start rabbitmq-server >>> >>>It seems to work: >>> >>> # rabbitmqctl status >>> Status of node rabbit@stack ... >>> [{pid,20946}, >>> {running_applications, >>> [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"}, >>> ... >>> >>>After manually starting rabbit and re-running >>>instack-install-undercloud, the process is able to successfully create >>>the rabbitmq_user resources and completes successfully. >> >>Are you on RHEL 7.1 or CentOS 7? I'll try to reproduce locally and see if I can >>get to the bottom of it. >> >>> >>>-- >>>Lars Kellogg-Stedman <lars(a)redhat.com&gt; | larsks @ {freenode,twitter,github} >>>Cloud Engineering / OpenStack | http://blog.oddbit.com/ >> >> >> >>>_______________________________________________ >>>Rdo-list mailing list &gt;&gt;&gt;Rdo-list(a)redhat.com >>>https://www.redhat.com/mailman/listinfo/rdo-list >>> >>>To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >>-- >>-- James Slagle >>-- >> >>_______________________________________________ >>Rdo-list mailing list &gt;&gt;Rdo-list(a)redhat.com >>https://www.redhat.com/mailman/listinfo/rdo-list >> >>To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> > >_______________________________________________ >Rdo-list mailing list &gt;Rdo-list(a)redhat.com >https://www.redhat.com/mailman/listinfo/rdo-list > >To unsubscribe: rdo-list-unsubscribe(a)redhat.com >