[Rdo-list] rdo-manager failures: instack-install-undercloud failing for non-obvious reasons

[Jiří Stránský]

On 21.4.2015 23:00, James Slagle wrote:
> On Tue, Apr 21, 2015 at 04:37:26PM -0400, Lars Kellogg-Stedman wrote:
>> Running "instack-install-undercloud" is failing for me:
>>
>>    + echo 'puppet apply exited with exit code 6'
>>    puppet apply exited with exit code 6
>>    + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
>>    + exit 6
>>    [2015-04-21 20:13:20,426] (os-refresh-config) [ERROR] during configure
>>    phase. [Command '['dib-run-parts',
>>    '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit
>>    status 6]
>>
>> Unfortunately, the failure doesn't provide much in the way of useful
>> information.  If I scroll up several pages, I find:
>>
>>    Notice: /Stage[main]/Rabbitmq::Install::Rabbitmqadmin/File[/usr/local/bin/rabbitmqadmin]/ensure: defined content as '{md5}63d7331e825c865a97b7a8d1299841ff'
>>    Error: /Stage[main]/Main/Rabbitmq_user[neutron]: Could not evaluate: Command is still failing after 180 seconds expired!
>>    Error: /Stage[main]/Main/Rabbitmq_user[heat]: Could not evaluate: Command is still failing after 180 seconds expired!
>>    Error: /Stage[main]/Main/Rabbitmq_user[ceilometer]: Could not evaluate: Command is still failing after 180 seconds expired!
>>    Error: /Stage[main]/Main/Rabbitmq_user[nova]: Could not evaluate: Command is still failing after 180 seconds expired!
>>    Error: /Stage[main]/Main/Rabbitmq_vhost[/]: Could not evaluate: Command is still failing after 180 seconds expired!
>>
>> But again, that doesn't really tell me what is failing either (a
>> command is still failing? Which command?).
>
> Unfortunately we're pretty much at the mercy of puppet and all of the external
> puppet modules here in terms of its helpful output, and the point at which it
> chooses to stop applying after an error is encountered. Perhaps some people
> more familiar with puppet might chime in here on how to improve this.

Changing "puppet apply" to "puppet apply -d" here [1] should give you 
more output including the commands which are being run at each step. 
(Hoping i've found the right spot, i'm not very familiar with 
instack-undercloud.) Perhaps an env variable could be added to switch on 
the debug output? (It shouldn't be on by default i guess because Puppet 
prints a lot of stuff then, including potentially sensitive info.)

Regarding the problem as a whole, the hostname/domain issue you outlined 
in another e-mail might be a good clue. It certainly wouldn't be the 
first time i've seen problems with Puppet caused by FQDN settings. In 
general it's a good idea to verify that `facter fqdn` prints the same 
thing as `hostname -f` before running Puppet.

Cheers

J.

[1] 
https://github.com/rdo-management/instack-undercloud/blob/6f75c8dc3c37d489763b7310a7b57d00e1e70da2/elements/puppet-stack-config/os-refresh-config/configure.d/50-puppet-stack-config#L7

>
>>
>> It looks like rabbitmq is having some problems:
>>
>>    [stack at localhost ~]$ sudo rabbitmqctl status
>>    Status of node rabbit at localhost ...
>>    Error: unable to connect to node rabbit at localhost: nodedown
>>
>>    DIAGNOSTICS
>>    ===========
>>
>>    attempted to contact: [rabbit at localhost]
>>
>>    rabbit at localhost:
>>    * connected to epmd (port 4369) on localhost
>>    * epmd reports node 'rabbit' running on port 25672
>>    * TCP connection succeeded but Erlang distribution failed
>>    * suggestion: hostname mismatch?
>>    * suggestion: is the cookie set correctly?
>>
>>    current node details:
>>    - node name: rabbitmqctl20640 at stack
>>    - home dir: /var/lib/rabbitmq
>>    - cookie hash: 4DA3U2yua3rw7wYLr+PbiQ==
>>
>> If I manually stop and then start rabbitmq:
>>
>>      sudo systemctl stop rabbitmq-server
>>      sudo systemctl start rabbitmq-server
>>
>> It seems to work:
>>
>>    # rabbitmqctl status
>>    Status of node rabbit at stack ...
>>    [{pid,20946},
>>     {running_applications,
>>         [{rabbitmq_management,"RabbitMQ Management Console","3.3.5"},
>>    ...
>>
>> After manually starting rabbit and re-running
>> instack-install-undercloud, the process is able to successfully create
>> the rabbitmq_user resources and completes successfully.
>
> Are you on RHEL 7.1 or CentOS 7? I'll try to reproduce locally and see if I can
> get to the bottom of it.
>
>>
>> --
>> Lars Kellogg-Stedman <lars at redhat.com> | larsks @ {freenode,twitter,github}
>> Cloud Engineering / OpenStack          | http://blog.oddbit.com/
>
>
>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
> --
> -- James Slagle
> --
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com
>