[Rdo-list] devstack on F20 (cinder: No suitable rootwrap found)

[Perry Myers]

On 29/12/13 17:23, Perry Myers wrote:
> It looks like devstack hasn't been completely vetted on F20 yet (when I
> ran it it printed out a warning message and asked me to override w/
> FORCE=yes), but I went ahead and decided to try it anyhow.

Ok, consistently getting past the cinder rootwrap missing issue now that
I am installing libxslt-devel manually.

But now I'm completely hung on:

> nova --os-password XXXXXXXX --os-username admin --os-tenant-name admin x509-create-cert /home/admin/devstack/accrc/admin/admin-pk.pem /home/admin/devstack/accrc/admin/admin-cert.pem

the vm is mostly idle as far as I can see.  top shows nova-api at the
top of the list, but only 1 or 2% CPU utilization

Hm.  I guess not surprising, given this in nova-api logs:

> 2013-12-29 22:59:35.760 INFO nova.openstack.common.rpc.common [req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] Reconnecting to AMQP server on localhost:5672
> 2013-12-29 22:59:38.769 ERROR nova.openstack.common.rpc.common [req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] AMQP server on localhost:5672 is unreachable: Socket closed. Trying again in 30 seconds.

Ok, rabbitmq is running and is listening on port 5672

> [admin at fm ~]$ sudo lsof -i tcp -n -P | grep LISTEN | grep 5672
> beam.smp  14278 rabbitmq   16u  IPv6  80540      0t0  TCP *:5672 (LISTEN)

And it is responding to incoming traffic (telnet localhost 5672 resulted
in a connection)

Well, checking the rest of the service logs, all of them are unable to
connect to the rabbitmq service.  So it's not just a nova problem.

Restarting the rabbitmq-server.service doesn't seem to allow the other
services (like nova-api) to reconnect to the rabbit server.  Nor does
restarting each of the individual OpenStack services

Looks like a credential issue:

> =ERROR REPORT==== 29-Dec-2013::23:54:52 ===
> closing AMQP connection <0.1336.0> (127.0.0.1:36009 -> 127.0.0.1:5672):
> {handshake_error,starting,0,
>                  {amqp_error,access_refused,
>                              "AMQPLAIN login refused: user 'guest' - invalid credentials",
>                              'connection.start_ok'}}

In devstack you are prompted for the RabbitMQ password... I have tried
entering a password for this manually (i.e. smth like 'password') and
I've tried leaving it blank, in which case devstack will randomly
generate a password for you.  In both cases, the password does propagate
to the proper config files (like nova.conf).

So... I manually ran:

> [root at fm log]# rabbitmqctl change_password guest XXXXXXXX

And that cleared everything up.

So it seems like devstack is not setting the guest password for rabbitmq
correctly?

Ok, it's trying to and failing:

> + echo_summary 'Starting RabbitMQ'
> + [[ -t 3 ]]
> + echo -e Starting RabbitMQ
> Starting RabbitMQ
> + is_fedora
> + [[ -z Fedora ]]
> + '[' Fedora = Fedora ']'
> + restart_service rabbitmq-server
> + is_ubuntu
> + [[ -z rpm ]]
> + '[' rpm = deb ']'
> + sudo /sbin/service rabbitmq-server restart
> Redirecting to /bin/systemctl restart  rabbitmq-server.service
> + sudo rabbitmqctl change_password guest password
> Changing password for user "guest" ...
> Error: unable to connect to node rabbit at fm: nodedown
> 
> DIAGNOSTICS
> ===========
> 
> nodes in question: [rabbit at fm]
> 
> hosts, their running nodes and ports:
> - fm: [{rabbitmqctl14804,38208}]
> 
> current node details:
> - node name: rabbitmqctl14804 at fm
> - home dir: /var/lib/rabbitmq
> - cookie hash: c13n3S7iTV2JbeQLNCLWvQ==
> 

The above rabbitmqctl command is called from lib/rpc_backend and is done
right after a rabbitmq restart.

I found that I had to restart the rabbitmq server manually myself w/

systemctl restart rabbitmq-server.service

before rabbitmqctl commands would work properly

So this doesn't seem like a race, but it does appear that the service
restart being done in the restart_rpc_backend function is not working
properly.  It's restarting the service, but the service doesn't come
back up properly.

I also tried putting a second restart in the restart function along with
various sleeps.  It seems like there is something specific about the
restart being issued from stack.sh...

Anyone else seeing this?

Perry