[Rdo-list] devstack on F20 (cinder: No suitable rootwrap found)

Perry Myers pmyers at redhat.com
Mon Dec 30 04:38:52 UTC 2013


On 29/12/13 17:23, Perry Myers wrote:
> It looks like devstack hasn't been completely vetted on F20 yet (when I
> ran it it printed out a warning message and asked me to override w/
> FORCE=yes), but I went ahead and decided to try it anyhow.

Ok, consistently getting past the cinder rootwrap missing issue now that
I am installing libxslt-devel manually.

But now I'm completely hung on:

> nova --os-password XXXXXXXX --os-username admin --os-tenant-name admin x509-create-cert /home/admin/devstack/accrc/admin/admin-pk.pem /home/admin/devstack/accrc/admin/admin-cert.pem

the vm is mostly idle as far as I can see.  top shows nova-api at the
top of the list, but only 1 or 2% CPU utilization

Hm.  I guess not surprising, given this in nova-api logs:

> 2013-12-29 22:59:35.760 INFO nova.openstack.common.rpc.common [req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] Reconnecting to AMQP server on localhost:5672
> 2013-12-29 22:59:38.769 ERROR nova.openstack.common.rpc.common [req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] AMQP server on localhost:5672 is unreachable: Socket closed. Trying again in 30 seconds.

Ok, rabbitmq is running and is listening on port 5672

> [admin at fm ~]$ sudo lsof -i tcp -n -P | grep LISTEN | grep 5672
> beam.smp  14278 rabbitmq   16u  IPv6  80540      0t0  TCP *:5672 (LISTEN)

And it is responding to incoming traffic (telnet localhost 5672 resulted
in a connection)

Well, checking the rest of the service logs, all of them are unable to
connect to the rabbitmq service.  So it's not just a nova problem.

Restarting the rabbitmq-server.service doesn't seem to allow the other
services (like nova-api) to reconnect to the rabbit server.  Nor does
restarting each of the individual OpenStack services

Looks like a credential issue:

> =ERROR REPORT==== 29-Dec-2013::23:54:52 ===
> closing AMQP connection <0.1336.0> (127.0.0.1:36009 -> 127.0.0.1:5672):
> {handshake_error,starting,0,
>                  {amqp_error,access_refused,
>                              "AMQPLAIN login refused: user 'guest' - invalid credentials",
>                              'connection.start_ok'}}

In devstack you are prompted for the RabbitMQ password... I have tried
entering a password for this manually (i.e. smth like 'password') and
I've tried leaving it blank, in which case devstack will randomly
generate a password for you.  In both cases, the password does propagate
to the proper config files (like nova.conf).

So... I manually ran:

> [root at fm log]# rabbitmqctl change_password guest XXXXXXXX

And that cleared everything up.

So it seems like devstack is not setting the guest password for rabbitmq
correctly?

Ok, it's trying to and failing:

> + echo_summary 'Starting RabbitMQ'
> + [[ -t 3 ]]
> + echo -e Starting RabbitMQ
> Starting RabbitMQ
> + is_fedora
> + [[ -z Fedora ]]
> + '[' Fedora = Fedora ']'
> + restart_service rabbitmq-server
> + is_ubuntu
> + [[ -z rpm ]]
> + '[' rpm = deb ']'
> + sudo /sbin/service rabbitmq-server restart
> Redirecting to /bin/systemctl restart  rabbitmq-server.service
> + sudo rabbitmqctl change_password guest password
> Changing password for user "guest" ...
> Error: unable to connect to node rabbit at fm: nodedown
> 
> DIAGNOSTICS
> ===========
> 
> nodes in question: [rabbit at fm]
> 
> hosts, their running nodes and ports:
> - fm: [{rabbitmqctl14804,38208}]
> 
> current node details:
> - node name: rabbitmqctl14804 at fm
> - home dir: /var/lib/rabbitmq
> - cookie hash: c13n3S7iTV2JbeQLNCLWvQ==
> 

The above rabbitmqctl command is called from lib/rpc_backend and is done
right after a rabbitmq restart.

I found that I had to restart the rabbitmq server manually myself w/

systemctl restart rabbitmq-server.service

before rabbitmqctl commands would work properly

So this doesn't seem like a race, but it does appear that the service
restart being done in the restart_rpc_backend function is not working
properly.  It's restarting the service, but the service doesn't come
back up properly.

I also tried putting a second restart in the restart function along with
various sleeps.  It seems like there is something specific about the
restart being issued from stack.sh...

Anyone else seeing this?

Perry




More information about the dev mailing list