10:38 p.m.
On 29/12/13 17:23, Perry Myers wrote:
It looks like devstack hasn't been completely vetted on F20 yet
(when I
ran it it printed out a warning message and asked me to override w/
FORCE=yes), but I went ahead and decided to try it anyhow.
Ok, consistently getting past the cinder rootwrap missing issue now that
I am installing libxslt-devel manually.
But now I'm completely hung on:
nova --os-password XXXXXXXX --os-username admin --os-tenant-name
admin x509-create-cert /home/admin/devstack/accrc/admin/admin-pk.pem
/home/admin/devstack/accrc/admin/admin-cert.pem
the vm is mostly idle as far as I can see. top shows nova-api at the
top of the list, but only 1 or 2% CPU utilization
Hm. I guess not surprising, given this in nova-api logs:
2013-12-29 22:59:35.760 INFO nova.openstack.common.rpc.common
[req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] Reconnecting to AMQP server on
localhost:5672
2013-12-29 22:59:38.769 ERROR nova.openstack.common.rpc.common
[req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] AMQP server on localhost:5672 is
unreachable: Socket closed. Trying again in 30 seconds.
Ok, rabbitmq is running and is listening on port 5672
[admin@fm ~]$ sudo lsof -i tcp -n -P | grep LISTEN | grep 5672
beam.smp 14278 rabbitmq 16u IPv6 80540 0t0 TCP *:5672 (LISTEN)
And it is responding to incoming traffic (telnet localhost 5672 resulted
in a connection)
Well, checking the rest of the service logs, all of them are unable to
connect to the rabbitmq service. So it's not just a nova problem.
Restarting the rabbitmq-server.service doesn't seem to allow the other
services (like nova-api) to reconnect to the rabbit server. Nor does
restarting each of the individual OpenStack services
Looks like a credential issue:
=ERROR REPORT==== 29-Dec-2013::23:54:52 ===
closing AMQP connection <0.1336.0> (127.0.0.1:36009 -> 127.0.0.1:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"AMQPLAIN login refused: user 'guest' - invalid
credentials",
'connection.start_ok'}}
In devstack you are prompted for the RabbitMQ password... I have tried
entering a password for this manually (i.e. smth like 'password') and
I've tried leaving it blank, in which case devstack will randomly
generate a password for you. In both cases, the password does propagate
to the proper config files (like nova.conf).
So... I manually ran:
[root@fm log]# rabbitmqctl change_password guest XXXXXXXX
And that cleared everything up.
So it seems like devstack is not setting the guest password for rabbitmq
correctly?
Ok, it's trying to and failing:
+ echo_summary 'Starting RabbitMQ'
+ [[ -t 3 ]]
+ echo -e Starting RabbitMQ
Starting RabbitMQ
+ is_fedora
+ [[ -z Fedora ]]
+ '[' Fedora = Fedora ']'
+ restart_service rabbitmq-server
+ is_ubuntu
+ [[ -z rpm ]]
+ '[' rpm = deb ']'
+ sudo /sbin/service rabbitmq-server restart
Redirecting to /bin/systemctl restart rabbitmq-server.service
+ sudo rabbitmqctl change_password guest password
Changing password for user "guest" ...
Error: unable to connect to node rabbit@fm: nodedown
DIAGNOSTICS
===========
nodes in question: [rabbit@fm]
hosts, their running nodes and ports:
- fm: [{rabbitmqctl14804,38208}]
current node details:
- node name: rabbitmqctl14804@fm
- home dir: /var/lib/rabbitmq
- cookie hash: c13n3S7iTV2JbeQLNCLWvQ==
The above rabbitmqctl command is called from lib/rpc_backend and is done
right after a rabbitmq restart.
I found that I had to restart the rabbitmq server manually myself w/
systemctl restart rabbitmq-server.service
before rabbitmqctl commands would work properly
So this doesn't seem like a race, but it does appear that the service
restart being done in the restart_rpc_backend function is not working
properly. It's restarting the service, but the service doesn't come
back up properly.
I also tried putting a second restart in the restart function along with
various sleeps. It seems like there is something specific about the
restart being issued from stack.sh...
Anyone else seeing this?
Perry