4:23 p.m.
It looks like devstack hasn't been completely vetted on F20 yet (when I
ran it it printed out a warning message and asked me to override w/
FORCE=yes), but I went ahead and decided to try it anyhow.
++ get_rootwrap_location cinder
++ local module=cinder
+++ get_python_exec_prefix
+++ is_fedora
+++ [[ -z Fedora ]]
+++ '[' Fedora = Fedora ']'
+++ echo /usr/bin
++ echo /usr/bin/cinder-rootwrap
+ CINDER_ROOTWRAP=/usr/bin/cinder-rootwrap
+ [[ ! -x /usr/bin/cinder-rootwrap ]]
++ get_rootwrap_location oslo
++ local module=oslo
+++ get_python_exec_prefix
+++ is_fedora
+++ [[ -z Fedora ]]
+++ '[' Fedora = Fedora ']'
+++ echo /usr/bin
++ echo /usr/bin/oslo-rootwrap
+ CINDER_ROOTWRAP=/usr/bin/oslo-rootwrap
+ [[ ! -x /usr/bin/oslo-rootwrap ]]
+ die 180 'No suitable rootwrap found.'
+ local exitcode=0
+ set +o xtrace
[Call Trace]
./stack.sh:686:configure_cinder
/home/admin/devstack/lib/cinder:180:die
[ERROR] /home/admin/devstack/lib/cinder:180 No suitable rootwrap found.
I looked further up in the stack and saw some compilation errors related
to some compilation that requires libxml2-devel and libxslt-devel.
Turned out that I didn't have libxslt-devel installed on the VM I was
running this on. Installed that, reran stack.sh and it seemed to get
further.
Should we be checking for the presence of that package and installing it
in devstack if it is not already installed?
(fwiw, I don't have a clue as to why the failed gcc compile due to
missing devel libs would result in a missing cinder rootwrap, perhaps
someone can explain?) :)
As an aside, I also gave devstack on RHEL 6.5 a run and that seemed to
go fine as far as my cursory checking could tell (using nova networking)
Glad to see that devstack/RHEL is not falling over :)
10:38 p.m.
On 29/12/13 17:23, Perry Myers wrote:
It looks like devstack hasn't been completely vetted on F20 yet
(when I
ran it it printed out a warning message and asked me to override w/
FORCE=yes), but I went ahead and decided to try it anyhow.
Ok, consistently getting past the cinder rootwrap missing issue now that
I am installing libxslt-devel manually.
But now I'm completely hung on:
nova --os-password XXXXXXXX --os-username admin --os-tenant-name
admin x509-create-cert /home/admin/devstack/accrc/admin/admin-pk.pem
/home/admin/devstack/accrc/admin/admin-cert.pem
the vm is mostly idle as far as I can see. top shows nova-api at the
top of the list, but only 1 or 2% CPU utilization
Hm. I guess not surprising, given this in nova-api logs:
2013-12-29 22:59:35.760 INFO nova.openstack.common.rpc.common
[req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] Reconnecting to AMQP server on
localhost:5672
2013-12-29 22:59:38.769 ERROR nova.openstack.common.rpc.common
[req-832266ce-9d66-41e2-ac03-3376b383f2ab admin admin] AMQP server on localhost:5672 is
unreachable: Socket closed. Trying again in 30 seconds.
Ok, rabbitmq is running and is listening on port 5672
[admin@fm ~]$ sudo lsof -i tcp -n -P | grep LISTEN | grep 5672
beam.smp 14278 rabbitmq 16u IPv6 80540 0t0 TCP *:5672 (LISTEN)
And it is responding to incoming traffic (telnet localhost 5672 resulted
in a connection)
Well, checking the rest of the service logs, all of them are unable to
connect to the rabbitmq service. So it's not just a nova problem.
Restarting the rabbitmq-server.service doesn't seem to allow the other
services (like nova-api) to reconnect to the rabbit server. Nor does
restarting each of the individual OpenStack services
Looks like a credential issue:
=ERROR REPORT==== 29-Dec-2013::23:54:52 ===
closing AMQP connection <0.1336.0> (127.0.0.1:36009 -> 127.0.0.1:5672):
{handshake_error,starting,0,
{amqp_error,access_refused,
"AMQPLAIN login refused: user 'guest' - invalid
credentials",
'connection.start_ok'}}
In devstack you are prompted for the RabbitMQ password... I have tried
entering a password for this manually (i.e. smth like 'password') and
I've tried leaving it blank, in which case devstack will randomly
generate a password for you. In both cases, the password does propagate
to the proper config files (like nova.conf).
So... I manually ran:
[root@fm log]# rabbitmqctl change_password guest XXXXXXXX
And that cleared everything up.
So it seems like devstack is not setting the guest password for rabbitmq
correctly?
Ok, it's trying to and failing:
+ echo_summary 'Starting RabbitMQ'
+ [[ -t 3 ]]
+ echo -e Starting RabbitMQ
Starting RabbitMQ
+ is_fedora
+ [[ -z Fedora ]]
+ '[' Fedora = Fedora ']'
+ restart_service rabbitmq-server
+ is_ubuntu
+ [[ -z rpm ]]
+ '[' rpm = deb ']'
+ sudo /sbin/service rabbitmq-server restart
Redirecting to /bin/systemctl restart rabbitmq-server.service
+ sudo rabbitmqctl change_password guest password
Changing password for user "guest" ...
Error: unable to connect to node rabbit@fm: nodedown
DIAGNOSTICS
===========
nodes in question: [rabbit@fm]
hosts, their running nodes and ports:
- fm: [{rabbitmqctl14804,38208}]
current node details:
- node name: rabbitmqctl14804@fm
- home dir: /var/lib/rabbitmq
- cookie hash: c13n3S7iTV2JbeQLNCLWvQ==
The above rabbitmqctl command is called from lib/rpc_backend and is done
right after a rabbitmq restart.
I found that I had to restart the rabbitmq server manually myself w/
systemctl restart rabbitmq-server.service
before rabbitmqctl commands would work properly
So this doesn't seem like a race, but it does appear that the service
restart being done in the restart_rpc_backend function is not working
properly. It's restarting the service, but the service doesn't come
back up properly.
I also tried putting a second restart in the restart function along with
various sleeps. It seems like there is something specific about the
restart being issued from stack.sh...
Anyone else seeing this?
Perry
8:16 a.m.
On 29/12/13 17:23 -0500, Perry Myers wrote:
It looks like devstack hasn't been completely vetted on F20 yet
(when I
ran it it printed out a warning message and asked me to override w/
FORCE=yes), but I went ahead and decided to try it anyhow.
> ++ get_rootwrap_location cinder
> ++ local module=cinder
> +++ get_python_exec_prefix
> +++ is_fedora
> +++ [[ -z Fedora ]]
> +++ '[' Fedora = Fedora ']'
> +++ echo /usr/bin
> ++ echo /usr/bin/cinder-rootwrap
> + CINDER_ROOTWRAP=/usr/bin/cinder-rootwrap
> + [[ ! -x /usr/bin/cinder-rootwrap ]]
> ++ get_rootwrap_location oslo
> ++ local module=oslo
> +++ get_python_exec_prefix
> +++ is_fedora
> +++ [[ -z Fedora ]]
> +++ '[' Fedora = Fedora ']'
> +++ echo /usr/bin
> ++ echo /usr/bin/oslo-rootwrap
> + CINDER_ROOTWRAP=/usr/bin/oslo-rootwrap
> + [[ ! -x /usr/bin/oslo-rootwrap ]]
> + die 180 'No suitable rootwrap found.'
> + local exitcode=0
> + set +o xtrace
> [Call Trace]
> ./stack.sh:686:configure_cinder
> /home/admin/devstack/lib/cinder:180:die
> [ERROR] /home/admin/devstack/lib/cinder:180 No suitable rootwrap found.
I looked further up in the stack and saw some compilation errors related
to some compilation that requires libxml2-devel and libxslt-devel.
Turned out that I didn't have libxslt-devel installed on the VM I was
running this on. Installed that, reran stack.sh and it seemed to get
further.
Should we be checking for the presence of that package and installing it
in devstack if it is not already installed?
This is weird, though. We have libxml2-devel as a 'general' dependency
in devstack[0]. I checked and the package is still there. I'll give
this a try on a fresh f20.
[0] https://github.com/openstack-dev/devstack/blob/master/files/rpms/general#L10
(fwiw, I don't have a clue as to why the failed gcc compile due
to
missing devel libs would result in a missing cinder rootwrap, perhaps
someone can explain?) :)
Oslo's incubator install failed because of the missing lxml
dependency, which resulted in the oslo-rootwrap entry_point not being
created. This caused cinder configuration process to fail after trying
to use the not existent oslo-rootwrap bin.
Cheers,
FF
--
@flaper87
Flavio Percoco
11:12 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/12/13 09:16, Flavio Percoco wrote:
On 29/12/13 17:23 -0500, Perry Myers wrote:
> It looks like devstack hasn't been completely vetted on F20 yet
> (when I ran it it printed out a warning message and asked me to
> override w/ FORCE=yes), but I went ahead and decided to try it
> anyhow.
>
>> ++ get_rootwrap_location cinder ++ local module=cinder +++
>> get_python_exec_prefix +++ is_fedora +++ [[ -z Fedora ]] +++
>> '[' Fedora = Fedora ']' +++ echo /usr/bin ++ echo
>> /usr/bin/cinder-rootwrap +
>> CINDER_ROOTWRAP=/usr/bin/cinder-rootwrap + [[ ! -x
>> /usr/bin/cinder-rootwrap ]] ++ get_rootwrap_location oslo ++
>> local module=oslo +++ get_python_exec_prefix +++ is_fedora +++
>> [[ -z Fedora ]] +++ '[' Fedora = Fedora ']' +++ echo /usr/bin
>> ++ echo /usr/bin/oslo-rootwrap +
>> CINDER_ROOTWRAP=/usr/bin/oslo-rootwrap + [[ ! -x
>> /usr/bin/oslo-rootwrap ]] + die 180 'No suitable rootwrap
>> found.' + local exitcode=0 + set +o xtrace [Call Trace]
>> ./stack.sh:686:configure_cinder
>> /home/admin/devstack/lib/cinder:180:die [ERROR]
>> /home/admin/devstack/lib/cinder:180 No suitable rootwrap
>> found.
>
> I looked further up in the stack and saw some compilation errors
> related to some compilation that requires libxml2-devel and
> libxslt-devel.
>
> Turned out that I didn't have libxslt-devel installed on the VM I
> was running this on. Installed that, reran stack.sh and it
> seemed to get further.
>
> Should we be checking for the presence of that package and
> installing it in devstack if it is not already installed?
This is weird, though. We have libxml2-devel as a 'general'
dependency in devstack[0]. I checked and the package is still
there. I'll give this a try on a fresh f20.
It wasn't libxml2-devel that was missing. It was libxslt-devel that
was missing.
Though I do see both libxml2-devel AND libxslt-devel in the general
dependencies list. I wonder why it's not being installed...
+++ for line in '$(<${fname})' +++ [[ libxml2-devel #
dist:rhel6
[2] =~ NOPRIME ]] +++ package='libxml2-devel ' +++ inst_pkg=1 +++
[[ libxml2-devel # dist:rhel6 [2] =~ (.*)#.*dist:([^ ]*) ]] +++
package='libxml2-devel ' +++ distros=rhel6 +++ [[ ! rhel6 =~ f20
]] +++ inst_pkg=0 +++ [[ libxml2-devel # dist:rhel6 [2] =~
(.*)#.*testonly.* ]] +++ [[ 0 = 1 ]] +++ for line in
'$(<${fname})' +++ [[ libxslt-devel # dist:rhel6 [2] =~ NOPRIME ]]
+++ package='libxslt-devel ' +++ inst_pkg=1 +++ [[ libxslt-devel #
dist:rhel6 [2] =~ (.*)#.*dist:([^ ]*) ]] +++ package='libxslt-devel
' +++ distros=rhel6 +++ [[ ! rhel6 =~ f20 ]] +++ inst_pkg=0 +++ [[
libxslt-devel # dist:rhel6 [2] =~ (.*)#.*testonly.* ]] +++ [[ 0 = 1
]]
Hm, it looks like it's not being installed because the dependency list
says:
libxml2-devel # dist:rhel6 [2]
libxslt-devel # dist:rhel6 [2]
And I think this means that these deps should only be installed on
rhel6 machines. Perhaps we need to remove dist:rhel6 from those lines?
[0]
https://github.com/openstack-dev/devstack/blob/master/files/rpms/general#L10
> (fwiw, I don't have a clue as to why the failed gcc compile due
> to missing devel libs would result in a missing cinder rootwrap,
> perhaps someone can explain?) :)
Oslo's incubator install failed because of the missing lxml
dependency, which resulted in the oslo-rootwrap entry_point not
being created. This caused cinder configuration process to fail
after trying to use the not existent oslo-rootwrap bin.
Ah, thanks for the explanation :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlLBqZQACgkQxdKLkeZeTz2uIACbB6t+WsvSjNv/6ajvHH00stc8
9S4An0nKYRj2R4MuliJ2ikWRUCGRYw7I
=7nSQ
-----END PGP SIGNATURE-----
3776
days inactive
3777
days old
3 comments
2 participants
participants (2)
-
Flavio Percoco -
Perry Myers