[rdo-list] Issues with Ceph 0.94.6 on Mitaka and updating to 0.94.9 (Trunk promotion)

[David Moreau Simard]

Hi,

The Mitaka pipeline hasn't promoted in >70 days now [1]. That's a huge problem.
It looks like jobs are reliably passing recently except for one which
I could not figure out until now.

The issue is the following in puppet-openstack-integration scenario001:
Error: /Stage[main]/Openstack_integration::Provision/Glance_image[cirros]/ensure:
change from absent to present failed: Command: 'openstack ["image",
"create", "--format", "shell", ["cirros", "--public",
"--container-format=bare", "--disk-format=qcow2",
"--file=/tmp/openstack/tempest/cirros-0.3.4-x86_64-disk.img"]]' has
been running for more than 170 seconds

For some reasons still unknown, we are unable to reproduce this error
outside of the ci.centos.org Mitaka promotion pipeline.
Against seemingly the same packages, CentOS version, trunk
repositories and everything else, we don't encounter this issue in the
OpenStack gate.
When trying to reproduce the problem manually locally, either on
virtual machines or on a ci.centos.org duffy node, we usually run into
other issues (ceph key injection errors).

Anyway, I launched a Mitaka job manually and disabled node release --
it predictably failed with the same Cirros image error [2].
When trying to troubleshoot directly on the node, it's obvious that
the issue is Ceph that is inexplicably hanging. There are no obvious
errors in logs.
Even bypassing OpenStack completely, some Ceph commands have a hard
time finishing and doing a "rbd create" manually hangs.

I tried a variety of things before ending up tentatively updating Ceph
to 0.94.9 which I recently built in CBS and the issue instantly
disappeared.
I could create images manually with "rbd create", create images with
glance and so on.

I posted about needing a hand testing 0.94.9 on centos-devel [3].
I would appreciate a spot check ASAP as it looks like everything else
is green for Mitaka right now and we're long overdue for a release.

Thanks !

[1]: https://dashboards.rdoproject.org/rdo-dev
[2]: https://ci.centos.org/job/weirdo-mitaka-promote-puppet-openstack-scenario001/532/
[3]: https://lists.centos.org/pipermail/centos-devel/2017-February/015655.html

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]