[rdo-list] Issues with Ceph 0.94.6 on Mitaka and updating to 0.94.9 (Trunk promotion)

Hi, The Mitaka pipeline hasn't promoted in >70 days now [1]. That's a huge problem. It looks like jobs are reliably passing recently except for one which I could not figure out until now. The issue is the following in puppet-openstack-integration scenario001: Error: /Stage[main]/Openstack_integration::Provision/Glance_image[cirros]/ensure: change from absent to present failed: Command: 'openstack ["image", "create", "--format", "shell", ["cirros", "--public", "--container-format=bare", "--disk-format=qcow2", "--file=/tmp/openstack/tempest/cirros-0.3.4-x86_64-disk.img"]]' has been running for more than 170 seconds For some reasons still unknown, we are unable to reproduce this error outside of the ci.centos.org Mitaka promotion pipeline. Against seemingly the same packages, CentOS version, trunk repositories and everything else, we don't encounter this issue in the OpenStack gate. When trying to reproduce the problem manually locally, either on virtual machines or on a ci.centos.org duffy node, we usually run into other issues (ceph key injection errors). Anyway, I launched a Mitaka job manually and disabled node release -- it predictably failed with the same Cirros image error [2]. When trying to troubleshoot directly on the node, it's obvious that the issue is Ceph that is inexplicably hanging. There are no obvious errors in logs. Even bypassing OpenStack completely, some Ceph commands have a hard time finishing and doing a "rbd create" manually hangs. I tried a variety of things before ending up tentatively updating Ceph to 0.94.9 which I recently built in CBS and the issue instantly disappeared. I could create images manually with "rbd create", create images with glance and so on. I posted about needing a hand testing 0.94.9 on centos-devel [3]. I would appreciate a spot check ASAP as it looks like everything else is green for Mitaka right now and we're long overdue for a release. Thanks ! [1]: https://dashboards.rdoproject.org/rdo-dev [2]: https://ci.centos.org/job/weirdo-mitaka-promote-puppet-openstack-scenario... [3]: https://lists.centos.org/pipermail/centos-devel/2017-February/015655.html David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter]