[rdo-list] Issues with Ceph 0.94.6 on Mitaka and updating to 0.94.9 (Trunk promotion)

[David Moreau Simard]

Mitaka has now promoted [1]. Yay.

[1]: https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo-delorean-promote-mitaka/962/

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]


On Wed, Feb 1, 2017 at 6:55 PM, David Moreau Simard <dms at redhat.com> wrote:
> Hi,
>
> The Mitaka pipeline hasn't promoted in >70 days now [1]. That's a huge problem.
> It looks like jobs are reliably passing recently except for one which
> I could not figure out until now.
>
> The issue is the following in puppet-openstack-integration scenario001:
> Error: /Stage[main]/Openstack_integration::Provision/Glance_image[cirros]/ensure:
> change from absent to present failed: Command: 'openstack ["image",
> "create", "--format", "shell", ["cirros", "--public",
> "--container-format=bare", "--disk-format=qcow2",
> "--file=/tmp/openstack/tempest/cirros-0.3.4-x86_64-disk.img"]]' has
> been running for more than 170 seconds
>
> For some reasons still unknown, we are unable to reproduce this error
> outside of the ci.centos.org Mitaka promotion pipeline.
> Against seemingly the same packages, CentOS version, trunk
> repositories and everything else, we don't encounter this issue in the
> OpenStack gate.
> When trying to reproduce the problem manually locally, either on
> virtual machines or on a ci.centos.org duffy node, we usually run into
> other issues (ceph key injection errors).
>
> Anyway, I launched a Mitaka job manually and disabled node release --
> it predictably failed with the same Cirros image error [2].
> When trying to troubleshoot directly on the node, it's obvious that
> the issue is Ceph that is inexplicably hanging. There are no obvious
> errors in logs.
> Even bypassing OpenStack completely, some Ceph commands have a hard
> time finishing and doing a "rbd create" manually hangs.
>
> I tried a variety of things before ending up tentatively updating Ceph
> to 0.94.9 which I recently built in CBS and the issue instantly
> disappeared.
> I could create images manually with "rbd create", create images with
> glance and so on.
>
> I posted about needing a hand testing 0.94.9 on centos-devel [3].
> I would appreciate a spot check ASAP as it looks like everything else
> is green for Mitaka right now and we're long overdue for a release.
>
> Thanks !
>
> [1]: https://dashboards.rdoproject.org/rdo-dev
> [2]: https://ci.centos.org/job/weirdo-mitaka-promote-puppet-openstack-scenario001/532/
> [3]: https://lists.centos.org/pipermail/centos-devel/2017-February/015655.html
>
> David Moreau Simard
> Senior Software Engineer | Openstack RDO
>
> dmsimard = [irc, github, twitter]