[Rdo-list] Improving RDO continuous integration/testing

[David Moreau Simard]

Hi,

Continuous integration jobs for RDO trunk (liberty) are not in good
shape right now [1] and they also provide poor coverage.
The only test that is run is tempest.scenario.test_server_basic_ops
[2] which adds a ssh-keypair.

As part of our efforts to speed up the release process of RDO and
improve the quality and stability of what we ship, I am working to
improve the CI.
The good news is that I've managed to get promising results with
khaleesi+rdo+packstack+aio with selinux permissive locally (selinux
enforced being blocked right now [3]) so we are on the right track.

As I've been trying to improve test coverage, a good first step would
be to enforce tempest smoke tests.
However, I've noticed that khaleesi uses a fork of tempest [4] and
this generated a failure [5] on a test that has since been fixed
upstream [6].

I'm very concerned about testing trunk RDO against a fork of tempest.
We should be testing trunk RDO against trunk tempest.
Running against a fork means we might lack some important changes to
test coverage or can unnecessarily encounter failures which have
already been resolved upstream.

My understanding is that Red Hat maintains a fork of tempest to run
test suites against products which have a longer release and support
cycles and that is fine.
Should we switch RDO CI testing to the upstream branches ?

Thanks,

[1]: https://prod-rdojenkins.rhcloud.com/view/RDO-Liberty-Delorean-Trunk/
[2]: https://github.com/openstack/tempest/blob/master/tempest/scenario/test_server_basic_ops.py
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1249685
[4]: https://github.com/redhat-openstack/khaleesi-settings/blob/master/settings/tester/tempest/setup/git.yml#L4
[5]: http://paste.openstack.org/show/463360/
[6]: https://github.com/openstack/tempest/commit/986b9e6fda6c76806a329bd53f5f73f557da903e#diff-06c9ec2d3ed36f96bfce6f75e82a4f45

David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]