[rdo-list] Future of Tempest packaging in RDO

[Chandan kumar]

Hi,

Tempest is the OpenStack integration testing suite which is used by
upstream CI to test functionality against each patchset submitted.

RDO provides:
* RPM of modified version of tempest containing the additional script
for generating tempest config and configuring tempest easily[VI].
* test sub packages and tempest-service packages for all OpenStack
services tempest plugins

These were consumed by upstream puppet CI, Weirdo, and OOOQ CI.

Currently, below are the lists of problem associated with it:

[1]. In upstream puppet CI,OpenStack puppet services modules are
tested using openstack-puppet-integration[I].
     which installs tempest from git source and tempest-services
plugins are installed through rpm using puppet-tempest.

     The problem here is it allows mixing installing tempest from git
and plugins as RPM which is not a configuration that should be
supported by RDO.
     and one of the side effects is you can't use tempest plugins from
pure packages unless installing separately tempest.

[2]. In OOOQ CI, It uses RDO tempest rpm that installs all the test
sub packages.

     The tempest rpm is currently outdated in terms of Upstream
Tempest and some of the functionality which works with the tempest
rpm,
     might not work with upstream so we are again ending up filling bugs.

[3.] Circular dependency problems with tempest [II].
The tempest rpm installs all the tempest plugins and test sub packages.
Suppose you install the undercloud, try to run tempest from it. You
have many packages with the common code for the various services (say:
python-ceilometer) which
expose the entry point for code which is not available (as it is in
python-ceilometer-tests).When you try to run testr list-tests (or
other discovery wrappers) and you get tons
of errors, until you figure out and install all the required -tests package.
This was and is confusing for many people and lead to bugs again [III].

[4.] Problems in installing and testing a tempest plugins separately.
For now, only two (designate and horizon) tempest-plugin packages
require openstack-tempest [IV][V].
So for installing and testing a tempest plugin, we will again end up
with the problem [3]

Another case is that Tempest plugins differ from each other in terms
of behaviour: some of them need to be specifically disabled if
installed
but not used, others require specific configuration to run (or not).
It causes another problem while testing it.

These are the above lists of problems we found in current tempest rpm
and others might have some more
We are looking forward to discussing the same in order to find a
better solution for everyone.

Links:
[I]. https://github.com/openstack/puppet-openstack-integration/blob/master/run_tests.sh#L85
[II]. https://review.rdoproject.org/r/#/c/1780/
[III]. https://bugzilla.redhat.com/show_bug.cgi?id=1335541
[IV]. https://github.com/rdo-packages/designate-tempest-plugin-distgit/blob/rpm-master/python-designate-tests-tempest.spec#L21
[V]. https://review.rdoproject.org/r/#/c/1820/
[VI]. https://review.gerrithub.io/#/c/266365


Thanks,
Chandan Kumar