There has been a lot of interest in running tempest against real RDO clusters. Having an rpm would make that a lot easier. There are several issues peculiar to tempest that need to be resolved. 1. The way tempest configures itself and does test discovery depends heavily on the tests being run from a directory containing the tests. 2. Unlike OpenStack python client libraries, tempest changes from release to release in incompatible ways, so you need "havana tempest" to test a havana cluster and an "icehouse tempest" to test icehouse. The tempest group has little interest in changing either of these behaviors. Additionally, it would be desirable if a tempest user could install tempest rpms to test different RDO versions on the same machine. Here is a proposal for how this could work and what the user experience would be. Dealing with these tempest issues suggests that the the tempest code should be copied to /var/lib/tempest/{4.0,5.0, etc.} and the user should configure a separate directory for each OpenStack cluster to be tested. Each directory needs to contain: .testr.conf an etc directory containing the tempest.conf and logging.conf files a symlink to the tempest test modules for the appropriate version a copy of the test run scripts that are in the tools directory of tempest To help the user create such directories, there should be a global executable "configure-tempest-directory" that takes an optional version. If multiple versions are present in /var/lib/tempest and no version is specified then the user will be asked which version to configure. User experience: 1. Install tempest rpm: yum install tempest-4.0 2. Run configure-tempest-directory 3. Make changes to tempest.conf to match the cluster being tested (and possibly logging.conf and .testr.conf as well) 4. Run tempest with desired test selection using tools/pretty_tox_serial.sh or tools/ pretty_tox_serial Does any one have any comments/suggestions about this? -David _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list

On 02/03/2014 10:50 AM, David Kranz wrote: > There has been a lot of interest in running tempest against real RDO > clusters. Having an rpm would make that a lot easier. There are several > issues peculiar to tempest that need to be resolved. > > 1. The way tempest configures itself and does test discovery depends > heavily on the tests being run from a directory containing the tests. > 2. Unlike OpenStack python client libraries, tempest changes from > release to release in incompatible ways, so you need "havana tempest" to > test a havana cluster and an "icehouse tempest" to test icehouse. > > The tempest group has little interest in changing either of these > behaviors. Additionally, it would be desirable if a tempest user could > install tempest rpms to test different RDO versions on the same machine. > Here is a proposal for how this could work and what the user experience > would be. > > Dealing with these tempest issues suggests that the the tempest code > should be copied to /var/lib/tempest/{4.0,5.0, etc.} and the user should > configure a separate directory for each OpenStack cluster to be tested. > Each directory needs to contain: > > .testr.conf > an etc directory containing the tempest.conf and logging.conf files > a symlink to the tempest test modules for the appropriate version > a copy of the test run scripts that are in the tools directory of tempest > > To help the user create such directories, there should be a global > executable "configure-tempest-directory" that takes an optional version. > If multiple versions are present in /var/lib/tempest and no version is > specified then the user will be asked which version to configure. Would it make sense to make the package name itself to have it? This way, you could say install tempest for grizzly and update tempest for havana - e.g.: yum install tempest-grizzly yum update -y tempest-havana ... and the RPMs would not obsolete each other. If we use RPM versioning, "yum update" will blow away testing environments of "older" tempest versions. We could perhaps do it using sub-rpms - which would allow us to add and remove tempest suites as we move along: * ship no files except perhaps license in 'tempest' itself and stand-up environment bits * use subpackages for tempest-grizzly/tempest-havana/tempest-icehouse/... * get a special exception to *not* make tempest-* child RPMs require a fully-versioned tempest base package all the time just in case you wanted to have older tempest for a given release e.g. tempest-2.0.0 tempest-grizzly-1.0.0 tempest-havana-2.0.0 ... could all be installed, and if done right, you could 'yum update -y tempest-grizzly' to 3.0.0 without breaking the tempest-havana-2.0.0 package. Just a thought. > User experience: > > 1. Install tempest rpm: yum install tempest-4.0 The above would mean: yum install -y tempest-havana yum install -y tempest-grizzly etc. > 2. Run configure-tempest-directory > 3. Make changes to tempest.conf to match the cluster being tested (and > possibly logging.conf and .testr.conf as well) > 4. Run tempest with desired test selection using > tools/pretty_tox_serial.sh or tools/ pretty_tox_serial > > Does any one have any comments/suggestions about this? > -- Lon

On 02/03/2014 03:55 PM, David Kranz wrote: > On 02/03/2014 02:27 PM, Lon Hohberger wrote: >> On 02/03/2014 10:50 AM, David Kranz wrote: >>> There has been a lot of interest in running tempest against real RDO >>> clusters. Having an rpm would make that a lot easier. There are several >>> issues peculiar to tempest that need to be resolved. >>> >>> 1. The way tempest configures itself and does test discovery depends >>> heavily on the tests being run from a directory containing the tests. >>> 2. Unlike OpenStack python client libraries, tempest changes from >>> release to release in incompatible ways, so you need "havana >>> tempest" to >>> test a havana cluster and an "icehouse tempest" to test icehouse. >>> >>> The tempest group has little interest in changing either of these >>> behaviors. Additionally, it would be desirable if a tempest user could >>> install tempest rpms to test different RDO versions on the same >>> machine. >>> Here is a proposal for how this could work and what the user experience >>> would be. >>> >>> Dealing with these tempest issues suggests that the the tempest code >>> should be copied to /var/lib/tempest/{4.0,5.0, etc.} and the user >>> should >>> configure a separate directory for each OpenStack cluster to be tested. >>> Each directory needs to contain: >>> >>> .testr.conf >>> an etc directory containing the tempest.conf and logging.conf files >>> a symlink to the tempest test modules for the appropriate version >>> a copy of the test run scripts that are in the tools directory of >>> tempest >>> >>> To help the user create such directories, there should be a global >>> executable "configure-tempest-directory" that takes an optional >>> version. >>> If multiple versions are present in /var/lib/tempest and no version is >>> specified then the user will be asked which version to configure. >> Would it make sense to make the package name itself to have it? >> >> This way, you could say install tempest for grizzly and update tempest >> for havana - e.g.: >> >> yum install tempest-grizzly >> yum update -y tempest-havana >> >> ... and the RPMs would not obsolete each other. If we use RPM >> versioning, "yum update" will blow away testing environments of "older" >> tempest versions. >> >> We could perhaps do it using sub-rpms - which would allow us to add and >> remove tempest suites as we move along: >> >> * ship no files except perhaps license in 'tempest' itself >> and stand-up environment bits >> * use subpackages for >> tempest-grizzly/tempest-havana/tempest-icehouse/... >> * get a special exception to *not* make tempest-* child >> RPMs require a fully-versioned tempest base package all >> the time just in case you wanted to have older tempest >> for a given release >> e.g. tempest-2.0.0 >> tempest-grizzly-1.0.0 >> tempest-havana-2.0.0 >> >> ... could all be installed, and if done right, you could >> 'yum update -y tempest-grizzly' to 3.0.0 without breaking >> the tempest-havana-2.0.0 package. >> >> Just a thought. >> >>> User experience: >>> >>> 1. Install tempest rpm: yum install tempest-4.0 >> The above would mean: >> >> yum install -y tempest-havana >> yum install -y tempest-grizzly >> >> etc. >> >> >>> 2. Run configure-tempest-directory >>> 3. Make changes to tempest.conf to match the cluster being tested (and >>> possibly logging.conf and .testr.conf as well) >>> 4. Run tempest with desired test selection using >>> tools/pretty_tox_serial.sh or tools/ pretty_tox_serial >>> >>> Does any one have any comments/suggestions about this? >>> >> -- Lon >> > Lon, I was not clear enough but I was proposing something very similar > to your suggestion, just without subpackages which I don't understand > well. Now that I understand the rules a little better and that the > base package names should not have things that look like version > numbers, I propose: > > package-names: tempest-havana.{version stuff}, > tempest-icehouse.{version stuff} > > And a tempest-base package on which they all depend that creates > "/var/lib/tempest" and the configure-tempest-directory script. Perhaps this complexity is not even needed. We can just use /var/lib/tempest-{havana,icehouse} and configure-tempest-{havana,icehouse}-directory scripts. The rpms for havana and icehouse are just separate, end of story. > > -David

_______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list