[rdo-list] Proposition to get more CI promotions

[Frédéric Lepied]

On 08/16/2016 12:26 PM, Javier Pena wrote:
>
> ----- Original Message -----
>> On Sat, Aug 13, 2016 at 11:21 AM, Frédéric Lepied
>> <frederic.lepied at redhat.com> wrote:
>>> Hi,
>>>
>>> Our CI promotion system is as if we were running downhill: we cannot
>>> stop and the only way to fix issues is to move forward by getting new
>>> fixed packages. And it usually takes days before we can get the fixes
>>> and by the time we get them we can have other issues appearing and so
>>> never being able to have a promotion.
>>>
>>> I would like to propose an improved way of working to get more CI
>>> promotions. When we have failures in our tests, we do like we do
>>> today: debug the issues and find the commits that are causing the
>>> regression and working with upstream or fixing packaging issues to
>>> solve the regression.
>>>
>>> The proposed improvement to get more CI promotions is, while we wait
>>> for the fixes to be ready, to get the oldest commit that is currently
>>> causing an issue from the current analysis and to try the previous
>>> commits in reverse order to promote before the issues appear. With the
>>> database of DLRN we have all the information to be able to implement
>>> this backward tries and have more chances to promote.
>>>
>>> I'm in holidays next week but I wanted to bring this idea before been
>>> completely off. WDYT?
>> In the past we already applied this approach a couple of times for
>> upstream issues that took longer that desired to be fixed (these cases
>> lead us to move to the current u-c pinning for libraries). IMO, this
>> has some drawbacks:
>> - Breaks the consistency principle we try to enforce among packages in
>> RDO repos.
>> - Requires manual definition of the versions or commits for each package.
> Well, if I understood Frédéric's proposal correctly, that's not what we'd do, just go back to the list of previous commits and start trying backwards. They don't even have to be for the same package.
>
> Using an example, let's say we have the following commits, from newest to oldest:
>
> - Commit 2 to openstack-nova (consistent)
> - Commit 2 to openstack-cinder (consistent)
> - Commit 1 to openstack-cinder (consistent)
> - Commit 1 to openstack-keystone (consistent, previous promoted commit)
>
> Just by luck, when the promotion job is started, it tests commit 2 to openstack-nova, and fails due to a new issue. If we can go back just one commit, to "commit 2 to openstack-cinder", and it is passes CI, we could still have a consistent repo promoted, just not the very last one at the time of running the CI job.
>
> Actually, this makes sense to me. By doing this, we could promote the last known good+consistent repo, while we fix the latest issue.
>

Yes that was the intent. Let's see how we could implement this.
--
Fred - May the Source be with you