[rdo-list] Proposition to get more CI promotions

[Alfredo Moralejo Alonso]

On Tue, Aug 16, 2016 at 12:26 PM, Javier Pena <javier.pena at redhat.com> wrote:
>
>
> ----- Original Message -----
>> On Sat, Aug 13, 2016 at 11:21 AM, Frédéric Lepied
>> <frederic.lepied at redhat.com> wrote:
>> > Hi,
>> >
>> > Our CI promotion system is as if we were running downhill: we cannot
>> > stop and the only way to fix issues is to move forward by getting new
>> > fixed packages. And it usually takes days before we can get the fixes
>> > and by the time we get them we can have other issues appearing and so
>> > never being able to have a promotion.
>> >
>> > I would like to propose an improved way of working to get more CI
>> > promotions. When we have failures in our tests, we do like we do
>> > today: debug the issues and find the commits that are causing the
>> > regression and working with upstream or fixing packaging issues to
>> > solve the regression.
>> >
>> > The proposed improvement to get more CI promotions is, while we wait
>> > for the fixes to be ready, to get the oldest commit that is currently
>> > causing an issue from the current analysis and to try the previous
>> > commits in reverse order to promote before the issues appear. With the
>> > database of DLRN we have all the information to be able to implement
>> > this backward tries and have more chances to promote.
>> >
>> > I'm in holidays next week but I wanted to bring this idea before been
>> > completely off. WDYT?
>>
>> In the past we already applied this approach a couple of times for
>> upstream issues that took longer that desired to be fixed (these cases
>> lead us to move to the current u-c pinning for libraries). IMO, this
>> has some drawbacks:
>> - Breaks the consistency principle we try to enforce among packages in
>> RDO repos.
>> - Requires manual definition of the versions or commits for each package.
>
> Well, if I understood Frédéric's proposal correctly, that's not what we'd do, just go back to the list of previous commits and start trying backwards. They don't even have to be for the same package.
>
> Using an example, let's say we have the following commits, from newest to oldest:
>
> - Commit 2 to openstack-nova (consistent)
> - Commit 2 to openstack-cinder (consistent)
> - Commit 1 to openstack-cinder (consistent)
> - Commit 1 to openstack-keystone (consistent, previous promoted commit)
>
> Just by luck, when the promotion job is started, it tests commit 2 to openstack-nova, and fails due to a new issue. If we can go back just one commit, to "commit 2 to openstack-cinder", and it is passes CI, we could still have a consistent repo promoted, just not the very last one at the time of running the CI job.
>
> Actually, this makes sense to me. By doing this, we could promote the last known good+consistent repo, while we fix the latest issue.
>

OK, i understood "try the previous commits in reverse order", he was
referring to the same package but re-reading the mail,  you are
probably right about the proposal. This approach is easier and better,
as we don't break consistency and don't need any trick in dlrn side.
However, it will not work if issues of different packages overlap (as
we had last week, in fact we were discussing on irc about setting a
specific hash but at that time there was not "sane" repo), but
probably may help in some cases.

>> - When a package is pinned to a specific commit or version, we stop
>> packaging and testing any new commit, so we are delaying and piling
>> the detection of new issues that may appear in the package.
>> - Currently, DLRN doesn't manage properly moving back to a previous
>> commit whenever a new one has been built. It requires manual
>> error-prone tasks to be performed on the dlrn instance (anyway it'd be
>> probably a good idea to implement this us.
>>
>> IMO, we shouldn't use this as a systematic approach for managing
>> versions in our repos to enforce promotions but only as last resort
>> for exceptional cases. Anyway, it's probably a good idea to improve
>> dlrn to implement this use case in a easier and cleaner way.
>>
>
> This requires some good thinking. Right now, the only way I know is to fiddle with the DLRN database, which is quite scary and can lead to inconsistencies. Maybe we can come up with a better approach.
>

Yeap, it's probably not an easy change but i think it's still worthy
to do it as I'm pretty sure we'll find scenarios were we'll need it,
don't you think so?

> Regards,
> Javier
>
>>
>> > --
>> > Fred - May the Source be with you
>> >
>> >
>> > _______________________________________________
>> > rdo-list mailing list
>> > rdo-list at redhat.com
>> > https://www.redhat.com/mailman/listinfo/rdo-list
>> >
>> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com