[rdo-list] Proposition to get more CI promotions

Tue Aug 16 11:32:24 UTC 2016

----- Original Message -----
> On Tue, Aug 16, 2016 at 12:26 PM, Javier Pena <javier.pena at redhat.com> wrote:
> >
> >
> > ----- Original Message -----
> >> On Sat, Aug 13, 2016 at 11:21 AM, Frédéric Lepied
> >> <frederic.lepied at redhat.com> wrote:
> >> > Hi,
> >> >
> >> > Our CI promotion system is as if we were running downhill: we cannot
> >> > stop and the only way to fix issues is to move forward by getting new
> >> > fixed packages. And it usually takes days before we can get the fixes
> >> > and by the time we get them we can have other issues appearing and so
> >> > never being able to have a promotion.
> >> >
> >> > I would like to propose an improved way of working to get more CI
> >> > promotions. When we have failures in our tests, we do like we do
> >> > today: debug the issues and find the commits that are causing the
> >> > regression and working with upstream or fixing packaging issues to
> >> > solve the regression.
> >> >
> >> > The proposed improvement to get more CI promotions is, while we wait
> >> > for the fixes to be ready, to get the oldest commit that is currently
> >> > causing an issue from the current analysis and to try the previous
> >> > commits in reverse order to promote before the issues appear. With the
> >> > database of DLRN we have all the information to be able to implement
> >> > this backward tries and have more chances to promote.
> >> >
> >> > I'm in holidays next week but I wanted to bring this idea before been
> >> > completely off. WDYT?
> >>
> >> In the past we already applied this approach a couple of times for
> >> upstream issues that took longer that desired to be fixed (these cases
> >> lead us to move to the current u-c pinning for libraries). IMO, this
> >> has some drawbacks:
> >> - Breaks the consistency principle we try to enforce among packages in
> >> RDO repos.
> >> - Requires manual definition of the versions or commits for each package.
> >
> > Well, if I understood Frédéric's proposal correctly, that's not what we'd
> > do, just go back to the list of previous commits and start trying
> > backwards. They don't even have to be for the same package.
> >
> > Using an example, let's say we have the following commits, from newest to
> > oldest:
> >
> > - Commit 2 to openstack-nova (consistent)
> > - Commit 2 to openstack-cinder (consistent)
> > - Commit 1 to openstack-cinder (consistent)
> > - Commit 1 to openstack-keystone (consistent, previous promoted commit)
> >
> > Just by luck, when the promotion job is started, it tests commit 2 to
> > openstack-nova, and fails due to a new issue. If we can go back just one
> > commit, to "commit 2 to openstack-cinder", and it is passes CI, we could
> > still have a consistent repo promoted, just not the very last one at the
> > time of running the CI job.
> >
> > Actually, this makes sense to me. By doing this, we could promote the last
> > known good+consistent repo, while we fix the latest issue.
> >
> 
> OK, i understood "try the previous commits in reverse order", he was
> referring to the same package but re-reading the mail,  you are
> probably right about the proposal. This approach is easier and better,
> as we don't break consistency and don't need any trick in dlrn side.
> However, it will not work if issues of different packages overlap (as
> we had last week, in fact we were discussing on irc about setting a
> specific hash but at that time there was not "sane" repo), but
> probably may help in some cases. 

Yes, well, with multiple failures it will go to the last commit before any of the two started happening, which is ok. I just wonder how we could detect if a repo is consistent, but that's a minor implementation detail.

> 
> >> - When a package is pinned to a specific commit or version, we stop
> >> packaging and testing any new commit, so we are delaying and piling
> >> the detection of new issues that may appear in the package.
> >> - Currently, DLRN doesn't manage properly moving back to a previous
> >> commit whenever a new one has been built. It requires manual
> >> error-prone tasks to be performed on the dlrn instance (anyway it'd be
> >> probably a good idea to implement this us.
> >>
> >> IMO, we shouldn't use this as a systematic approach for managing
> >> versions in our repos to enforce promotions but only as last resort
> >> for exceptional cases. Anyway, it's probably a good idea to improve
> >> dlrn to implement this use case in a easier and cleaner way.
> >>
> >
> > This requires some good thinking. Right now, the only way I know is to
> > fiddle with the DLRN database, which is quite scary and can lead to
> > inconsistencies. Maybe we can come up with a better approach.
> >
> 
> Yeap, it's probably not an easy change but i think it's still worthy
> to do it as I'm pretty sure we'll find scenarios were we'll need it,
> don't you think so?
> 

With the current approach (using the upper-constraints for master), we should only need to go backwards if there's an upstream revert due to a bug, but yeah, that's a situation we'd like to have covered.

> > Regards,
> > Javier
> >
> >>
> >> > --
> >> > Fred - May the Source be with you
> >> >
> >> >
> >> > _______________________________________________
> >> > rdo-list mailing list
> >> > rdo-list at redhat.com
> >> > https://www.redhat.com/mailman/listinfo/rdo-list
> >> >
> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
> >>
> >> _______________________________________________
> >> rdo-list mailing list
> >> rdo-list at redhat.com
> >> https://www.redhat.com/mailman/listinfo/rdo-list
> >>
> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
> 
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
> 
> To unsubscribe: rdo-list-unsubscribe at redhat.com