5:26 a.m.
----- Original Message -----
On Sat, Aug 13, 2016 at 11:21 AM, Frédéric Lepied
<frederic.lepied(a)redhat.com> wrote:
> Hi,
>
> Our CI promotion system is as if we were running downhill: we cannot
> stop and the only way to fix issues is to move forward by getting new
> fixed packages. And it usually takes days before we can get the fixes
> and by the time we get them we can have other issues appearing and so
> never being able to have a promotion.
>
> I would like to propose an improved way of working to get more CI
> promotions. When we have failures in our tests, we do like we do
> today: debug the issues and find the commits that are causing the
> regression and working with upstream or fixing packaging issues to
> solve the regression.
>
> The proposed improvement to get more CI promotions is, while we wait
> for the fixes to be ready, to get the oldest commit that is currently
> causing an issue from the current analysis and to try the previous
> commits in reverse order to promote before the issues appear. With the
> database of DLRN we have all the information to be able to implement
> this backward tries and have more chances to promote.
>
> I'm in holidays next week but I wanted to bring this idea before been
> completely off. WDYT?
In the past we already applied this approach a couple of times for
upstream issues that took longer that desired to be fixed (these cases
lead us to move to the current u-c pinning for libraries). IMO, this
has some drawbacks:
- Breaks the consistency principle we try to enforce among packages in
RDO repos.
- Requires manual definition of the versions or commits for each package.
Well, if I understood Frédéric's proposal correctly, that's not what we'd do,
just go back to the list of previous commits and start trying backwards. They don't
even have to be for the same package.
Using an example, let's say we have the following commits, from newest to oldest:
- Commit 2 to openstack-nova (consistent)
- Commit 2 to openstack-cinder (consistent)
- Commit 1 to openstack-cinder (consistent)
- Commit 1 to openstack-keystone (consistent, previous promoted commit)
Just by luck, when the promotion job is started, it tests commit 2 to openstack-nova, and
fails due to a new issue. If we can go back just one commit, to "commit 2 to
openstack-cinder", and it is passes CI, we could still have a consistent repo
promoted, just not the very last one at the time of running the CI job.
Actually, this makes sense to me. By doing this, we could promote the last known
good+consistent repo, while we fix the latest issue.
- When a package is pinned to a specific commit or version, we stop
packaging and testing any new commit, so we are delaying and piling
the detection of new issues that may appear in the package.
- Currently, DLRN doesn't manage properly moving back to a previous
commit whenever a new one has been built. It requires manual
error-prone tasks to be performed on the dlrn instance (anyway it'd be
probably a good idea to implement this us.
IMO, we shouldn't use this as a systematic approach for managing
versions in our repos to enforce promotions but only as last resort
for exceptional cases. Anyway, it's probably a good idea to improve
dlrn to implement this use case in a easier and cleaner way.
This requires some good thinking. Right now, the only way I know is to fiddle with the
DLRN database, which is quite scary and can lead to inconsistencies. Maybe we can come up
with a better approach.
Regards,
Javier
> --
> Fred - May the Source be with you
>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com