[Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling

Thu Feb 4 10:30:42 UTC 2016

Hi Gabriele,

I add rdo-list as CC to that mail as the discussion is around
RPM Factory and Gitnetics. This may interest the RDO community.

For the context as the discussion started in the softwarefactory-dev ML:
https://www.redhat.com/archives/softwarefactory-dev/2016-February/msg00003.html

Le 03/02/2016 11:44, Gabriele Cerami a écrit :
> On 03 Feb, Fabien Boucher wrote:
>>
>> So to summarize Gitnetic will help for such cases:
>> - we have a *-patches branch with some "not yet/never" included upstream changes
>> and where we want to cherry-pick upstream changes into it.
>> - we want to cherry-pick upstream changes into a "mirror" repository where we already
>> have included "not yet/never" included upstream changes.
>>
>> Gitnetic is then able to take actions when the cherry-pick is not possible, like
>> warning the maintainer via a notification like a Gerrit review, ...
>>
>> Let me know if I miss understood.
> 
> Yes, gitnetics will try the cherry-pick and launch unit tests on the
> merged result. If the cherry pick is not possible, it will ask for human
> interaction in the form of comments on a gerrit review.
> It would also be able to include upstream changes into mirror
> repository on non -patches branches. This is not necessary for most of
> the upstream projects, because the master branches are tested by delorean, but
> delorean is not testing any local patch. It's the goal of rdo, offering
> packages from unpatched repositories, but it's not clear yet if for some
> opm repositories for example, we will be able to ship unpatched upstream
> releases.

For the case of the current stable RDO release "Liberty" there ~ 25 patches
in different projects like ironic, swift, horizon, opm, ... and patches
are applied on top of unmodified upstream project release AFAIK.

In RPM Factory patches are handled in form of Gerrit review (never merged) and
tox unit tests are run on each patches of the patches chain each time there is a
change in the chain. Ideally the use of a patches chain containing a patch
decreasing the upstream quality (the project no longer pass the unit test suite)
should be prohibited but at least a maintainer will know if the patch he's
including in the packaging is decreasing the quality.

Furthermore this is, I think, useful when a new minor release of an upstream
project is created, like a new tag in stable/liberty. In that case the maintainer
may want to reset the liberty-patches branch on that new tag then rebase the patches
chain and then the rebase will produce a new patchset and trigger a new run of unit test
for each patches of the chain.

About the master version of RDO, AFAIK yes Delorean does not use any patches when
trying to use rpm-master distgit to test the packaging against each upstream changes
so there is not need to run unit test in that case IMO.

About OPM, It seems there is an effort to split it and in that case I'm wondering why
eventual patches cannot be managed as explained above expect we replace
tox test by the right ones for Puppet.

>> Actually we have already imported all projects from rdoinfo (rdo.yml) into RPM factory.
>> This has been done by a tool called python-sfrdo. Let's take
>> an example: Nova:
> 
> Yes, sorry, with "manual import" I meant that the mirror repository is
> maintained manually, I did not know about the periodic job. But then
> again nothing is testing local patches with new upstream changes, when
> the mirror is synchronized.

Yes at sync time changes coming from upstream are not tested because
this is a mirror so unit test has been already run on the upstream CI
(at least for OpenStack projects). And, at the moment a maintainer
rebase a patch or a chain of patches, unit test will then be run on each patches
of the chain.

After, in the case we want to sync be doing a cherry-pick of each upstream changes
on top of a modified version of a "mirror" repo/branch, yes I agree it is
safer to run unit test during the sync and I understand Gitnetics will
help in that way. 

> We are yet not sure how much we want to test these upstream changes.
> Gitnetics tests even before merging...
> 
>> We haven't yet thought to tightly integrate such options in the dashboard.
>> We first try to implement RPM Factory features in a separate tool
>> called python-sfrdo and evaluate after if there are generic enough to backport
>> them in Software Factory.
> 
> Gitnetics is written in python too, and it uses yaml configuration file
> to handle all the various options for the projects it has to replicate.
> 
>> But yes I think we can find a way to let a PTL/CORE member of a packaging
>> project to select the mechanism that will handle the sync of the "mirror" repo:
>>
>> Classical: like we have today in RPM Factory: Naive sync of master and stable/*
>> periodically.
>> Gitnetic: Classical is deactivated and Gitnetic managed the sync of branches
>> from upstream in a smarter manner taking in account "not yet/never" included upstream changes
>> has been inserted in the branches.
> 
> Yes, something like this, maybe with more options (gitnetics
> configuration file contains branch mappings e.g. stable/kilo upstream is
> replicated downstream)

Alright do you think it possible to have such a workflow : ?
- Gitnetics is triggered on a slave by a periodic job to sync (by cherry-picking)
  each "mirror" repositories (configured to be managed by Gitnetics).
- Each upstream changes are one by one cherry-picked on top of the liberty-patches
  branch and triggering a unit test job attached to that project/branch (already
  the case with RPM Factory). If unit test pass then the Gerrit review is merged.
- When a cherry-pick cannot apply a patch on top of the branch then Gitnetics create
  the notification for warning the maintainer about the sync issue and let him take
  an action

What do you think ?  

Best regards
Fabien

> _______________________________________________
> Softwarefactory-dev mailing list
> Softwarefactory-dev at redhat.com
> https://www.redhat.com/mailman/listinfo/softwarefactory-dev
> 