Re: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
4:30 a.m.
Hi Gabriele,
I add rdo-list as CC to that mail as the discussion is around
RPM Factory and Gitnetics. This may interest the RDO community.
For the context as the discussion started in the softwarefactory-dev ML:
https://www.redhat.com/archives/softwarefactory-dev/2016-February/msg0000...
Le 03/02/2016 11:44, Gabriele Cerami a écrit :
On 03 Feb, Fabien Boucher wrote:
>
> So to summarize Gitnetic will help for such cases:
> - we have a *-patches branch with some "not yet/never" included upstream
changes
> and where we want to cherry-pick upstream changes into it.
> - we want to cherry-pick upstream changes into a "mirror" repository where
we already
> have included "not yet/never" included upstream changes.
>
> Gitnetic is then able to take actions when the cherry-pick is not possible, like
> warning the maintainer via a notification like a Gerrit review, ...
>
> Let me know if I miss understood.
Yes, gitnetics will try the cherry-pick and launch unit tests on the
merged result. If the cherry pick is not possible, it will ask for human
interaction in the form of comments on a gerrit review.
It would also be able to include upstream changes into mirror
repository on non -patches branches. This is not necessary for most of
the upstream projects, because the master branches are tested by delorean, but
delorean is not testing any local patch. It's the goal of rdo, offering
packages from unpatched repositories, but it's not clear yet if for some
opm repositories for example, we will be able to ship unpatched upstream
releases.
For the case of the current stable RDO release "Liberty" there ~ 25 patches
in different projects like ironic, swift, horizon, opm, ... and patches
are applied on top of unmodified upstream project release AFAIK.
In RPM Factory patches are handled in form of Gerrit review (never merged) and
tox unit tests are run on each patches of the patches chain each time there is a
change in the chain. Ideally the use of a patches chain containing a patch
decreasing the upstream quality (the project no longer pass the unit test suite)
should be prohibited but at least a maintainer will know if the patch he's
including in the packaging is decreasing the quality.
Furthermore this is, I think, useful when a new minor release of an upstream
project is created, like a new tag in stable/liberty. In that case the maintainer
may want to reset the liberty-patches branch on that new tag then rebase the patches
chain and then the rebase will produce a new patchset and trigger a new run of unit test
for each patches of the chain.
About the master version of RDO, AFAIK yes Delorean does not use any patches when
trying to use rpm-master distgit to test the packaging against each upstream changes
so there is not need to run unit test in that case IMO.
About OPM, It seems there is an effort to split it and in that case I'm wondering why
eventual patches cannot be managed as explained above expect we replace
tox test by the right ones for Puppet.
> Actually we have already imported all projects from rdoinfo
(rdo.yml) into RPM factory.
> This has been done by a tool called python-sfrdo. Let's take
> an example: Nova:
Yes, sorry, with "manual import" I meant that the mirror repository is
maintained manually, I did not know about the periodic job. But then
again nothing is testing local patches with new upstream changes, when
the mirror is synchronized.
Yes at sync time changes coming from upstream are not tested because
this is a mirror so unit test has been already run on the upstream CI
(at least for OpenStack projects). And, at the moment a maintainer
rebase a patch or a chain of patches, unit test will then be run on each patches
of the chain.
After, in the case we want to sync be doing a cherry-pick of each upstream changes
on top of a modified version of a "mirror" repo/branch, yes I agree it is
safer to run unit test during the sync and I understand Gitnetics will
help in that way.
We are yet not sure how much we want to test these upstream changes.
Gitnetics tests even before merging...
> We haven't yet thought to tightly integrate such options in the dashboard.
> We first try to implement RPM Factory features in a separate tool
> called python-sfrdo and evaluate after if there are generic enough to backport
> them in Software Factory.
Gitnetics is written in python too, and it uses yaml configuration file
to handle all the various options for the projects it has to replicate.
> But yes I think we can find a way to let a PTL/CORE member of a packaging
> project to select the mechanism that will handle the sync of the "mirror"
repo:
>
> Classical: like we have today in RPM Factory: Naive sync of master and stable/*
> periodically.
> Gitnetic: Classical is deactivated and Gitnetic managed the sync of branches
> from upstream in a smarter manner taking in account "not yet/never"
included upstream changes
> has been inserted in the branches.
Yes, something like this, maybe with more options (gitnetics
configuration file contains branch mappings e.g. stable/kilo upstream is
replicated downstream)
Alright do you think it possible to have such a workflow : ?
- Gitnetics is triggered on a slave by a periodic job to sync (by cherry-picking)
each "mirror" repositories (configured to be managed by Gitnetics).
- Each upstream changes are one by one cherry-picked on top of the liberty-patches
branch and triggering a unit test job attached to that project/branch (already
the case with RPM Factory). If unit test pass then the Gerrit review is merged.
- When a cherry-pick cannot apply a patch on top of the branch then Gitnetics create
the notification for warning the maintainer about the sync issue and let him take
an action
What do you think ?
Best regards
Fabien
_______________________________________________
Softwarefactory-dev mailing list
Softwarefactory-dev(a)redhat.com
https://www.redhat.com/mailman/listinfo/softwarefactory-dev
5:31 a.m.
New subject: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
I do not have the whole context, so this is just few comments:
1. we're trying to split the OPM behemoth and goal is to converge to
the point where we don't need any downstream patches.
2. until then, we can just use our own mirrors instead of upstream
3. depending how much time, it'll take, it would be worth considering
having vanilla puppet modules packages available for testing
Moreover associated w/ delorean, it will help us to catch glitches as
soon as puppet modules gets broken.
4. I don't like cherry-pick upstream commits on top of own branches,
it only encourage us to widen the gap with upstream.
If we aim to be the first (as in excellence) vanilla distribution of
OpenStack, no exception for puppet modules. If upstream model is
broken, for us, then fix it.
We've been working with Emilien (upstream Puppet Module PTL) to fix
upstream CI w/ RDO, so it's doable.
Regards,
H.
8:41 a.m.
New subject: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
+1, let's split OPM into a single package per module (and then a
meta-package to install them all) like we agreed and let it stay
vanilla.
If anything breaks, let's address it.
Until we have a mean to test a vanilla OPM, we do not have a way to
see what is really necessary.
Upstream puppet openstack uses vanilla puppet modules and it works just fine.
If there are downstream patches necessary for RDO Manager or
Packstack, our CI or frequent testing (such as test days) should be
able to identify the issues.
David Moreau Simard
Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
On Thu, Feb 4, 2016 at 6:31 AM, Haïkel <hguemar(a)fedoraproject.org> wrote:
I do not have the whole context, so this is just few comments:
1. we're trying to split the OPM behemoth and goal is to converge to
the point where we don't need any downstream patches.
2. until then, we can just use our own mirrors instead of upstream
3. depending how much time, it'll take, it would be worth considering
having vanilla puppet modules packages available for testing
Moreover associated w/ delorean, it will help us to catch glitches as
soon as puppet modules gets broken.
4. I don't like cherry-pick upstream commits on top of own branches,
it only encourage us to widen the gap with upstream.
If we aim to be the first (as in excellence) vanilla distribution of
OpenStack, no exception for puppet modules. If upstream model is
broken, for us, then fix it.
We've been working with Emilien (upstream Puppet Module PTL) to fix
upstream CI w/ RDO, so it's doable.
Regards,
H.
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
9:49 a.m.
New subject: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
On 02/04/2016 12:31 PM, Haïkel wrote:
I do not have the whole context, so this is just few comments:
1. we're trying to split the OPM behemoth and goal is to converge to
the point where we don't need any downstream patches.
2. until then, we can just use our own mirrors instead of upstream
3. depending how much time, it'll take, it would be worth considering
having vanilla puppet modules packages available for testing
Moreover associated w/ delorean, it will help us to catch glitches as
soon as puppet modules gets broken.
4. I don't like cherry-pick upstream commits on top of own branches,
it only encourage us to widen the gap with upstream.
While we're pushing "upstream first", we'll probably have custom
patches, for the long term maintenance process.
I don't say "I want to have downstream-only patches", I just say "we
need to be able to backport upstream patches to our downstream branches,
because not all upstream projects accepts backports to stable branches.
For example in Puppet modules: Upstream accepts backports until
stable/kilo but not after. What if we have a case where we need to do so?
Big +1 to reduce to 0 the number of downstream-only patches, but we
might want to be able to have some process allowing us to do it for
exceptions.
If we aim to be the first (as in excellence) vanilla distribution of
OpenStack, no exception for puppet modules. If upstream model is
broken, for us, then fix it.
We've been working with Emilien (upstream Puppet Module PTL) to fix
upstream CI w/ RDO, so it's doable.
Yeah, it's doable, though the upstream backport policy is not the same
as we have in RDO I guess.
--
Emilien Macchi
11:09 a.m.
New subject: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
2016-02-04 16:49 GMT+01:00 Emilien Macchi <emilien(a)redhat.com>:
While we're pushing "upstream first", we'll probably have custom
patches, for the long term maintenance process.
I don't say "I want to have downstream-only patches", I just say "we
need to be able to backport upstream patches to our downstream branches,
because not all upstream projects accepts backports to stable branches.
For example in Puppet modules: Upstream accepts backports until
stable/kilo but not after. What if we have a case where we need to do so?
RDO lifecycle currently mirrors upstream, so it should seldom but we
can manage that properly.
Big +1 to reduce to 0 the number of downstream-only patches, but we
might want to be able to have some process allowing us to do it for
exceptions.
We have few exceptions already, and rdopkg automatically rebase your
patchsets for updates.
It's more about rebasing downstream patches on top of upstream
branches, rather than cherry-picking upstream patches on top of
downstream patches.
The amount of work to fix rebase conflicts is sensibly the same or
even lesser than rebasing. And it makes much easier to track
downstream only patches and keeping their number low.
I know there was good reasons in the past to do that, but our
long-term goal is to do better.
Yeah, it's doable, though the upstream backport policy is not the same
as we have in RDO I guess.
--
Emilien Macchi
In short, it's fine to have downstream patches as long as we keep
their number as low as possible.
8:06 a.m.
New subject: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
Hi,
Apologies for cross-posting.
I don't like replies branching, so I'm answering to all replies in one
big email.
On Thu, 2016-02-04 at 11:30 +0100, Fabien Boucher wrote:
In RPM Factory patches are handled in form of Gerrit review (never
merged) and
If the patches are never merged how do you know they don't create
conflicts, what is the merge base of the patches ? patches are never
rebased on each other but only to a common base ?
Since SF is package-centered, I know this is less relevant, because all
the patches end in the package at some point, and you test that.
About the master version of RDO, AFAIK yes Delorean does not use any
patches when
trying to use rpm-master distgit to test the packaging against each
upstream changes
so there is not need to run unit test in that case IMO.
Unit tests, no. Acceptance tests, yes. Upstream CI will never be enough
because all the projects always download bleeding edge versions for
dependencies that may not be present, or have different version in the
distro packages.
After, in the case we want to sync be doing a cherry-pick of each
upstream changes
on top of a modified version of a "mirror" repo/branch, yes I agree
it is
safer to run unit test during the sync and I understand Gitnetics
will
help in that way.
not on top, and not unit tests. Gitnetics, in one of its mode, merges
clean upstream branch, with local patches branch, and create a third
branch that is the merge between the two. On the other mode, just
proposes (without merging) backports to a -patches branch.
Currently OPM-CI is running upstream acceptance tests, but forcing
modules to use dependencies available from distro packages, then
something called stability tests, to see if modules are deploying stuff
like we expect them do to. (usually catch when a variable changes its
default)
Alright do you think it possible to have such a workflow : ?
- Gitnetics is triggered on a slave by a periodic job to sync (by
cherry-picking)
each "mirror" repositories (configured to be managed by Gitnetics).
- Each upstream changes are one by one cherry-picked on top of the
liberty-patches
branch and triggering a unit test job attached to that
project/branch (already
the case with RPM Factory). If unit test pass then the Gerrit
review is merged.
- When a cherry-pick cannot apply a patch on top of the branch then
Gitnetics create
the notification for warning the maintainer about the sync issue
and let him take
an action
This is exactly what happens, except gitnetics can run on both periodic
jobs and triggered by gerrit event, and in its "lock and backports"
mode there is no automatic merging, all is tested in temporary branch
and only if unit tests pass, the change is proposed as backport to the
branch. If something fails, a developer may just add DISCARD as comment
to the paused review, and everything will be canceled.
On Thu, 2016-02-04 at 12:31 +0100, Haïkel wrote:
I do not have the whole context, so this is just few comments:
1. we're trying to split the OPM behemoth and goal is to converge to
the point where we don't need any downstream patches.
Not having downstream patches is a great goal to have, not having a
backup plan for that one time that is really needed and you don't have
the infrastructure to handle it, is a bad move
2. until then, we can just use our own mirrors instead of upstream
3. depending how much time, it'll take, it would be worth considering
having vanilla puppet modules packages available for testing
Moreover associated w/ delorean, it will help us to catch glitches as
soon as puppet modules gets broken.
One of the big things I'm trying to undestand here, is how much test do
we want, on OPM and in general.
OPM-CI is currently trying to run tests after each upstream change is
merged, but before the same change is merged downstream. After this, a
package may be created and tested with the rest of the puppet modules,
but testing before merge catches errors very early in the chain, and we
immediately know which package caused the havoc, because we are testing
only one, not the entire set of modules at once.
4. I don't like cherry-pick upstream commits on top of own
branches,
it only encourage us to widen the gap with upstream.
I think everyone here is talking about cherry-picking in -patches
branches. Main branches downstream are always an exact copy of the
upstream equivalent branch
If we aim to be the first (as in excellence) vanilla distribution of
OpenStack, no exception for puppet modules. If upstream model is
broken, for us, then fix it.
Can you be more specific in what you consider upstream model, and in
what way this is going to be broken ?
Anyway, I expect to be corrected by some of the opm developers, but I
don't think vanilla puppet modules in rdo is realistic goal.
We've been working with Emilien (upstream Puppet Module PTL) to
fix
upstream CI w/ RDO, so it's doable.
Only 22 modules out of 59 present in OPM package are from upstream
openstack. The rest is scattered around the galaxy. Fixing upstream CI
in openstack may not be enough.
On Thu, 2016-02-04 at 16:49 +0100, Emilien Macchi wrote:
While we're pushing "upstream first", we'll
probably have custom
patches, for the long term maintenance process.
I don't say "I want to have downstream-only patches", I just say "we
need to be able to backport upstream patches to our downstream
branches,
because not all upstream projects accepts backports to stable
branches.
For example in Puppet modules: Upstream accepts backports until
stable/kilo but not after. What if we have a case where we need to do
so?
Other reasons to have a plan B when "no downstream patches" is not an
applicable policy.
Fabien, can we begin to transfer gitnetics jobs to SF ? I'd like to
know what happens when the two meet.
9:17 a.m.
New subject: [Rdo-list] [Softwarefactory-dev] Gitnetics integration, or how can we automate upstream changes polling
Hi,
Le 05/02/2016 15:06, Gabriele Cerami a écrit :
If the patches are never merged how do you know they don't
create
conflicts, what is the merge base of the patches ? patches are never
rebased on each other but only to a common base ?
In the RPM Factory context let's say we have a project "A" packaged for
Liberty.
"A" is packaged at version 1.0 and 2 patches are needed. Then we have
liberty-patches
branch reset on tag 1.0 and :
- patch 1 (gerrit review) has liberty-patches HEAD as parent
- patch 2 (gerrit review) has patch 1 as parent.
So the base is liberty-patches HEAD (or the tag 1.0). If you
"git review -d <gerrit_id of patch 2>" -> you fetch tag 1.0 + p1 + p2
already applied.
Since SF is package-centered, I know this is less relevant, because
all
the patches end in the package at some point, and you test that.
> About the master version of RDO, AFAIK yes Delorean does not use any
> patches when
> trying to use rpm-master distgit to test the packaging against each
> upstream changes
> so there is not need to run unit test in that case IMO.
Unit tests, no. Acceptance tests, yes. Upstream CI will never be enough
because all the projects always download bleeding edge versions for
dependencies that may not be present, or have different version in the
distro packages.
Yes sure if a project or a puppet module embed a functional test suite
(like Beaker for a puppet module) then we should definitely run them each time
a change is proposed again a distgit.
For instance when a maintainer, of one of the future splited OPM, proposes on new change
(Gerrit review) on the distgit then the RPM of the puppet module is built and
is installed on a fresh test node (dependencies, configured in the RPM, are installed)
and then the acceptance tests (Beaker) will run. I think we can easily
give an adequate feedback to the packager especially dependencies issues.
After all this is one of the job of a package maintainer -> dealing with
package dependencies :)
In the context of RPM Factory, note this behavior can only work for stable
branches rdo-liberty, "rdo-mitaka', not with rpm-master.
> 2. until then, we can just use our own mirrors instead of
upstream
> 3. depending how much time, it'll take, it would be worth considering
> having vanilla puppet modules packages available for testing
> Moreover associated w/ delorean, it will help us to catch glitches as
> soon as puppet modules gets broken.
One of the big things I'm trying to undestand here, is how much test do
we want, on OPM and in general.
OPM-CI is currently trying to run tests after each upstream change is
merged, but before the same change is merged downstream. After this, a
package may be created and tested with the rest of the puppet modules,
but testing before merge catches errors very early in the chain, and we
immediately know which package caused the havoc, because we are testing
only one, not the entire set of modules at once.
Should splited OPM be managed the way as the other RDO packages ?
I mean Delorean uses the rpm-master to test packaging of each OPM packages
and the CI tags the repo as "current-passed-ci" when validation jobs pass.
Fabien, can we begin to transfer gitnetics jobs to SF ? I'd like to
know what happens when the two meet.
Yes sure we can discuss how to test both together. Let's ping me on
IRC.
Cheers,
Fabien
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
2999
days inactive
3003
days old
6 comments
5 participants
participants (5)
-
David Moreau Simard -
Emilien Macchi -
Fabien Boucher -
Gabriele Cerami -
Haïkel