11:37 a.m.
So I started an etherpad to discuss why we have so much downstream
patches in Puppet modules.
https://etherpad.openstack.org/p/opm-patches
In my opinion, we should follow some best practices:
* upstream first. If you find a bug, submit the patch upstream, wait for
at least a positive review from a core and also successful CI jobs. Then
you can backport it downstream if urgent.
* backport it to stable branches when needed. The patch we want is in
master and not stable? It's too easy to backport it in OPM. Do the
backport in upstream/stable first, it will help to stay updated with
upstream.
* don't change default parameters, don't override them. Our installers
are able to override any parameter so do not hardcode this kind of change.
* keep up with upstream: if you have an upstream patch under review that
is already in OPM: keep it alive and make sure it lands as soon as possible.
UPSTREAM FIRST please please please (I'll send you cookies if you want).
If you have any question about an upstream patch, please join
#puppet-openstack (freenode) and talk to the group. We're doing reviews
every day and it's not difficult to land a patch.
In the meantime, I would like to justify each of our backports in the
etherpad and clean-up a maximum of them.
Thank you for reading so far,
--
Emilien Macchi
11:43 a.m.
+1 to upstream first.
Also, if any downstream modifications are deemed necessary, I'm
convinced we should be maintaining actual patches, not entire forked
repositories but I think that's another topic.
RPM packaging spec files have built-in mechanisms to pull patches and
we should leverage it.
David Moreau Simard
Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
On Tue, Jan 12, 2016 at 12:37 PM, Emilien Macchi <emilien(a)redhat.com> wrote:
So I started an etherpad to discuss why we have so much downstream
patches in Puppet modules.
https://etherpad.openstack.org/p/opm-patches
In my opinion, we should follow some best practices:
* upstream first. If you find a bug, submit the patch upstream, wait for
at least a positive review from a core and also successful CI jobs. Then
you can backport it downstream if urgent.
* backport it to stable branches when needed. The patch we want is in
master and not stable? It's too easy to backport it in OPM. Do the
backport in upstream/stable first, it will help to stay updated with
upstream.
* don't change default parameters, don't override them. Our installers
are able to override any parameter so do not hardcode this kind of change.
* keep up with upstream: if you have an upstream patch under review that
is already in OPM: keep it alive and make sure it lands as soon as possible.
UPSTREAM FIRST please please please (I'll send you cookies if you want).
If you have any question about an upstream patch, please join
#puppet-openstack (freenode) and talk to the group. We're doing reviews
every day and it's not difficult to land a patch.
In the meantime, I would like to justify each of our backports in the
etherpad and clean-up a maximum of them.
Thank you for reading so far,
--
Emilien Macchi
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
12:16 p.m.
Also, the way we're packaging OPM is really bad.
* we have no SHA1 for each module we have in OPM
* we are not able to validate each module
* package tarball is not pure. All other OpenStack RPMS take upstream
tarball so we can easily compare but in OPM... no way to do it.
Those issues are really critical, I would like to hear from OPM folks,
and find solutions that we will work on during the following weeks.
Thanks
On 01/12/2016 12:37 PM, Emilien Macchi wrote:
So I started an etherpad to discuss why we have so much downstream
patches in Puppet modules.
https://etherpad.openstack.org/p/opm-patches
In my opinion, we should follow some best practices:
* upstream first. If you find a bug, submit the patch upstream, wait for
at least a positive review from a core and also successful CI jobs. Then
you can backport it downstream if urgent.
* backport it to stable branches when needed. The patch we want is in
master and not stable? It's too easy to backport it in OPM. Do the
backport in upstream/stable first, it will help to stay updated with
upstream.
* don't change default parameters, don't override them. Our installers
are able to override any parameter so do not hardcode this kind of change.
* keep up with upstream: if you have an upstream patch under review that
is already in OPM: keep it alive and make sure it lands as soon as possible.
UPSTREAM FIRST please please please (I'll send you cookies if you want).
If you have any question about an upstream patch, please join
#puppet-openstack (freenode) and talk to the group. We're doing reviews
every day and it's not difficult to land a patch.
In the meantime, I would like to justify each of our backports in the
etherpad and clean-up a maximum of them.
Thank you for reading so far,
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
Emilien Macchi
3:53 p.m.
On 01/12/2016 01:16 PM, Emilien Macchi wrote:
Also, the way we're packaging OPM is really bad.
* we have no SHA1 for each module we have in OPM
* we are not able to validate each module
* package tarball is not pure. All other OpenStack RPMS take upstream
tarball so we can easily compare but in OPM... no way to do it.
Those issues are really critical, I would like to hear from OPM folks,
and find solutions that we will work on during the following weeks.
I have 2 proposals, maybe wrong but I wanted to share.
Solution #1 - Forks + Puppetfile
* Forking all our Puppet modules in https://github.com/redhat-openstack/
* Apply our custom patches in a specific branch for each module
* Create a Puppetfile per RDO/OSP version that track SHA1 from repos
* Create a script (with r10k for example) that checkout all modules and
build RPM.
Solution #2 - Forks + one RPM per module
* Forking all our Puppet modules in https://github.com/redhat-openstack/
* Apply our custom patches in a specific branch for each module
* Create a RPM for each module
Of course, I don't take in consideration CI work so I'm willing to
suggestions.
Feedback is welcome here!
Thanks
On 01/12/2016 12:37 PM, Emilien Macchi wrote:
> So I started an etherpad to discuss why we have so much downstream
> patches in Puppet modules.
>
> https://etherpad.openstack.org/p/opm-patches
>
> In my opinion, we should follow some best practices:
>
> * upstream first. If you find a bug, submit the patch upstream, wait for
> at least a positive review from a core and also successful CI jobs. Then
> you can backport it downstream if urgent.
> * backport it to stable branches when needed. The patch we want is in
> master and not stable? It's too easy to backport it in OPM. Do the
> backport in upstream/stable first, it will help to stay updated with
> upstream.
> * don't change default parameters, don't override them. Our installers
> are able to override any parameter so do not hardcode this kind of change.
> * keep up with upstream: if you have an upstream patch under review that
> is already in OPM: keep it alive and make sure it lands as soon as possible.
>
> UPSTREAM FIRST please please please (I'll send you cookies if you want).
>
> If you have any question about an upstream patch, please join
> #puppet-openstack (freenode) and talk to the group. We're doing reviews
> every day and it's not difficult to land a patch.
>
> In the meantime, I would like to justify each of our backports in the
> etherpad and clean-up a maximum of them.
>
> Thank you for reading so far,
>
>
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
Emilien Macchi
6:54 p.m.
On Tue, 2016-01-12 at 13:16 -0500, Emilien Macchi wrote:
I have 2 proposals, maybe wrong but I wanted to share.
Solution #1 - Forks + Puppetfile
* Forking all our Puppet modules in
https://github.com/redhat-openstack/
* Apply our custom patches in a specific branch for each module
* Create a Puppetfile per RDO/OSP version that track SHA1 from repos
* Create a script (with r10k for example) that checkout all modules
and
build RPM.
Solution #2 - Forks + one RPM per module
* Forking all our Puppet modules in
https://github.com/redhat-openstack/
* Apply our custom patches in a specific branch for each module
* Create a RPM for each module
All upstream modules are already forked in
https://github.com/rdo-puppet-modules, not under redhat-openstack
because of the way gerrithub deals with github replication.
Modules are forked, because of the way opm has been handled until now.
Here are some informations on these modules, and how CI handles them.
- Each fork branch {branch} is an exact and automatically updated copy
of upstream module branch {branch}
- Each fork may contain a {branch}-patches to host all the patches for
the branch {branch}
- Each fork is updated after every upstream change is submitted, and
after testing every time the result of the merge between {branch} and
{branch}-patches on a centos+rdo environment. From this successful and
tested merge, a {branch}-tag branch is updated, and can be taken to be
packaged.
So, in a way we already have the first two points of every proposal.
{branch}-patches for the modules should be populated with all the
needed patches though, but to do this it should be enough to propose
the patches as reviews in gerrithub for the corresponding project, on
the branch {branch}-patches. CI should do the rest (test merge with
{branch}, test results and update {branch}-tag, then submit)
Raw material is there and available for whatever packaging solution is
preferred, but I think at this point, one RPM per module with maybe a
metapackage to rule them all is the best solution, now that modules
update is automated.
3 p.m.
On 14/01/16 01:54 +0100, Gabriele Cerami wrote:
On Tue, 2016-01-12 at 13:16 -0500, Emilien Macchi wrote:
> I have 2 proposals, maybe wrong but I wanted to share.
>
> Solution #1 - Forks + Puppetfile
>
> * Forking all our Puppet modules in
> https://github.com/redhat-openstack/
> * Apply our custom patches in a specific branch for each module
> * Create a Puppetfile per RDO/OSP version that track SHA1 from repos
> * Create a script (with r10k for example) that checkout all modules
> and
> build RPM.
>
> Solution #2 - Forks + one RPM per module
> * Forking all our Puppet modules in
> https://github.com/redhat-openstack/
> * Apply our custom patches in a specific branch for each module
> * Create a RPM for each module
All upstream modules are already forked in
https://github.com/rdo-puppet-modules, not under redhat-openstack
because of the way gerrithub deals with github replication.
Modules are forked, because of the way opm has been handled until now.
Here are some informations on these modules, and how CI handles them.
- Each fork branch {branch} is an exact and automatically updated copy
of upstream module branch {branch}
- Each fork may contain a {branch}-patches to host all the patches for
the branch {branch}
I could be wrong, but I think this is consistent with what opm
_used_
to do, which itself was _inconsistent_ with the rest of rdo (not sure
how, or if this relates to pure upstream). So, for liberty, for
example, we have upstream-liberty, which is _exactly_ what was in
stable/liberty for the various openstack puppet modules at the time of
the last (yes, manual) sync. Then we (opm) have stable/liberty, which
has everything from upstream-liberty, with whatever patches have been
added on top of it. If I am correct that is needed for rdo, but
inconsistent with what you have described, how hard would it be to
alter the rdo-puppet-modules stuff to work with the new reality?
- Each fork is updated after every upstream change is submitted, and
after testing every time the result of the merge between {branch} and
{branch}-patches on a centos+rdo environment. From this successful and
tested merge, a {branch}-tag branch is updated, and can be taken to be
packaged.
So, in a way we already have the first two points of every proposal.
{branch}-patches for the modules should be populated with all the
needed patches though, but to do this it should be enough to propose
the patches as reviews in gerrithub for the corresponding project, on
the branch {branch}-patches. CI should do the rest (test merge with
{branch}, test results and update {branch}-tag, then submit)
Raw material is there and available for whatever packaging solution is
preferred, but I think at this point, one RPM per module with maybe a
metapackage to rule them all is the best solution, now that modules
update is automated.
+1000 to individual packages, and opm as metapackage only.
-j
3:18 p.m.
On Thu, Jan 14, 2016 at 4:00 PM, Jason Guiditta <jguiditt(a)redhat.com> wrote:
If I am correct that is needed for rdo
Can you define "RDO" ?
Packstack ? RDO manager ? Both ? Something else entirely ?
I'd really, really like to run both with pure upstream modules to see
what - if anything - breaks and draw conclusions from that as to:
1) Is it the consumer's fault (i.e, Packstack or RDO Manager that
aren't passing the proper settings to puppet modules and the puppet
module is patched instead ?)
2) Is it the puppet module's fault (i.e, upstream wrongfully refused
to merge/backport a necessary patch ?)
3) Something else ?
I'm convinced that we must keep OPM without any patches in RDO (OSP is
another matter) even if it requires a little bit a work.
Sure, we need to put visibility on the work that needs to be done to
accomplish this but we can't do that before trying it out first.
I can only assume that consumers and users of RDO do not expect
midstream patches in their packages.
We don't ship anything custom for Nova, Cinder, etc - OPM shouldn't be
any different.
David Moreau Simard
Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
5:36 p.m.
On Thu, 2016-01-14 at 16:18 -0500, David Moreau Simard wrote:
I can only assume that consumers and users of RDO do not expect
midstream patches in their packages.
We don't ship anything custom for Nova, Cinder, etc - OPM shouldn't
be
any different.
OPM is a package with content taken from 59 repositories. Only 22 of
those 59 come from openstack, the remaining 37 upstream repositories do
not even have a gerrit instance for contribution, most of them do not
have CI, almost all of them are tested only in ubuntu.
I think it's enough for it to be treated differently.
7:56 a.m.
I can only assume that consumers and users of RDO do not expect
midstream patches in their packages.
We don't ship anything custom for Nova, Cinder, etc - OPM shouldn't be
any different.
You have a point here, the problem is that sometimes reviews take a lot of
time to be accepted and there are bugs that have to be solved really fast.
That's why with some exceptions there are a lot of patches into OPM's
patches repo, this patches stay there until the upstream patch is accepted.
In RDO there should be constant rebases since there's no code freeze
like in OSP so patches that are alreay in upstream are deleted more often.
From my point of view whe sould not have any fork of the puppet
modules,
we should be using the upstream release of the module and add patches when
needed.
Cheers,
Ivan
8:33 a.m.
On 01/18/2016 08:56 AM, Ivan Chavero wrote:
> I can only assume that consumers and users of RDO do not expect
> midstream patches in their packages.
> We don't ship anything custom for Nova, Cinder, etc - OPM shouldn't be
> any different.
You have a point here, the problem is that sometimes reviews take a lot of
time to be accepted and there are bugs that have to be solved really fast.
What? Are you saying Puppet OpenStack patches take time to be merged?
You can ask to the contributors, if your patch is passing CI, I'm sure
it's landed in the same day or the next 2 days maximum.
By the way, we have a weekly meeting where we do bug and review triage.
If you have any patch that needs attention, please come-up and we'll
help it to make it.
So please, don't say it takes time to merge Puppet patches upstream.
That's why with some exceptions there are a lot of patches into
OPM's
patches repo, this patches stay there until the upstream patch is accepted.
In RDO there should be constant rebases since there's no code freeze
like in OSP so patches that are alreay in upstream are deleted more often.
>From my point of view whe sould not have any fork of the puppet modules,
we should be using the upstream release of the module and add patches when needed.
Cheers,
Ivan
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
Emilien Macchi
9:22 a.m.
----- Mensaje original -----
De: "Emilien Macchi" <emilien(a)redhat.com>
Para: rdo-list(a)redhat.com
Enviados: Lunes, 18 de Enero 2016 8:33:39
Asunto: Re: [Rdo-list] OPM downstream patches
On 01/18/2016 08:56 AM, Ivan Chavero wrote:
>
>> I can only assume that consumers and users of RDO do not expect
>> midstream patches in their packages.
>> We don't ship anything custom for Nova, Cinder, etc - OPM shouldn't be
>> any different.
>
> You have a point here, the problem is that sometimes reviews take a lot of
> time to be accepted and there are bugs that have to be solved really fast.
What? Are you saying Puppet OpenStack patches take time to be merged?
You can ask to the contributors, if your patch is passing CI, I'm sure
it's landed in the same day or the next 2 days maximum.
I wasn't criticizing the review process i was justifying the existence of
patches on the RPM package.
My point was that sometimes you can't wait for the review process to finish
By the way, we have a weekly meeting where we do bug and review
triage.
If you have any patch that needs attention, please come-up and we'll
help it to make it.
So please, don't say it takes time to merge Puppet patches upstream.
I think that you misunderstood my comment i said **sometimes**
Cheers,
Ivan
8:13 p.m.
On 19/01/16 02:22, Ivan Chavero wrote:
----- Mensaje original -----
> De: "Emilien Macchi" <emilien(a)redhat.com>
> Para: rdo-list(a)redhat.com
> Enviados: Lunes, 18 de Enero 2016 8:33:39
> Asunto: Re: [Rdo-list] OPM downstream patches
>
>
>
> On 01/18/2016 08:56 AM, Ivan Chavero wrote:
>>
>>> I can only assume that consumers and users of RDO do not expect
>>> midstream patches in their packages.
>>> We don't ship anything custom for Nova, Cinder, etc - OPM shouldn't
be
>>> any different.
>>
>> You have a point here, the problem is that sometimes reviews take a lot of
>> time to be accepted and there are bugs that have to be solved really fast.
>
> What? Are you saying Puppet OpenStack patches take time to be merged?
> You can ask to the contributors, if your patch is passing CI, I'm sure
> it's landed in the same day or the next 2 days maximum.
I wasn't criticizing the review process i was justifying the existence of
patches on the RPM package.
My point was that sometimes you can't wait for the review process to finish
Thank you Ivan, this is a very important feedback.
We all know Emilien is working very hard to make sure things are moving.
Meanwhile sometimes patches take longer than 2 days, that's for sure!
> By the way, we have a weekly meeting where we do bug and review
triage.
> If you have any patch that needs attention, please come-up and we'll
> help it to make it.
>
> So please, don't say it takes time to merge Puppet patches upstream.
>
I think that you misunderstood my comment i said **sometimes**
Cheers,
Ivan
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
5:26 p.m.
On Thu, 2016-01-14 at 16:00 -0500, Jason Guiditta wrote:
I could be wrong, but I think this is consistent with what opm
_used_
to do, which itself was _inconsistent_ with the rest of rdo (not sure
how, or if this relates to pure upstream). So, for liberty, for
example, we have upstream-liberty, which is _exactly_ what was in
stable/liberty for the various openstack puppet modules at the time
of
the last (yes, manual) sync. Then we (opm) have stable/liberty,
which
has everything from upstream-liberty, with whatever patches have been
added on top of it. If I am correct that is needed for rdo, but
inconsistent with what you have described, how hard would it be to
alter the rdo-puppet-modules stuff to work with the new reality?
OPM has never used rdopkg for its update, if it's what you mean with
"this is needed for rdo": the patches are applied directly to the
midstream repo, and not saved in patch files to be applied to the
package. I don't know why OPM had this "privilege" and if we're going
to change it in the future, but this manual process of merging upstream
-liberty with the patches it's what has been automated in OPMCI.
For example the module puppet-nova in rdo-puppet-modules, contains
three branches for every upstream branch:
- stable/liberty (tracks upstream/stable/liberty)
- stable/liberty-patches
- stable/liberty-tag (created with git merge liberty liberty-patches)
what you call stable/liberty in rdo-puppet-modules/puppet-nova is
called liberty-tag. It contains upstream liberty branch to the latest
revision merged with all the patches present in liberty-patches (and if
a patch added to liberty-patches is finally merged upstream, it is
automatically removed from liberty-patches).
The scripts that handle OPMCI have been modified recently (in the hope
that they could be used to handle other repositories as well), to
handle repositories much closely to what rdopkg requires, so it may
possible to alter to a new reality (if I understood what it means)
(At the moment OPM CI is configured to follow only master branches from
upstream so you will not really find a liberty branch, but the concept
is the same for any upstream branch)
5:51 p.m.
OPM has never used rdopkg for its update, if it's what you mean
with
"this is needed for rdo":
This (last branch renaming in redhat-openstack/openstack-puppet-modules
repo) was needed for the current Delorean/rdoinfo setup where "master"
and "stable/liberty" were expected as source branches to build from.
There's work in progress to expand rdoinfo extension to handle
deviations per project and release[1].
Cheers,
Alan
[1] https://trello.com/c/KxKtVkTz/62-restructure-rdoinfo-for-new-needs
7:58 a.m.
----- Mensaje original -----
De: "Gabriele Cerami" <gcerami(a)redhat.com>
Para: rdo-list(a)redhat.com
Enviados: Jueves, 14 de Enero 2016 17:26:01
Asunto: Re: [Rdo-list] OPM downstream patches
On Thu, 2016-01-14 at 16:00 -0500, Jason Guiditta wrote:
> I could be wrong, but I think this is consistent with what opm _used_
> to do, which itself was _inconsistent_ with the rest of rdo (not sure
> how, or if this relates to pure upstream). So, for liberty, for
> example, we have upstream-liberty, which is _exactly_ what was in
> stable/liberty for the various openstack puppet modules at the time
> of
> the last (yes, manual) sync. Then we (opm) have stable/liberty,
> which
> has everything from upstream-liberty, with whatever patches have been
> added on top of it. If I am correct that is needed for rdo, but
> inconsistent with what you have described, how hard would it be to
> alter the rdo-puppet-modules stuff to work with the new reality?
OPM has never used rdopkg for its update, if it's what you mean with
"this is needed for rdo": the patches are applied directly to the
midstream repo, and not saved in patch files to be applied to the
package. I don't know why OPM had this "privilege" and if we're going
to change it in the future, but this manual process of merging upstream
-liberty with the patches it's what has been automated in OPMCI.
rdopkg has been used in OPM for a while, i'm not sure if they are still using
it through...
For example the module puppet-nova in rdo-puppet-modules, contains
three branches for every upstream branch:
- stable/liberty (tracks upstream/stable/liberty)
- stable/liberty-patches
- stable/liberty-tag (created with git merge liberty liberty-patches)
what you call stable/liberty in rdo-puppet-modules/puppet-nova is
called liberty-tag. It contains upstream liberty branch to the latest
revision merged with all the patches present in liberty-patches (and if
a patch added to liberty-patches is finally merged upstream, it is
automatically removed from liberty-patches).
The scripts that handle OPMCI have been modified recently (in the hope
that they could be used to handle other repositories as well), to
handle repositories much closely to what rdopkg requires, so it may
possible to alter to a new reality (if I understood what it means)
(At the moment OPM CI is configured to follow only master branches from
upstream so you will not really find a liberty branch, but the concept
is the same for any upstream branch)
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
10:30 p.m.
On 13/01/16 08:53, Emilien Macchi wrote:
On 01/12/2016 01:16 PM, Emilien Macchi wrote:
> Also, the way we're packaging OPM is really bad.
>
> * we have no SHA1 for each module we have in OPM
> * we are not able to validate each module
> * package tarball is not pure. All other OpenStack RPMS take upstream
> tarball so we can easily compare but in OPM... no way to do it.
>
> Those issues are really critical, I would like to hear from OPM folks,
> and find solutions that we will work on during the following weeks.
I have 2 proposals, maybe wrong but I wanted to share.
Solution #1 - Forks + Puppetfile
* Forking all our Puppet modules in https://github.com/redhat-openstack/
* Apply our custom patches in a specific branch for each module
* Create a Puppetfile per RDO/OSP version that track SHA1 from repos
* Create a script (with r10k for example) that checkout all modules and
build RPM.
Solution #2 - Forks + one RPM per module
* Forking all our Puppet modules in https://github.com/redhat-openstack/
* Apply our custom patches in a specific branch for each module
* Create a RPM for each module
Of course, I don't take in consideration CI work so I'm willing to
suggestions.
Feedback is welcome here!
So far, we seems to agree to:
- Do upstream work as much as possible and even more to reduce the
backlog of downstream patches.
- Keep Puppet related patches the closest to its upstream source
- Keep non Puppet related patches close the package (RPM) when needed
- Limit forking
- Have 1 package per module
- Use meta/wrapper packages
- There are two groups of modules, the Openstack and non Openstack ones.
My input:
- Confusion between RDO and OSP
- There is an obvious structural complexity with OPM and also related
to Openstack structure itself. I think this is part of the challenge
we need to address in order to reduce the complexity factor.
- Creating a Bugzilla every time a patch needs to be submitted and/or
backported to any RDO or OSP (or other distro) is a workflow burden
(politely said!).
- For validation purpose each installer needs each module to point to a
specific version (commit). This cannot be done when using a common
OPM.
I believe it's important to help with the RDO/OSP confusion and get some
facts about RDO and OSP relationship.
I believe (please jump in if needed):
- RDO and OSP are both opensource projects on top of Openstack.
- RDO is the community version, more cutting edge but unsupported.
- OSP is the Enterprise version, certified and supported.
- OSP inheriting from RDO (downstream).
The latter is the important part because the life cycle work flow is
supposed to go from RDO to OSP. Historically, it has not always happened
mainly because scarce resources were working on either one at a time.
That has created some confusion too.
Back to OPM, Openstack Puppet Modules repo was initially created as a
common denominator placeholder between Packstack and Quickstack related
installers - Quickstack is github repo name containing the puppet core
used by Foreman/Red Hat Satellite 6, Staypuft.
At the time, there wasn't enough resources to handle puppet modules for
each installer's team. So all modules ended up in the same bucket, here
was sadly born 'OPM'. Another temporary solution to last.
The bad was that RDO-Manager/OSP-Director installers started taping into
it too! ;)
The important change that happened in the meantime was
Openstack puppet projects moved under Openstack main tent.
This offers Openstack puppet modules to directly benefit from the stable
branches structure. I understand making use of this is already happening
but we have the opportunity to use it to reorganize the whole OPM structure.
I think we would gain of having each Openstack Puppet module to extend
its current branch structure in a 'distro sub-branch' way, such as:
master
+ +
| |
| +--> RDO9 +--> OSP9
|
back|port
|
+--> stable/liberty +--> RDO8 +--> OSP8
|
|
+--> stable/kilo +--> RDO7 +--> OSP7
The structural complexity could be reduced by benefiting from limiting
forks, as describe above, using branch for each related Openstack distro
(RDO/OSP/etc). If this is not possible then each Opentack installer has
to fork every Openstack modules it needs.
For, the non Openstack puppet modules, because there isn't any link to
Openstack branches. A correspond fork has to be created.
Hence, a solution #3:
- For each installer
- Create a meta/wrapper package
- For each required/desired Openstack Puppet module
- If no 'distro sub-branches' available
- Module's source repo is forked
- Openstack equivalent branches structure are created
- A RPM package is created
- Non Puppet patches are applied if needed
- Add module to list of meta/wrapper package
- For each required/desired non Openstack Puppet module
- Module's source repo is forked
- Openstack equivalent branches structure are created
- Puppet patches are applied in correspond branch if needed
- A RPM package is created
- Non Puppet patches are applied if needed
- Add module to list of meta/wrapper package
-Gilles
> Thanks
>
> On 01/12/2016 12:37 PM, Emilien Macchi wrote:
>> So I started an etherpad to discuss why we have so much downstream
>> patches in Puppet modules.
>>
>> https://etherpad.openstack.org/p/opm-patches
>>
>> In my opinion, we should follow some best practices:
>>
>> * upstream first. If you find a bug, submit the patch upstream, wait for
>> at least a positive review from a core and also successful CI jobs. Then
>> you can backport it downstream if urgent.
>> * backport it to stable branches when needed. The patch we want is in
>> master and not stable? It's too easy to backport it in OPM. Do the
>> backport in upstream/stable first, it will help to stay updated with
>> upstream.
>> * don't change default parameters, don't override them. Our installers
>> are able to override any parameter so do not hardcode this kind of change.
>> * keep up with upstream: if you have an upstream patch under review that
>> is already in OPM: keep it alive and make sure it lands as soon as possible.
>>
>> UPSTREAM FIRST please please please (I'll send you cookies if you want).
>>
>> If you have any question about an upstream patch, please join
>> #puppet-openstack (freenode) and talk to the group. We're doing reviews
>> every day and it's not difficult to land a patch.
>>
>> In the meantime, I would like to justify each of our backports in the
>> etherpad and clean-up a maximum of them.
>>
>> Thank you for reading so far,
>>
>>
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>
>
>
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
12:26 p.m.
On Fri, 2016-01-15 at 15:30 +1100, Gilles Dubreuil wrote:
Hi,
thanks for the historical background, it was enlightening.
The structural complexity could be reduced by benefiting from
limiting
forks, as describe above, using branch for each related Openstack
distro
(RDO/OSP/etc). If this is not possible then each Opentack installer
has
to fork every Openstack modules it needs.
Why are forks so bad ?
Do not resist the fork. Embrace it.
I'll try to clarify some of the choices in the current OPM CI.
We forked every modules because we needed a place to put the patches
for that module, and the result of the merge with the main branch.
The fork is not static, rdo-puppet-modules/puppet-nova/master is kept
in sync with openstack/puppet-nova/master constantly and automatically,
and master equivalent on rdo-puppet-modules/puppet-nova is kept really
just as a cursor to mark the point where we have tested upstream
changes with our patches.
We could as well use master-patches from midstream and master from
upstream to make the final merge, not touching midstream master at all.
So do not consider forks a problem, since their maintenance burden is
quite lifted at the moment, at least for OPM.
Regarding the naming issues, the OPMCI scripts are pretty flexible, we
should be able to change the naming scheme with minimal efforts, and
the scripts are already supporting branch mappings (master in upstream
can be tracked as stable/liberty in midstream for example)
Use the fork!
12:45 p.m.
- Confusion between RDO and OSP
...
| +--> RDO9 +--> OSP9
One clarification: RDO is not using version numbers, it's using upstream
release names i.e. RDO Kilo not RDO7 etc.
RDO strives to be vanilla upstream sources packaged in RPM hence we're
keeping upstream release versioning.
Cheers,
Alan
8:09 a.m.
I think we would gain of having each Openstack Puppet module to
extend
its current branch structure in a 'distro sub-branch' way, such as:
master
+ +
| |
| +--> RDO9 +--> OSP9
|
back|port
|
+--> stable/liberty +--> RDO8 +--> OSP8
|
|
+--> stable/kilo +--> RDO7 +--> OSP7
This should be done already for the official OpenStack puppet modules
The structural complexity could be reduced by benefiting from limiting
forks, as describe above, using branch for each related Openstack distro
(RDO/OSP/etc). If this is not possible then each Opentack installer has
to fork every Openstack modules it needs.
For, the non Openstack puppet modules, because there isn't any link to
Openstack branches. A correspond fork has to be created.
Hence, a solution #3:
- For each installer
- Create a meta/wrapper package
- For each required/desired Openstack Puppet module
- If no 'distro sub-branches' available
- Module's source repo is forked
- Openstack equivalent branches structure are created
- A RPM package is created
- Non Puppet patches are applied if needed
- Add module to list of meta/wrapper package
- For each required/desired non Openstack Puppet module
- Module's source repo is forked
- Openstack equivalent branches structure are created
- Puppet patches are applied in correspond branch if needed
- A RPM package is created
- Non Puppet patches are applied if needed
- Add module to list of meta/wrapper package
There should be only one OPM package and the installers should be modified
to work with it. To maintain an OPM package for earch installer would be
a burden for the installer developers and could generate confusion for the
people that uses only OPM without any installer.
Also, taking in account that every eight months or so there are new
iniciative to create a new installer we will end up maintaining a slightly
different OPM package for who knows how many installers we will end up with
Cheers,
Ivan
8:20 p.m.
On 19/01/16 01:09, Ivan Chavero wrote:
> I think we would gain of having each Openstack Puppet module to extend
> its current branch structure in a 'distro sub-branch' way, such as:
>
> master
> + +
> | |
> | +--> RDO9 +--> OSP9
> |
> back|port
> |
> +--> stable/liberty +--> RDO8 +--> OSP8
> |
> |
> +--> stable/kilo +--> RDO7 +--> OSP7
This should be done already for the official OpenStack puppet modules
Really, I wish. Do you have any pointer?
>
>
> The structural complexity could be reduced by benefiting from limiting
> forks, as describe above, using branch for each related Openstack distro
> (RDO/OSP/etc). If this is not possible then each Opentack installer has
> to fork every Openstack modules it needs.
>
> For, the non Openstack puppet modules, because there isn't any link to
> Openstack branches. A correspond fork has to be created.
>
> Hence, a solution #3:
>
> - For each installer
> - Create a meta/wrapper package
> - For each required/desired Openstack Puppet module
> - If no 'distro sub-branches' available
> - Module's source repo is forked
> - Openstack equivalent branches structure are created
> - A RPM package is created
> - Non Puppet patches are applied if needed
> - Add module to list of meta/wrapper package
> - For each required/desired non Openstack Puppet module
> - Module's source repo is forked
> - Openstack equivalent branches structure are created
> - Puppet patches are applied in correspond branch if needed
> - A RPM package is created
> - Non Puppet patches are applied if needed
> - Add module to list of meta/wrapper package
There should be only one OPM package and the installers should be modified
to work with it. To maintain an OPM package for earch installer would be
a burden for the installer developers and could generate confusion for the
people that uses only OPM without any installer.
Also, taking in account that every eight months or so there are new
iniciative to create a new installer we will end up maintaining a slightly
different OPM package for who knows how many installers we will end up with
Cheers,
Ivan
10:36 a.m.
On Tue, 2016-01-12 at 13:16 -0500, Emilien Macchi wrote:
Also, the way we're packaging OPM is really bad.
* we have no SHA1 for each module we have in OPM
/usr/share/openstack-puppet/Puppetfile
* we are not able to validate each module
you have Puppetfile
and our own patches as patches to tar
* package tarball is not pure. All other OpenStack RPMS take
upstream
tarball so we can easily compare but in OPM... no way to do it.
Tarballs are always
taken from github releases:
https://github.com/redhat-openstack/openstack-puppet-modules/releases
And yes, dropping single package and creating metapackage is the way to
go.
Those issues are really critical, I would like to hear from OPM
folks,
and find solutions that we will work on during the following weeks.
Thanks
On 01/12/2016 12:37 PM, Emilien Macchi wrote:
> So I started an etherpad to discuss why we have so much downstream
> patches in Puppet modules.
>
> https://etherpad.openstack.org/p/opm-patches
>
> In my opinion, we should follow some best practices:
>
> * upstream first. If you find a bug, submit the patch upstream,
> wait for
> at least a positive review from a core and also successful CI jobs.
> Then
> you can backport it downstream if urgent.
> * backport it to stable branches when needed. The patch we want is
> in
> master and not stable? It's too easy to backport it in OPM. Do the
> backport in upstream/stable first, it will help to stay updated
> with
> upstream.
> * don't change default parameters, don't override them. Our
> installers
> are able to override any parameter so do not hardcode this kind of
> change.
> * keep up with upstream: if you have an upstream patch under review
> that
> is already in OPM: keep it alive and make sure it lands as soon as
> possible.
>
> UPSTREAM FIRST please please please (I'll send you cookies if you
> want).
>
> If you have any question about an upstream patch, please join
> #puppet-openstack (freenode) and talk to the group. We're doing
> reviews
> every day and it's not difficult to land a patch.
>
> In the meantime, I would like to justify each of our backports in
> the
> etherpad and clean-up a maximum of them.
>
> Thank you for reading so far,
>
>
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
3:04 p.m.
On 01/13/2016 11:36 AM, Lukas Bezdicka wrote:
On Tue, 2016-01-12 at 13:16 -0500, Emilien Macchi wrote:
> Also, the way we're packaging OPM is really bad.
>
> * we have no SHA1 for each module we have in OPM
/usr/share/openstack-puppet/Puppetfile
Nope, this file is not containing SHA1 for the actual modules we have in
OPM. This file was used (and is 100% useless now) by SpinalStack.
> * we are not able to validate each module
you have Puppetfile and our own patches as patches to tar
No, see my previous comment.
> * package tarball is not pure. All other OpenStack RPMS take
upstream
> tarball so we can easily compare but in OPM... no way to do it.
Tarballs are always taken from github releases:
https://github.com/redhat-openstack/openstack-puppet-modules/releases
You can't pull a tarball per module. Modules have been introduced by a
RAW copy-paste.
And yes, dropping single package and creating metapackage is the way
to
go.
I would like to see a more constructive reply from people actually
authors on OPM.
We want to build a plan for the future release that will radically
change the way we build & ship OPM.
>
> Those issues are really critical, I would like to hear from OPM
> folks,
> and find solutions that we will work on during the following weeks.
>
> Thanks
>
> On 01/12/2016 12:37 PM, Emilien Macchi wrote:
>> So I started an etherpad to discuss why we have so much downstream
>> patches in Puppet modules.
>>
>> https://etherpad.openstack.org/p/opm-patches
>>
>> In my opinion, we should follow some best practices:
>>
>> * upstream first. If you find a bug, submit the patch upstream,
>> wait for
>> at least a positive review from a core and also successful CI jobs.
>> Then
>> you can backport it downstream if urgent.
>> * backport it to stable branches when needed. The patch we want is
>> in
>> master and not stable? It's too easy to backport it in OPM. Do the
>> backport in upstream/stable first, it will help to stay updated
>> with
>> upstream.
>> * don't change default parameters, don't override them. Our
>> installers
>> are able to override any parameter so do not hardcode this kind of
>> change.
>> * keep up with upstream: if you have an upstream patch under review
>> that
>> is already in OPM: keep it alive and make sure it lands as soon as
>> possible.
>>
>> UPSTREAM FIRST please please please (I'll send you cookies if you
>> want).
>>
>> If you have any question about an upstream patch, please join
>> #puppet-openstack (freenode) and talk to the group. We're doing
>> reviews
>> every day and it's not difficult to land a patch.
>>
>> In the meantime, I would like to justify each of our backports in
>> the
>> etherpad and clean-up a maximum of them.
>>
>> Thank you for reading so far,
>>
>>
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
Emilien Macchi
7:43 a.m.
----- Mensaje original -----
De: "Lukas Bezdicka" <lbezdick(a)redhat.com>
Para: "Emilien Macchi" <emilien(a)redhat.com>,
"Rdo-list(a)redhat.com" <rdo-list(a)redhat.com>
Enviados: Miércoles, 13 de Enero 2016 10:36:30
Asunto: Re: [Rdo-list] OPM downstream patches
On Tue, 2016-01-12 at 13:16 -0500, Emilien Macchi wrote:
> Also, the way we're packaging OPM is really bad.
>
> * we have no SHA1 for each module we have in OPM
/usr/share/openstack-puppet/Puppetfile
> * we are not able to validate each module
you have Puppetfile and our own patches as patches to tar
> * package tarball is not pure. All other OpenStack RPMS take upstream
> tarball so we can easily compare but in OPM... no way to do it.
Tarballs are always taken from github releases:
https://github.com/redhat-openstack/openstack-puppet-modules/releases
And yes, dropping single package and creating metapackage is the way to
go.
+1
This would simplify a lot the puppet module upgrading.
>
> Those issues are really critical, I would like to hear from OPM
> folks,
> and find solutions that we will work on during the following weeks.
>
> Thanks
>
> On 01/12/2016 12:37 PM, Emilien Macchi wrote:
> > So I started an etherpad to discuss why we have so much downstream
> > patches in Puppet modules.
> >
> > https://etherpad.openstack.org/p/opm-patches
> >
> > In my opinion, we should follow some best practices:
> >
> > * upstream first. If you find a bug, submit the patch upstream,
> > wait for
> > at least a positive review from a core and also successful CI jobs.
> > Then
> > you can backport it downstream if urgent.
> > * backport it to stable branches when needed. The patch we want is
> > in
> > master and not stable? It's too easy to backport it in OPM. Do the
> > backport in upstream/stable first, it will help to stay updated
> > with
> > upstream.
> > * don't change default parameters, don't override them. Our
> > installers
> > are able to override any parameter so do not hardcode this kind of
> > change.
> > * keep up with upstream: if you have an upstream patch under review
> > that
> > is already in OPM: keep it alive and make sure it lands as soon as
> > possible.
> >
> > UPSTREAM FIRST please please please (I'll send you cookies if you
> > want).
> >
> > If you have any question about an upstream patch, please join
> > #puppet-openstack (freenode) and talk to the group. We're doing
> > reviews
> > every day and it's not difficult to land a patch.
> >
> > In the meantime, I would like to justify each of our backports in
> > the
> > etherpad and clean-up a maximum of them.
> >
> > Thank you for reading so far,
> >
> >
> >
> > _______________________________________________
> > Rdo-list mailing list
> > Rdo-list(a)redhat.com
> > https://www.redhat.com/mailman/listinfo/rdo-list
> >
> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com
> >
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
_______________________________________________
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
2:18 p.m.
On 01/12/2016 12:37 PM, Emilien Macchi wrote:
So I started an etherpad to discuss why we have so much downstream
patches in Puppet modules.
https://etherpad.openstack.org/p/opm-patches
In my opinion, we should follow some best practices:
* upstream first. If you find a bug, submit the patch upstream, wait for
at least a positive review from a core and also successful CI jobs. Then
you can backport it downstream if urgent.
* backport it to stable branches when needed. The patch we want is in
master and not stable? It's too easy to backport it in OPM. Do the
backport in upstream/stable first, it will help to stay updated with
upstream.
* don't change default parameters, don't override them. Our installers
are able to override any parameter so do not hardcode this kind of change.
* keep up with upstream: if you have an upstream patch under review that
is already in OPM: keep it alive and make sure it lands as soon as possible.
UPSTREAM FIRST please please please (I'll send you cookies if you want).
If you have any question about an upstream patch, please join
#puppet-openstack (freenode) and talk to the group. We're doing reviews
every day and it's not difficult to land a patch.
In the meantime, I would like to justify each of our backports in the
etherpad and clean-up a maximum of them.
Thank you for reading so far,
Wow. Lot of thoughts, lot of interest, I like it!
So this thread is having lot of ideas, proposals, facts, etc.
I'm opening https://etherpad.openstack.org/p/rdo-opm and gathering the
data from this thread.
Feel free to open it and contribute,
I hope from this etherpad we'll find a plan for the next steps.
Thanks,
--
Emilien Macchi
3022
days inactive
3029
days old
23 comments
8 participants
participants (8)
-
Alan Pevec -
David Moreau Simard -
Emilien Macchi -
Gabriele Cerami -
Gilles Dubreuil -
Ivan Chavero -
Jason Guiditta -
Lukas Bezdicka