[Rdo-list] OPM downstream patches

Fri Jan 15 04:30:24 UTC 2016

On 13/01/16 08:53, Emilien Macchi wrote:
> 
> 
> On 01/12/2016 01:16 PM, Emilien Macchi wrote:
>> Also, the way we're packaging OPM is really bad.
>>
>> * we have no SHA1 for each module we have in OPM
>> * we are not able to validate each module
>> * package tarball is not pure. All other OpenStack RPMS take upstream
>> tarball so we can easily compare but in OPM... no way to do it.
>>
>> Those issues are really critical, I would like to hear from OPM folks,
>> and find solutions that we will work on during the following weeks.
> 
> I have 2 proposals, maybe wrong but I wanted to share.
> 
> Solution #1 - Forks + Puppetfile
> 
> * Forking all our Puppet modules in https://github.com/redhat-openstack/
> * Apply our custom patches in a specific branch for each module
> * Create a Puppetfile per RDO/OSP version that track SHA1 from repos
> * Create a script (with r10k for example) that checkout all modules and
> build RPM.
> 
> Solution #2 - Forks + one RPM per module
> * Forking all our Puppet modules in https://github.com/redhat-openstack/
> * Apply our custom patches in a specific branch for each module
> * Create a RPM for each module
> 
> Of course, I don't take in consideration CI work so I'm willing to
> suggestions.
> 
> Feedback is welcome here!
> 

So far, we seems to agree to:
- Do upstream work as much as possible and even more to reduce the
  backlog of downstream patches.
- Keep Puppet related patches the closest to its upstream source
- Keep non Puppet related patches close the package (RPM) when needed
- Limit forking
- Have 1 package per module
- Use meta/wrapper packages
- There are two groups of modules, the Openstack and non Openstack ones.

My input:
- Confusion between RDO and OSP
- There is an obvious structural complexity with OPM and also related
  to Openstack structure itself. I think this is part of the challenge
  we need to address in order to reduce the complexity factor.
- Creating a Bugzilla every time a patch needs to be submitted and/or
  backported to any RDO or OSP (or other distro) is a workflow burden
  (politely said!).
- For validation purpose each installer needs each module to point to a
  specific version (commit). This cannot be done when using a common
  OPM.

I believe it's important to help with the RDO/OSP confusion and get some
facts about RDO and OSP relationship.

I believe (please jump in if needed):
- RDO and OSP are both opensource projects on top of Openstack.
- RDO is the community version, more cutting edge but unsupported.
- OSP is the Enterprise version, certified and supported.
- OSP inheriting from RDO (downstream).

The latter is the important part because the life cycle work flow is
supposed to go from RDO to OSP. Historically, it has not always happened
mainly because scarce resources were working on either one at a time.
That has created some confusion too.

Back to OPM, Openstack Puppet Modules repo was initially created as a
common denominator placeholder between Packstack and Quickstack related
installers - Quickstack is github repo name containing the puppet core
used by Foreman/Red Hat Satellite 6, Staypuft.

At the time, there wasn't enough resources to handle puppet modules for
each installer's team. So all modules ended up in the same bucket, here
was sadly born 'OPM'. Another temporary solution to last.

The bad was that RDO-Manager/OSP-Director installers started taping into
it too! ;)

The important change that happened in the meantime was
Openstack puppet projects moved under Openstack main tent.

This offers Openstack puppet modules to directly benefit from the stable
branches structure. I understand making use of this is already happening
but we have the opportunity to use it to reorganize the whole OPM structure.

I think we would gain of having each Openstack Puppet module to extend
its current branch structure in a 'distro sub-branch' way, such as:

  master
    + +
    | |
    | +--> RDO9 +--> OSP9
    |
back|port
    |
    +--> stable/liberty +--> RDO8 +--> OSP8
    |
    |
    +--> stable/kilo +--> RDO7 +--> OSP7

The structural complexity could be reduced by benefiting from limiting
forks, as describe above, using branch for each related Openstack distro
(RDO/OSP/etc). If this is not possible then each Opentack installer has
to fork every Openstack modules it needs.

For, the non Openstack puppet modules, because there isn't any link to
Openstack branches. A correspond fork has to be created.

Hence, a solution #3:

- For each installer
 - Create a meta/wrapper package
 - For each required/desired Openstack Puppet module
   - If no 'distro sub-branches' available
     - Module's source repo is forked
     - Openstack equivalent branches structure are created
   - A RPM package is created
     - Non Puppet patches are applied if needed
   - Add module to list of meta/wrapper package
 - For each required/desired non Openstack Puppet module
   - Module's source repo is forked
   - Openstack equivalent branches structure are created
   - Puppet patches are applied in correspond branch if needed
   - A RPM package is created
     - Non Puppet patches are applied if needed
   - Add module to list of meta/wrapper package

-Gilles

>> Thanks
>>
>> On 01/12/2016 12:37 PM, Emilien Macchi wrote:
>>> So I started an etherpad to discuss why we have so much downstream
>>> patches in Puppet modules.
>>>
>>> https://etherpad.openstack.org/p/opm-patches
>>>
>>> In my opinion, we should follow some best practices:
>>>
>>> * upstream first. If you find a bug, submit the patch upstream, wait for
>>> at least a positive review from a core and also successful CI jobs. Then
>>> you can backport it downstream if urgent.
>>> * backport it to stable branches when needed. The patch we want is in
>>> master and not stable? It's too easy to backport it in OPM. Do the
>>> backport in upstream/stable first, it will help to stay updated with
>>> upstream.
>>> * don't change default parameters, don't override them. Our installers
>>> are able to override any parameter so do not hardcode this kind of change.
>>> * keep up with upstream: if you have an upstream patch under review that
>>> is already in OPM: keep it alive and make sure it lands as soon as possible.
>>>
>>> UPSTREAM FIRST please please please (I'll send you cookies if you want).
>>>
>>> If you have any question about an upstream patch, please join
>>> #puppet-openstack (freenode) and talk to the group. We're doing reviews
>>> every day and it's not difficult to land a patch.
>>>
>>> In the meantime, I would like to justify each of our backports in the
>>> etherpad and clean-up a maximum of them.
>>>
>>> Thank you for reading so far,
>>>
>>>
>>>
>>> _______________________________________________
>>> Rdo-list mailing list
>>> Rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>
>>
>>
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
> 
> 
> 
> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
> 
> To unsubscribe: rdo-list-unsubscribe at redhat.com
> 