[Rdo-list] reconsidering midstream repos

[Hugh O. Brock]

On Thu, May 28, 2015 at 10:34:29AM +0200, Dmitry Tantsur wrote:
> On 05/26/2015 09:06 PM, James Slagle wrote:
> >On Mon, May 25, 2015 at 02:55:57PM +0200, Hugh O. Brock wrote:
> >>Seems like the midstream repos are causing us a lot of pain with little
> >>gain, at least in some cases. (For example it appears the t-h-t
> >>midstream exists to carry a single patch that enables mongodb on
> >>Centos.) Is it worth discussing whether we can eliminate some of these,
> >>especially for upstreams like t-h-t that aren't tightly tied to the
> >>OpenStack release schedule?
> >>
> >>/me ducks flying bricks
> >
> >No flying bricks from me anyway :)
> >
> >I agree with what you're saying, but would like to agree on what we're actually
> >meaning.  I think it's great that pretty much exactly what we said would happen
> >has happened: as Kilo nears completion, the patches that we're having to carry
> >for RDO-Manager is approaching 0.
> 
> For the record: it's not the case for Ironic, Ironic client and even
> discoverd. Also 1 Nova patch won't go away.
> 
> >
> >But before we say, let's kill the midstream repos...let's consider why we even
> >have them to begin with.
> >
> >We didn't come up with the idea to fork a bunch of projects because it would
> >make our *development* lives for RDO-Manager easier. In fact, it's exactly the
> >opposite, they cause pain.
> >
> >So why do we have them? Quite simply it's because of the demand that we show,
> >commit to, and support features in RDO-Manager prior to them necessarily
> >merging in upstream OpenStack projects.
> >
> >If we can lift that requirement, we could likely do away with the midstream
> >repos entirely.  I know this is rdo-list, but in the spirit of being open, if
> >we don't have midstream repos for RDO-Manager, then we shouldn't have them for
> >OSP either, and we should only be committing to features as they land in
> >upstream OpenStack.
> >
> >It doesn't make sense to me to drop the midstream repos for RDO-Manager, but
> >then turn around and set them up privately for OSP. At least with them setup
> >for RDO, we're doing the development of the management product in the open. In
> >fact if we were to split like this, I think the pain would only intensify.
> >
> >It's pragmatic to say that anyone looking to productize OpenStack is going to
> >have their own bug fixes/etc that they need, and maybe even some features. But
> >setting up private forks as a general development principal to drive OpenStack
> >based product development should be avoided at all costs.
> >
> >So yes, I'm +1 on dropping the midstream repos if they're no longer needed.
> >
> >But I'm -1 on then turning around and setting them up privately so that we can
> >commit code to private forks because that code hasn't landed in upstream
> >OpenStack yet. If we can agree that this isn't needed, then I think we can do
> >without the midstream. Otherwise, if we're going to insist on forks to "land"
> >features prior to them merging upstream, let's at least keep them public.

I very much agree on *not* making things worse by doing a bunch of
secret-squirrel stuff in private repos. And you are right that the
big-picture purpose of having the midstream repos is in fact to
demonstrate (and build on) features we believe will land upstream,
before they have actually landed. The "slower" the upstream project is,
the more we need to consider taking the pain of having a midstream repo
for that project.

What is giving me pause is that it seems like we have midstream repos
for projects that are not "slow" at all. The tripleo heat templates are
a great example of this; there is little or no delay for our commits on
this project, and in fact we benefit by pushing there first because of
the tripleo-CI coverage. So why have a midstream repo for it, why not
work entirely upstream? The same thing goes for the os-* tools, and a
number of other repos I believe.

I think we probably do benefit from maintaining midstreams of Heat and
Ironic (I *think*), but I question the rest of them. Let me know please
if I'm way off-base here.

> >There's a couple other facets to the discussion as well:
> >
> >The midstream repos is a place where we can preview where the management
> >product is headed. For example, the Heat breakpoint work was available from the
> >midstream repos much earlier than it ended up merging into Heat/heatclient.
> >Some devs/users might find this sort of thing really useful, and could provide
> >some early feedback for RDO-Manager.
> >
> >The midstream repos are also a place where we can quickly land reverts to
> >unblock work. The Neutron regression that broke provisioning via Ironic this
> >past cycle immediately comes to mind...it took a couple of weeks before the
> >upstream revert even landed in Neutron. While that was broken, we carried the
> >proposed revert in the midstream repos so that people could install a Neutron
> >that actually worked for RDO-Manager.

Agree, this makes sense. So is it correct to say that the more unwieldy
an upstream project is, the more likely it is we need a mid-stream repo
for it?

Thanks,
--Hugh

-- 
== Hugh Brock, hbrock at redhat.com                                   ==
== Senior Engineering Manager, Cloud Engineering                   ==
== RDO Manager: Install, configure, and scale OpenStack            ==
== http://rdoproject.org                                           ==

"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey