[Rdo-list] Integration of MidoNet into RDO Manager

[Mike Orazi]

On 07/30/2015 08:17 AM, Sandro Mathys wrote:
> On Thu, Jul 30, 2015 at 7:39 PM, Haïkel <hguemar at fedoraproject.org> wrote:
>>> That should be no problem. Does this have to be a single person or can
>> it be a team?
>>
>> fine with me
>>
>>> Our biggest headache right now (like with any Java software) in terms of the Fedora
>> Packaging Guidelines are tons of bundled jars - so that would be a no-go, right?
>>
>> Currently, yes, but it'll change in the near future.
>> Starting Mitaka, we're considering to switch over CentOS dist-git as
>> our main source
>> for packaging and we'll be able to provide bundling libraries
>> exception for RDO packages.
>> I plan to discuss at Flock with bstinson and kbsingh about our plans,
>> and how to enable
>> contributors to have access to git & build system safely.
> 
> To be honest, we were hoping to be integrated before Mitaka - but then
> again, unbundling everything would be such a huge effort that it would
> probably take us much longer. So this would still be great news for
> us.
> 
> Just let me know when and where you guys meet if you want me to join
> you so I can maybe weigh in with an outside view.
> 
>> We don't have yet a reviewing process for such packages but it's
>> mostly a matter of timing.
>>
>> Only clients and oslo libraries will be shipped in Fedora 24, Fedora
>> users could still use
>> RDO Delorean packages (which are unsigned).
>>
>> Regards,
>> H.
>>
> On Thu, Jul 30, 2015 at 8:22 PM, Hugh O. Brock <hbrock at redhat.com> wrote:
>> On Thu, Jul 30, 2015 at 07:14:54AM -0400, Perry Myers wrote:
>>>> It's combined APT and RPM, but you're right, there's currently no
>>>> source packages for either.
>>>>
>>>> Out of curiosity, why do people need to be able to rebuild them (in
>>>> the scenario we're discussing - integrating MidoNet with RDO Manager)?
>>>
>>> I don't think they strictly need this ability. Providing packages that
>>> are rebuildable by the community is a nice thing to do, but it shouldn't
>>> be a strict requirement in a case like this, I believe.
> 
> To be clear: we do plan to release source packages but our current
> build process does not allow for it, so it will take a while to get
> there - which is why we were hoping it was not strictly necessary.
> 
>>> If anything I'd focus on providing a rebuildable RPM for the Neutron
>>> plugin itself, but perhaps skip that for the SDN solution that is paired
>>> with it.
> 
> That should be absolutely no problem at all, the plugin has different
> processes, etc. and the package is in quite good shape.
> 
>>>>>>>
>>>>>>> Are packages hosted on the developer's repositories acceptable for
>>>>>>> integration into RDO Manager? We need to unbundle a couple of things
>>>>>>> (or maybe more than just a couple) before we can package them for
>>>>>>> inclusion into the proper repository.
>>>>>> Typically, using a COPR is just a transition step to getting packages
>>>>>> into Fedora;  RDO very much follows the Fedora model.
>>>>>>
>>>>>> The individual packages themselves must be submitted, reviewed, and
>>>>>> maintained.  RDO manager is the last step of the process, and will only
>>>>>> work with RDOmanaged packages.
>>>>
>>>> So all our packages would have to be part of the RDO repository (which
>>>> I trust follows the EPEL/Fedora Guidelines)?
>>>
>>> I think having the packages part of a repo available on RDO would make
>>> the end user experience a lot easier, but I don't think there is/should
>>> be a strict requirement that anything integrated with RDO and RDO
>>> Manager _must_ be hosted on the RDO site.
> 
> Agreed. Basically, we'd like to first use our external repo until
> we're ready for inclusion in the RDO repo (with bundled jars).
> 
>>> We might want to differentiate between (for example) the Neutron plugin
>>> packages and the SDN solution, for that.
>>>
>>> I also want to make sure folks understand that I don't think that
>>> getting packages like this 'into Fedora' should be a requirement.
>>>
>>> Right now we are tied to getting things into Fedora since we don't have
>>> another place to host spec files, etc, but we're working on decoupling
>>> from this so that we can move to a model where we are 'on Fedora'
>>> instead of 'in Fedora'
> 
> That makes sense. So I guess this would work well with what I said
> above: use an external repo at first until 1) our packages (or rather
> build processes) are in a better shape (i.e. use packaging tools) and
> 2) RDO is not "in Fedora" anymore.
> 
>>> But to the extent that we can, we try to follow Fedora packaging
>>> guidelines as a best practice, but can/will make exceptions where it
>>> makes sense.
> 
> I really hope we won't need any other exception than "bundled
> (fat)jars", and don't think we will.
> 
>>>>>>> Speaking of repositories, once we're ready to package them properly
>>>>>>> for inclusion, which repository would be the (most?) proper one for
>>>>>>> RDO Manager? RDO? CentOS (wherever Cloud SIG packages go)? EPEL?
>>>>>> Its usually easiest to start with Fedora for all packaging.  Once they
>>>>>> are accepted into Fedora, figuring out how to get them into the
>>>>>> appropriate other locations will follow.
>>>>
>>>> Well, for our packages, Fedora and EL would be fairly different. The
>>>> MidoNet core is written in Java/Scala, so much more (tools, deps) is
>>>> missing from EL, e.g. gradle and of course lots of artifacts. So we
>>>> should target EPEL, I guess.
>>>
>>> I wouldn't follow Adam's advice here (starting with Fedora). Especially
>>> for the SDN solution which is Java based. That would lead to a lot of
>>> pain and overhead.
>>>
>>>>>> Thus far, RDO manager has been focused on upstream OpenStack and the
>>>>>> necessary pieces from the base OS that need to be updated to support
>>>>>> it.  While it should be possible to have an add-on like MidoNet, I don't
>>>>>> know how the rest of the community would feel about it being required to
>>>>>> be part of RDO.  My thought is that, so long as it A) is under a
>>>>>> sufficient license and B) provides real value beyond what is available
>>>>>> from Neutron, it should be possible to include, so long as including it
>>>>>> does not impact people currently developing and deploying RDO.
>>>>>>
>>>>>>
>>>>>> Is there any move to get MidoNet into upstream OpenStack?
>>>>>
>>>>> Aren't we talking about the midonet neutron plug-in which was already part of neutron-proper but as a result of the plug-in decomposition effort now has it's own repo under the openstack namespace ( http://git.openstack.org/cgit/openstack/networking-midonet/ ). I'm really unclear on what the issue is here versus other RDO-Manager integration efforts - perhaps you can clarify?
>>>>
>>>> Yes, that's what we're talking about, thanks for providing the reference :)
>>>>
>>>> However, I'm a bit confused now - surely, integration would not just
>>>> include the plugin but also the SDN solution the plugin is leveraging,
>>>> right?
>>>
>>> That's actually a good question. I think typically we've been thinking
>>> about integration of the plugin only, because that is co-located on
>>> compute nodes and there might be controller node software that needs to
>>> be bundled for certain SDN solutions as well.
>>>
>>> But I can see where deployment of the SDN solution itself by RDO Manager
>>> could be very useful and would make the end user experience much easier.
>>> But I'd have to defer to folks on that team as to how and if this could
>>> be done :)
> 
> Well, one of our main differences e.g. from the default plugin/SDN
> solution (ovs) is, that we don't have a central network node but
> instead an agent on all compute nodes. So I'm not sure I would fully
> agree with that.
> 

I think this is probably a good opportunity for us to think about how to
represent alternate deployment architectures.  I added a few folks
explicitly to the cc: who I think likely already have some ideas on this
front.

>> I think we would be anxious to help with this. We want RDO Manager to be
>> a one-stop shop for all kinds of deployments -- the more participation
>> we have, the better.
> 
> Cool, thanks! :)
> 
>>>>> Thanks,
>>>>>
>>>>> Steve
>>>>>
>>>>
>>>> On Wed, Jul 29, 2015 at 3:35 AM, Haïkel <hguemar at fedoraproject.org> wrote:
>>>>> I don't know much about MidoNet, but in order to get accepted in RDO
>>>>>
>>>>> 1. licensing should be compatible with RDO
>>>>
>>>> Apache Software License 2.0
>>>>
>>>>> 2. it has to be an upstreamed effort
>>>>
>>>> Like with most Neutron plugins, the plugin itself is upstream but the
>>>> rest isn't.
>>>>
>>>> Neutron MidoNet Plugin (as shared by Steve above already):
>>>> http://git.openstack.org/cgit/openstack/networking-midonet/
>>>>
>>>> MidoNet:
>>>> https://github.com/midonet/midonet/
>>>>
>>>>
>>>>> 3. an identified maintainer (RDO Eng. will focus on maintaining "core"
>>>>> projects, and we won't be maintaining all neutron drivers)
>>>>
>>>> That should be no problem. Does this have to be a single person or can
>>>> it be a team?
>>>
>>> A single person can be a team :)
>>>
>>>>> 4. provide packages conforming Fedora guidelines
>>>>
>>>> Okay, so - as Adam and Steve already hinted - non-conforming packages
>>>> (hosted on our own repo) are not acceptable then? Our biggest headache
>>>> right now (like with any Java software) in terms of the Fedora
>>>> Packaging Guidelines are tons of bundled jars - so that would be a
>>>> no-go, right?
>>>
>>> I think we need to relax the requirements on #4 here. For the Neutron
>>> plugin, I think we can/should conform to Fedora packaging guidelines,
>>> and definitely get that package hosted on RDO directly.
>>>
>>> For the SDN solution in Java, I think we should either allow:
>>> * Relaxed packaging guidelines, recognizing that this is an SDN
>>>   solution, and we host Midonet RPMs on RDO site
>>>   (i.e. allow jar bundling for this)
>>> * Or allow RDO Manager to pull packages from your repos, not hosted on
>>>   RDO site
>>>
>>> I think either path should be acceptable.
>>
>> Yes, this is exactly right. We want the neutron plugin to be packaged
>> such that we can build it into the overcloud images, but the SDN
>> solution can easily be added post-build with virt-customize or
>> equivalent. I would never recommend anyone go through the full java
>> un-bundling process without a very, very good reason.
> 
> Right, as mentioned above, I think both the plugin and the MidoNet
> Agent should be included in the overcloud images as nova-compute will
> require the agent - unless there's some workaround for that but I'd
> rather avoid workarounds. I had to ask one of our tech guys, but he
> agrees the that the rest can be added post-build.
> 

Agreed.  I'm happy to help with the virt-customize steps when we get to
that part.  I'll likely have something banged together on the rdo wiki
by then that can serve as a simple scenario-based guide to adding small
bits of content into the overcloud images.


>>>>> 5. take responsibility for CI
>>>>
>>>> Of course.
>>>
>>> With assistance from the CI team in RDO, of course, to get you started.
> 
> Great, I'm sure our CI guys will appreciate that very much.
> 
>>>>> As far as these conditions are respected, there should be no problems
>>>>> in accepting MidoNet in RDO.
>>>>
>>>> Getting all these deps packaged will really be a major effort so if
>>>> that's required, it will take a while. Otherwise, I see no obstacles
>>>> :)
>>>>
>>>>> In brief, what we require is commitment and taking responsibility of
>>>>> contributed packages.
>>>>
>>>> Sure, that's in our best interest anyway :)
>>>>
>>>>> Sandro, we'll both be at Flock in few weeks, so I'll be glad to
>>>>> discuss with you about it.
>>>>
>>>> Sure, let's do that :)
>>>
>>> Sounds like a plan.
>>>
>>> Sandro, glad to see you engaging with us in the community here, and
>>> looking forward to seeing how this integration works!
>>>
>>> Thanks,
>>>
>>> Perry
>>
>> +1 from me, we're really looking forward to getting Midonet into RDO Manager!
> 
> +1, and likewise! Glad you're open to this :)
> 
> Thanks,
> Sandro
> 
> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
> 
> To unsubscribe: rdo-list-unsubscribe at redhat.com
> 

I'm really excited about this as well!  If you would like some more real
time interaction particularly on the rdo-manager front I'd be happy to
chat on #rdo on freenode (nick == morazi).  If I don't have answers for
you, I'm pretty sure I'll be able to find folks that can help out.

Thanks in advance,
- Mike