[Rdo-list] Integration of MidoNet into RDO Manager

[Sandro Mathys]

On Thu, Jul 30, 2015 at 7:39 PM, Haïkel <hguemar at fedoraproject.org> wrote:
>> That should be no problem. Does this have to be a single person or can
> it be a team?
>
> fine with me
>
>> Our biggest headache right now (like with any Java software) in terms of the Fedora
> Packaging Guidelines are tons of bundled jars - so that would be a no-go, right?
>
> Currently, yes, but it'll change in the near future.
> Starting Mitaka, we're considering to switch over CentOS dist-git as
> our main source
> for packaging and we'll be able to provide bundling libraries
> exception for RDO packages.
> I plan to discuss at Flock with bstinson and kbsingh about our plans,
> and how to enable
> contributors to have access to git & build system safely.

To be honest, we were hoping to be integrated before Mitaka - but then
again, unbundling everything would be such a huge effort that it would
probably take us much longer. So this would still be great news for
us.

Just let me know when and where you guys meet if you want me to join
you so I can maybe weigh in with an outside view.

> We don't have yet a reviewing process for such packages but it's
> mostly a matter of timing.
>
> Only clients and oslo libraries will be shipped in Fedora 24, Fedora
> users could still use
> RDO Delorean packages (which are unsigned).
>
> Regards,
> H.
>
On Thu, Jul 30, 2015 at 8:22 PM, Hugh O. Brock <hbrock at redhat.com> wrote:
> On Thu, Jul 30, 2015 at 07:14:54AM -0400, Perry Myers wrote:
>> > It's combined APT and RPM, but you're right, there's currently no
>> > source packages for either.
>> >
>> > Out of curiosity, why do people need to be able to rebuild them (in
>> > the scenario we're discussing - integrating MidoNet with RDO Manager)?
>>
>> I don't think they strictly need this ability. Providing packages that
>> are rebuildable by the community is a nice thing to do, but it shouldn't
>> be a strict requirement in a case like this, I believe.

To be clear: we do plan to release source packages but our current
build process does not allow for it, so it will take a while to get
there - which is why we were hoping it was not strictly necessary.

>> If anything I'd focus on providing a rebuildable RPM for the Neutron
>> plugin itself, but perhaps skip that for the SDN solution that is paired
>> with it.

That should be absolutely no problem at all, the plugin has different
processes, etc. and the package is in quite good shape.

>> >>>>
>> >>>> Are packages hosted on the developer's repositories acceptable for
>> >>>> integration into RDO Manager? We need to unbundle a couple of things
>> >>>> (or maybe more than just a couple) before we can package them for
>> >>>> inclusion into the proper repository.
>> >>> Typically, using a COPR is just a transition step to getting packages
>> >>> into Fedora;  RDO very much follows the Fedora model.
>> >>>
>> >>> The individual packages themselves must be submitted, reviewed, and
>> >>> maintained.  RDO manager is the last step of the process, and will only
>> >>> work with RDOmanaged packages.
>> >
>> > So all our packages would have to be part of the RDO repository (which
>> > I trust follows the EPEL/Fedora Guidelines)?
>>
>> I think having the packages part of a repo available on RDO would make
>> the end user experience a lot easier, but I don't think there is/should
>> be a strict requirement that anything integrated with RDO and RDO
>> Manager _must_ be hosted on the RDO site.

Agreed. Basically, we'd like to first use our external repo until
we're ready for inclusion in the RDO repo (with bundled jars).

>> We might want to differentiate between (for example) the Neutron plugin
>> packages and the SDN solution, for that.
>>
>> I also want to make sure folks understand that I don't think that
>> getting packages like this 'into Fedora' should be a requirement.
>>
>> Right now we are tied to getting things into Fedora since we don't have
>> another place to host spec files, etc, but we're working on decoupling
>> from this so that we can move to a model where we are 'on Fedora'
>> instead of 'in Fedora'

That makes sense. So I guess this would work well with what I said
above: use an external repo at first until 1) our packages (or rather
build processes) are in a better shape (i.e. use packaging tools) and
2) RDO is not "in Fedora" anymore.

>> But to the extent that we can, we try to follow Fedora packaging
>> guidelines as a best practice, but can/will make exceptions where it
>> makes sense.

I really hope we won't need any other exception than "bundled
(fat)jars", and don't think we will.

>> >>>> Speaking of repositories, once we're ready to package them properly
>> >>>> for inclusion, which repository would be the (most?) proper one for
>> >>>> RDO Manager? RDO? CentOS (wherever Cloud SIG packages go)? EPEL?
>> >>> Its usually easiest to start with Fedora for all packaging.  Once they
>> >>> are accepted into Fedora, figuring out how to get them into the
>> >>> appropriate other locations will follow.
>> >
>> > Well, for our packages, Fedora and EL would be fairly different. The
>> > MidoNet core is written in Java/Scala, so much more (tools, deps) is
>> > missing from EL, e.g. gradle and of course lots of artifacts. So we
>> > should target EPEL, I guess.
>>
>> I wouldn't follow Adam's advice here (starting with Fedora). Especially
>> for the SDN solution which is Java based. That would lead to a lot of
>> pain and overhead.
>>
>> >>> Thus far, RDO manager has been focused on upstream OpenStack and the
>> >>> necessary pieces from the base OS that need to be updated to support
>> >>> it.  While it should be possible to have an add-on like MidoNet, I don't
>> >>> know how the rest of the community would feel about it being required to
>> >>> be part of RDO.  My thought is that, so long as it A) is under a
>> >>> sufficient license and B) provides real value beyond what is available
>> >>> from Neutron, it should be possible to include, so long as including it
>> >>> does not impact people currently developing and deploying RDO.
>> >>>
>> >>>
>> >>> Is there any move to get MidoNet into upstream OpenStack?
>> >>
>> >> Aren't we talking about the midonet neutron plug-in which was already part of neutron-proper but as a result of the plug-in decomposition effort now has it's own repo under the openstack namespace ( http://git.openstack.org/cgit/openstack/networking-midonet/ ). I'm really unclear on what the issue is here versus other RDO-Manager integration efforts - perhaps you can clarify?
>> >
>> > Yes, that's what we're talking about, thanks for providing the reference :)
>> >
>> > However, I'm a bit confused now - surely, integration would not just
>> > include the plugin but also the SDN solution the plugin is leveraging,
>> > right?
>>
>> That's actually a good question. I think typically we've been thinking
>> about integration of the plugin only, because that is co-located on
>> compute nodes and there might be controller node software that needs to
>> be bundled for certain SDN solutions as well.
>>
>> But I can see where deployment of the SDN solution itself by RDO Manager
>> could be very useful and would make the end user experience much easier.
>> But I'd have to defer to folks on that team as to how and if this could
>> be done :)

Well, one of our main differences e.g. from the default plugin/SDN
solution (ovs) is, that we don't have a central network node but
instead an agent on all compute nodes. So I'm not sure I would fully
agree with that.

> I think we would be anxious to help with this. We want RDO Manager to be
> a one-stop shop for all kinds of deployments -- the more participation
> we have, the better.

Cool, thanks! :)

>> >> Thanks,
>> >>
>> >> Steve
>> >>
>> >
>> > On Wed, Jul 29, 2015 at 3:35 AM, Haïkel <hguemar at fedoraproject.org> wrote:
>> >> I don't know much about MidoNet, but in order to get accepted in RDO
>> >>
>> >> 1. licensing should be compatible with RDO
>> >
>> > Apache Software License 2.0
>> >
>> >> 2. it has to be an upstreamed effort
>> >
>> > Like with most Neutron plugins, the plugin itself is upstream but the
>> > rest isn't.
>> >
>> > Neutron MidoNet Plugin (as shared by Steve above already):
>> > http://git.openstack.org/cgit/openstack/networking-midonet/
>> >
>> > MidoNet:
>> > https://github.com/midonet/midonet/
>> >
>> >
>> >> 3. an identified maintainer (RDO Eng. will focus on maintaining "core"
>> >> projects, and we won't be maintaining all neutron drivers)
>> >
>> > That should be no problem. Does this have to be a single person or can
>> > it be a team?
>>
>> A single person can be a team :)
>>
>> >> 4. provide packages conforming Fedora guidelines
>> >
>> > Okay, so - as Adam and Steve already hinted - non-conforming packages
>> > (hosted on our own repo) are not acceptable then? Our biggest headache
>> > right now (like with any Java software) in terms of the Fedora
>> > Packaging Guidelines are tons of bundled jars - so that would be a
>> > no-go, right?
>>
>> I think we need to relax the requirements on #4 here. For the Neutron
>> plugin, I think we can/should conform to Fedora packaging guidelines,
>> and definitely get that package hosted on RDO directly.
>>
>> For the SDN solution in Java, I think we should either allow:
>> * Relaxed packaging guidelines, recognizing that this is an SDN
>>   solution, and we host Midonet RPMs on RDO site
>>   (i.e. allow jar bundling for this)
>> * Or allow RDO Manager to pull packages from your repos, not hosted on
>>   RDO site
>>
>> I think either path should be acceptable.
>
> Yes, this is exactly right. We want the neutron plugin to be packaged
> such that we can build it into the overcloud images, but the SDN
> solution can easily be added post-build with virt-customize or
> equivalent. I would never recommend anyone go through the full java
> un-bundling process without a very, very good reason.

Right, as mentioned above, I think both the plugin and the MidoNet
Agent should be included in the overcloud images as nova-compute will
require the agent - unless there's some workaround for that but I'd
rather avoid workarounds. I had to ask one of our tech guys, but he
agrees the that the rest can be added post-build.

>> >> 5. take responsibility for CI
>> >
>> > Of course.
>>
>> With assistance from the CI team in RDO, of course, to get you started.

Great, I'm sure our CI guys will appreciate that very much.

>> >> As far as these conditions are respected, there should be no problems
>> >> in accepting MidoNet in RDO.
>> >
>> > Getting all these deps packaged will really be a major effort so if
>> > that's required, it will take a while. Otherwise, I see no obstacles
>> > :)
>> >
>> >> In brief, what we require is commitment and taking responsibility of
>> >> contributed packages.
>> >
>> > Sure, that's in our best interest anyway :)
>> >
>> >> Sandro, we'll both be at Flock in few weeks, so I'll be glad to
>> >> discuss with you about it.
>> >
>> > Sure, let's do that :)
>>
>> Sounds like a plan.
>>
>> Sandro, glad to see you engaging with us in the community here, and
>> looking forward to seeing how this integration works!
>>
>> Thanks,
>>
>> Perry
>
> +1 from me, we're really looking forward to getting Midonet into RDO Manager!

+1, and likewise! Glad you're open to this :)

Thanks,
Sandro