[rdo-list] TripleO UI Packaging Strategy

Tue Jul 19 10:37:16 UTC 2016

On Fri, Jul 15, 2016 at 6:29 PM, Haïkel <hguemar at fedoraproject.org> wrote:
> 2016-07-15 17:31 GMT+02:00 Dougal Matthews <dougal at redhat.com>:
>> On 15 July 2016 at 16:27, David Moreau Simard <dms at redhat.com> wrote:
>>>
>>> I think TripleO-UI can draw from a lot of the work that has been done
>>> in Horizon packaging [1] (adding mrunge).
>>>
>>> You can see that most of the libraries are made available through
>>> xstatic python packages, for example jquery.
>>> If there are missing libraries they need to be highlighted so we can
>>> package them.
>>
>>
>> Due to the UI being built in react and using the npm ecosystem I think it
>> has over 800 dependent packages. I'm not sure that doing them all
>> individually is realistic.
>>
>
> Realistic or not, we are *compelled* to review licensing for each
> dependencies we ship.
> I'm not sure we'll be able to ship this in time for GA, considering
> that we passed M2.
>
> We can relax unbunding rules to a certain point but we also need to
> review the minifying toolchain even if it's not yet packaged.
> We're not allowed to ship minified javascript through minifiers that
> are not acceptable (e.g Google Closure compiler).
> There's a limit of how we can reduce the reviewing churn.

What can be interesting in situations like this is to attempt to
explain what's an absolute requirement versus where exceptions might
be acceptable.

For everyone's reference, RDO's packaging guidelines are here:

https://www.rdoproject.org/documentation/rdo-packaging-guidelines/

and there is a brief mention of RDO allowing for exceptions to
Fedora's strict unbundling guidelines:

https://fedoraproject.org/wiki/Bundled_Libraries

Some more reading on Fedora's unbundling policy:
https://lwn.net/Articles/660429/

Anyway, taking a step back beyond unbundling for one minute, we should
always require that packages are built in an entirely reproducible way
from source so that, for example, we always have the ability to
rebuilt the package with a minor fix and know that there won't be
side-effects other than the change we intended to make.

Some of the implications of that:

* The toolchain you use to build the package must also follow this
basic rule - because fixing an issue might entail fixing the toolchain
and rebuilding the package

* Everything[1] must be rebuilt from source that is uploaded to the
build system - we don't use pre-built artifacts supplied by upstream.

* It should be possible to review (e.g. from a security POV) all of
the source used to build the final package, and know that everything
was built from that source code. If you can't imagine us actually
doing this detailed a review, then think of it as us requiring the
ability to do a spot-check review.

* The buildsystem should not have access to the internet during a
build, helping to ensure that the build process isn't using source
files from outside of the buildsystem.

Many of Fedora's packaging guidelines are very useful to achieve
consistency for the user, like those around how packages are named or
versioned, or where files should be installed. Other guidelines
enforce a policies like Fedora should be built "exclusively from Free
and Open Source software", and to ensure the wishes of the original
copyright holders are respected.

Back to unbundling ... as Haikel says, this is the one area where a
relaxing of the Fedora packaging guidelines might make sense in this
case, particularly for expediency.

Having each dependency individually packaged has a number of benefits:

* The correlation between package version and upstream dependency
version is clear
* It's easier to track when a a new upstream version is available
* It's easier to identify whether a given package is affected by a
published security issue
* Updates for users are smaller - they may only have to download and
update a small individual package rather than a large package
containing everything
* Package maintenance can be shared with others - if multiple
applications all use the same dependency, it's better if they don't
each package that dependency
* On-disk and in-memory space can be reduced if multiple applications
share the same dependency

However, many of these benefits are probably quite small in the case
of TripleO UI compared to the likely overhead of individually
packaging everyting, and compared to the benefit of including TripleO
UI in the distribution.

The exception to this argument that individual packaging is of little
benefit is that AIUI we are moving away from openstack-puppet-modules
being one giant package with many puppet modules bundled, towards
individual packages? The reason for this is that the maintenance
overhead with a single giant o-p-m package was actually greater than
using automation to maintain the individual packages? Is this likely
to also be true in the TripleO UI case?

I guess I don't even have a basic understanding of what's going on
here - for example, it looks like Horizon bundles the Angular
framework in its upstream git and tarball, but TripleO UI doesn't do
anything like that with React?

The starting point for the thread was that the package users install
would just contain the minified version of the javascript ... which I
guess would be fine so long as the minification happened in the
buildsystem, using only "source" artifacts or artifacts built from
source.

Hope that helps,
Mark.

[1] - there are interesting nuances to this - e.g. for C projects, the
upstream maintainer will usually include configure.in, Makefile.in,
etc. files in their tarballs. These are likely generated with a
different version of autoconf and automake that are in the build
system. If, in future, we had to modify configure.ac and Makefile.am,
and then regenerate configure.in and Makefile.in using autoconf and
automake, we run the risk of introducing changes caused by differences
in the autotools versions. AIUI, this is still an unresolved debate in
Fedora, and left to discretion of packagers:
https://fedoraproject.org/wiki/PackagingDrafts/AutoConf

>>> [1]: https://github.com/rdo-packages/horizon-distgit
>>>
>>> David Moreau Simard
>>> Senior Software Engineer | Openstack RDO
>>>
>>> dmsimard = [irc, github, twitter]
>>>
>>>
>>> On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist at redhat.com> wrote:
>>> > Hey everyone - we are trying to think about our packaging strategy for
>>> > the TripleO UI and would like some feedback.  Feel free to yell
>>> > regarding the details as this is high priority.
>>> >
>>> > The plan:
>>> >
>>> > 1.) Create a spec file for the RPM that includes the pre-compiled
>>> > (minified, production ready) javascript application.
>>> > 2.) Push new repository to review RDO repositories
>>> > RTFM:
>>> >
>>> > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-package-to-rdo-trunk
>>> > 3.) Have people review said package here:
>>> > https://review.rdoproject.org/r/#/q/status:open
>>> > 4.) Add info to
>>> > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml
>>> > 5.) Package appears in trunk delorean
>>> >
>>> >
>>> > We talked a little and we are thinking that the UI will be able to be
>>> > installed without the dependency of mistral and zaqar since those are
>>> > connected services rather than binary dependencies.
>>> >
>>> > We are going to try that as a first pass and then iterate.
>>> >
>>> > We are targeting next week for this work and already have the beginning
>>> > of #1, so I am confident we'll be able to begin iterating on the
>>> > packaging setup.
>>> >
>>> > Please let me know if you have any questions.
>>> >
>>> > Thanks!
>>> > Jason
>>> > --
>>> > Jason E. Rist
>>> > Senior Software Engineer
>>> > OpenStack User Interfaces
>>> > Red Hat, Inc.
>>> > openuc: +1.972.707.6408
>>> > mobile: +1.720.256.3933
>>> > Freenode: jrist
>>> > github/twitter: knowncitizen
>>> >
>>> > _______________________________________________
>>> > rdo-list mailing list
>>> > rdo-list at redhat.com
>>> > https://www.redhat.com/mailman/listinfo/rdo-list
>>> >
>>> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>
>>> _______________________________________________
>>> rdo-list mailing list
>>> rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
>>
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com