[rdo-list] TripleO UI Packaging Strategy

Tue Jul 19 13:02:14 UTC 2016

2016-07-19 12:37 GMT+02:00 Mark McLoughlin <markmc at redhat.com>:
> On Fri, Jul 15, 2016 at 6:29 PM, Haïkel <hguemar at fedoraproject.org> wrote:
>> 2016-07-15 17:31 GMT+02:00 Dougal Matthews <dougal at redhat.com>:
>>> On 15 July 2016 at 16:27, David Moreau Simard <dms at redhat.com> wrote:
>>>>
>>>> I think TripleO-UI can draw from a lot of the work that has been done
>>>> in Horizon packaging [1] (adding mrunge).
>>>>
>>>> You can see that most of the libraries are made available through
>>>> xstatic python packages, for example jquery.
>>>> If there are missing libraries they need to be highlighted so we can
>>>> package them.
>>>
>>>
>>> Due to the UI being built in react and using the npm ecosystem I think it
>>> has over 800 dependent packages. I'm not sure that doing them all
>>> individually is realistic.
>>>
>>
>> Realistic or not, we are *compelled* to review licensing for each
>> dependencies we ship.
>> I'm not sure we'll be able to ship this in time for GA, considering
>> that we passed M2.
>>
>> We can relax unbunding rules to a certain point but we also need to
>> review the minifying toolchain even if it's not yet packaged.
>> We're not allowed to ship minified javascript through minifiers that
>> are not acceptable (e.g Google Closure compiler).
>> There's a limit of how we can reduce the reviewing churn.
>
> What can be interesting in situations like this is to attempt to
> explain what's an absolute requirement versus where exceptions might
> be acceptable.
>
>
> For everyone's reference, RDO's packaging guidelines are here:
>
> https://www.rdoproject.org/documentation/rdo-packaging-guidelines/
>
> and there is a brief mention of RDO allowing for exceptions to
> Fedora's strict unbundling guidelines:
>
> https://fedoraproject.org/wiki/Bundled_Libraries
>
> Some more reading on Fedora's unbundling policy:
> https://lwn.net/Articles/660429/
>

First, thanks for this complete summary.

This is a must-read, FESCO relaxed a bit Fedora's unbundling policy
while keeping balance between sanity and pragmatism.
Declaring bundled libraries allows tracking them more effectively for
security issues.

>
> Anyway, taking a step back beyond unbundling for one minute, we should
> always require that packages are built in an entirely reproducible way
> from source so that, for example, we always have the ability to
> rebuilt the package with a minor fix and know that there won't be
> side-effects other than the change we intended to make.
>

*nods*

> Some of the implications of that:
>
> * The toolchain you use to build the package must also follow this
> basic rule - because fixing an issue might entail fixing the toolchain
> and rebuilding the package
>
> * Everything[1] must be rebuilt from source that is uploaded to the
> build system - we don't use pre-built artifacts supplied by upstream.
>
> * It should be possible to review (e.g. from a security POV) all of
> the source used to build the final package, and know that everything
> was built from that source code. If you can't imagine us actually
> doing this detailed a review, then think of it as us requiring the
> ability to do a spot-check review.
>
> * The buildsystem should not have access to the internet during a
> build, helping to ensure that the build process isn't using source
> files from outside of the buildsystem.
>
> Many of Fedora's packaging guidelines are very useful to achieve
> consistency for the user, like those around how packages are named or
> versioned, or where files should be installed. Other guidelines
> enforce a policies like Fedora should be built "exclusively from Free
> and Open Source software", and to ensure the wishes of the original
> copyright holders are respected.
>

Yes, we still lean back on those guidelines, as they were designed by
the very same legal team behind RDO.
We can have exceptions, but this has to be reviewed by lawyers.

>
> Back to unbundling ... as Haikel says, this is the one area where a
> relaxing of the Fedora packaging guidelines might make sense in this
> case, particularly for expediency.
>
> Having each dependency individually packaged has a number of benefits:
>
> * The correlation between package version and upstream dependency
> version is clear
> * It's easier to track when a a new upstream version is available
> * It's easier to identify whether a given package is affected by a
> published security issue
> * Updates for users are smaller - they may only have to download and
> update a small individual package rather than a large package
> containing everything
> * Package maintenance can be shared with others - if multiple
> applications all use the same dependency, it's better if they don't
> each package that dependency
> * On-disk and in-memory space can be reduced if multiple applications
> share the same dependency
>
> However, many of these benefits are probably quite small in the case
> of TripleO UI compared to the likely overhead of individually
> packaging everyting, and compared to the benefit of including TripleO
> UI in the distribution.
>
> The exception to this argument that individual packaging is of little
> benefit is that AIUI we are moving away from openstack-puppet-modules
> being one giant package with many puppet modules bundled, towards
> individual packages? The reason for this is that the maintenance
> overhead with a single giant o-p-m package was actually greater than
> using automation to maintain the individual packages? Is this likely
> to also be true in the TripleO UI case?
>
>
> I guess I don't even have a basic understanding of what's going on
> here - for example, it looks like Horizon bundles the Angular
> framework in its upstream git and tarball, but TripleO UI doesn't do
> anything like that with React?
>

Mattias can probably explain it better than I do, but Horizon relies
on Xstatic project.
Xstatic wraps web assets in lightweight python modules for easier
consumption by python applications.
https://xstatic.readthedocs.io/en/latest/https://xstatic.readthedocs.io/en/latest/

So the angular library in the git repo is likely for development
purpose and is not used in the RPM package.
Web assets packaging is quite painful, so their bundling has been
tolerated by most distributions, anyway.

Maybe it's worth that Tripleo-UI tries to see how Horizon handles web
assets, but it could already be the case.

> The starting point for the thread was that the package users install
> would just contain the minified version of the javascript ... which I
> guess would be fine so long as the minification happened in the
> buildsystem, using only "source" artifacts or artifacts built from
> source.
>
>
> Hope that helps,
> Mark.
>

Yes, that's the goal.
Depending the schedule, we may tolerate to use minified sources if we
haven't finished by Newton GA for a very limited time
But it requires that we're actively working on packaging and fixing
the minification toolchain to use alternative component with proper
licensing (if needed).

I suspect that non-free component in the minification toolchain would
also conflict with OpenStack licensing, but web assets are not tracked
as well as python requirements by release team.
So there's little to no licensing check upstream.

H.

> [1] - there are interesting nuances to this - e.g. for C projects, the
> upstream maintainer will usually include configure.in, Makefile.in,
> etc. files in their tarballs. These are likely generated with a
> different version of autoconf and automake that are in the build
> system. If, in future, we had to modify configure.ac and Makefile.am,
> and then regenerate configure.in and Makefile.in using autoconf and
> automake, we run the risk of introducing changes caused by differences
> in the autotools versions. AIUI, this is still an unresolved debate in
> Fedora, and left to discretion of packagers:
> https://fedoraproject.org/wiki/PackagingDrafts/AutoConf
>
>
>
>>>> [1]: https://github.com/rdo-packages/horizon-distgit
>>>>
>>>> David Moreau Simard
>>>> Senior Software Engineer | Openstack RDO
>>>>
>>>> dmsimard = [irc, github, twitter]
>>>>
>>>>
>>>> On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist at redhat.com> wrote:
>>>> > Hey everyone - we are trying to think about our packaging strategy for
>>>> > the TripleO UI and would like some feedback.  Feel free to yell
>>>> > regarding the details as this is high priority.
>>>> >
>>>> > The plan:
>>>> >
>>>> > 1.) Create a spec file for the RPM that includes the pre-compiled
>>>> > (minified, production ready) javascript application.
>>>> > 2.) Push new repository to review RDO repositories
>>>> > RTFM:
>>>> >
>>>> > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-package-to-rdo-trunk
>>>> > 3.) Have people review said package here:
>>>> > https://review.rdoproject.org/r/#/q/status:open
>>>> > 4.) Add info to
>>>> > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml
>>>> > 5.) Package appears in trunk delorean
>>>> >
>>>> >
>>>> > We talked a little and we are thinking that the UI will be able to be
>>>> > installed without the dependency of mistral and zaqar since those are
>>>> > connected services rather than binary dependencies.
>>>> >
>>>> > We are going to try that as a first pass and then iterate.
>>>> >
>>>> > We are targeting next week for this work and already have the beginning
>>>> > of #1, so I am confident we'll be able to begin iterating on the
>>>> > packaging setup.
>>>> >
>>>> > Please let me know if you have any questions.
>>>> >
>>>> > Thanks!
>>>> > Jason
>>>> > --
>>>> > Jason E. Rist
>>>> > Senior Software Engineer
>>>> > OpenStack User Interfaces
>>>> > Red Hat, Inc.
>>>> > openuc: +1.972.707.6408
>>>> > mobile: +1.720.256.3933
>>>> > Freenode: jrist
>>>> > github/twitter: knowncitizen
>>>> >
>>>> > _______________________________________________
>>>> > rdo-list mailing list
>>>> > rdo-list at redhat.com
>>>> > https://www.redhat.com/mailman/listinfo/rdo-list
>>>> >
>>>> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>>
>>>> _______________________________________________
>>>> rdo-list mailing list
>>>> rdo-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>
>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>
>>>
>>>
>>> _______________________________________________
>>> rdo-list mailing list
>>> rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com