[rdo-list] TripleO UI Packaging Strategy

Florian Fuchs flfuchs at redhat.com
Tue Jul 19 14:08:28 UTC 2016 

On Tue, Jul 19, 2016 at 15:02:14PM +0200, Haïkel wrote:
>2016-07-19 12:37 GMT+02:00 Mark McLoughlin <markmc at redhat.com>:
>> On Fri, Jul 15, 2016 at 6:29 PM, Haïkel <hguemar at fedoraproject.org> wrote:
>>> 2016-07-15 17:31 GMT+02:00 Dougal Matthews <dougal at redhat.com>:
>>>> On 15 July 2016 at 16:27, David Moreau Simard <dms at redhat.com> wrote:
>>>>>
>>>>> I think TripleO-UI can draw from a lot of the work that has been done
>>>>> in Horizon packaging [1] (adding mrunge).
>>>>>
>>>>> You can see that most of the libraries are made available through
>>>>> xstatic python packages, for example jquery.
>>>>> If there are missing libraries they need to be highlighted so we can
>>>>> package them.
>>>>
>>>>
>>>> Due to the UI being built in react and using the npm ecosystem I think it
>>>> has over 800 dependent packages. I'm not sure that doing them all
>>>> individually is realistic.
>>>>
>>>
>>> Realistic or not, we are *compelled* to review licensing for each
>>> dependencies we ship.
>>> I'm not sure we'll be able to ship this in time for GA, considering
>>> that we passed M2.
>>>
>>> We can relax unbunding rules to a certain point but we also need to
>>> review the minifying toolchain even if it's not yet packaged.
>>> We're not allowed to ship minified javascript through minifiers that
>>> are not acceptable (e.g Google Closure compiler).
>>> There's a limit of how we can reduce the reviewing churn.
>>
>> What can be interesting in situations like this is to attempt to
>> explain what's an absolute requirement versus where exceptions might
>> be acceptable.
>>
>>
>> For everyone's reference, RDO's packaging guidelines are here:
>>
>> https://www.rdoproject.org/documentation/rdo-packaging-guidelines/
>>
>> and there is a brief mention of RDO allowing for exceptions to
>> Fedora's strict unbundling guidelines:
>>
>> https://fedoraproject.org/wiki/Bundled_Libraries
>>
>> Some more reading on Fedora's unbundling policy:
>> https://lwn.net/Articles/660429/
>>
>
>First, thanks for this complete summary.
>
>This is a must-read, FESCO relaxed a bit Fedora's unbundling policy
>while keeping balance between sanity and pragmatism.
>Declaring bundled libraries allows tracking them more effectively for
>security issues.
>
>>
>> Anyway, taking a step back beyond unbundling for one minute, we should
>> always require that packages are built in an entirely reproducible way
>> from source so that, for example, we always have the ability to
>> rebuilt the package with a minor fix and know that there won't be
>> side-effects other than the change we intended to make.
>>
>
>*nods*
>
>
>> Some of the implications of that:
>>
>> * The toolchain you use to build the package must also follow this
>> basic rule - because fixing an issue might entail fixing the toolchain
>> and rebuilding the package
>>
>> * Everything[1] must be rebuilt from source that is uploaded to the
>> build system - we don't use pre-built artifacts supplied by upstream.
>>
>> * It should be possible to review (e.g. from a security POV) all of
>> the source used to build the final package, and know that everything
>> was built from that source code. If you can't imagine us actually
>> doing this detailed a review, then think of it as us requiring the
>> ability to do a spot-check review.
>>
>> * The buildsystem should not have access to the internet during a
>> build, helping to ensure that the build process isn't using source
>> files from outside of the buildsystem.
>>
>> Many of Fedora's packaging guidelines are very useful to achieve
>> consistency for the user, like those around how packages are named or
>> versioned, or where files should be installed. Other guidelines
>> enforce a policies like Fedora should be built "exclusively from Free
>> and Open Source software", and to ensure the wishes of the original
>> copyright holders are respected.
>>
>
>Yes, we still lean back on those guidelines, as they were designed by
>the very same legal team behind RDO.
>We can have exceptions, but this has to be reviewed by lawyers.
>
>>
>> Back to unbundling ... as Haikel says, this is the one area where a
>> relaxing of the Fedora packaging guidelines might make sense in this
>> case, particularly for expediency.
>>
>> Having each dependency individually packaged has a number of benefits:
>>
>> * The correlation between package version and upstream dependency
>> version is clear
>> * It's easier to track when a a new upstream version is available
>> * It's easier to identify whether a given package is affected by a
>> published security issue
>> * Updates for users are smaller - they may only have to download and
>> update a small individual package rather than a large package
>> containing everything
>> * Package maintenance can be shared with others - if multiple
>> applications all use the same dependency, it's better if they don't
>> each package that dependency
>> * On-disk and in-memory space can be reduced if multiple applications
>> share the same dependency
>>
>> However, many of these benefits are probably quite small in the case
>> of TripleO UI compared to the likely overhead of individually
>> packaging everyting, and compared to the benefit of including TripleO
>> UI in the distribution.
>>
>> The exception to this argument that individual packaging is of little
>> benefit is that AIUI we are moving away from openstack-puppet-modules
>> being one giant package with many puppet modules bundled, towards
>> individual packages? The reason for this is that the maintenance
>> overhead with a single giant o-p-m package was actually greater than
>> using automation to maintain the individual packages? Is this likely
>> to also be true in the TripleO UI case?
>>
>>
>> I guess I don't even have a basic understanding of what's going on
>> here - for example, it looks like Horizon bundles the Angular
>> framework in its upstream git and tarball, but TripleO UI doesn't do
>> anything like that with React?
>>
>
>Mattias can probably explain it better than I do, but Horizon relies
>on Xstatic project.
>Xstatic wraps web assets in lightweight python modules for easier
>consumption by python applications.
>https://xstatic.readthedocs.io/en/latest/https://xstatic.readthedocs.io/en/latest/
>
>So the angular library in the git repo is likely for development
>purpose and is not used in the RPM package.
>Web assets packaging is quite painful, so their bundling has been
>tolerated by most distributions, anyway.
>
>
>Maybe it's worth that Tripleo-UI tries to see how Horizon handles web
>assets, but it could already be the case.
>
>> The starting point for the thread was that the package users install
>> would just contain the minified version of the javascript ... which I
>> guess would be fine so long as the minification happened in the
>> buildsystem, using only "source" artifacts or artifacts built from
>> source.
>>
>>
>> Hope that helps,
>> Mark.
>>
>
>Yes, that's the goal.
>Depending the schedule, we may tolerate to use minified sources if we
>haven't finished by Newton GA for a very limited time
>But it requires that we're actively working on packaging and fixing
>the minification toolchain to use alternative component with proper
>licensing (if needed).


So a couple of questions, assuming/suggesting the following workflow:


1. In a first step, we make sure the build tools and dependencies exist as
node modules on the build system, so it can compile the target JS files from
it. We make sure all dependencies have compatible licenses.

2. On each new tripleo-ui release, the build system compiles new target JS
files using the dependencies from step 1.

3. The build system adds the compiled files to the new package (which is
otherwise based on the tripleo-ui distgit repo).


Questions:

- Is this workflow plausible/acceptable/feasible?

- If it is, would that flow be good acceptable for now only, or even
  permanently, given that all sources are free software and the build is
  transparent and reproducible?

- Would version changes for dependencies have to be reviewed separately or
  could they be updated with each new build based on the version information
  in the upstream repo's package.json file?

- Could steps 1. and 2. be combined, so tools and dependencies are updated
  and installed on each new release? (Assuming dependency changes are
  reviewed beforehand.)


Thanks for clarifying!
Florian


>
>I suspect that non-free component in the minification toolchain would
>also conflict with OpenStack licensing, but web assets are not tracked
>as well as python requirements by release team.
>So there's little to no licensing check upstream.
>
>H.
>
>
>> [1] - there are interesting nuances to this - e.g. for C projects, the
>> upstream maintainer will usually include configure.in, Makefile.in,
>> etc. files in their tarballs. These are likely generated with a
>> different version of autoconf and automake that are in the build
>> system. If, in future, we had to modify configure.ac and Makefile.am,
>> and then regenerate configure.in and Makefile.in using autoconf and
>> automake, we run the risk of introducing changes caused by differences
>> in the autotools versions. AIUI, this is still an unresolved debate in
>> Fedora, and left to discretion of packagers:
>> https://fedoraproject.org/wiki/PackagingDrafts/AutoConf
>>
>>
>>
>>>>> [1]: https://github.com/rdo-packages/horizon-distgit
>>>>>
>>>>> David Moreau Simard
>>>>> Senior Software Engineer | Openstack RDO
>>>>>
>>>>> dmsimard = [irc, github, twitter]
>>>>>
>>>>>
>>>>> On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist at redhat.com> wrote:
>>>>> > Hey everyone - we are trying to think about our packaging strategy for
>>>>> > the TripleO UI and would like some feedback.  Feel free to yell
>>>>> > regarding the details as this is high priority.
>>>>> >
>>>>> > The plan:
>>>>> >
>>>>> > 1.) Create a spec file for the RPM that includes the pre-compiled
>>>>> > (minified, production ready) javascript application.
>>>>> > 2.) Push new repository to review RDO repositories
>>>>> > RTFM:
>>>>> >
>>>>> > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-package-to-rdo-trunk
>>>>> > 3.) Have people review said package here:
>>>>> > https://review.rdoproject.org/r/#/q/status:open
>>>>> > 4.) Add info to
>>>>> > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml
>>>>> > 5.) Package appears in trunk delorean
>>>>> >
>>>>> >
>>>>> > We talked a little and we are thinking that the UI will be able to be
>>>>> > installed without the dependency of mistral and zaqar since those are
>>>>> > connected services rather than binary dependencies.
>>>>> >
>>>>> > We are going to try that as a first pass and then iterate.
>>>>> >
>>>>> > We are targeting next week for this work and already have the beginning
>>>>> > of #1, so I am confident we'll be able to begin iterating on the
>>>>> > packaging setup.
>>>>> >
>>>>> > Please let me know if you have any questions.
>>>>> >
>>>>> > Thanks!
>>>>> > Jason
>>>>> > --
>>>>> > Jason E. Rist
>>>>> > Senior Software Engineer
>>>>> > OpenStack User Interfaces
>>>>> > Red Hat, Inc.
>>>>> > openuc: +1.972.707.6408
>>>>> > mobile: +1.720.256.3933
>>>>> > Freenode: jrist
>>>>> > github/twitter: knowncitizen
>>>>> >
>>>>> > _______________________________________________
>>>>> > rdo-list mailing list
>>>>> > rdo-list at redhat.com
>>>>> > https://www.redhat.com/mailman/listinfo/rdo-list
>>>>> >
>>>>> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>>>
>>>>> _______________________________________________
>>>>> rdo-list mailing list
>>>>> rdo-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>
>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rdo-list mailing list
>>>> rdo-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>
>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>
>>> _______________________________________________
>>> rdo-list mailing list
>>> rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
>_______________________________________________
>rdo-list mailing list
>rdo-list at redhat.com
>https://www.redhat.com/mailman/listinfo/rdo-list
>
>To unsubscribe: rdo-list-unsubscribe at redhat.com

More information about the dev mailing list