I think TripleO-UI can draw from a lot of the work that has been done in Horizon packaging [1] (adding mrunge). You can see that most of the libraries are made available through xstatic python packages, for example jquery. If there are missing libraries they need to be highlighted so we can package them.

David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist(a)redhat.com&gt; wrote: > Hey everyone - we are trying to think about our packaging strategy for > the TripleO UI and would like some feedback. Feel free to yell > regarding the details as this is high priority. > > The plan: > > 1.) Create a spec file for the RPM that includes the pre-compiled > (minified, production ready) javascript application. > 2.) Push new repository to review RDO repositories > RTFM: > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-... > 3.) Have people review said package here: > https://review.rdoproject.org/r/#/q/status:open > 4.) Add info to > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml > 5.) Package appears in trunk delorean > > > We talked a little and we are thinking that the UI will be able to be > installed without the dependency of mistral and zaqar since those are > connected services rather than binary dependencies. > > We are going to try that as a first pass and then iterate. > > We are targeting next week for this work and already have the beginning > of #1, so I am confident we'll be able to begin iterating on the > packaging setup. > > Please let me know if you have any questions. > > Thanks! > Jason > -- > Jason E. Rist > Senior Software Engineer > OpenStack User Interfaces > Red Hat, Inc. > openuc: +1.972.707.6408 > mobile: +1.720.256.3933 > Freenode: jrist > github/twitter: knowncitizen > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On 15 July 2016 at 16:27, David Moreau Simard <dms(a)redhat.com&gt; wrote: > > I think TripleO-UI can draw from a lot of the work that has been done > in Horizon packaging [1] (adding mrunge). > > You can see that most of the libraries are made available through > xstatic python packages, for example jquery. > If there are missing libraries they need to be highlighted so we can > package them. Due to the UI being built in react and using the npm ecosystem I think it has over 800 dependent packages. I'm not sure that doing them all individually is realistic.

> [1]: https://github.com/rdo-packages/horizon-distgit > > David Moreau Simard > Senior Software Engineer | Openstack RDO > > dmsimard = [irc, github, twitter] > > > On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist(a)redhat.com&gt; wrote: > > Hey everyone - we are trying to think about our packaging strategy for > > the TripleO UI and would like some feedback. Feel free to yell > > regarding the details as this is high priority. > > > > The plan: > > > > 1.) Create a spec file for the RPM that includes the pre-compiled > > (minified, production ready) javascript application. > > 2.) Push new repository to review RDO repositories > > RTFM: > > > > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-... > > 3.) Have people review said package here: > > https://review.rdoproject.org/r/#/q/status:open > > 4.) Add info to > > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml > > 5.) Package appears in trunk delorean > > > > > > We talked a little and we are thinking that the UI will be able to be > > installed without the dependency of mistral and zaqar since those are > > connected services rather than binary dependencies. > > > > We are going to try that as a first pass and then iterate. > > > > We are targeting next week for this work and already have the beginning > > of #1, so I am confident we'll be able to begin iterating on the > > packaging setup. > > > > Please let me know if you have any questions. > > > > Thanks! > > Jason > > -- > > Jason E. Rist > > Senior Software Engineer > > OpenStack User Interfaces > > Red Hat, Inc. > > openuc: +1.972.707.6408 > > mobile: +1.720.256.3933 > > Freenode: jrist > > github/twitter: knowncitizen > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list(a)redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

2016-07-15 17:31 GMT+02:00 Dougal Matthews <dougal(a)redhat.com&gt;: > On 15 July 2016 at 16:27, David Moreau Simard <dms(a)redhat.com&gt; wrote: >> >> I think TripleO-UI can draw from a lot of the work that has been done >> in Horizon packaging [1] (adding mrunge). >> >> You can see that most of the libraries are made available through >> xstatic python packages, for example jquery. >> If there are missing libraries they need to be highlighted so we can >> package them. > > > Due to the UI being built in react and using the npm ecosystem I think it > has over 800 dependent packages. I'm not sure that doing them all > individually is realistic. > Realistic or not, we are *compelled* to review licensing for each dependencies we ship. I'm not sure we'll be able to ship this in time for GA, considering that we passed M2. We can relax unbunding rules to a certain point but we also need to review the minifying toolchain even if it's not yet packaged. We're not allowed to ship minified javascript through minifiers that are not acceptable (e.g Google Closure compiler). There's a limit of how we can reduce the reviewing churn.

>> [1]: https://github.com/rdo-packages/horizon-distgit >> >> David Moreau Simard >> Senior Software Engineer | Openstack RDO >> >> dmsimard = [irc, github, twitter] >> >> >> On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist(a)redhat.com&gt; wrote: >> > Hey everyone - we are trying to think about our packaging strategy for >> > the TripleO UI and would like some feedback. Feel free to yell >> > regarding the details as this is high priority. >> > >> > The plan: >> > >> > 1.) Create a spec file for the RPM that includes the pre-compiled >> > (minified, production ready) javascript application. >> > 2.) Push new repository to review RDO repositories >> > RTFM: >> > >> > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-... >> > 3.) Have people review said package here: >> > https://review.rdoproject.org/r/#/q/status:open >> > 4.) Add info to >> > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml >> > 5.) Package appears in trunk delorean >> > >> > >> > We talked a little and we are thinking that the UI will be able to be >> > installed without the dependency of mistral and zaqar since those are >> > connected services rather than binary dependencies. >> > >> > We are going to try that as a first pass and then iterate. >> > >> > We are targeting next week for this work and already have the beginning >> > of #1, so I am confident we'll be able to begin iterating on the >> > packaging setup. >> > >> > Please let me know if you have any questions. >> > >> > Thanks! >> > Jason >> > -- >> > Jason E. Rist >> > Senior Software Engineer >> > OpenStack User Interfaces >> > Red Hat, Inc. >> > openuc: +1.972.707.6408 >> > mobile: +1.720.256.3933 >> > Freenode: jrist >> > github/twitter: knowncitizen >> > >> > _______________________________________________ >> > rdo-list mailing list >> > rdo-list(a)redhat.com >> > https://www.redhat.com/mailman/listinfo/rdo-list >> > >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

2016-07-19 12:37 GMT+02:00 Mark McLoughlin <markmc(a)redhat.com&gt;: > On Fri, Jul 15, 2016 at 6:29 PM, Haïkel <hguemar(a)fedoraproject.org&gt; wrote: >> 2016-07-15 17:31 GMT+02:00 Dougal Matthews <dougal(a)redhat.com&gt;: >>> On 15 July 2016 at 16:27, David Moreau Simard <dms(a)redhat.com&gt; wrote: >>>> >>>> I think TripleO-UI can draw from a lot of the work that has been done >>>> in Horizon packaging [1] (adding mrunge). >>>> >>>> You can see that most of the libraries are made available through >>>> xstatic python packages, for example jquery. >>>> If there are missing libraries they need to be highlighted so we can >>>> package them. >>> >>> >>> Due to the UI being built in react and using the npm ecosystem I think it >>> has over 800 dependent packages. I'm not sure that doing them all >>> individually is realistic. >>> >> >> Realistic or not, we are *compelled* to review licensing for each >> dependencies we ship. >> I'm not sure we'll be able to ship this in time for GA, considering >> that we passed M2. >> >> We can relax unbunding rules to a certain point but we also need to >> review the minifying toolchain even if it's not yet packaged. >> We're not allowed to ship minified javascript through minifiers that >> are not acceptable (e.g Google Closure compiler). >> There's a limit of how we can reduce the reviewing churn. > > What can be interesting in situations like this is to attempt to > explain what's an absolute requirement versus where exceptions might > be acceptable. > > > For everyone's reference, RDO's packaging guidelines are here: > > https://www.rdoproject.org/documentation/rdo-packaging-guidelines/ > > and there is a brief mention of RDO allowing for exceptions to > Fedora's strict unbundling guidelines: > > https://fedoraproject.org/wiki/Bundled_Libraries > > Some more reading on Fedora's unbundling policy: > https://lwn.net/Articles/660429/ > First, thanks for this complete summary. This is a must-read, FESCO relaxed a bit Fedora's unbundling policy while keeping balance between sanity and pragmatism. Declaring bundled libraries allows tracking them more effectively for security issues. > > Anyway, taking a step back beyond unbundling for one minute, we should > always require that packages are built in an entirely reproducible way > from source so that, for example, we always have the ability to > rebuilt the package with a minor fix and know that there won't be > side-effects other than the change we intended to make. > *nods* > Some of the implications of that: > > * The toolchain you use to build the package must also follow this > basic rule - because fixing an issue might entail fixing the toolchain > and rebuilding the package > > * Everything[1] must be rebuilt from source that is uploaded to the > build system - we don't use pre-built artifacts supplied by upstream. > > * It should be possible to review (e.g. from a security POV) all of > the source used to build the final package, and know that everything > was built from that source code. If you can't imagine us actually > doing this detailed a review, then think of it as us requiring the > ability to do a spot-check review. > > * The buildsystem should not have access to the internet during a > build, helping to ensure that the build process isn't using source > files from outside of the buildsystem. > > Many of Fedora's packaging guidelines are very useful to achieve > consistency for the user, like those around how packages are named or > versioned, or where files should be installed. Other guidelines > enforce a policies like Fedora should be built "exclusively from Free > and Open Source software", and to ensure the wishes of the original > copyright holders are respected. > Yes, we still lean back on those guidelines, as they were designed by the very same legal team behind RDO. We can have exceptions, but this has to be reviewed by lawyers. > > Back to unbundling ... as Haikel says, this is the one area where a > relaxing of the Fedora packaging guidelines might make sense in this > case, particularly for expediency. > > Having each dependency individually packaged has a number of benefits: > > * The correlation between package version and upstream dependency > version is clear > * It's easier to track when a a new upstream version is available > * It's easier to identify whether a given package is affected by a > published security issue > * Updates for users are smaller - they may only have to download and > update a small individual package rather than a large package > containing everything > * Package maintenance can be shared with others - if multiple > applications all use the same dependency, it's better if they don't > each package that dependency > * On-disk and in-memory space can be reduced if multiple applications > share the same dependency > > However, many of these benefits are probably quite small in the case > of TripleO UI compared to the likely overhead of individually > packaging everyting, and compared to the benefit of including TripleO > UI in the distribution. > > The exception to this argument that individual packaging is of little > benefit is that AIUI we are moving away from openstack-puppet-modules > being one giant package with many puppet modules bundled, towards > individual packages? The reason for this is that the maintenance > overhead with a single giant o-p-m package was actually greater than > using automation to maintain the individual packages? Is this likely > to also be true in the TripleO UI case? > > > I guess I don't even have a basic understanding of what's going on > here - for example, it looks like Horizon bundles the Angular > framework in its upstream git and tarball, but TripleO UI doesn't do > anything like that with React? > Mattias can probably explain it better than I do, but Horizon relies on Xstatic project. Xstatic wraps web assets in lightweight python modules for easier consumption by python applications. https://xstatic.readthedocs.io/en/latest/https://xstatic.readthedocs.io/e... So the angular library in the git repo is likely for development purpose and is not used in the RPM package. Web assets packaging is quite painful, so their bundling has been tolerated by most distributions, anyway. Maybe it's worth that Tripleo-UI tries to see how Horizon handles web assets, but it could already be the case. > The starting point for the thread was that the package users install > would just contain the minified version of the javascript ... which I > guess would be fine so long as the minification happened in the > buildsystem, using only "source" artifacts or artifacts built from > source. > > > Hope that helps, > Mark. > Yes, that's the goal. Depending the schedule, we may tolerate to use minified sources if we haven't finished by Newton GA for a very limited time But it requires that we're actively working on packaging and fixing the minification toolchain to use alternative component with proper licensing (if needed).

I suspect that non-free component in the minification toolchain would also conflict with OpenStack licensing, but web assets are not tracked as well as python requirements by release team. So there's little to no licensing check upstream. H. > [1] - there are interesting nuances to this - e.g. for C projects, the > upstream maintainer will usually include configure.in, Makefile.in, > etc. files in their tarballs. These are likely generated with a > different version of autoconf and automake that are in the build > system. If, in future, we had to modify configure.ac and Makefile.am, > and then regenerate configure.in and Makefile.in using autoconf and > automake, we run the risk of introducing changes caused by differences > in the autotools versions. AIUI, this is still an unresolved debate in > Fedora, and left to discretion of packagers: > https://fedoraproject.org/wiki/PackagingDrafts/AutoConf > > > >>>> [1]: https://github.com/rdo-packages/horizon-distgit >>>> >>>> David Moreau Simard >>>> Senior Software Engineer | Openstack RDO >>>> >>>> dmsimard = [irc, github, twitter] >>>> >>>> >>>> On Fri, Jul 15, 2016 at 10:56 AM, Jason Rist <jrist(a)redhat.com&gt; wrote: >>>> > Hey everyone - we are trying to think about our packaging strategy for >>>> > the TripleO UI and would like some feedback. Feel free to yell >>>> > regarding the details as this is high priority. >>>> > >>>> > The plan: >>>> > >>>> > 1.) Create a spec file for the RPM that includes the pre-compiled >>>> > (minified, production ready) javascript application. >>>> > 2.) Push new repository to review RDO repositories >>>> > RTFM: >>>> > >>>> > https://www.rdoproject.org/documentation/rdo-packaging/#how-to-add-a-new-... >>>> > 3.) Have people review said package here: >>>> > https://review.rdoproject.org/r/#/q/status:open >>>> > 4.) Add info to >>>> > https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml >>>> > 5.) Package appears in trunk delorean >>>> > >>>> > >>>> > We talked a little and we are thinking that the UI will be able to be >>>> > installed without the dependency of mistral and zaqar since those are >>>> > connected services rather than binary dependencies. >>>> > >>>> > We are going to try that as a first pass and then iterate. >>>> > >>>> > We are targeting next week for this work and already have the beginning >>>> > of #1, so I am confident we'll be able to begin iterating on the >>>> > packaging setup. >>>> > >>>> > Please let me know if you have any questions. >>>> > >>>> > Thanks! >>>> > Jason >>>> > -- >>>> > Jason E. Rist >>>> > Senior Software Engineer >>>> > OpenStack User Interfaces >>>> > Red Hat, Inc. >>>> > openuc: +1.972.707.6408 >>>> > mobile: +1.720.256.3933 >>>> > Freenode: jrist >>>> > github/twitter: knowncitizen >>>> > >>>> > _______________________________________________ >>>> > rdo-list mailing list >>>> > rdo-list(a)redhat.com >>>> > https://www.redhat.com/mailman/listinfo/rdo-list >>>> > >>>> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>> >>>> _______________________________________________ >>>> rdo-list mailing list >>>> rdo-list(a)redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>> >>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >>> >>> >>> _______________________________________________ >>> rdo-list mailing list >>> rdo-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

2016-07-19 16:08 GMT+02:00 Florian Fuchs <flfuchs(a)redhat.com&gt;: > > > So a couple of questions, assuming/suggesting the following workflow: > > > 1. In a first step, we make sure the build tools and dependencies exist as > node modules on the build system, so it can compile the target JS files from > it. We make sure all dependencies have compatible licenses. > > 2. On each new tripleo-ui release, the build system compiles new target JS > files using the dependencies from step 1. > > 3. The build system adds the compiled files to the new package (which is > otherwise based on the tripleo-ui distgit repo). > > > Questions: > > - Is this workflow plausible/acceptable/feasible? > Yes, though I'm not sure what you understand as build system. Build system has no internet access, and except baseOS, only things available are provided sources + dependencies declared as BuildRequires (must be packaged)

> - If it is, would that flow be good acceptable for now only, or even > permanently, given that all sources are free software and the build is > transparent and reproducible? > That's the stable state we want to reach. Depending the amount of work needed, we may tolerate temporary exceptions, but they have to be approved by RDO maintainers in our weekly meeting. > - Would version changes for dependencies have to be reviewed separately or > could they be updated with each new build based on the version information > in the upstream repo's package.json file? > Process is: * initial review when you introduce new package (except if it's packaged in Fedora as they have exemption from legal team) * update bumps are done directly without peer reviewing (well release wranglers and CI are checking sanity) > - Could steps 1. and 2. be combined, so tools and dependencies are updated > and installed on each new release? (Assuming dependency changes are > reviewed beforehand.) > > > Thanks for clarifying! > Florian > Up to a certain extent, while we tolerate bundling web assets, I prefer that we don't bundle the toolchain and try to keep it stable. Not that we'd enforce strict constraints on that, but remember that we have limited ressources to maintain the whole distribution. Regards, H. _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

2016-07-19 16:53 GMT+02:00 Honza Pokorny <hpokorny(a)redhat.com&gt;: > On 2016-07-19 16:29, Haďkel wrote: >> 2016-07-19 16:08 GMT+02:00 Florian Fuchs <flfuchs(a)redhat.com&gt;: >> > >> > >> > So a couple of questions, assuming/suggesting the following workflow: >> > >> > >> > 1. In a first step, we make sure the build tools and dependencies exist as >> > node modules on the build system, so it can compile the target JS files from >> > it. We make sure all dependencies have compatible licenses. >> > >> > 2. On each new tripleo-ui release, the build system compiles new target JS >> > files using the dependencies from step 1. >> > >> > 3. The build system adds the compiled files to the new package (which is >> > otherwise based on the tripleo-ui distgit repo). >> > >> > >> > Questions: >> > >> > - Is this workflow plausible/acceptable/feasible? >> > >> >> Yes, though I'm not sure what you understand as build system. >> Build system has no internet access, and except baseOS, only things >> available are provided sources + dependencies declared as >> BuildRequires (must be packaged) > > I guess this is the biggest problem: our build system is responsible for > both fetching any dependencies and for actually building the project. > Preferably, we'd like to run "npm install" from the RPM spec. This step > requires internet access, and therefore violates one of the rules. Since > our dependencies are fixed/pinned, there is a fair degree of certainty > that the dependencies will be the same each time the build is run[1]. > Npm install brings in sources, not artifacts. > There's an ongoing work to integrate language "native" packages with RPM ecosystem (so-called modularity workgroup in Fedora) In the future, build system will have access to an internal npm registry mirror (and pypi for python, etc.), but that's still an ongoing work and there are many issues to solve (notably reproducibility). For now, we either have to package dependencies or bundle them. Our build and delivery infrastructure is provided by CentOS, these are not constraints that we can bypass, nor we can maintain on our own. The landscape of software engineering was very different when GNU/Linux packaging and distributions were created, and it'll take time to adapt to modern software engineering (for the better or the worst) > What would be your preferred solution? Should we try and use xstatic? > We could also bundle our dependencies along with the source tarball --- > would that ease your mind? > xstatic would be nice but I'd prefer that you check with our horizon developers, first. They have more experience on that topic, especially Mattias who also maintained horizon packaging. Yes, direct dependencies can be bundled, we can tolerate minification toolchain (provided it complies with our licensing terms) bundling but that'd be temporary exception. > [1]: Yes, I know about the leftpad fiasco :) > >> >> > - If it is, would that flow be good acceptable for now only, or even >> > permanently, given that all sources are free software and the build is >> > transparent and reproducible? >> > >> >> That's the stable state we want to reach. >> Depending the amount of work needed, we may tolerate temporary >> exceptions, but they have to be approved by RDO maintainers in our >> weekly meeting. >> >> > - Would version changes for dependencies have to be reviewed separately or >> > could they be updated with each new build based on the version information >> > in the upstream repo's package.json file? >> > >> >> Process is: >> * initial review when you introduce new package (except if it's >> packaged in Fedora as they have exemption from legal team) >> * update bumps are done directly without peer reviewing (well release >> wranglers and CI are checking sanity) >> >> > - Could steps 1. and 2. be combined, so tools and dependencies are updated >> > and installed on each new release? (Assuming dependency changes are >> > reviewed beforehand.) >> > >> > >> > Thanks for clarifying! >> > Florian >> > >> >> Up to a certain extent, while we tolerate bundling web assets, I >> prefer that we don't bundle the toolchain and try to keep it stable. >> Not that we'd enforce strict constraints on that, but remember that we >> have limited ressources to maintain the whole distribution. >> >> Regards, >> H. >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com

2016-07-21 16:23 GMT+02:00 Honza Pokorny <honza(a)redhat.com&gt;: > There still seems to be some confusion about what we're saying, so let > me attempt to summarize: > > 1. bundling of npm dependencies (sources) undesirable but temporarily tolerated > > 2. bundling of build toolchain even more undesirable > > 3. all bundling of sources tolerated temporarily > > 4. start working on packaging build toolchain as soon as possible > > A modern javascript application (both frontend and nodejs) uses npm to > manage its dependencies and to build production/release versions from > sources. All of this configuration information is contained in the > package.json file. The "dependencies" section of that file contains a > list of direct application dependencies; the "devDependencies" section > contains a list of build/minification dependencies. Typically, one will > run "npm install" to fetch all dependencies: these will be placed in the > node_modules/ directory --- npm downloads sources along with artifacts > (e.g. if the package is written in coffee-script, it will contain both > the coffee-script sources and the compiled js). And, we plan to use npm > to also build the minified code (e.g. "npm run build"). > > What I propose to do is: > > 1. On release, run "npm install" to bring in all dependencies > 2. Create a tarball of node_modules > 3. Run "npm pack" to create a release package > 4. The tripleo-ui RPM spec will receive the package from 3. as Source0 > 5. The RPM build system has all of the application and build > dependencies in the node_modules.tar.gz file; it can build minified > dist files (release, production) without internet access > > Once we have this system in place and we can actually ship our code, we > can start working on unbundling those dependencies (i.e removing them > from the node_modules tarball, and publishing them as rpms that the > tripleo-ui.spec can request). > > How does this sound? Do I understand this correctly? > > Honza Pokorny > Looks ok as a first step. Difficulty is that as long as we have this bundling, we won't be able to build TripleO UI in DLRN (except by introducing a specific workaround). H. > On 2016-07-19 17:34, Haïkel wrote: >> 2016-07-19 16:53 GMT+02:00 Honza Pokorny <hpokorny(a)redhat.com&gt;: >> > On 2016-07-19 16:29, Haďkel wrote: >> >> 2016-07-19 16:08 GMT+02:00 Florian Fuchs <flfuchs(a)redhat.com&gt;: >> >> > >> >> > >> >> > So a couple of questions, assuming/suggesting the following workflow: >> >> > >> >> > >> >> > 1. In a first step, we make sure the build tools and dependencies exist as >> >> > node modules on the build system, so it can compile the target JS files from >> >> > it. We make sure all dependencies have compatible licenses. >> >> > >> >> > 2. On each new tripleo-ui release, the build system compiles new target JS >> >> > files using the dependencies from step 1. >> >> > >> >> > 3. The build system adds the compiled files to the new package (which is >> >> > otherwise based on the tripleo-ui distgit repo). >> >> > >> >> > >> >> > Questions: >> >> > >> >> > - Is this workflow plausible/acceptable/feasible? >> >> > >> >> >> >> Yes, though I'm not sure what you understand as build system. >> >> Build system has no internet access, and except baseOS, only things >> >> available are provided sources + dependencies declared as >> >> BuildRequires (must be packaged) >> > >> > I guess this is the biggest problem: our build system is responsible for >> > both fetching any dependencies and for actually building the project. >> > Preferably, we'd like to run "npm install" from the RPM spec. This step >> > requires internet access, and therefore violates one of the rules. Since >> > our dependencies are fixed/pinned, there is a fair degree of certainty >> > that the dependencies will be the same each time the build is run[1]. >> > Npm install brings in sources, not artifacts. >> > >> >> There's an ongoing work to integrate language "native" packages with >> RPM ecosystem (so-called modularity workgroup in Fedora) >> In the future, build system will have access to an internal npm >> registry mirror (and pypi for python, etc.), but that's still an >> ongoing work and there are many issues to solve (notably >> reproducibility). >> >> For now, we either have to package dependencies or bundle them. Our >> build and delivery infrastructure is provided by CentOS, these are not >> constraints that we can bypass, nor we can maintain on our own. >> >> The landscape of software engineering was very different when >> GNU/Linux packaging and distributions were created, and it'll take >> time to adapt to modern software engineering (for the better or the >> worst) >> >> > What would be your preferred solution? Should we try and use xstatic? >> > We could also bundle our dependencies along with the source tarball --- >> > would that ease your mind? >> > >> >> xstatic would be nice but I'd prefer that you check with our horizon >> developers, first. They have more experience on that topic, especially >> Mattias who also maintained horizon packaging. >> Yes, direct dependencies can be bundled, we can tolerate minification >> toolchain (provided it complies with our licensing terms) bundling but >> that'd be temporary exception. >> >> > [1]: Yes, I know about the leftpad fiasco :) >> > >> >> >> >> > - If it is, would that flow be good acceptable for now only, or even >> >> > permanently, given that all sources are free software and the build is >> >> > transparent and reproducible? >> >> > >> >> >> >> That's the stable state we want to reach. >> >> Depending the amount of work needed, we may tolerate temporary >> >> exceptions, but they have to be approved by RDO maintainers in our >> >> weekly meeting. >> >> >> >> > - Would version changes for dependencies have to be reviewed separately or >> >> > could they be updated with each new build based on the version information >> >> > in the upstream repo's package.json file? >> >> > >> >> >> >> Process is: >> >> * initial review when you introduce new package (except if it's >> >> packaged in Fedora as they have exemption from legal team) >> >> * update bumps are done directly without peer reviewing (well release >> >> wranglers and CI are checking sanity) >> >> >> >> > - Could steps 1. and 2. be combined, so tools and dependencies are updated >> >> > and installed on each new release? (Assuming dependency changes are >> >> > reviewed beforehand.) >> >> > >> >> > >> >> > Thanks for clarifying! >> >> > Florian >> >> > >> >> >> >> Up to a certain extent, while we tolerate bundling web assets, I >> >> prefer that we don't bundle the toolchain and try to keep it stable. >> >> Not that we'd enforce strict constraints on that, but remember that we >> >> have limited ressources to maintain the whole distribution. >> >> >> >> Regards, >> >> H. >> >> >> >> _______________________________________________ >> >> rdo-list mailing list >> >> rdo-list(a)redhat.com >> >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com

> * Currently, the build system uses nodejs 0.10.x, let's look at > upgrading that What is the purpose of this? I saw today Fedora 25 is using 6.x but RHEL/CentOS is still

Version : 0.10.42 Release : 4.el7

On 2016-07-23 04:26, Haďkel wrote: > @Jason: As I said during the mentioned call, we have nodejs 0.10.36 > build in CBS -I'm actually the one who built it- and there's no > problem in importing a LTS release (4.x or 6.x) > Mattias is of the opinion we should stick to 4.x which is the current LTS. > http://cbs.centos.org/koji/buildinfo?buildID=8185 > > I'm taking the action of importing a newer nodejs build in our > repository. Jakub and I will be providing support in > packaging/reviewing tripleo-ui. I'm confident we can make this happen early this week, thanks to Haikel and Jakup. As Jason said, if we can get some eyes on the infra patch and the packaging patch that would certainly speed things up. Haikel, if you could have a quick look at the spec file for tripleo-ui, I think that would make a nice starting point. Or, would you rather I submitted a format request for review to RDO at this point?

> Considering we'll be having many missing people due to summer > vacations, I associated Jakub to avoid being the SPOF in that process. > > As we want to have this sorted out before M3, I'd like to have status > during our weekly RDO meetings to follow up progress and solve > impediments. > > Regards, > H.

On 2016-07-25 16:23, Haïkel wrote: > 2016-07-25 16:17 GMT+02:00 Honza Pokorny <honza(a)redhat.com&gt;: > > On 2016-07-23 04:26, Haďkel wrote: > >> @Jason: As I said during the mentioned call, we have nodejs 0.10.36 > >> build in CBS -I'm actually the one who built it- and there's no > >> problem in importing a LTS release (4.x or 6.x) > >> Mattias is of the opinion we should stick to 4.x which is the current LTS. > >> http://cbs.centos.org/koji/buildinfo?buildID=8185 > >> > >> I'm taking the action of importing a newer nodejs build in our > >> repository. Jakub and I will be providing support in > >> packaging/reviewing tripleo-ui. > > > > I'm confident we can make this happen early this week, thanks to Haikel > > and Jakup. As Jason said, if we can get some eyes on the infra patch > > and the packaging patch that would certainly speed things up. > > > > Haikel, if you could have a quick look at the spec file for tripleo-ui, > > I think that would make a nice starting point. Or, would you rather I > > submitted a format request for review to RDO at this point? > > > > Both works for me. RDO review request: https://bugzilla.redhat.com/show_bug.cgi?id=1359950 Gerrit patch: https://review.openstack.org/#/c/344932/

> >> Considering we'll be having many missing people due to summer > >> vacations, I associated Jakub to avoid being the SPOF in that process. > >> > >> As we want to have this sorted out before M3, I'd like to have status > >> during our weekly RDO meetings to follow up progress and solve > >> impediments. > >> > >> Regards, > >> H. _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

2016-07-26 12:33 GMT+02:00 Mark McLoughlin <markmc(a)redhat.com&gt;: > On Tue, Jul 26, 2016 at 1:04 AM, Honza Pokorny <honza(a)redhat.com&gt; wrote: >> On 2016-07-25 16:23, Haïkel wrote: >>> 2016-07-25 16:17 GMT+02:00 Honza Pokorny <honza(a)redhat.com&gt;: >>> > On 2016-07-23 04:26, Haďkel wrote: >>> >> @Jason: As I said during the mentioned call, we have nodejs 0.10.36 >>> >> build in CBS -I'm actually the one who built it- and there's no >>> >> problem in importing a LTS release (4.x or 6.x) >>> >> Mattias is of the opinion we should stick to 4.x which is the current LTS. >>> >> http://cbs.centos.org/koji/buildinfo?buildID=8185 >>> >> >>> >> I'm taking the action of importing a newer nodejs build in our >>> >> repository. Jakub and I will be providing support in >>> >> packaging/reviewing tripleo-ui. >>> > >>> > I'm confident we can make this happen early this week, thanks to Haikel >>> > and Jakup. As Jason said, if we can get some eyes on the infra patch >>> > and the packaging patch that would certainly speed things up. >>> > >>> > Haikel, if you could have a quick look at the spec file for tripleo-ui, >>> > I think that would make a nice starting point. Or, would you rather I >>> > submitted a format request for review to RDO at this point? >>> > >>> >>> Both works for me. >> >> RDO review request: https://bugzilla.redhat.com/show_bug.cgi?id=1359950 >> >> Gerrit patch: https://review.openstack.org/#/c/344932/ > > Thanks, Honza. > > Just looking through the node_modules tarball very briefly, I think > it's going to take quite a bit of review to satisfy ourselves that > everything is really built from source in the build system, and that > we will actually be able to %patch the source for anything shipped in > the final RPM or anything used to build an artifact in the final RPM. > > For a start, I suggest deleting phantomjs-prebuilt and see what > happens. It contains an ELF executable. >

This is the only ELF binary, and it's only required for running tests, so we can drop it. Another good reason is that rebuilding recent webkit on EL7 would take some time.

node_modules.tar.gz should be in a separate tarballs, in order for us to include tripleo-ui in DLRN. Many libs are generated from coffeescript (800 exactly), some libs are duplicated (e.g lodash, source-map, express, base64 etc.) but fighting duplication will be a never-ending task with node apps.

First licensing checks: * React.js: should be ok but patents grant is still debated among legal

* known licenses: MIT, BSD 2 and 3 clauses, ISC, WTFPL 2.0, ASL 2.0, MPL 2.0 (w/ BSD 3 clauses which saves the hassle to figure out compatibility w/ GPL)

* unknown licenses: mostly generated files from coffeescript.

Regards, H. > Another minor observation - the tarball published by upstream should > have a tripleo-ui-$version base directory. > > Mark. > > > > >>> >> Considering we'll be having many missing people due to summer >>> >> vacations, I associated Jakub to avoid being the SPOF in that process. >>> >> >>> >> As we want to have this sorted out before M3, I'd like to have status >>> >> during our weekly RDO meetings to follow up progress and solve >>> >> impediments. >>> >> >>> >> Regards, >>> >> H. >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

There still seems to be some confusion about what we're saying, so let me attempt to summarize: 1. bundling of npm dependencies (sources) undesirable but temporarily tolerated 2. bundling of build toolchain even more undesirable 3. all bundling of sources tolerated temporarily 4. start working on packaging build toolchain as soon as possible A modern javascript application (both frontend and nodejs) uses npm to manage its dependencies and to build production/release versions from sources. All of this configuration information is contained in the package.json file. The "dependencies" section of that file contains a list of direct application dependencies; the "devDependencies" section contains a list of build/minification dependencies. Typically, one will run "npm install" to fetch all dependencies: these will be placed in the node_modules/ directory --- npm downloads sources along with artifacts (e.g. if the package is written in coffee-script, it will contain both the coffee-script sources and the compiled js). And, we plan to use npm to also build the minified code (e.g. "npm run build"). What I propose to do is: 1. On release, run "npm install" to bring in all dependencies 2. Create a tarball of node_modules 3. Run "npm pack" to create a release package

4. The tripleo-ui RPM spec will receive the package from 3. as Source0

5. The RPM build system has all of the application and build dependencies in the node_modules.tar.gz file; it can build minified dist files (release, production) without internet access Once we have this system in place and we can actually ship our code, we can start working on unbundling those dependencies (i.e removing them from the node_modules tarball, and publishing them as rpms that the tripleo-ui.spec can request). How does this sound? Do I understand this correctly? Honza Pokorny On 2016-07-19 17:34, Haïkel wrote: > 2016-07-19 16:53 GMT+02:00 Honza Pokorny <hpokorny(a)redhat.com&gt;: > > On 2016-07-19 16:29, Haďkel wrote: > >> 2016-07-19 16:08 GMT+02:00 Florian Fuchs <flfuchs(a)redhat.com&gt;: > >> > > >> > > >> > So a couple of questions, assuming/suggesting the following workflow: > >> > > >> > > >> > 1. In a first step, we make sure the build tools and dependencies exist as > >> > node modules on the build system, so it can compile the target JS files from > >> > it. We make sure all dependencies have compatible licenses. > >> > > >> > 2. On each new tripleo-ui release, the build system compiles new target JS > >> > files using the dependencies from step 1. > >> > > >> > 3. The build system adds the compiled files to the new package (which is > >> > otherwise based on the tripleo-ui distgit repo). > >> > > >> > > >> > Questions: > >> > > >> > - Is this workflow plausible/acceptable/feasible? > >> > > >> > >> Yes, though I'm not sure what you understand as build system. > >> Build system has no internet access, and except baseOS, only things > >> available are provided sources + dependencies declared as > >> BuildRequires (must be packaged) > > > > I guess this is the biggest problem: our build system is responsible for > > both fetching any dependencies and for actually building the project. > > Preferably, we'd like to run "npm install" from the RPM spec. This step > > requires internet access, and therefore violates one of the rules. Since > > our dependencies are fixed/pinned, there is a fair degree of certainty > > that the dependencies will be the same each time the build is run[1]. > > Npm install brings in sources, not artifacts. > > > > There's an ongoing work to integrate language "native" packages with > RPM ecosystem (so-called modularity workgroup in Fedora) > In the future, build system will have access to an internal npm > registry mirror (and pypi for python, etc.), but that's still an > ongoing work and there are many issues to solve (notably > reproducibility). > > For now, we either have to package dependencies or bundle them. Our > build and delivery infrastructure is provided by CentOS, these are not > constraints that we can bypass, nor we can maintain on our own. > > The landscape of software engineering was very different when > GNU/Linux packaging and distributions were created, and it'll take > time to adapt to modern software engineering (for the better or the > worst) > > > What would be your preferred solution? Should we try and use xstatic? > > We could also bundle our dependencies along with the source tarball --- > > would that ease your mind? > > > > xstatic would be nice but I'd prefer that you check with our horizon > developers, first. They have more experience on that topic, especially > Mattias who also maintained horizon packaging. > Yes, direct dependencies can be bundled, we can tolerate minification > toolchain (provided it complies with our licensing terms) bundling but > that'd be temporary exception. > > > [1]: Yes, I know about the leftpad fiasco :) > > > >> > >> > - If it is, would that flow be good acceptable for now only, or even > >> > permanently, given that all sources are free software and the build is > >> > transparent and reproducible? > >> > > >> > >> That's the stable state we want to reach. > >> Depending the amount of work needed, we may tolerate temporary > >> exceptions, but they have to be approved by RDO maintainers in our > >> weekly meeting. > >> > >> > - Would version changes for dependencies have to be reviewed separately or > >> > could they be updated with each new build based on the version information > >> > in the upstream repo's package.json file? > >> > > >> > >> Process is: > >> * initial review when you introduce new package (except if it's > >> packaged in Fedora as they have exemption from legal team) > >> * update bumps are done directly without peer reviewing (well release > >> wranglers and CI are checking sanity) > >> > >> > - Could steps 1. and 2. be combined, so tools and dependencies are updated > >> > and installed on each new release? (Assuming dependency changes are > >> > reviewed beforehand.) > >> > > >> > > >> > Thanks for clarifying! > >> > Florian > >> > > >> > >> Up to a certain extent, while we tolerate bundling web assets, I > >> prefer that we don't bundle the toolchain and try to keep it stable. > >> Not that we'd enforce strict constraints on that, but remember that we > >> have limited ressources to maintain the whole distribution. > >> > >> Regards, > >> H. > >> > >> _______________________________________________ > >> rdo-list mailing list > >> rdo-list(a)redhat.com > >> https://www.redhat.com/mailman/listinfo/rdo-list > >> > >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On 21/07/16 16:23, Honza Pokorny wrote: > There still seems to be some confusion about what we're saying, so let > me attempt to summarize: > > 1. bundling of npm dependencies (sources) undesirable but temporarily tolerated Taking the conversation from IRC here: I don't think we got an answer on this yet. If you're pulling all dependencies in, and compile a package then, you're basically creating something comparable to statically linked binaries: If a library has a security issue, you're going to rebuild the whole thing.

You mentioned somewhere else, dependencies are pinned: is that true for dependencies of dependencies as well? Or would I get a different tarball, when collecting all dependencies (and deps of deps) in a few weeks?

> node_modules/ directory --- npm downloads sources along with artifacts > (e.g. if the package is written in coffee-script, it will contain both > the coffee-script sources and the compiled js). And, we plan to use npm > to also build the minified code (e.g. "npm run build"). -- Matthias Runge <mrunge(a)redhat.com&gt; Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com