Re: [rdo-list] Multiple tools for deploying and testing TripleO

On Wed, Aug 3, 2016 at 8:45 PM, Emilien Macchi <emilien(a)redhat.com&gt; wrote: > On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; > wrote: > > > > > > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien(a)redhat.com&gt; > wrote: > >> > >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; > >> wrote: > >> > > >> > > >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle(a)redhat.com&gt; > >> > wrote: > >> >> > >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote: > >> >> > Please hear me out. > >> >> > TL;DR, Let's work upstream and make it awesome so that downstream > can > >> >> > be awesome. > >> >> > > >> >> > I've said this before but I'm going to re-iterate that I do not > >> >> > understand why there is so much effort spent around testing > TripleO > >> >> > downstream. > >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI > >> >> > proper. > >> >> > > >> >> > All this work should be done upstream to make TripleO and it's CI > >> >> > super awesome and this would trickle down for free downstream. > >> >> > > >> >> > The RDO Trunk testing pipeline is composed of two tools, today. > >> >> > The TripleO-Quickstart project [1] is a good example of an > initiative > >> >> > that started downstream but always had the intention of being > >> >> > proposed > >> >> > upstream [2] after being "incubated" and fleshed out. > >> >> > >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement > >> >> for > >> >> the > >> >> virtual environment setup done by instack-virt-setup. 3rd party CI > >> >> would > >> >> be > >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup > was > >> >> always > >> >> working. That was the extent of the CI scope defined in the spec. > That > >> >> work is > >> >> not yet completed (see work items in the spec). > >> >> > >> >> Now it seems it is a much more all encompassing > CI/automation/testing > >> >> project > >> >> that is competing in scope with tripleo-ci itself. > >> > > >> > > >> > IMHO you are correct here. There has been quite a bit of discussion > >> > about > >> > removing the parts > >> > of oooq that are outside of the original blueprint to replace > >> > instack-virt-setup w/ oooq. As usual there are many different > opinions > >> > here. I think there are a lot of RDO guys that would prefer a lot of > >> > the > >> > native oooq roles stay where they are, I think that is short sighted > >> > imho. > >> > I agree that anything outside of the blueprint be removed from oooq. > >> > This > >> > would hopefully allow the upstream to be more comfortable with oooq > and > >> > allow us to really start consolidating tools. > >> > > >> > Luckily for the users that still want to use oooq as a full > end-to-end > >> > solution the 3rd party roles can be used even after tearing out these > >> > native > >> > roles. > >> > > >> >> > >> >> > >> >> I'm all for consolidation of these types of tools *if* there is > >> >> interest. > >> > > >> > > >> > Roll call.. is there interest? +1 from me. > >> > > >> >> > >> >> > >> >> However, IMO, incubating these things downstream and then trying to > get > >> >> them > >> >> upstream or get upstream to adopt them is not ideal or a good > example. > >> >> The > >> >> same > >> >> topic came up and was pushed several times with khaleesi, and it > just > >> >> never > >> >> happened, it was continually DOA upstream. > >> > > >> > > >> > True, however that could be a result of the downstream perceiving > >> > barriers ( > >> > real or not ) in incubating projects in upstream openstack. > >> > > >> >> > >> >> > >> >> I think it would be fairly difficult to get tripleo-ci to wholesale > >> >> adopt > >> >> tripleo-quickstart at this stage. The separate irc channel from > >> >> #tripleo > >> >> is not > >> >> conducive to consolidation on tooling and direction imo. > >> > > >> > > >> > The irc channel is easily addressed. We do seem to generate an awful > >> > amount > >> > of chatter though :) > >> > > >> >> > >> >> > >> >> The scope of quickstart is actually not fully understood by myself. > >> >> I've > >> >> also > >> >> heard from some in the upstream TripleO community as well who are > >> >> confused > >> >> by > >> >> its direction and are facing similar difficulties using its > generated > >> >> bash > >> >> scripts that they'd be facing if they were just using TripleO > >> >> documentation > >> >> instead. > >> > > >> > > >> > The point of the generated bash scripts is to create rst > documentation > >> > and > >> > reusable scripts for the end user. Since the documentation and the > >> > generated scripts are equivalent I would expect the same errors, > >> > problems > >> > and issues. I see this as a good thing really. We *want* the CI to > hit > >> > the > >> > same issues as those who are following the doc. > >> > > >> >> > >> >> > >> >> I do think that this sort of problem lends itself easily to one off > >> >> implementations as is quite evidenced in this thread. Everyone/group > >> >> wants > >> >> and > >> >> needs to automate something in a different way. And imo, none of > these > >> >> tools > >> >> are building end-user or operator facing interfaces, so they're not > >> >> fully > >> >> focused on building something that "just works for everyone". Those > >> >> interfaces > >> >> should be developed in TripleO user facing tooling anyway > >> >> (tripleoclient/openstackclient/etc). > >> >> > >> >> So, I actually think it's ok in some degree that things have been > >> >> automated > >> >> differently in different tools. Anecdotally, I suspect many users of > >> >> TripleO in > >> >> production have their own automation tools as well. And none of the > >> >> implementations mentioned in this thread would likely meet their > needs > >> >> either. > >> > > >> > > >> > This is true.. without a tool in the upstream that addresses ci, > dev, > >> > test > >> > use cases across the development cycle this will continue to be the > >> > case. I > >> > suspect even with a perfect tool, it won't ever be perfect for > everyone. > >> > > >> >> > >> >> > >> >> However, if there is a desire to focus resources on consolidated > >> >> tooling > >> >> and > >> >> someone to drive it forward, then I definitely agree that the effort > >> >> needs > >> >> to > >> >> start upstream with a singular plan for tripleo-ci. From what I > gather, > >> >> that > >> >> would be some sort of alignment and reuse of tripleo-quickstart, and > >> >> then > >> >> we > >> >> could build from there. > >> > > >> > > >> > +1 > >> > > >> >> > >> >> > >> >> That could start as a discussion and plan within that community with > >> >> some > >> >> agreed on concensus around that plan. There was an initial thread on > >> >> openstack-dev related to this topic but it is stalled a bit. It > could > >> >> be > >> >> continually driven to resolution via specs, the tripleo meeting, > email > >> >> or > >> >> irc > >> >> discussion until a plan is formed. > >> > > >> > > >> > +1, I think the first step is to complete the original blueprint and > >> > move > >> > on from there. > >> > I think there has also been interest in having an in person meeting > at > >> > summit. > >> > > >> > Thanks! > >> > > >> >> > >> >> > >> >> -- > >> >> -- James Slagle > >> >> -- > >> >> > >> >> _______________________________________________ > >> >> rdo-list mailing list > >> >> rdo-list(a)redhat.com > >> >> https://www.redhat.com/mailman/listinfo/rdo-list > >> >> > >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > >> > > >> > > >> > > >> > _______________________________________________ > >> > rdo-list mailing list > >> > rdo-list(a)redhat.com > >> > https://www.redhat.com/mailman/listinfo/rdo-list > >> > > >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > >> > >> I like how the discussion goes though I have some personal (and > >> probably shared) feeling that I would like to share here, more or less > >> related. > >> > >> As a TripleO core developer, I have some frustration to see that a lot > >> of people are involved in making TripleO Quickstart better, while we > >> have a few people actually working on tripleo-ci tool and try to > >> maintain upstream CI stable. > >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that > >> actually gates TripleO, even if we don't like the tool. It is right > >> now, testing TripleO upstream, everything that is not tested in there > >> will probably break one day downstream CIs. > >> Yes we have this tooling discussion here and that's awesome, but words > >> are words. I would like to see some real engagement to help TripleO CI > >> to converge into something better and not only everyone working on > >> their side. > > > > > > You have a valid point and reason to be frustrated. > > Is the point here that everyone downstream should use tripleo.sh or that > > everyone should be focused on ci and testing at the tripleo level? > > Not everyone should use tripleo.sh. My point is that we should move > forward with a common tool, and stop enlarging the gap between tools. > We have created (and are still doing) a technical dept where we have > multiple tools with a ton of overlap, the more we wait, more difficult > it will be to clean this up. > +1 > > >> > >> > >> Some examples: > >> - TripleO Quickstart (downstream) CI has coverage for undercloud & > >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade > >> job and used to have a overcloud (minor) upgrade job (disabled now, > >> for some reasons related to our capacity to run jobs and also some > >> blockers into code itself). > >> - TripleO CI has some TripleO Heat templates that could also be re-use > >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to > >> THT, WIP here: https://review.openstack.org/350775). > >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't. > >> - (...) > > > > > > As others have mentioned, there are at least 5-10 tools in development > that > > are used to deploy tripleo in some CI fashion. Calling out > > tripleo-quickstart alone is not quite right imho. There are a number > of > > tripleo devs that burn cycles on their own ci tools and maybe that is > fine > > thing to do. > > I called quickstart because that's the one I see everyday but my > frustration is about all our tools in general. > I'm actually a OOOQ user and I like this tool, really. > But as you can see, I'm also working on tripleo-ci right now because I > want TripleO CI better and I haven't seen until now some interest to > converge. > James started something cool by trying to deploy an undercloud using > OOOQ from tripleo-ci. That's a start ! We need things like this, > prototyping convergence, and see what we can do. > +1 working in multiple ci systems is painful, I know your pain! I've been playing around w/ Jame's patch [1] to see if I can help. I like Jame's approach, I think I may be missing some setup steps that nodepool provides. [1] https://review.openstack.org/#/c/348530/ > > > TripleO-Quickstart is meant to replace instack-virt-setup which it does > > quite well. The only group that was actually running > instack-virt-setup > > was the RDO CI team, upstream had taken it out of the ci system. I > think > > it's not unfair to say gaps have been left for other teams to fill. > > Gotcha. It was just some examples. > > >> > >> > >> We have been having this discussion for a while now but we're still > >> not making much progress here, I feel like we're in statu quo. > >> James mentioned a blueprint, I like it. We need to engage some > >> upstream discussion about this major CI refactor, like we need with > >> specs and then we'll decide if whether or not we need to change the > >> tool, and how. > > > > > > Well, this would take some leadership imho. We need some people that > are > > familiar with the upstream, midstream and downstream requirements of CI. > > This was addressed at the production chain meetings initially but then > > pretty much ignored. The leaders responsible at the various stages of > a > > build (upstream -> downstream ) failed to take this issue on. Here we > are > > today. > > > > Would it be acceptable by anyone.. IF > > > > tripleo-quickstart replaced instack-virt-setup [1] and walked through > the > > undercloud install, then handed off to tripleo.sh to deploy, upgrade, > > update, scale, validate etc??? > > That's something we can try. > > > That these two tools *would* in fact be the the official CI tools of > tripleo > > at the upstream, RDO, and at least parts of the downstream? > > My opinion on this is that upstream and downstream CI should only differ > on: > * the packages (OSP vs RDO) > * the scenarios (downstream could have customer-specific things) > And that's it. Tools should remain the same IMHO. > > > Would that help to ease the current frustration around CI? Emilien what > do > > you think? > > I spent the last months working on composable roles and I have now > more time to work on CI; $topic is definitely something where I would > like to help. > woot, I'm excited that you are freeing up and can be more involved! Thanks as usual Emilien! > -- > Emilien Macchi >