Re: [rdo-list] Multiple tools for deploying and testing TripleO

Reading this through I can't help but feel that once again we are straying from the original intent of the discussion. I also share the same feeling as Emilien and believe we should focus our efforts on where it matters rather than having each group focus on one part trying to "win" over the other. My concern since day 1 was that no one has asked what are the requirements of each group and how does the tool serve that requirement. Simply put, the requirement was "we need a CI tool" while I feel that the question of "what is a CI tool?" or "what/who does it need to serve?" was never raised and was left to self interpretation. The result became that each group started working according to its own understanding of a "CI tool" and I feel that even now, going over the e-mail thread, this question is left unanswered.

I believe that in order to achieve a consensus around a tool, we should at least agree first on the needs. Some questions that come to mind: (feel free to pitch in more questions) 1. should the tool provide provisioning? if yes, what type? (virsh? openstack? foreman? etc..) --> this can be further discussed as it's quite a subject to cover 2. should the tool be Ansible based / other language? 3. where do we push / maintain this tool? (I believe 'upstream' is the answer but before it can officially accepted, we need a "midstream" place to hold it) 4. what other requirements does this tool needs to support? (does it need to include testing as well? what is the proper "plugin" mechanism to maintain, etc) Once we understand the above, we can ask: 1. "is oooq the right tool to use/push upstream?" (maybe one of the other projects like Octario, Weirdo, InfraRed, etc, has a better base / better suited for the job?) 2. "what are we looking for in such a tool?" (are we developing a tool for CI? for Automation? can it be shared across products/projects or dedicated Openstack tool? etc..)

From my view, we are telling the community to choose which "CI tool"

I propose that all CI groups meet up and present their tool to each other. After that, we should have an open discussion on the pros and cons of each tool allowing everyone to comment on the design openly. Then after everyone shared their thoughts, we can start improving / create a proper tool from the experience of everyone. Whether it will be oooq based, InfraRed based or Octario based, it really doesn't matter, as long as there is an open discussion (ego aside) and an honest decision between all parties on how we can improve, we can get to where everyone is aiming, a truly collaborative project by all groups. Thoughts?

On Thu, Aug 4, 2016 at 3:45 AM, Emilien Macchi <emilien(a)redhat.com&gt; wrote: > > On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; > wrote: > > > > > > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien(a)redhat.com&gt; > > wrote: > >> > >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; > >> wrote: > >> > > >> > > >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle(a)redhat.com&gt; > >> > wrote: > >> >> > >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote: > >> >> > Please hear me out. > >> >> > TL;DR, Let's work upstream and make it awesome so that downstream > >> >> > can > >> >> > be awesome. > >> >> > > >> >> > I've said this before but I'm going to re-iterate that I do not > >> >> > understand why there is so much effort spent around testing > >> >> > TripleO > >> >> > downstream. > >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI > >> >> > proper. > >> >> > > >> >> > All this work should be done upstream to make TripleO and it's CI > >> >> > super awesome and this would trickle down for free downstream. > >> >> > > >> >> > The RDO Trunk testing pipeline is composed of two tools, today. > >> >> > The TripleO-Quickstart project [1] is a good example of an > >> >> > initiative > >> >> > that started downstream but always had the intention of being > >> >> > proposed > >> >> > upstream [2] after being "incubated" and fleshed out. > >> >> > >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement > >> >> for > >> >> the > >> >> virtual environment setup done by instack-virt-setup. 3rd party CI > >> >> would > >> >> be > >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup > >> >> was > >> >> always > >> >> working. That was the extent of the CI scope defined in the spec. > >> >> That > >> >> work is > >> >> not yet completed (see work items in the spec). > >> >> > >> >> Now it seems it is a much more all encompassing > >> >> CI/automation/testing > >> >> project > >> >> that is competing in scope with tripleo-ci itself. > >> > > >> > > >> > IMHO you are correct here. There has been quite a bit of discussion > >> > about > >> > removing the parts > >> > of oooq that are outside of the original blueprint to replace > >> > instack-virt-setup w/ oooq. As usual there are many different > >> > opinions > >> > here. I think there are a lot of RDO guys that would prefer a lot of > >> > the > >> > native oooq roles stay where they are, I think that is short sighted > >> > imho. > >> > I agree that anything outside of the blueprint be removed from oooq. > >> > This > >> > would hopefully allow the upstream to be more comfortable with oooq > >> > and > >> > allow us to really start consolidating tools. > >> > > >> > Luckily for the users that still want to use oooq as a full > >> > end-to-end > >> > solution the 3rd party roles can be used even after tearing out these > >> > native > >> > roles. > >> > > >> >> > >> >> > >> >> I'm all for consolidation of these types of tools *if* there is > >> >> interest. > >> > > >> > > >> > Roll call.. is there interest? +1 from me. > >> > > >> >> > >> >> > >> >> However, IMO, incubating these things downstream and then trying to > >> >> get > >> >> them > >> >> upstream or get upstream to adopt them is not ideal or a good > >> >> example. > >> >> The > >> >> same > >> >> topic came up and was pushed several times with khaleesi, and it > >> >> just > >> >> never > >> >> happened, it was continually DOA upstream. > >> > > >> > > >> > True, however that could be a result of the downstream perceiving > >> > barriers ( > >> > real or not ) in incubating projects in upstream openstack. > >> > > >> >> > >> >> > >> >> I think it would be fairly difficult to get tripleo-ci to wholesale > >> >> adopt > >> >> tripleo-quickstart at this stage. The separate irc channel from > >> >> #tripleo > >> >> is not > >> >> conducive to consolidation on tooling and direction imo. > >> > > >> > > >> > The irc channel is easily addressed. We do seem to generate an awful > >> > amount > >> > of chatter though :) > >> > > >> >> > >> >> > >> >> The scope of quickstart is actually not fully understood by myself. > >> >> I've > >> >> also > >> >> heard from some in the upstream TripleO community as well who are > >> >> confused > >> >> by > >> >> its direction and are facing similar difficulties using its > >> >> generated > >> >> bash > >> >> scripts that they'd be facing if they were just using TripleO > >> >> documentation > >> >> instead. > >> > > >> > > >> > The point of the generated bash scripts is to create rst > >> > documentation > >> > and > >> > reusable scripts for the end user. Since the documentation and the > >> > generated scripts are equivalent I would expect the same errors, > >> > problems > >> > and issues. I see this as a good thing really. We *want* the CI to > >> > hit > >> > the > >> > same issues as those who are following the doc. > >> > > >> >> > >> >> > >> >> I do think that this sort of problem lends itself easily to one off > >> >> implementations as is quite evidenced in this thread. Everyone/group > >> >> wants > >> >> and > >> >> needs to automate something in a different way. And imo, none of > >> >> these > >> >> tools > >> >> are building end-user or operator facing interfaces, so they're not > >> >> fully > >> >> focused on building something that "just works for everyone". Those > >> >> interfaces > >> >> should be developed in TripleO user facing tooling anyway > >> >> (tripleoclient/openstackclient/etc). > >> >> > >> >> So, I actually think it's ok in some degree that things have been > >> >> automated > >> >> differently in different tools. Anecdotally, I suspect many users of > >> >> TripleO in > >> >> production have their own automation tools as well. And none of the > >> >> implementations mentioned in this thread would likely meet their > >> >> needs > >> >> either. > >> > > >> > > >> > This is true.. without a tool in the upstream that addresses ci, > >> > dev, > >> > test > >> > use cases across the development cycle this will continue to be the > >> > case. I > >> > suspect even with a perfect tool, it won't ever be perfect for > >> > everyone. > >> > > >> >> > >> >> > >> >> However, if there is a desire to focus resources on consolidated > >> >> tooling > >> >> and > >> >> someone to drive it forward, then I definitely agree that the effort > >> >> needs > >> >> to > >> >> start upstream with a singular plan for tripleo-ci. From what I > >> >> gather, > >> >> that > >> >> would be some sort of alignment and reuse of tripleo-quickstart, and > >> >> then > >> >> we > >> >> could build from there. > >> > > >> > > >> > +1 > >> > > >> >> > >> >> > >> >> That could start as a discussion and plan within that community with > >> >> some > >> >> agreed on concensus around that plan. There was an initial thread on > >> >> openstack-dev related to this topic but it is stalled a bit. It > >> >> could > >> >> be > >> >> continually driven to resolution via specs, the tripleo meeting, > >> >> email > >> >> or > >> >> irc > >> >> discussion until a plan is formed. > >> > > >> > > >> > +1, I think the first step is to complete the original blueprint and > >> > move > >> > on from there. > >> > I think there has also been interest in having an in person meeting > >> > at > >> > summit. > >> > > >> > Thanks! > >> > > >> >> > >> >> > >> >> -- > >> >> -- James Slagle > >> >> -- > >> >> > >> >> _______________________________________________ > >> >> rdo-list mailing list > >> >> rdo-list(a)redhat.com > >> >> https://www.redhat.com/mailman/listinfo/rdo-list > >> >> > >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > >> > > >> > > >> > > >> > _______________________________________________ > >> > rdo-list mailing list > >> > rdo-list(a)redhat.com > >> > https://www.redhat.com/mailman/listinfo/rdo-list > >> > > >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > >> > >> I like how the discussion goes though I have some personal (and > >> probably shared) feeling that I would like to share here, more or less > >> related. > >> > >> As a TripleO core developer, I have some frustration to see that a lot > >> of people are involved in making TripleO Quickstart better, while we > >> have a few people actually working on tripleo-ci tool and try to > >> maintain upstream CI stable. > >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that > >> actually gates TripleO, even if we don't like the tool. It is right > >> now, testing TripleO upstream, everything that is not tested in there > >> will probably break one day downstream CIs. > >> Yes we have this tooling discussion here and that's awesome, but words > >> are words. I would like to see some real engagement to help TripleO CI > >> to converge into something better and not only everyone working on > >> their side. > > > > > > You have a valid point and reason to be frustrated. > > Is the point here that everyone downstream should use tripleo.sh or that > > everyone should be focused on ci and testing at the tripleo level? > > Not everyone should use tripleo.sh. My point is that we should move > forward with a common tool, and stop enlarging the gap between tools. > We have created (and are still doing) a technical dept where we have > multiple tools with a ton of overlap, the more we wait, more difficult > it will be to clean this up. > > >> > >> > >> Some examples: > >> - TripleO Quickstart (downstream) CI has coverage for undercloud & > >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade > >> job and used to have a overcloud (minor) upgrade job (disabled now, > >> for some reasons related to our capacity to run jobs and also some > >> blockers into code itself). > >> - TripleO CI has some TripleO Heat templates that could also be re-use > >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to > >> THT, WIP here: https://review.openstack.org/350775). > >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't. > >> - (...) > > > > > > As others have mentioned, there are at least 5-10 tools in development > > that > > are used to deploy tripleo in some CI fashion. Calling out > > tripleo-quickstart alone is not quite right imho. There are a number > > of > > tripleo devs that burn cycles on their own ci tools and maybe that is > > fine > > thing to do. > > I called quickstart because that's the one I see everyday but my > frustration is about all our tools in general. > I'm actually a OOOQ user and I like this tool, really. > But as you can see, I'm also working on tripleo-ci right now because I > want TripleO CI better and I haven't seen until now some interest to > converge. > James started something cool by trying to deploy an undercloud using > OOOQ from tripleo-ci. That's a start ! We need things like this, > prototyping convergence, and see what we can do. > > > TripleO-Quickstart is meant to replace instack-virt-setup which it does > > quite well. The only group that was actually running > > instack-virt-setup > > was the RDO CI team, upstream had taken it out of the ci system. I > > think > > it's not unfair to say gaps have been left for other teams to fill. > > Gotcha. It was just some examples. > > >> > >> > >> We have been having this discussion for a while now but we're still > >> not making much progress here, I feel like we're in statu quo. > >> James mentioned a blueprint, I like it. We need to engage some > >> upstream discussion about this major CI refactor, like we need with > >> specs and then we'll decide if whether or not we need to change the > >> tool, and how. > > > > > > Well, this would take some leadership imho. We need some people that > > are > > familiar with the upstream, midstream and downstream requirements of CI. > > This was addressed at the production chain meetings initially but then > > pretty much ignored. The leaders responsible at the various stages of > > a > > build (upstream -> downstream ) failed to take this issue on. Here we > > are > > today. > > > > Would it be acceptable by anyone.. IF > > > > tripleo-quickstart replaced instack-virt-setup [1] and walked through > > the > > undercloud install, then handed off to tripleo.sh to deploy, upgrade, > > update, scale, validate etc??? > > That's something we can try. > > > That these two tools *would* in fact be the the official CI tools of > > tripleo > > at the upstream, RDO, and at least parts of the downstream? > > My opinion on this is that upstream and downstream CI should only differ > on: > * the packages (OSP vs RDO) > * the scenarios (downstream could have customer-specific things) > And that's it. Tools should remain the same IMHO. > > > Would that help to ease the current frustration around CI? Emilien what > > do > > you think? > > I spent the last months working on composable roles and I have now > more time to work on CI; $topic is definitely something where I would > like to help. > -- > Emilien Macchi > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com -- Tal Kammer Associate manager, automation and infrastracture, Openstack platform. Red Hat Israel Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659 _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com