Re: [rdo-list] Multiple tools for deploying and testing TripleO

Resurrecting a old thread... On Thu, Aug 4, 2016 at 5:10 AM, Tal Kammer <tkammer(a)redhat.com&gt; wrote: > Reading this through I can't help but feel that once again we are straying > from the original intent of the discussion. > I also share the same feeling as Emilien and believe we should focus our > efforts on where it matters rather than having each group focus on one part > trying to "win" over the other. > > My concern since day 1 was that no one has asked what are the requirements > of each group and how does the tool serve that requirement. Simply put, the > requirement was "we need a CI tool" while I feel that the question of "what > is a CI tool?" or "what/who does it need to serve?" was never raised and was > left to self interpretation. > The result became that each group started working according to its own > understanding of a "CI tool" and I feel that even now, going over the e-mail > thread, this question is left unanswered. Wouldn't things be better if we as a collective team standardize on a "CI Tool"? > > I believe that in order to achieve a consensus around a tool, we should at > least agree first on the needs. > Some questions that come to mind: (feel free to pitch in more questions) > 1. should the tool provide provisioning? if yes, what type? (virsh? > openstack? foreman? etc..) --> this can be further discussed as it's quite a > subject to cover > 2. should the tool be Ansible based / other language? > 3. where do we push / maintain this tool? (I believe 'upstream' is the > answer but before it can officially accepted, we need a "midstream" place to > hold it) > 4. what other requirements does this tool needs to support? (does it need to > include testing as well? what is the proper "plugin" mechanism to maintain, > etc) > > Once we understand the above, we can ask: > 1. "is oooq the right tool to use/push upstream?" (maybe one of the other > projects like Octario, Weirdo, InfraRed, etc, has a better base / better > suited for the job?) > 2. "what are we looking for in such a tool?" (are we developing a tool for > CI? for Automation? can it be shared across products/projects or dedicated > Openstack tool? etc..) Khaleesi, InfraRed, and OOOQ, what is next? Within the scale and performance team we have been developing performance CI which is integrated with oooq. Since a different group uses InfraRed, they cannot take advantage of the work we have already done with oooq -- and this is much larger then just running tests. We have playbooks to monitor performance metrics of the under/overcloud so you can trend if memory usage is growing with each release/puddle/poodle (or whatever we call them now). From my view, we are telling the community to choose which "CI tool" to go with. Instead we should converge on a single tool, and improve on that tool - fill whatever gaps exist. > > I propose that all CI groups meet up and present their tool to each other. > After that, we should have an open discussion on the pros and cons of each > tool allowing everyone to comment on the design openly. > Then after everyone shared their thoughts, we can start improving / create a > proper tool from the experience of everyone. > Whether it will be oooq based, InfraRed based or Octario based, it really > doesn't matter, as long as there is an open discussion (ego aside) and an > honest decision between all parties on how we can improve, we can get to > where everyone is aiming, a truly collaborative project by all groups. > > Thoughts? I agree, agenda's aside, egos left at the door. Let's come to some conclusion on what as a community we move forward with.

> > On Thu, Aug 4, 2016 at 3:45 AM, Emilien Macchi <emilien(a)redhat.com&gt; wrote: >> >> On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; >> wrote: >> > >> > >> > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien(a)redhat.com&gt; >> > wrote: >> >> >> >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; >> >> wrote: >> >> > >> >> > >> >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle(a)redhat.com&gt; >> >> > wrote: >> >> >> >> >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote: >> >> >> > Please hear me out. >> >> >> > TL;DR, Let's work upstream and make it awesome so that downstream >> >> >> > can >> >> >> > be awesome. >> >> >> > >> >> >> > I've said this before but I'm going to re-iterate that I do not >> >> >> > understand why there is so much effort spent around testing >> >> >> > TripleO >> >> >> > downstream. >> >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI >> >> >> > proper. >> >> >> > >> >> >> > All this work should be done upstream to make TripleO and it's CI >> >> >> > super awesome and this would trickle down for free downstream. >> >> >> > >> >> >> > The RDO Trunk testing pipeline is composed of two tools, today. >> >> >> > The TripleO-Quickstart project [1] is a good example of an >> >> >> > initiative >> >> >> > that started downstream but always had the intention of being >> >> >> > proposed >> >> >> > upstream [2] after being "incubated" and fleshed out. >> >> >> >> >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement >> >> >> for >> >> >> the >> >> >> virtual environment setup done by instack-virt-setup. 3rd party CI >> >> >> would >> >> >> be >> >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup >> >> >> was >> >> >> always >> >> >> working. That was the extent of the CI scope defined in the spec. >> >> >> That >> >> >> work is >> >> >> not yet completed (see work items in the spec). >> >> >> >> >> >> Now it seems it is a much more all encompassing >> >> >> CI/automation/testing >> >> >> project >> >> >> that is competing in scope with tripleo-ci itself. >> >> > >> >> > >> >> > IMHO you are correct here. There has been quite a bit of discussion >> >> > about >> >> > removing the parts >> >> > of oooq that are outside of the original blueprint to replace >> >> > instack-virt-setup w/ oooq. As usual there are many different >> >> > opinions >> >> > here. I think there are a lot of RDO guys that would prefer a lot of >> >> > the >> >> > native oooq roles stay where they are, I think that is short sighted >> >> > imho. >> >> > I agree that anything outside of the blueprint be removed from oooq. >> >> > This >> >> > would hopefully allow the upstream to be more comfortable with oooq >> >> > and >> >> > allow us to really start consolidating tools. >> >> > >> >> > Luckily for the users that still want to use oooq as a full >> >> > end-to-end >> >> > solution the 3rd party roles can be used even after tearing out these >> >> > native >> >> > roles. >> >> > >> >> >> >> >> >> >> >> >> I'm all for consolidation of these types of tools *if* there is >> >> >> interest. >> >> > >> >> > >> >> > Roll call.. is there interest? +1 from me. >> >> > >> >> >> >> >> >> >> >> >> However, IMO, incubating these things downstream and then trying to >> >> >> get >> >> >> them >> >> >> upstream or get upstream to adopt them is not ideal or a good >> >> >> example. >> >> >> The >> >> >> same >> >> >> topic came up and was pushed several times with khaleesi, and it >> >> >> just >> >> >> never >> >> >> happened, it was continually DOA upstream. >> >> > >> >> > >> >> > True, however that could be a result of the downstream perceiving >> >> > barriers ( >> >> > real or not ) in incubating projects in upstream openstack. >> >> > >> >> >> >> >> >> >> >> >> I think it would be fairly difficult to get tripleo-ci to wholesale >> >> >> adopt >> >> >> tripleo-quickstart at this stage. The separate irc channel from >> >> >> #tripleo >> >> >> is not >> >> >> conducive to consolidation on tooling and direction imo. >> >> > >> >> > >> >> > The irc channel is easily addressed. We do seem to generate an awful >> >> > amount >> >> > of chatter though :) >> >> > >> >> >> >> >> >> >> >> >> The scope of quickstart is actually not fully understood by myself. >> >> >> I've >> >> >> also >> >> >> heard from some in the upstream TripleO community as well who are >> >> >> confused >> >> >> by >> >> >> its direction and are facing similar difficulties using its >> >> >> generated >> >> >> bash >> >> >> scripts that they'd be facing if they were just using TripleO >> >> >> documentation >> >> >> instead. >> >> > >> >> > >> >> > The point of the generated bash scripts is to create rst >> >> > documentation >> >> > and >> >> > reusable scripts for the end user. Since the documentation and the >> >> > generated scripts are equivalent I would expect the same errors, >> >> > problems >> >> > and issues. I see this as a good thing really. We *want* the CI to >> >> > hit >> >> > the >> >> > same issues as those who are following the doc. >> >> > >> >> >> >> >> >> >> >> >> I do think that this sort of problem lends itself easily to one off >> >> >> implementations as is quite evidenced in this thread. Everyone/group >> >> >> wants >> >> >> and >> >> >> needs to automate something in a different way. And imo, none of >> >> >> these >> >> >> tools >> >> >> are building end-user or operator facing interfaces, so they're not >> >> >> fully >> >> >> focused on building something that "just works for everyone". Those >> >> >> interfaces >> >> >> should be developed in TripleO user facing tooling anyway >> >> >> (tripleoclient/openstackclient/etc). >> >> >> >> >> >> So, I actually think it's ok in some degree that things have been >> >> >> automated >> >> >> differently in different tools. Anecdotally, I suspect many users of >> >> >> TripleO in >> >> >> production have their own automation tools as well. And none of the >> >> >> implementations mentioned in this thread would likely meet their >> >> >> needs >> >> >> either. >> >> > >> >> > >> >> > This is true.. without a tool in the upstream that addresses ci, >> >> > dev, >> >> > test >> >> > use cases across the development cycle this will continue to be the >> >> > case. I >> >> > suspect even with a perfect tool, it won't ever be perfect for >> >> > everyone. >> >> > >> >> >> >> >> >> >> >> >> However, if there is a desire to focus resources on consolidated >> >> >> tooling >> >> >> and >> >> >> someone to drive it forward, then I definitely agree that the effort >> >> >> needs >> >> >> to >> >> >> start upstream with a singular plan for tripleo-ci. From what I >> >> >> gather, >> >> >> that >> >> >> would be some sort of alignment and reuse of tripleo-quickstart, and >> >> >> then >> >> >> we >> >> >> could build from there. >> >> > >> >> > >> >> > +1 >> >> > >> >> >> >> >> >> >> >> >> That could start as a discussion and plan within that community with >> >> >> some >> >> >> agreed on concensus around that plan. There was an initial thread on >> >> >> openstack-dev related to this topic but it is stalled a bit. It >> >> >> could >> >> >> be >> >> >> continually driven to resolution via specs, the tripleo meeting, >> >> >> email >> >> >> or >> >> >> irc >> >> >> discussion until a plan is formed. >> >> > >> >> > >> >> > +1, I think the first step is to complete the original blueprint and >> >> > move >> >> > on from there. >> >> > I think there has also been interest in having an in person meeting >> >> > at >> >> > summit. >> >> > >> >> > Thanks! >> >> > >> >> >> >> >> >> >> >> >> -- >> >> >> -- James Slagle >> >> >> -- >> >> >> >> >> >> _______________________________________________ >> >> >> rdo-list mailing list >> >> >> rdo-list(a)redhat.com >> >> >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> >> >> >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > rdo-list mailing list >> >> > rdo-list(a)redhat.com >> >> > https://www.redhat.com/mailman/listinfo/rdo-list >> >> > >> >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >> >> >> I like how the discussion goes though I have some personal (and >> >> probably shared) feeling that I would like to share here, more or less >> >> related. >> >> >> >> As a TripleO core developer, I have some frustration to see that a lot >> >> of people are involved in making TripleO Quickstart better, while we >> >> have a few people actually working on tripleo-ci tool and try to >> >> maintain upstream CI stable. >> >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that >> >> actually gates TripleO, even if we don't like the tool. It is right >> >> now, testing TripleO upstream, everything that is not tested in there >> >> will probably break one day downstream CIs. >> >> Yes we have this tooling discussion here and that's awesome, but words >> >> are words. I would like to see some real engagement to help TripleO CI >> >> to converge into something better and not only everyone working on >> >> their side. >> > >> > >> > You have a valid point and reason to be frustrated. >> > Is the point here that everyone downstream should use tripleo.sh or that >> > everyone should be focused on ci and testing at the tripleo level? >> >> Not everyone should use tripleo.sh. My point is that we should move >> forward with a common tool, and stop enlarging the gap between tools. >> We have created (and are still doing) a technical dept where we have >> multiple tools with a ton of overlap, the more we wait, more difficult >> it will be to clean this up. >> >> >> >> >> >> >> Some examples: >> >> - TripleO Quickstart (downstream) CI has coverage for undercloud & >> >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade >> >> job and used to have a overcloud (minor) upgrade job (disabled now, >> >> for some reasons related to our capacity to run jobs and also some >> >> blockers into code itself). >> >> - TripleO CI has some TripleO Heat templates that could also be re-use >> >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to >> >> THT, WIP here: https://review.openstack.org/350775). >> >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't. >> >> - (...) >> > >> > >> > As others have mentioned, there are at least 5-10 tools in development >> > that >> > are used to deploy tripleo in some CI fashion. Calling out >> > tripleo-quickstart alone is not quite right imho. There are a number >> > of >> > tripleo devs that burn cycles on their own ci tools and maybe that is >> > fine >> > thing to do. >> >> I called quickstart because that's the one I see everyday but my >> frustration is about all our tools in general. >> I'm actually a OOOQ user and I like this tool, really. >> But as you can see, I'm also working on tripleo-ci right now because I >> want TripleO CI better and I haven't seen until now some interest to >> converge. >> James started something cool by trying to deploy an undercloud using >> OOOQ from tripleo-ci. That's a start ! We need things like this, >> prototyping convergence, and see what we can do. >> >> > TripleO-Quickstart is meant to replace instack-virt-setup which it does >> > quite well. The only group that was actually running >> > instack-virt-setup >> > was the RDO CI team, upstream had taken it out of the ci system. I >> > think >> > it's not unfair to say gaps have been left for other teams to fill. >> >> Gotcha. It was just some examples. >> >> >> >> >> >> >> We have been having this discussion for a while now but we're still >> >> not making much progress here, I feel like we're in statu quo. >> >> James mentioned a blueprint, I like it. We need to engage some >> >> upstream discussion about this major CI refactor, like we need with >> >> specs and then we'll decide if whether or not we need to change the >> >> tool, and how. >> > >> > >> > Well, this would take some leadership imho. We need some people that >> > are >> > familiar with the upstream, midstream and downstream requirements of CI. >> > This was addressed at the production chain meetings initially but then >> > pretty much ignored. The leaders responsible at the various stages of >> > a >> > build (upstream -> downstream ) failed to take this issue on. Here we >> > are >> > today. >> > >> > Would it be acceptable by anyone.. IF >> > >> > tripleo-quickstart replaced instack-virt-setup [1] and walked through >> > the >> > undercloud install, then handed off to tripleo.sh to deploy, upgrade, >> > update, scale, validate etc??? >> >> That's something we can try. >> >> > That these two tools *would* in fact be the the official CI tools of >> > tripleo >> > at the upstream, RDO, and at least parts of the downstream? >> >> My opinion on this is that upstream and downstream CI should only differ >> on: >> * the packages (OSP vs RDO) >> * the scenarios (downstream could have customer-specific things) >> And that's it. Tools should remain the same IMHO. >> >> > Would that help to ease the current frustration around CI? Emilien what >> > do >> > you think? >> >> I spent the last months working on composable roles and I have now >> more time to work on CI; $topic is definitely something where I would >> like to help. >> -- >> Emilien Macchi >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > > -- > Tal Kammer > Associate manager, automation and infrastracture, Openstack platform. > Red Hat Israel > Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659 > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com