Re: [rdo-list] Multiple tools for deploying and testing TripleO

On Thu, Oct 13, 2016 at 8:19 PM, Joe Talerico <jtaleric(a)redhat.com&gt; wrote: > Resurrecting a old thread... > > On Thu, Aug 4, 2016 at 5:10 AM, Tal Kammer <tkammer(a)redhat.com&gt; wrote: >> Reading this through I can't help but feel that once again we are straying >> from the original intent of the discussion. >> I also share the same feeling as Emilien and believe we should focus our >> efforts on where it matters rather than having each group focus on one part >> trying to "win" over the other. >> >> My concern since day 1 was that no one has asked what are the requirements >> of each group and how does the tool serve that requirement. Simply put, the >> requirement was "we need a CI tool" while I feel that the question of "what >> is a CI tool?" or "what/who does it need to serve?" was never raised and was >> left to self interpretation. >> The result became that each group started working according to its own >> understanding of a "CI tool" and I feel that even now, going over the e-mail >> thread, this question is left unanswered. > > Wouldn't things be better if we as a collective team standardize on a "CI Tool"? > >> >> I believe that in order to achieve a consensus around a tool, we should at >> least agree first on the needs. >> Some questions that come to mind: (feel free to pitch in more questions) >> 1. should the tool provide provisioning? if yes, what type? (virsh? >> openstack? foreman? etc..) --> this can be further discussed as it's quite a >> subject to cover >> 2. should the tool be Ansible based / other language? >> 3. where do we push / maintain this tool? (I believe 'upstream' is the >> answer but before it can officially accepted, we need a "midstream" place to >> hold it) >> 4. what other requirements does this tool needs to support? (does it need to >> include testing as well? what is the proper "plugin" mechanism to maintain, >> etc) >> >> Once we understand the above, we can ask: >> 1. "is oooq the right tool to use/push upstream?" (maybe one of the other >> projects like Octario, Weirdo, InfraRed, etc, has a better base / better >> suited for the job?) >> 2. "what are we looking for in such a tool?" (are we developing a tool for >> CI? for Automation? can it be shared across products/projects or dedicated >> Openstack tool? etc..) > > Khaleesi, InfraRed, and OOOQ, what is next? > > Within the scale and performance team we have been developing > performance CI which is integrated with oooq. Since a different group > uses InfraRed, they cannot take advantage of the work we have already > done with oooq -- and this is much larger then just running tests. We > have playbooks to monitor performance metrics of the under/overcloud > so you can trend if memory usage is growing with each > release/puddle/poodle (or whatever we call them now). > > From my view, we are telling the community to choose which "CI tool" > to go with. Instead we should converge on a single tool, and improve > on that tool - fill whatever gaps exist. > >> >> I propose that all CI groups meet up and present their tool to each other. >> After that, we should have an open discussion on the pros and cons of each >> tool allowing everyone to comment on the design openly. >> Then after everyone shared their thoughts, we can start improving / create a >> proper tool from the experience of everyone. >> Whether it will be oooq based, InfraRed based or Octario based, it really >> doesn't matter, as long as there is an open discussion (ego aside) and an >> honest decision between all parties on how we can improve, we can get to >> where everyone is aiming, a truly collaborative project by all groups. >> >> Thoughts? > > I agree, agenda's aside, egos left at the door. Let's come to some > conclusion on what as a community we move forward with. Joe, you are welcome to:

1) Join our TripleO design session about CI (Barcelona) - https://etherpad.openstack.org/p/ocata-tripleo 2) Contribute to the specs that Wes is proposing about OOOQ: https://review.openstack.org/#/c/386250/ We're all aware about this challenge and we consider it as an essential topic in our roadmap.

Thanks for your feedback, > >> >> On Thu, Aug 4, 2016 at 3:45 AM, Emilien Macchi <emilien(a)redhat.com&gt; wrote: >>> >>> On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; >>> wrote: >>> > >>> > >>> > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien(a)redhat.com&gt; >>> > wrote: >>> >> >>> >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin(a)redhat.com&gt; >>> >> wrote: >>> >> > >>> >> > >>> >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle(a)redhat.com&gt; >>> >> > wrote: >>> >> >> >>> >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote: >>> >> >> > Please hear me out. >>> >> >> > TL;DR, Let's work upstream and make it awesome so that downstream >>> >> >> > can >>> >> >> > be awesome. >>> >> >> > >>> >> >> > I've said this before but I'm going to re-iterate that I do not >>> >> >> > understand why there is so much effort spent around testing >>> >> >> > TripleO >>> >> >> > downstream. >>> >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI >>> >> >> > proper. >>> >> >> > >>> >> >> > All this work should be done upstream to make TripleO and it's CI >>> >> >> > super awesome and this would trickle down for free downstream. >>> >> >> > >>> >> >> > The RDO Trunk testing pipeline is composed of two tools, today. >>> >> >> > The TripleO-Quickstart project [1] is a good example of an >>> >> >> > initiative >>> >> >> > that started downstream but always had the intention of being >>> >> >> > proposed >>> >> >> > upstream [2] after being "incubated" and fleshed out. >>> >> >> >>> >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement >>> >> >> for >>> >> >> the >>> >> >> virtual environment setup done by instack-virt-setup. 3rd party CI >>> >> >> would >>> >> >> be >>> >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup >>> >> >> was >>> >> >> always >>> >> >> working. That was the extent of the CI scope defined in the spec. >>> >> >> That >>> >> >> work is >>> >> >> not yet completed (see work items in the spec). >>> >> >> >>> >> >> Now it seems it is a much more all encompassing >>> >> >> CI/automation/testing >>> >> >> project >>> >> >> that is competing in scope with tripleo-ci itself. >>> >> > >>> >> > >>> >> > IMHO you are correct here. There has been quite a bit of discussion >>> >> > about >>> >> > removing the parts >>> >> > of oooq that are outside of the original blueprint to replace >>> >> > instack-virt-setup w/ oooq. As usual there are many different >>> >> > opinions >>> >> > here. I think there are a lot of RDO guys that would prefer a lot of >>> >> > the >>> >> > native oooq roles stay where they are, I think that is short sighted >>> >> > imho. >>> >> > I agree that anything outside of the blueprint be removed from oooq. >>> >> > This >>> >> > would hopefully allow the upstream to be more comfortable with oooq >>> >> > and >>> >> > allow us to really start consolidating tools. >>> >> > >>> >> > Luckily for the users that still want to use oooq as a full >>> >> > end-to-end >>> >> > solution the 3rd party roles can be used even after tearing out these >>> >> > native >>> >> > roles. >>> >> > >>> >> >> >>> >> >> >>> >> >> I'm all for consolidation of these types of tools *if* there is >>> >> >> interest. >>> >> > >>> >> > >>> >> > Roll call.. is there interest? +1 from me. >>> >> > >>> >> >> >>> >> >> >>> >> >> However, IMO, incubating these things downstream and then trying to >>> >> >> get >>> >> >> them >>> >> >> upstream or get upstream to adopt them is not ideal or a good >>> >> >> example. >>> >> >> The >>> >> >> same >>> >> >> topic came up and was pushed several times with khaleesi, and it >>> >> >> just >>> >> >> never >>> >> >> happened, it was continually DOA upstream. >>> >> > >>> >> > >>> >> > True, however that could be a result of the downstream perceiving >>> >> > barriers ( >>> >> > real or not ) in incubating projects in upstream openstack. >>> >> > >>> >> >> >>> >> >> >>> >> >> I think it would be fairly difficult to get tripleo-ci to wholesale >>> >> >> adopt >>> >> >> tripleo-quickstart at this stage. The separate irc channel from >>> >> >> #tripleo >>> >> >> is not >>> >> >> conducive to consolidation on tooling and direction imo. >>> >> > >>> >> > >>> >> > The irc channel is easily addressed. We do seem to generate an awful >>> >> > amount >>> >> > of chatter though :) >>> >> > >>> >> >> >>> >> >> >>> >> >> The scope of quickstart is actually not fully understood by myself. >>> >> >> I've >>> >> >> also >>> >> >> heard from some in the upstream TripleO community as well who are >>> >> >> confused >>> >> >> by >>> >> >> its direction and are facing similar difficulties using its >>> >> >> generated >>> >> >> bash >>> >> >> scripts that they'd be facing if they were just using TripleO >>> >> >> documentation >>> >> >> instead. >>> >> > >>> >> > >>> >> > The point of the generated bash scripts is to create rst >>> >> > documentation >>> >> > and >>> >> > reusable scripts for the end user. Since the documentation and the >>> >> > generated scripts are equivalent I would expect the same errors, >>> >> > problems >>> >> > and issues. I see this as a good thing really. We *want* the CI to >>> >> > hit >>> >> > the >>> >> > same issues as those who are following the doc. >>> >> > >>> >> >> >>> >> >> >>> >> >> I do think that this sort of problem lends itself easily to one off >>> >> >> implementations as is quite evidenced in this thread. Everyone/group >>> >> >> wants >>> >> >> and >>> >> >> needs to automate something in a different way. And imo, none of >>> >> >> these >>> >> >> tools >>> >> >> are building end-user or operator facing interfaces, so they're not >>> >> >> fully >>> >> >> focused on building something that "just works for everyone". Those >>> >> >> interfaces >>> >> >> should be developed in TripleO user facing tooling anyway >>> >> >> (tripleoclient/openstackclient/etc). >>> >> >> >>> >> >> So, I actually think it's ok in some degree that things have been >>> >> >> automated >>> >> >> differently in different tools. Anecdotally, I suspect many users of >>> >> >> TripleO in >>> >> >> production have their own automation tools as well. And none of the >>> >> >> implementations mentioned in this thread would likely meet their >>> >> >> needs >>> >> >> either. >>> >> > >>> >> > >>> >> > This is true.. without a tool in the upstream that addresses ci, >>> >> > dev, >>> >> > test >>> >> > use cases across the development cycle this will continue to be the >>> >> > case. I >>> >> > suspect even with a perfect tool, it won't ever be perfect for >>> >> > everyone. >>> >> > >>> >> >> >>> >> >> >>> >> >> However, if there is a desire to focus resources on consolidated >>> >> >> tooling >>> >> >> and >>> >> >> someone to drive it forward, then I definitely agree that the effort >>> >> >> needs >>> >> >> to >>> >> >> start upstream with a singular plan for tripleo-ci. From what I >>> >> >> gather, >>> >> >> that >>> >> >> would be some sort of alignment and reuse of tripleo-quickstart, and >>> >> >> then >>> >> >> we >>> >> >> could build from there. >>> >> > >>> >> > >>> >> > +1 >>> >> > >>> >> >> >>> >> >> >>> >> >> That could start as a discussion and plan within that community with >>> >> >> some >>> >> >> agreed on concensus around that plan. There was an initial thread on >>> >> >> openstack-dev related to this topic but it is stalled a bit. It >>> >> >> could >>> >> >> be >>> >> >> continually driven to resolution via specs, the tripleo meeting, >>> >> >> email >>> >> >> or >>> >> >> irc >>> >> >> discussion until a plan is formed. >>> >> > >>> >> > >>> >> > +1, I think the first step is to complete the original blueprint and >>> >> > move >>> >> > on from there. >>> >> > I think there has also been interest in having an in person meeting >>> >> > at >>> >> > summit. >>> >> > >>> >> > Thanks! >>> >> > >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> -- James Slagle >>> >> >> -- >>> >> >> >>> >> >> _______________________________________________ >>> >> >> rdo-list mailing list >>> >> >> rdo-list(a)redhat.com >>> >> >> https://www.redhat.com/mailman/listinfo/rdo-list >>> >> >> >>> >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> > >>> >> > >>> >> > >>> >> > _______________________________________________ >>> >> > rdo-list mailing list >>> >> > rdo-list(a)redhat.com >>> >> > https://www.redhat.com/mailman/listinfo/rdo-list >>> >> > >>> >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> >>> >> I like how the discussion goes though I have some personal (and >>> >> probably shared) feeling that I would like to share here, more or less >>> >> related. >>> >> >>> >> As a TripleO core developer, I have some frustration to see that a lot >>> >> of people are involved in making TripleO Quickstart better, while we >>> >> have a few people actually working on tripleo-ci tool and try to >>> >> maintain upstream CI stable. >>> >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that >>> >> actually gates TripleO, even if we don't like the tool. It is right >>> >> now, testing TripleO upstream, everything that is not tested in there >>> >> will probably break one day downstream CIs. >>> >> Yes we have this tooling discussion here and that's awesome, but words >>> >> are words. I would like to see some real engagement to help TripleO CI >>> >> to converge into something better and not only everyone working on >>> >> their side. >>> > >>> > >>> > You have a valid point and reason to be frustrated. >>> > Is the point here that everyone downstream should use tripleo.sh or that >>> > everyone should be focused on ci and testing at the tripleo level? >>> >>> Not everyone should use tripleo.sh. My point is that we should move >>> forward with a common tool, and stop enlarging the gap between tools. >>> We have created (and are still doing) a technical dept where we have >>> multiple tools with a ton of overlap, the more we wait, more difficult >>> it will be to clean this up. >>> >>> >> >>> >> >>> >> Some examples: >>> >> - TripleO Quickstart (downstream) CI has coverage for undercloud & >>> >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade >>> >> job and used to have a overcloud (minor) upgrade job (disabled now, >>> >> for some reasons related to our capacity to run jobs and also some >>> >> blockers into code itself). >>> >> - TripleO CI has some TripleO Heat templates that could also be re-use >>> >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to >>> >> THT, WIP here: https://review.openstack.org/350775). >>> >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't. >>> >> - (...) >>> > >>> > >>> > As others have mentioned, there are at least 5-10 tools in development >>> > that >>> > are used to deploy tripleo in some CI fashion. Calling out >>> > tripleo-quickstart alone is not quite right imho. There are a number >>> > of >>> > tripleo devs that burn cycles on their own ci tools and maybe that is >>> > fine >>> > thing to do. >>> >>> I called quickstart because that's the one I see everyday but my >>> frustration is about all our tools in general. >>> I'm actually a OOOQ user and I like this tool, really. >>> But as you can see, I'm also working on tripleo-ci right now because I >>> want TripleO CI better and I haven't seen until now some interest to >>> converge. >>> James started something cool by trying to deploy an undercloud using >>> OOOQ from tripleo-ci. That's a start ! We need things like this, >>> prototyping convergence, and see what we can do. >>> >>> > TripleO-Quickstart is meant to replace instack-virt-setup which it does >>> > quite well. The only group that was actually running >>> > instack-virt-setup >>> > was the RDO CI team, upstream had taken it out of the ci system. I >>> > think >>> > it's not unfair to say gaps have been left for other teams to fill. >>> >>> Gotcha. It was just some examples. >>> >>> >> >>> >> >>> >> We have been having this discussion for a while now but we're still >>> >> not making much progress here, I feel like we're in statu quo. >>> >> James mentioned a blueprint, I like it. We need to engage some >>> >> upstream discussion about this major CI refactor, like we need with >>> >> specs and then we'll decide if whether or not we need to change the >>> >> tool, and how. >>> > >>> > >>> > Well, this would take some leadership imho. We need some people that >>> > are >>> > familiar with the upstream, midstream and downstream requirements of CI. >>> > This was addressed at the production chain meetings initially but then >>> > pretty much ignored. The leaders responsible at the various stages of >>> > a >>> > build (upstream -> downstream ) failed to take this issue on. Here we >>> > are >>> > today. >>> > >>> > Would it be acceptable by anyone.. IF >>> > >>> > tripleo-quickstart replaced instack-virt-setup [1] and walked through >>> > the >>> > undercloud install, then handed off to tripleo.sh to deploy, upgrade, >>> > update, scale, validate etc??? >>> >>> That's something we can try. >>> >>> > That these two tools *would* in fact be the the official CI tools of >>> > tripleo >>> > at the upstream, RDO, and at least parts of the downstream? >>> >>> My opinion on this is that upstream and downstream CI should only differ >>> on: >>> * the packages (OSP vs RDO) >>> * the scenarios (downstream could have customer-specific things) >>> And that's it. Tools should remain the same IMHO. >>> >>> > Would that help to ease the current frustration around CI? Emilien what >>> > do >>> > you think? >>> >>> I spent the last months working on composable roles and I have now >>> more time to work on CI; $topic is definitely something where I would >>> like to help. >>> -- >>> Emilien Macchi >>> >>> _______________________________________________ >>> rdo-list mailing list >>> rdo-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >> >> >> >> -- >> Tal Kammer >> Associate manager, automation and infrastracture, Openstack platform. >> Red Hat Israel >> Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659 >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com -- Emilien Macchi