[rdo-list] Multiple tools for deploying and testing TripleO

Fri Oct 14 01:10:51 UTC 2016

On Thu, Oct 13, 2016 at 8:54 PM, Emilien Macchi <emilien at redhat.com> wrote:
> On Thu, Oct 13, 2016 at 8:19 PM, Joe Talerico <jtaleric at redhat.com> wrote:
>> Resurrecting a old thread...
>>
>> On Thu, Aug 4, 2016 at 5:10 AM, Tal Kammer <tkammer at redhat.com> wrote:
>>> Reading this through I can't help but feel that once again we are straying
>>> from the original intent of the discussion.
>>> I also share the same feeling as Emilien and believe we should focus our
>>> efforts on where it matters rather than having each group focus on one part
>>> trying to "win" over the other.
>>>
>>> My concern since day 1 was that no one has asked what are the requirements
>>> of each group and how does the tool serve that requirement. Simply put, the
>>> requirement was "we need a CI tool" while I feel that the question of "what
>>> is a CI tool?" or "what/who does it need to serve?" was never raised and was
>>> left to self interpretation.
>>> The result became that each group started working according to its own
>>> understanding of a "CI tool" and I feel that even now, going over the e-mail
>>> thread, this question is left unanswered.
>>
>> Wouldn't things be better if we as a collective team standardize on a "CI Tool"?
>>
>>>
>>> I believe that in order to achieve a consensus around a tool, we should at
>>> least agree first on the needs.
>>> Some questions that come to mind: (feel free to pitch in more questions)
>>> 1. should the tool provide provisioning? if yes, what type? (virsh?
>>> openstack? foreman? etc..) --> this can be further discussed as it's quite a
>>> subject to cover
>>> 2. should the tool be Ansible based / other language?
>>> 3. where do we push / maintain this tool? (I believe 'upstream' is the
>>> answer but before it can officially accepted, we need a "midstream" place to
>>> hold it)
>>> 4. what other requirements does this tool needs to support? (does it need to
>>> include testing as well? what is the proper "plugin" mechanism to maintain,
>>> etc)
>>>
>>> Once we understand the above, we can ask:
>>> 1. "is oooq the right tool to use/push upstream?" (maybe one of the other
>>> projects like Octario, Weirdo, InfraRed, etc, has a better base / better
>>> suited for the job?)
>>> 2. "what are we looking for in such a tool?" (are we developing a tool for
>>> CI? for Automation? can it be shared across products/projects or dedicated
>>> Openstack tool? etc..)
>>
>> Khaleesi, InfraRed, and OOOQ, what is next?
>>
>> Within the scale and performance team we have been developing
>> performance CI which is integrated with oooq. Since a different group
>> uses InfraRed, they cannot take advantage of the work we have already
>> done with oooq -- and this is much larger then just running tests. We
>> have playbooks to monitor performance metrics of the under/overcloud
>> so you can trend if memory usage is growing with each
>> release/puddle/poodle (or whatever we call them now).
>>
>> From my view, we are telling the community to choose which "CI tool"
>> to go with. Instead we should converge on a single tool, and improve
>> on that tool - fill whatever gaps exist.
>>
>>>
>>> I propose that all CI groups meet up and present their tool to each other.
>>> After that, we should have an open discussion on the pros and cons of each
>>> tool allowing everyone to comment on the design openly.
>>> Then after everyone shared their thoughts, we can start improving / create a
>>> proper tool from the experience of everyone.
>>> Whether it will be oooq based, InfraRed based or Octario based, it really
>>> doesn't matter, as long as there is an open discussion (ego aside) and an
>>> honest decision between all parties on how we can improve, we can get to
>>> where everyone is aiming, a truly collaborative project by all groups.
>>>
>>> Thoughts?
>>
>> I agree, agenda's aside, egos left at the door. Let's come to some
>> conclusion on what as a community we move forward with.
>
> Joe, you are welcome to:

Thanks for the offer Emilien.

>
> 1) Join our TripleO design session about CI (Barcelona) -
> https://etherpad.openstack.org/p/ocata-tripleo
> 2) Contribute to the specs that Wes is proposing about OOOQ:
> https://review.openstack.org/#/c/386250/
>
> We're all aware about this challenge and we consider it as an
> essential topic in our roadmap.

Glad to hear that, looking forward to progress being made here.

Joe

> Thanks for your feedback,
>
>>
>>>
>>> On Thu, Aug 4, 2016 at 3:45 AM, Emilien Macchi <emilien at redhat.com> wrote:
>>>>
>>>> On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin at redhat.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien at redhat.com>
>>>> > wrote:
>>>> >>
>>>> >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin at redhat.com>
>>>> >> wrote:
>>>> >> >
>>>> >> >
>>>> >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle at redhat.com>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote:
>>>> >> >> > Please hear me out.
>>>> >> >> > TL;DR, Let's work upstream and make it awesome so that downstream
>>>> >> >> > can
>>>> >> >> > be awesome.
>>>> >> >> >
>>>> >> >> > I've said this before but I'm going to re-iterate that I do not
>>>> >> >> > understand why there is so much effort spent around testing
>>>> >> >> > TripleO
>>>> >> >> > downstream.
>>>> >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI
>>>> >> >> > proper.
>>>> >> >> >
>>>> >> >> > All this work should be done upstream to make TripleO and it's CI
>>>> >> >> > super awesome and this would trickle down for free downstream.
>>>> >> >> >
>>>> >> >> > The RDO Trunk testing pipeline is composed of two tools, today.
>>>> >> >> > The TripleO-Quickstart project [1] is a good example of an
>>>> >> >> > initiative
>>>> >> >> > that started downstream but always had the intention of being
>>>> >> >> > proposed
>>>> >> >> > upstream [2] after being "incubated" and fleshed out.
>>>> >> >>
>>>> >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement
>>>> >> >> for
>>>> >> >> the
>>>> >> >> virtual environment setup done by instack-virt-setup. 3rd party CI
>>>> >> >> would
>>>> >> >> be
>>>> >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup
>>>> >> >> was
>>>> >> >> always
>>>> >> >> working. That was the extent of the CI scope defined in the spec.
>>>> >> >> That
>>>> >> >> work is
>>>> >> >> not yet completed (see work items in the spec).
>>>> >> >>
>>>> >> >> Now it seems it is a much more all encompassing
>>>> >> >> CI/automation/testing
>>>> >> >> project
>>>> >> >> that is competing in scope with tripleo-ci itself.
>>>> >> >
>>>> >> >
>>>> >> > IMHO you are correct here.  There has been quite a bit of discussion
>>>> >> > about
>>>> >> > removing the parts
>>>> >> > of oooq that are outside of the original blueprint to replace
>>>> >> > instack-virt-setup w/ oooq.   As usual there are many different
>>>> >> > opinions
>>>> >> > here.  I think there are a lot of RDO guys that would prefer a lot of
>>>> >> > the
>>>> >> > native oooq roles stay where they are,  I think that is short sighted
>>>> >> > imho.
>>>> >> > I agree that anything outside of the blueprint be removed from oooq.
>>>> >> > This
>>>> >> > would hopefully allow the upstream to be more comfortable with oooq
>>>> >> > and
>>>> >> > allow us to really start consolidating tools.
>>>> >> >
>>>> >> > Luckily for the users that still want to use oooq as a full
>>>> >> > end-to-end
>>>> >> > solution the 3rd party roles can be used even after tearing out these
>>>> >> > native
>>>> >> > roles.
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> I'm all for consolidation of these types of tools *if* there is
>>>> >> >> interest.
>>>> >> >
>>>> >> >
>>>> >> > Roll call.. is there interest?   +1 from me.
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> However, IMO, incubating these things downstream and then trying to
>>>> >> >> get
>>>> >> >> them
>>>> >> >> upstream or get upstream to adopt them is not ideal or a good
>>>> >> >> example.
>>>> >> >> The
>>>> >> >> same
>>>> >> >> topic came up and was pushed several times with khaleesi, and it
>>>> >> >> just
>>>> >> >> never
>>>> >> >> happened, it was continually DOA upstream.
>>>> >> >
>>>> >> >
>>>> >> > True, however that could be a result of the downstream perceiving
>>>> >> > barriers (
>>>> >> > real or not ) in incubating projects in upstream openstack.
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> I think it would be fairly difficult to get tripleo-ci to wholesale
>>>> >> >> adopt
>>>> >> >> tripleo-quickstart at this stage. The separate irc channel from
>>>> >> >> #tripleo
>>>> >> >> is not
>>>> >> >> conducive to consolidation on tooling and direction imo.
>>>> >> >
>>>> >> >
>>>> >> > The irc channel is easily addressed.  We do seem to generate an awful
>>>> >> > amount
>>>> >> > of chatter though :)
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> The scope of quickstart is actually not fully understood by myself.
>>>> >> >> I've
>>>> >> >> also
>>>> >> >> heard from some in the upstream TripleO community as well who are
>>>> >> >> confused
>>>> >> >> by
>>>> >> >> its direction and are facing similar difficulties using its
>>>> >> >> generated
>>>> >> >> bash
>>>> >> >> scripts that they'd be facing if they were just using TripleO
>>>> >> >> documentation
>>>> >> >> instead.
>>>> >> >
>>>> >> >
>>>> >> > The point of the generated bash scripts is to create rst
>>>> >> > documentation
>>>> >> > and
>>>> >> > reusable scripts for the end user.  Since the documentation and the
>>>> >> > generated scripts are equivalent I would expect the same errors,
>>>> >> > problems
>>>> >> > and issues.  I see this as a good thing really.  We *want* the CI to
>>>> >> > hit
>>>> >> > the
>>>> >> > same issues as those who are following the doc.
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> I do think that this sort of problem lends itself easily to one off
>>>> >> >> implementations as is quite evidenced in this thread. Everyone/group
>>>> >> >> wants
>>>> >> >> and
>>>> >> >> needs to automate something in a different way. And imo, none of
>>>> >> >> these
>>>> >> >> tools
>>>> >> >> are building end-user or operator facing interfaces, so they're not
>>>> >> >> fully
>>>> >> >> focused on building something that "just works for everyone". Those
>>>> >> >> interfaces
>>>> >> >> should be developed in TripleO user facing tooling anyway
>>>> >> >> (tripleoclient/openstackclient/etc).
>>>> >> >>
>>>> >> >> So, I actually think it's ok in some degree that things have been
>>>> >> >> automated
>>>> >> >> differently in different tools. Anecdotally, I suspect many users of
>>>> >> >> TripleO in
>>>> >> >> production have their own automation tools as well. And none of the
>>>> >> >> implementations mentioned in this thread would likely meet their
>>>> >> >> needs
>>>> >> >> either.
>>>> >> >
>>>> >> >
>>>> >> > This is true..  without a tool in the upstream that addresses ci,
>>>> >> > dev,
>>>> >> > test
>>>> >> > use cases across the development cycle this will continue to be the
>>>> >> > case.  I
>>>> >> > suspect even with a perfect tool, it won't ever be perfect for
>>>> >> > everyone.
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> However, if there is a desire to focus resources on consolidated
>>>> >> >> tooling
>>>> >> >> and
>>>> >> >> someone to drive it forward, then I definitely agree that the effort
>>>> >> >> needs
>>>> >> >> to
>>>> >> >> start upstream with a singular plan for tripleo-ci. From what I
>>>> >> >> gather,
>>>> >> >> that
>>>> >> >> would be some sort of alignment and reuse of tripleo-quickstart, and
>>>> >> >> then
>>>> >> >> we
>>>> >> >> could build from there.
>>>> >> >
>>>> >> >
>>>> >> > +1
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> That could start as a discussion and plan within that community with
>>>> >> >> some
>>>> >> >> agreed on concensus around that plan. There was an initial thread on
>>>> >> >> openstack-dev related to this topic but it is stalled a bit. It
>>>> >> >> could
>>>> >> >> be
>>>> >> >> continually driven to resolution via specs, the tripleo meeting,
>>>> >> >> email
>>>> >> >> or
>>>> >> >> irc
>>>> >> >> discussion until a plan is formed.
>>>> >> >
>>>> >> >
>>>> >> > +1,  I think the first step is to complete the original blueprint and
>>>> >> > move
>>>> >> > on from there.
>>>> >> > I think there has also been interest in having an in person meeting
>>>> >> > at
>>>> >> > summit.
>>>> >> >
>>>> >> > Thanks!
>>>> >> >
>>>> >> >>
>>>> >> >>
>>>> >> >> --
>>>> >> >> -- James Slagle
>>>> >> >> --
>>>> >> >>
>>>> >> >> _______________________________________________
>>>> >> >> rdo-list mailing list
>>>> >> >> rdo-list at redhat.com
>>>> >> >> https://www.redhat.com/mailman/listinfo/rdo-list
>>>> >> >>
>>>> >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > _______________________________________________
>>>> >> > rdo-list mailing list
>>>> >> > rdo-list at redhat.com
>>>> >> > https://www.redhat.com/mailman/listinfo/rdo-list
>>>> >> >
>>>> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>> >>
>>>> >> I like how the discussion goes though I have some personal (and
>>>> >> probably shared) feeling that I would like to share here, more or less
>>>> >> related.
>>>> >>
>>>> >> As a TripleO core developer, I have some frustration to see that a lot
>>>> >> of people are involved in making TripleO Quickstart better, while we
>>>> >> have a few people actually working on tripleo-ci tool and try to
>>>> >> maintain upstream CI stable.
>>>> >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that
>>>> >> actually gates TripleO, even if we don't like the tool. It is right
>>>> >> now, testing TripleO upstream, everything that is not tested in there
>>>> >> will probably break one day downstream CIs.
>>>> >> Yes we have this tooling discussion here and that's awesome, but words
>>>> >> are words. I would like to see some real engagement to help TripleO CI
>>>> >> to converge into something better and not only everyone working on
>>>> >> their side.
>>>> >
>>>> >
>>>> > You have a valid point and reason to be frustrated.
>>>> > Is the point here that everyone downstream should use tripleo.sh or that
>>>> > everyone should be focused on ci and testing at the tripleo level?
>>>>
>>>> Not everyone should use tripleo.sh. My point is that we should move
>>>> forward with a common tool, and stop enlarging the gap between tools.
>>>> We have created (and are still doing) a technical dept where we have
>>>> multiple tools with a ton of overlap, the more we wait, more difficult
>>>> it will be to clean this up.
>>>>
>>>> >>
>>>> >>
>>>> >> Some examples:
>>>> >> - TripleO Quickstart (downstream) CI has coverage for undercloud &
>>>> >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade
>>>> >> job and used to have a overcloud (minor) upgrade job (disabled now,
>>>> >> for some reasons related to our capacity to run jobs and also some
>>>> >> blockers into code itself).
>>>> >> - TripleO CI has some TripleO Heat templates that could also be re-use
>>>> >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to
>>>> >> THT, WIP here: https://review.openstack.org/350775).
>>>> >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't.
>>>> >> - (...)
>>>> >
>>>> >
>>>> > As others have mentioned, there are at least 5-10 tools in development
>>>> > that
>>>> > are used to deploy tripleo in some CI fashion.  Calling out
>>>> > tripleo-quickstart alone is not quite right imho.   There are a number
>>>> > of
>>>> > tripleo devs that burn cycles on their own ci tools and maybe that is
>>>> > fine
>>>> > thing to do.
>>>>
>>>> I called quickstart because that's the one I see everyday but my
>>>> frustration is about all our tools in general.
>>>> I'm actually a OOOQ user and I like this tool, really.
>>>> But as you can see, I'm also working on tripleo-ci right now because I
>>>> want TripleO CI better and I haven't seen until now some interest to
>>>> converge.
>>>> James started something cool by trying to deploy an undercloud using
>>>> OOOQ from tripleo-ci. That's a start ! We need things like this,
>>>> prototyping convergence, and see what we can do.
>>>>
>>>> > TripleO-Quickstart is meant to replace instack-virt-setup which it does
>>>> > quite well.   The only group that was actually running
>>>> > instack-virt-setup
>>>> > was the RDO CI team, upstream had taken it out of the ci system.  I
>>>> > think
>>>> > it's not unfair to say gaps have been left for other teams to fill.
>>>>
>>>> Gotcha. It was just some examples.
>>>>
>>>> >>
>>>> >>
>>>> >> We have been having this discussion for a while now but we're still
>>>> >> not making much progress here, I feel like we're in statu quo.
>>>> >> James mentioned a blueprint, I like it. We need to engage some
>>>> >> upstream discussion about this major CI refactor, like we need with
>>>> >> specs and then we'll decide if whether or not we need to change the
>>>> >> tool, and how.
>>>> >
>>>> >
>>>> > Well, this would take some leadership imho.  We need some people that
>>>> > are
>>>> > familiar with the upstream, midstream and downstream requirements of CI.
>>>> > This was addressed at the production chain meetings initially but then
>>>> > pretty much ignored.   The leaders responsible at the various stages of
>>>> > a
>>>> > build (upstream -> downstream ) failed to take this issue on.  Here we
>>>> > are
>>>> > today.
>>>> >
>>>> > Would it be acceptable by anyone.. IF
>>>> >
>>>> > tripleo-quickstart  replaced instack-virt-setup [1] and walked through
>>>> > the
>>>> > undercloud install, then handed off to tripleo.sh to deploy, upgrade,
>>>> > update, scale, validate etc???
>>>>
>>>> That's something we can try.
>>>>
>>>> > That these two tools *would* in fact be the the official CI tools of
>>>> > tripleo
>>>> > at the upstream, RDO, and at least parts of the downstream?
>>>>
>>>> My opinion on this is that upstream and downstream CI should only differ
>>>> on:
>>>> * the packages (OSP vs RDO)
>>>> * the scenarios (downstream could have customer-specific things)
>>>> And that's it. Tools should remain the same IMHO.
>>>>
>>>> > Would that help to ease the current frustration around CI? Emilien what
>>>> > do
>>>> > you think?
>>>>
>>>> I spent the last months working on composable roles and I have now
>>>> more time to work on CI; $topic is definitely something where I would
>>>> like to help.
>>>> --
>>>> Emilien Macchi
>>>>
>>>> _______________________________________________
>>>> rdo-list mailing list
>>>> rdo-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>
>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>
>>>
>>>
>>>
>>> --
>>> Tal Kammer
>>> Associate manager, automation and infrastracture, Openstack platform.
>>> Red Hat Israel
>>> Automation group mojo: https://mojo.redhat.com/docs/DOC-1011659
>>>
>>> _______________________________________________
>>> rdo-list mailing list
>>> rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
>
>
> --
> Emilien Macchi