[rdo-list] Multiple tools for deploying and testing TripleO

Thu Aug 4 20:24:50 UTC 2016

On Thu, Aug 4, 2016 at 2:55 PM, Wesley Hayutin <whayutin at redhat.com> wrote:

>
>
>
> On Thu, Aug 4, 2016 at 9:43 AM, Wesley Hayutin <whayutin at redhat.com>
> wrote:
>
>>
>>
>> On Wed, Aug 3, 2016 at 8:45 PM, Emilien Macchi <emilien at redhat.com>
>> wrote:
>>
>>> On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin at redhat.com>
>>> wrote:
>>> >
>>> >
>>> > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien at redhat.com>
>>> wrote:
>>> >>
>>> >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin at redhat.com>
>>> >> wrote:
>>> >> >
>>> >> >
>>> >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle at redhat.com>
>>> >> > wrote:
>>> >> >>
>>> >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard
>>> wrote:
>>> >> >> > Please hear me out.
>>> >> >> > TL;DR, Let's work upstream and make it awesome so that
>>> downstream can
>>> >> >> > be awesome.
>>> >> >> >
>>> >> >> > I've said this before but I'm going to re-iterate that I do not
>>> >> >> > understand why there is so much effort spent around testing
>>> TripleO
>>> >> >> > downstream.
>>> >> >> > By downstream, I mean anything that isn't in TripleO or
>>> TripleO-CI
>>> >> >> > proper.
>>> >> >> >
>>> >> >> > All this work should be done upstream to make TripleO and it's CI
>>> >> >> > super awesome and this would trickle down for free downstream.
>>> >> >> >
>>> >> >> > The RDO Trunk testing pipeline is composed of two tools, today.
>>> >> >> > The TripleO-Quickstart project [1] is a good example of an
>>> initiative
>>> >> >> > that started downstream but always had the intention of being
>>> >> >> > proposed
>>> >> >> > upstream [2] after being "incubated" and fleshed out.
>>> >> >>
>>> >> >> tripleo-quickstart was proposed to upstream TripleO as a
>>> replacement
>>> >> >> for
>>> >> >> the
>>> >> >> virtual environment setup done by instack-virt-setup. 3rd party CI
>>> >> >> would
>>> >> >> be
>>> >> >> used to gate tripleo-quickstart so that we'd be sure the virt
>>> setup was
>>> >> >> always
>>> >> >> working. That was the extent of the CI scope defined in the spec.
>>> That
>>> >> >> work is
>>> >> >> not yet completed (see work items in the spec).
>>> >> >>
>>> >> >> Now it seems it is a much more all encompassing
>>> CI/automation/testing
>>> >> >> project
>>> >> >> that is competing in scope with tripleo-ci itself.
>>> >> >
>>> >> >
>>> >> > IMHO you are correct here.  There has been quite a bit of discussion
>>> >> > about
>>> >> > removing the parts
>>> >> > of oooq that are outside of the original blueprint to replace
>>> >> > instack-virt-setup w/ oooq.   As usual there are many different
>>> opinions
>>> >> > here.  I think there are a lot of RDO guys that would prefer a lot
>>> of
>>> >> > the
>>> >> > native oooq roles stay where they are,  I think that is short
>>> sighted
>>> >> > imho.
>>> >> > I agree that anything outside of the blueprint be removed from oooq.
>>> >> > This
>>> >> > would hopefully allow the upstream to be more comfortable with oooq
>>> and
>>> >> > allow us to really start consolidating tools.
>>> >> >
>>> >> > Luckily for the users that still want to use oooq as a full
>>> end-to-end
>>> >> > solution the 3rd party roles can be used even after tearing out
>>> these
>>> >> > native
>>> >> > roles.
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> I'm all for consolidation of these types of tools *if* there is
>>> >> >> interest.
>>> >> >
>>> >> >
>>> >> > Roll call.. is there interest?   +1 from me.
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> However, IMO, incubating these things downstream and then trying
>>> to get
>>> >> >> them
>>> >> >> upstream or get upstream to adopt them is not ideal or a good
>>> example.
>>> >> >> The
>>> >> >> same
>>> >> >> topic came up and was pushed several times with khaleesi, and it
>>> just
>>> >> >> never
>>> >> >> happened, it was continually DOA upstream.
>>> >> >
>>> >> >
>>> >> > True, however that could be a result of the downstream perceiving
>>> >> > barriers (
>>> >> > real or not ) in incubating projects in upstream openstack.
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> I think it would be fairly difficult to get tripleo-ci to wholesale
>>> >> >> adopt
>>> >> >> tripleo-quickstart at this stage. The separate irc channel from
>>> >> >> #tripleo
>>> >> >> is not
>>> >> >> conducive to consolidation on tooling and direction imo.
>>> >> >
>>> >> >
>>> >> > The irc channel is easily addressed.  We do seem to generate an
>>> awful
>>> >> > amount
>>> >> > of chatter though :)
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> The scope of quickstart is actually not fully understood by myself.
>>> >> >> I've
>>> >> >> also
>>> >> >> heard from some in the upstream TripleO community as well who are
>>> >> >> confused
>>> >> >> by
>>> >> >> its direction and are facing similar difficulties using its
>>> generated
>>> >> >> bash
>>> >> >> scripts that they'd be facing if they were just using TripleO
>>> >> >> documentation
>>> >> >> instead.
>>> >> >
>>> >> >
>>> >> > The point of the generated bash scripts is to create rst
>>> documentation
>>> >> > and
>>> >> > reusable scripts for the end user.  Since the documentation and the
>>> >> > generated scripts are equivalent I would expect the same errors,
>>> >> > problems
>>> >> > and issues.  I see this as a good thing really.  We *want* the CI
>>> to hit
>>> >> > the
>>> >> > same issues as those who are following the doc.
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> I do think that this sort of problem lends itself easily to one off
>>> >> >> implementations as is quite evidenced in this thread.
>>> Everyone/group
>>> >> >> wants
>>> >> >> and
>>> >> >> needs to automate something in a different way. And imo, none of
>>> these
>>> >> >> tools
>>> >> >> are building end-user or operator facing interfaces, so they're not
>>> >> >> fully
>>> >> >> focused on building something that "just works for everyone". Those
>>> >> >> interfaces
>>> >> >> should be developed in TripleO user facing tooling anyway
>>> >> >> (tripleoclient/openstackclient/etc).
>>> >> >>
>>> >> >> So, I actually think it's ok in some degree that things have been
>>> >> >> automated
>>> >> >> differently in different tools. Anecdotally, I suspect many users
>>> of
>>> >> >> TripleO in
>>> >> >> production have their own automation tools as well. And none of the
>>> >> >> implementations mentioned in this thread would likely meet their
>>> needs
>>> >> >> either.
>>> >> >
>>> >> >
>>> >> > This is true..  without a tool in the upstream that addresses ci,
>>> dev,
>>> >> > test
>>> >> > use cases across the development cycle this will continue to be the
>>> >> > case.  I
>>> >> > suspect even with a perfect tool, it won't ever be perfect for
>>> everyone.
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> However, if there is a desire to focus resources on consolidated
>>> >> >> tooling
>>> >> >> and
>>> >> >> someone to drive it forward, then I definitely agree that the
>>> effort
>>> >> >> needs
>>> >> >> to
>>> >> >> start upstream with a singular plan for tripleo-ci. From what I
>>> gather,
>>> >> >> that
>>> >> >> would be some sort of alignment and reuse of tripleo-quickstart,
>>> and
>>> >> >> then
>>> >> >> we
>>> >> >> could build from there.
>>> >> >
>>> >> >
>>> >> > +1
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> That could start as a discussion and plan within that community
>>> with
>>> >> >> some
>>> >> >> agreed on concensus around that plan. There was an initial thread
>>> on
>>> >> >> openstack-dev related to this topic but it is stalled a bit. It
>>> could
>>> >> >> be
>>> >> >> continually driven to resolution via specs, the tripleo meeting,
>>> email
>>> >> >> or
>>> >> >> irc
>>> >> >> discussion until a plan is formed.
>>> >> >
>>> >> >
>>> >> > +1,  I think the first step is to complete the original blueprint
>>> and
>>> >> > move
>>> >> > on from there.
>>> >> > I think there has also been interest in having an in person meeting
>>> at
>>> >> > summit.
>>> >> >
>>> >> > Thanks!
>>> >> >
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> -- James Slagle
>>> >> >> --
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> rdo-list mailing list
>>> >> >> rdo-list at redhat.com
>>> >> >> https://www.redhat.com/mailman/listinfo/rdo-list
>>> >> >>
>>> >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>> >> >
>>> >> >
>>> >> >
>>> >> > _______________________________________________
>>> >> > rdo-list mailing list
>>> >> > rdo-list at redhat.com
>>> >> > https://www.redhat.com/mailman/listinfo/rdo-list
>>> >> >
>>> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>> >>
>>> >> I like how the discussion goes though I have some personal (and
>>> >> probably shared) feeling that I would like to share here, more or less
>>> >> related.
>>> >>
>>> >> As a TripleO core developer, I have some frustration to see that a lot
>>> >> of people are involved in making TripleO Quickstart better, while we
>>> >> have a few people actually working on tripleo-ci tool and try to
>>> >> maintain upstream CI stable.
>>> >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that
>>> >> actually gates TripleO, even if we don't like the tool. It is right
>>> >> now, testing TripleO upstream, everything that is not tested in there
>>> >> will probably break one day downstream CIs.
>>> >> Yes we have this tooling discussion here and that's awesome, but words
>>> >> are words. I would like to see some real engagement to help TripleO CI
>>> >> to converge into something better and not only everyone working on
>>> >> their side.
>>> >
>>> >
>>> > You have a valid point and reason to be frustrated.
>>> > Is the point here that everyone downstream should use tripleo.sh or
>>> that
>>> > everyone should be focused on ci and testing at the tripleo level?
>>>
>>> Not everyone should use tripleo.sh. My point is that we should move
>>> forward with a common tool, and stop enlarging the gap between tools.
>>> We have created (and are still doing) a technical dept where we have
>>> multiple tools with a ton of overlap, the more we wait, more difficult
>>> it will be to clean this up.
>>>
>>
>> +1
>>
>>>
>>> >>
>>> >>
>>> >> Some examples:
>>> >> - TripleO Quickstart (downstream) CI has coverage for undercloud &
>>> >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade
>>> >> job and used to have a overcloud (minor) upgrade job (disabled now,
>>> >> for some reasons related to our capacity to run jobs and also some
>>> >> blockers into code itself).
>>> >> - TripleO CI has some TripleO Heat templates that could also be re-use
>>> >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to
>>> >> THT, WIP here: https://review.openstack.org/350775).
>>> >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart
>>> doesn't.
>>> >> - (...)
>>> >
>>> >
>>> > As others have mentioned, there are at least 5-10 tools in development
>>> that
>>> > are used to deploy tripleo in some CI fashion.  Calling out
>>> > tripleo-quickstart alone is not quite right imho.   There are a number
>>> of
>>> > tripleo devs that burn cycles on their own ci tools and maybe that is
>>> fine
>>> > thing to do.
>>>
>>> I called quickstart because that's the one I see everyday but my
>>> frustration is about all our tools in general.
>>> I'm actually a OOOQ user and I like this tool, really.
>>> But as you can see, I'm also working on tripleo-ci right now because I
>>> want TripleO CI better and I haven't seen until now some interest to
>>> converge.
>>> James started something cool by trying to deploy an undercloud using
>>> OOOQ from tripleo-ci. That's a start ! We need things like this,
>>> prototyping convergence, and see what we can do.
>>>
>>
>> +1 working in multiple ci systems is painful, I know your pain!
>> I've been playing around w/ Jame's patch [1] to see if I can help.
>> I like Jame's approach, I think I may be missing some setup steps that
>> nodepool provides.
>>
>> [1] https://review.openstack.org/#/c/348530/
>>
>>
>>>
>>> > TripleO-Quickstart is meant to replace instack-virt-setup which it does
>>> > quite well.   The only group that was actually running
>>> instack-virt-setup
>>> > was the RDO CI team, upstream had taken it out of the ci system.  I
>>> think
>>> > it's not unfair to say gaps have been left for other teams to fill.
>>>
>>> Gotcha. It was just some examples.
>>>
>>> >>
>>> >>
>>> >> We have been having this discussion for a while now but we're still
>>> >> not making much progress here, I feel like we're in statu quo.
>>> >> James mentioned a blueprint, I like it. We need to engage some
>>> >> upstream discussion about this major CI refactor, like we need with
>>> >> specs and then we'll decide if whether or not we need to change the
>>> >> tool, and how.
>>> >
>>> >
>>> > Well, this would take some leadership imho.  We need some people that
>>> are
>>> > familiar with the upstream, midstream and downstream requirements of
>>> CI.
>>> > This was addressed at the production chain meetings initially but then
>>> > pretty much ignored.   The leaders responsible at the various stages
>>> of a
>>> > build (upstream -> downstream ) failed to take this issue on.  Here we
>>> are
>>> > today.
>>> >
>>> > Would it be acceptable by anyone.. IF
>>> >
>>> > tripleo-quickstart  replaced instack-virt-setup [1] and walked through
>>> the
>>> > undercloud install, then handed off to tripleo.sh to deploy, upgrade,
>>> > update, scale, validate etc???
>>>
>>> That's something we can try.
>>>
>>> > That these two tools *would* in fact be the the official CI tools of
>>> tripleo
>>> > at the upstream, RDO, and at least parts of the downstream?
>>>
>>> My opinion on this is that upstream and downstream CI should only differ
>>> on:
>>> * the packages (OSP vs RDO)
>>> * the scenarios (downstream could have customer-specific things)
>>> And that's it. Tools should remain the same IMHO.
>>>
>>> > Would that help to ease the current frustration around CI? Emilien
>>> what do
>>> > you think?
>>>
>>> I spent the last months working on composable roles and I have now
>>> more time to work on CI; $topic is definitely something where I would
>>> like to help.
>>>
>>
>> woot, I'm excited that you are freeing up and can be more involved!
>> Thanks as usual Emilien!
>>
>>
>>> --
>>> Emilien Macchi
>>>
>>
>>
> I would add one thing.. If there are folks out there that rely on CI or CI
> tools and need to be part of this process than please speak up!
> If you have tools, ideas, requirements now's a pretty good time to be
> verbose about it.  Some of you already have, some have not.
>

After the quickstart fiasco and the uncertainty around infrared, what I
really want to see is a simple, adjustable/configurable and useful
replacement for instack-virt (which is the only working dpeloyment tool I
currently have). I have several issues with instack-virt, but none of them
are critical enough to care much, since my setups tend to be very short
lived, and I prefer a faster deployment to features atm.

Any script that will do what instack-virt does, but also will allow me to
adjust the parameters of the deployed VMs (disk/s; cpu; ram; NICs), and be
able to predict the VMs' IP addresses without a cumbersome framework around
it will be perfect.

>
> Thanks
>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20160804/e73ab0d1/attachment.html>