[rdo-list] Multiple tools for deploying and testing TripleO

Thu Aug 4 18:55:11 UTC 2016

On Thu, Aug 4, 2016 at 9:43 AM, Wesley Hayutin <whayutin at redhat.com> wrote:

>
>
> On Wed, Aug 3, 2016 at 8:45 PM, Emilien Macchi <emilien at redhat.com> wrote:
>
>> On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin at redhat.com>
>> wrote:
>> >
>> >
>> > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien at redhat.com>
>> wrote:
>> >>
>> >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin at redhat.com>
>> >> wrote:
>> >> >
>> >> >
>> >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle at redhat.com>
>> >> > wrote:
>> >> >>
>> >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote:
>> >> >> > Please hear me out.
>> >> >> > TL;DR, Let's work upstream and make it awesome so that downstream
>> can
>> >> >> > be awesome.
>> >> >> >
>> >> >> > I've said this before but I'm going to re-iterate that I do not
>> >> >> > understand why there is so much effort spent around testing
>> TripleO
>> >> >> > downstream.
>> >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI
>> >> >> > proper.
>> >> >> >
>> >> >> > All this work should be done upstream to make TripleO and it's CI
>> >> >> > super awesome and this would trickle down for free downstream.
>> >> >> >
>> >> >> > The RDO Trunk testing pipeline is composed of two tools, today.
>> >> >> > The TripleO-Quickstart project [1] is a good example of an
>> initiative
>> >> >> > that started downstream but always had the intention of being
>> >> >> > proposed
>> >> >> > upstream [2] after being "incubated" and fleshed out.
>> >> >>
>> >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement
>> >> >> for
>> >> >> the
>> >> >> virtual environment setup done by instack-virt-setup. 3rd party CI
>> >> >> would
>> >> >> be
>> >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup
>> was
>> >> >> always
>> >> >> working. That was the extent of the CI scope defined in the spec.
>> That
>> >> >> work is
>> >> >> not yet completed (see work items in the spec).
>> >> >>
>> >> >> Now it seems it is a much more all encompassing
>> CI/automation/testing
>> >> >> project
>> >> >> that is competing in scope with tripleo-ci itself.
>> >> >
>> >> >
>> >> > IMHO you are correct here.  There has been quite a bit of discussion
>> >> > about
>> >> > removing the parts
>> >> > of oooq that are outside of the original blueprint to replace
>> >> > instack-virt-setup w/ oooq.   As usual there are many different
>> opinions
>> >> > here.  I think there are a lot of RDO guys that would prefer a lot of
>> >> > the
>> >> > native oooq roles stay where they are,  I think that is short sighted
>> >> > imho.
>> >> > I agree that anything outside of the blueprint be removed from oooq.
>> >> > This
>> >> > would hopefully allow the upstream to be more comfortable with oooq
>> and
>> >> > allow us to really start consolidating tools.
>> >> >
>> >> > Luckily for the users that still want to use oooq as a full
>> end-to-end
>> >> > solution the 3rd party roles can be used even after tearing out these
>> >> > native
>> >> > roles.
>> >> >
>> >> >>
>> >> >>
>> >> >> I'm all for consolidation of these types of tools *if* there is
>> >> >> interest.
>> >> >
>> >> >
>> >> > Roll call.. is there interest?   +1 from me.
>> >> >
>> >> >>
>> >> >>
>> >> >> However, IMO, incubating these things downstream and then trying to
>> get
>> >> >> them
>> >> >> upstream or get upstream to adopt them is not ideal or a good
>> example.
>> >> >> The
>> >> >> same
>> >> >> topic came up and was pushed several times with khaleesi, and it
>> just
>> >> >> never
>> >> >> happened, it was continually DOA upstream.
>> >> >
>> >> >
>> >> > True, however that could be a result of the downstream perceiving
>> >> > barriers (
>> >> > real or not ) in incubating projects in upstream openstack.
>> >> >
>> >> >>
>> >> >>
>> >> >> I think it would be fairly difficult to get tripleo-ci to wholesale
>> >> >> adopt
>> >> >> tripleo-quickstart at this stage. The separate irc channel from
>> >> >> #tripleo
>> >> >> is not
>> >> >> conducive to consolidation on tooling and direction imo.
>> >> >
>> >> >
>> >> > The irc channel is easily addressed.  We do seem to generate an awful
>> >> > amount
>> >> > of chatter though :)
>> >> >
>> >> >>
>> >> >>
>> >> >> The scope of quickstart is actually not fully understood by myself.
>> >> >> I've
>> >> >> also
>> >> >> heard from some in the upstream TripleO community as well who are
>> >> >> confused
>> >> >> by
>> >> >> its direction and are facing similar difficulties using its
>> generated
>> >> >> bash
>> >> >> scripts that they'd be facing if they were just using TripleO
>> >> >> documentation
>> >> >> instead.
>> >> >
>> >> >
>> >> > The point of the generated bash scripts is to create rst
>> documentation
>> >> > and
>> >> > reusable scripts for the end user.  Since the documentation and the
>> >> > generated scripts are equivalent I would expect the same errors,
>> >> > problems
>> >> > and issues.  I see this as a good thing really.  We *want* the CI to
>> hit
>> >> > the
>> >> > same issues as those who are following the doc.
>> >> >
>> >> >>
>> >> >>
>> >> >> I do think that this sort of problem lends itself easily to one off
>> >> >> implementations as is quite evidenced in this thread. Everyone/group
>> >> >> wants
>> >> >> and
>> >> >> needs to automate something in a different way. And imo, none of
>> these
>> >> >> tools
>> >> >> are building end-user or operator facing interfaces, so they're not
>> >> >> fully
>> >> >> focused on building something that "just works for everyone". Those
>> >> >> interfaces
>> >> >> should be developed in TripleO user facing tooling anyway
>> >> >> (tripleoclient/openstackclient/etc).
>> >> >>
>> >> >> So, I actually think it's ok in some degree that things have been
>> >> >> automated
>> >> >> differently in different tools. Anecdotally, I suspect many users of
>> >> >> TripleO in
>> >> >> production have their own automation tools as well. And none of the
>> >> >> implementations mentioned in this thread would likely meet their
>> needs
>> >> >> either.
>> >> >
>> >> >
>> >> > This is true..  without a tool in the upstream that addresses ci,
>> dev,
>> >> > test
>> >> > use cases across the development cycle this will continue to be the
>> >> > case.  I
>> >> > suspect even with a perfect tool, it won't ever be perfect for
>> everyone.
>> >> >
>> >> >>
>> >> >>
>> >> >> However, if there is a desire to focus resources on consolidated
>> >> >> tooling
>> >> >> and
>> >> >> someone to drive it forward, then I definitely agree that the effort
>> >> >> needs
>> >> >> to
>> >> >> start upstream with a singular plan for tripleo-ci. From what I
>> gather,
>> >> >> that
>> >> >> would be some sort of alignment and reuse of tripleo-quickstart, and
>> >> >> then
>> >> >> we
>> >> >> could build from there.
>> >> >
>> >> >
>> >> > +1
>> >> >
>> >> >>
>> >> >>
>> >> >> That could start as a discussion and plan within that community with
>> >> >> some
>> >> >> agreed on concensus around that plan. There was an initial thread on
>> >> >> openstack-dev related to this topic but it is stalled a bit. It
>> could
>> >> >> be
>> >> >> continually driven to resolution via specs, the tripleo meeting,
>> email
>> >> >> or
>> >> >> irc
>> >> >> discussion until a plan is formed.
>> >> >
>> >> >
>> >> > +1,  I think the first step is to complete the original blueprint and
>> >> > move
>> >> > on from there.
>> >> > I think there has also been interest in having an in person meeting
>> at
>> >> > summit.
>> >> >
>> >> > Thanks!
>> >> >
>> >> >>
>> >> >>
>> >> >> --
>> >> >> -- James Slagle
>> >> >> --
>> >> >>
>> >> >> _______________________________________________
>> >> >> rdo-list mailing list
>> >> >> rdo-list at redhat.com
>> >> >> https://www.redhat.com/mailman/listinfo/rdo-list
>> >> >>
>> >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > rdo-list mailing list
>> >> > rdo-list at redhat.com
>> >> > https://www.redhat.com/mailman/listinfo/rdo-list
>> >> >
>> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>> >>
>> >> I like how the discussion goes though I have some personal (and
>> >> probably shared) feeling that I would like to share here, more or less
>> >> related.
>> >>
>> >> As a TripleO core developer, I have some frustration to see that a lot
>> >> of people are involved in making TripleO Quickstart better, while we
>> >> have a few people actually working on tripleo-ci tool and try to
>> >> maintain upstream CI stable.
>> >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that
>> >> actually gates TripleO, even if we don't like the tool. It is right
>> >> now, testing TripleO upstream, everything that is not tested in there
>> >> will probably break one day downstream CIs.
>> >> Yes we have this tooling discussion here and that's awesome, but words
>> >> are words. I would like to see some real engagement to help TripleO CI
>> >> to converge into something better and not only everyone working on
>> >> their side.
>> >
>> >
>> > You have a valid point and reason to be frustrated.
>> > Is the point here that everyone downstream should use tripleo.sh or that
>> > everyone should be focused on ci and testing at the tripleo level?
>>
>> Not everyone should use tripleo.sh. My point is that we should move
>> forward with a common tool, and stop enlarging the gap between tools.
>> We have created (and are still doing) a technical dept where we have
>> multiple tools with a ton of overlap, the more we wait, more difficult
>> it will be to clean this up.
>>
>
> +1
>
>>
>> >>
>> >>
>> >> Some examples:
>> >> - TripleO Quickstart (downstream) CI has coverage for undercloud &
>> >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade
>> >> job and used to have a overcloud (minor) upgrade job (disabled now,
>> >> for some reasons related to our capacity to run jobs and also some
>> >> blockers into code itself).
>> >> - TripleO CI has some TripleO Heat templates that could also be re-use
>> >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to
>> >> THT, WIP here: https://review.openstack.org/350775).
>> >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't.
>> >> - (...)
>> >
>> >
>> > As others have mentioned, there are at least 5-10 tools in development
>> that
>> > are used to deploy tripleo in some CI fashion.  Calling out
>> > tripleo-quickstart alone is not quite right imho.   There are a number
>> of
>> > tripleo devs that burn cycles on their own ci tools and maybe that is
>> fine
>> > thing to do.
>>
>> I called quickstart because that's the one I see everyday but my
>> frustration is about all our tools in general.
>> I'm actually a OOOQ user and I like this tool, really.
>> But as you can see, I'm also working on tripleo-ci right now because I
>> want TripleO CI better and I haven't seen until now some interest to
>> converge.
>> James started something cool by trying to deploy an undercloud using
>> OOOQ from tripleo-ci. That's a start ! We need things like this,
>> prototyping convergence, and see what we can do.
>>
>
> +1 working in multiple ci systems is painful, I know your pain!
> I've been playing around w/ Jame's patch [1] to see if I can help.
> I like Jame's approach, I think I may be missing some setup steps that
> nodepool provides.
>
> [1] https://review.openstack.org/#/c/348530/
>
>
>>
>> > TripleO-Quickstart is meant to replace instack-virt-setup which it does
>> > quite well.   The only group that was actually running
>> instack-virt-setup
>> > was the RDO CI team, upstream had taken it out of the ci system.  I
>> think
>> > it's not unfair to say gaps have been left for other teams to fill.
>>
>> Gotcha. It was just some examples.
>>
>> >>
>> >>
>> >> We have been having this discussion for a while now but we're still
>> >> not making much progress here, I feel like we're in statu quo.
>> >> James mentioned a blueprint, I like it. We need to engage some
>> >> upstream discussion about this major CI refactor, like we need with
>> >> specs and then we'll decide if whether or not we need to change the
>> >> tool, and how.
>> >
>> >
>> > Well, this would take some leadership imho.  We need some people that
>> are
>> > familiar with the upstream, midstream and downstream requirements of CI.
>> > This was addressed at the production chain meetings initially but then
>> > pretty much ignored.   The leaders responsible at the various stages of
>> a
>> > build (upstream -> downstream ) failed to take this issue on.  Here we
>> are
>> > today.
>> >
>> > Would it be acceptable by anyone.. IF
>> >
>> > tripleo-quickstart  replaced instack-virt-setup [1] and walked through
>> the
>> > undercloud install, then handed off to tripleo.sh to deploy, upgrade,
>> > update, scale, validate etc???
>>
>> That's something we can try.
>>
>> > That these two tools *would* in fact be the the official CI tools of
>> tripleo
>> > at the upstream, RDO, and at least parts of the downstream?
>>
>> My opinion on this is that upstream and downstream CI should only differ
>> on:
>> * the packages (OSP vs RDO)
>> * the scenarios (downstream could have customer-specific things)
>> And that's it. Tools should remain the same IMHO.
>>
>> > Would that help to ease the current frustration around CI? Emilien what
>> do
>> > you think?
>>
>> I spent the last months working on composable roles and I have now
>> more time to work on CI; $topic is definitely something where I would
>> like to help.
>>
>
> woot, I'm excited that you are freeing up and can be more involved!
> Thanks as usual Emilien!
>
>
>> --
>> Emilien Macchi
>>
>
>
I would add one thing.. If there are folks out there that rely on CI or CI
tools and need to be part of this process than please speak up!
If you have tools, ideas, requirements now's a pretty good time to be
verbose about it.  Some of you already have, some have not.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20160804/1860044b/attachment.html>