[rdo-list] Multiple tools for deploying and testing TripleO

[Emilien Macchi]

On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin at redhat.com> wrote:
>
>
> On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien at redhat.com> wrote:
>>
>> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin at redhat.com>
>> wrote:
>> >
>> >
>> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle at redhat.com>
>> > wrote:
>> >>
>> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote:
>> >> > Please hear me out.
>> >> > TL;DR, Let's work upstream and make it awesome so that downstream can
>> >> > be awesome.
>> >> >
>> >> > I've said this before but I'm going to re-iterate that I do not
>> >> > understand why there is so much effort spent around testing TripleO
>> >> > downstream.
>> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI
>> >> > proper.
>> >> >
>> >> > All this work should be done upstream to make TripleO and it's CI
>> >> > super awesome and this would trickle down for free downstream.
>> >> >
>> >> > The RDO Trunk testing pipeline is composed of two tools, today.
>> >> > The TripleO-Quickstart project [1] is a good example of an initiative
>> >> > that started downstream but always had the intention of being
>> >> > proposed
>> >> > upstream [2] after being "incubated" and fleshed out.
>> >>
>> >> tripleo-quickstart was proposed to upstream TripleO as a replacement
>> >> for
>> >> the
>> >> virtual environment setup done by instack-virt-setup. 3rd party CI
>> >> would
>> >> be
>> >> used to gate tripleo-quickstart so that we'd be sure the virt setup was
>> >> always
>> >> working. That was the extent of the CI scope defined in the spec. That
>> >> work is
>> >> not yet completed (see work items in the spec).
>> >>
>> >> Now it seems it is a much more all encompassing CI/automation/testing
>> >> project
>> >> that is competing in scope with tripleo-ci itself.
>> >
>> >
>> > IMHO you are correct here.  There has been quite a bit of discussion
>> > about
>> > removing the parts
>> > of oooq that are outside of the original blueprint to replace
>> > instack-virt-setup w/ oooq.   As usual there are many different opinions
>> > here.  I think there are a lot of RDO guys that would prefer a lot of
>> > the
>> > native oooq roles stay where they are,  I think that is short sighted
>> > imho.
>> > I agree that anything outside of the blueprint be removed from oooq.
>> > This
>> > would hopefully allow the upstream to be more comfortable with oooq and
>> > allow us to really start consolidating tools.
>> >
>> > Luckily for the users that still want to use oooq as a full end-to-end
>> > solution the 3rd party roles can be used even after tearing out these
>> > native
>> > roles.
>> >
>> >>
>> >>
>> >> I'm all for consolidation of these types of tools *if* there is
>> >> interest.
>> >
>> >
>> > Roll call.. is there interest?   +1 from me.
>> >
>> >>
>> >>
>> >> However, IMO, incubating these things downstream and then trying to get
>> >> them
>> >> upstream or get upstream to adopt them is not ideal or a good example.
>> >> The
>> >> same
>> >> topic came up and was pushed several times with khaleesi, and it just
>> >> never
>> >> happened, it was continually DOA upstream.
>> >
>> >
>> > True, however that could be a result of the downstream perceiving
>> > barriers (
>> > real or not ) in incubating projects in upstream openstack.
>> >
>> >>
>> >>
>> >> I think it would be fairly difficult to get tripleo-ci to wholesale
>> >> adopt
>> >> tripleo-quickstart at this stage. The separate irc channel from
>> >> #tripleo
>> >> is not
>> >> conducive to consolidation on tooling and direction imo.
>> >
>> >
>> > The irc channel is easily addressed.  We do seem to generate an awful
>> > amount
>> > of chatter though :)
>> >
>> >>
>> >>
>> >> The scope of quickstart is actually not fully understood by myself.
>> >> I've
>> >> also
>> >> heard from some in the upstream TripleO community as well who are
>> >> confused
>> >> by
>> >> its direction and are facing similar difficulties using its generated
>> >> bash
>> >> scripts that they'd be facing if they were just using TripleO
>> >> documentation
>> >> instead.
>> >
>> >
>> > The point of the generated bash scripts is to create rst documentation
>> > and
>> > reusable scripts for the end user.  Since the documentation and the
>> > generated scripts are equivalent I would expect the same errors,
>> > problems
>> > and issues.  I see this as a good thing really.  We *want* the CI to hit
>> > the
>> > same issues as those who are following the doc.
>> >
>> >>
>> >>
>> >> I do think that this sort of problem lends itself easily to one off
>> >> implementations as is quite evidenced in this thread. Everyone/group
>> >> wants
>> >> and
>> >> needs to automate something in a different way. And imo, none of these
>> >> tools
>> >> are building end-user or operator facing interfaces, so they're not
>> >> fully
>> >> focused on building something that "just works for everyone". Those
>> >> interfaces
>> >> should be developed in TripleO user facing tooling anyway
>> >> (tripleoclient/openstackclient/etc).
>> >>
>> >> So, I actually think it's ok in some degree that things have been
>> >> automated
>> >> differently in different tools. Anecdotally, I suspect many users of
>> >> TripleO in
>> >> production have their own automation tools as well. And none of the
>> >> implementations mentioned in this thread would likely meet their needs
>> >> either.
>> >
>> >
>> > This is true..  without a tool in the upstream that addresses ci, dev,
>> > test
>> > use cases across the development cycle this will continue to be the
>> > case.  I
>> > suspect even with a perfect tool, it won't ever be perfect for everyone.
>> >
>> >>
>> >>
>> >> However, if there is a desire to focus resources on consolidated
>> >> tooling
>> >> and
>> >> someone to drive it forward, then I definitely agree that the effort
>> >> needs
>> >> to
>> >> start upstream with a singular plan for tripleo-ci. From what I gather,
>> >> that
>> >> would be some sort of alignment and reuse of tripleo-quickstart, and
>> >> then
>> >> we
>> >> could build from there.
>> >
>> >
>> > +1
>> >
>> >>
>> >>
>> >> That could start as a discussion and plan within that community with
>> >> some
>> >> agreed on concensus around that plan. There was an initial thread on
>> >> openstack-dev related to this topic but it is stalled a bit. It could
>> >> be
>> >> continually driven to resolution via specs, the tripleo meeting, email
>> >> or
>> >> irc
>> >> discussion until a plan is formed.
>> >
>> >
>> > +1,  I think the first step is to complete the original blueprint and
>> > move
>> > on from there.
>> > I think there has also been interest in having an in person meeting at
>> > summit.
>> >
>> > Thanks!
>> >
>> >>
>> >>
>> >> --
>> >> -- James Slagle
>> >> --
>> >>
>> >> _______________________________________________
>> >> rdo-list mailing list
>> >> rdo-list at redhat.com
>> >> https://www.redhat.com/mailman/listinfo/rdo-list
>> >>
>> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
>> >
>> >
>> >
>> > _______________________________________________
>> > rdo-list mailing list
>> > rdo-list at redhat.com
>> > https://www.redhat.com/mailman/listinfo/rdo-list
>> >
>> > To unsubscribe: rdo-list-unsubscribe at redhat.com
>>
>> I like how the discussion goes though I have some personal (and
>> probably shared) feeling that I would like to share here, more or less
>> related.
>>
>> As a TripleO core developer, I have some frustration to see that a lot
>> of people are involved in making TripleO Quickstart better, while we
>> have a few people actually working on tripleo-ci tool and try to
>> maintain upstream CI stable.
>> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that
>> actually gates TripleO, even if we don't like the tool. It is right
>> now, testing TripleO upstream, everything that is not tested in there
>> will probably break one day downstream CIs.
>> Yes we have this tooling discussion here and that's awesome, but words
>> are words. I would like to see some real engagement to help TripleO CI
>> to converge into something better and not only everyone working on
>> their side.
>
>
> You have a valid point and reason to be frustrated.
> Is the point here that everyone downstream should use tripleo.sh or that
> everyone should be focused on ci and testing at the tripleo level?

Not everyone should use tripleo.sh. My point is that we should move
forward with a common tool, and stop enlarging the gap between tools.
We have created (and are still doing) a technical dept where we have
multiple tools with a ton of overlap, the more we wait, more difficult
it will be to clean this up.

>>
>>
>> Some examples:
>> - TripleO Quickstart (downstream) CI has coverage for undercloud &
>> overcloud upgrades while TripleO CI freshly has a undercloud upgrade
>> job and used to have a overcloud (minor) upgrade job (disabled now,
>> for some reasons related to our capacity to run jobs and also some
>> blockers into code itself).
>> - TripleO CI has some TripleO Heat templates that could also be re-use
>> by TripleO Quickstart (I'm working on moving them from tripleo-ci to
>> THT, WIP here: https://review.openstack.org/350775).
>> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't.
>> - (...)
>
>
> As others have mentioned, there are at least 5-10 tools in development that
> are used to deploy tripleo in some CI fashion.  Calling out
> tripleo-quickstart alone is not quite right imho.   There are a number of
> tripleo devs that burn cycles on their own ci tools and maybe that is fine
> thing to do.

I called quickstart because that's the one I see everyday but my
frustration is about all our tools in general.
I'm actually a OOOQ user and I like this tool, really.
But as you can see, I'm also working on tripleo-ci right now because I
want TripleO CI better and I haven't seen until now some interest to
converge.
James started something cool by trying to deploy an undercloud using
OOOQ from tripleo-ci. That's a start ! We need things like this,
prototyping convergence, and see what we can do.

> TripleO-Quickstart is meant to replace instack-virt-setup which it does
> quite well.   The only group that was actually running instack-virt-setup
> was the RDO CI team, upstream had taken it out of the ci system.  I think
> it's not unfair to say gaps have been left for other teams to fill.

Gotcha. It was just some examples.

>>
>>
>> We have been having this discussion for a while now but we're still
>> not making much progress here, I feel like we're in statu quo.
>> James mentioned a blueprint, I like it. We need to engage some
>> upstream discussion about this major CI refactor, like we need with
>> specs and then we'll decide if whether or not we need to change the
>> tool, and how.
>
>
> Well, this would take some leadership imho.  We need some people that are
> familiar with the upstream, midstream and downstream requirements of CI.
> This was addressed at the production chain meetings initially but then
> pretty much ignored.   The leaders responsible at the various stages of a
> build (upstream -> downstream ) failed to take this issue on.  Here we are
> today.
>
> Would it be acceptable by anyone.. IF
>
> tripleo-quickstart  replaced instack-virt-setup [1] and walked through the
> undercloud install, then handed off to tripleo.sh to deploy, upgrade,
> update, scale, validate etc???

That's something we can try.

> That these two tools *would* in fact be the the official CI tools of tripleo
> at the upstream, RDO, and at least parts of the downstream?

My opinion on this is that upstream and downstream CI should only differ on:
* the packages (OSP vs RDO)
* the scenarios (downstream could have customer-specific things)
And that's it. Tools should remain the same IMHO.

> Would that help to ease the current frustration around CI? Emilien what do
> you think?

I spent the last months working on composable roles and I have now
more time to work on CI; $topic is definitely something where I would
like to help.
-- 
Emilien Macchi