[rdo-list] Multiple tools for deploying and testing TripleO

Thu Aug 4 13:43:50 UTC 2016

On Wed, Aug 3, 2016 at 8:45 PM, Emilien Macchi <emilien at redhat.com> wrote:

> On Wed, Aug 3, 2016 at 7:38 PM, Wesley Hayutin <whayutin at redhat.com>
> wrote:
> >
> >
> > On Wed, Aug 3, 2016 at 6:19 PM, Emilien Macchi <emilien at redhat.com>
> wrote:
> >>
> >> On Wed, Aug 3, 2016 at 1:33 PM, Wesley Hayutin <whayutin at redhat.com>
> >> wrote:
> >> >
> >> >
> >> > On Wed, Aug 3, 2016 at 12:51 PM, James Slagle <jslagle at redhat.com>
> >> > wrote:
> >> >>
> >> >> On Wed, Aug 03, 2016 at 11:36:57AM -0400, David Moreau Simard wrote:
> >> >> > Please hear me out.
> >> >> > TL;DR, Let's work upstream and make it awesome so that downstream
> can
> >> >> > be awesome.
> >> >> >
> >> >> > I've said this before but I'm going to re-iterate that I do not
> >> >> > understand why there is so much effort spent around testing TripleO
> >> >> > downstream.
> >> >> > By downstream, I mean anything that isn't in TripleO or TripleO-CI
> >> >> > proper.
> >> >> >
> >> >> > All this work should be done upstream to make TripleO and it's CI
> >> >> > super awesome and this would trickle down for free downstream.
> >> >> >
> >> >> > The RDO Trunk testing pipeline is composed of two tools, today.
> >> >> > The TripleO-Quickstart project [1] is a good example of an
> initiative
> >> >> > that started downstream but always had the intention of being
> >> >> > proposed
> >> >> > upstream [2] after being "incubated" and fleshed out.
> >> >>
> >> >> tripleo-quickstart was proposed to upstream TripleO as a replacement
> >> >> for
> >> >> the
> >> >> virtual environment setup done by instack-virt-setup. 3rd party CI
> >> >> would
> >> >> be
> >> >> used to gate tripleo-quickstart so that we'd be sure the virt setup
> was
> >> >> always
> >> >> working. That was the extent of the CI scope defined in the spec.
> That
> >> >> work is
> >> >> not yet completed (see work items in the spec).
> >> >>
> >> >> Now it seems it is a much more all encompassing CI/automation/testing
> >> >> project
> >> >> that is competing in scope with tripleo-ci itself.
> >> >
> >> >
> >> > IMHO you are correct here.  There has been quite a bit of discussion
> >> > about
> >> > removing the parts
> >> > of oooq that are outside of the original blueprint to replace
> >> > instack-virt-setup w/ oooq.   As usual there are many different
> opinions
> >> > here.  I think there are a lot of RDO guys that would prefer a lot of
> >> > the
> >> > native oooq roles stay where they are,  I think that is short sighted
> >> > imho.
> >> > I agree that anything outside of the blueprint be removed from oooq.
> >> > This
> >> > would hopefully allow the upstream to be more comfortable with oooq
> and
> >> > allow us to really start consolidating tools.
> >> >
> >> > Luckily for the users that still want to use oooq as a full end-to-end
> >> > solution the 3rd party roles can be used even after tearing out these
> >> > native
> >> > roles.
> >> >
> >> >>
> >> >>
> >> >> I'm all for consolidation of these types of tools *if* there is
> >> >> interest.
> >> >
> >> >
> >> > Roll call.. is there interest?   +1 from me.
> >> >
> >> >>
> >> >>
> >> >> However, IMO, incubating these things downstream and then trying to
> get
> >> >> them
> >> >> upstream or get upstream to adopt them is not ideal or a good
> example.
> >> >> The
> >> >> same
> >> >> topic came up and was pushed several times with khaleesi, and it just
> >> >> never
> >> >> happened, it was continually DOA upstream.
> >> >
> >> >
> >> > True, however that could be a result of the downstream perceiving
> >> > barriers (
> >> > real or not ) in incubating projects in upstream openstack.
> >> >
> >> >>
> >> >>
> >> >> I think it would be fairly difficult to get tripleo-ci to wholesale
> >> >> adopt
> >> >> tripleo-quickstart at this stage. The separate irc channel from
> >> >> #tripleo
> >> >> is not
> >> >> conducive to consolidation on tooling and direction imo.
> >> >
> >> >
> >> > The irc channel is easily addressed.  We do seem to generate an awful
> >> > amount
> >> > of chatter though :)
> >> >
> >> >>
> >> >>
> >> >> The scope of quickstart is actually not fully understood by myself.
> >> >> I've
> >> >> also
> >> >> heard from some in the upstream TripleO community as well who are
> >> >> confused
> >> >> by
> >> >> its direction and are facing similar difficulties using its generated
> >> >> bash
> >> >> scripts that they'd be facing if they were just using TripleO
> >> >> documentation
> >> >> instead.
> >> >
> >> >
> >> > The point of the generated bash scripts is to create rst documentation
> >> > and
> >> > reusable scripts for the end user.  Since the documentation and the
> >> > generated scripts are equivalent I would expect the same errors,
> >> > problems
> >> > and issues.  I see this as a good thing really.  We *want* the CI to
> hit
> >> > the
> >> > same issues as those who are following the doc.
> >> >
> >> >>
> >> >>
> >> >> I do think that this sort of problem lends itself easily to one off
> >> >> implementations as is quite evidenced in this thread. Everyone/group
> >> >> wants
> >> >> and
> >> >> needs to automate something in a different way. And imo, none of
> these
> >> >> tools
> >> >> are building end-user or operator facing interfaces, so they're not
> >> >> fully
> >> >> focused on building something that "just works for everyone". Those
> >> >> interfaces
> >> >> should be developed in TripleO user facing tooling anyway
> >> >> (tripleoclient/openstackclient/etc).
> >> >>
> >> >> So, I actually think it's ok in some degree that things have been
> >> >> automated
> >> >> differently in different tools. Anecdotally, I suspect many users of
> >> >> TripleO in
> >> >> production have their own automation tools as well. And none of the
> >> >> implementations mentioned in this thread would likely meet their
> needs
> >> >> either.
> >> >
> >> >
> >> > This is true..  without a tool in the upstream that addresses ci, dev,
> >> > test
> >> > use cases across the development cycle this will continue to be the
> >> > case.  I
> >> > suspect even with a perfect tool, it won't ever be perfect for
> everyone.
> >> >
> >> >>
> >> >>
> >> >> However, if there is a desire to focus resources on consolidated
> >> >> tooling
> >> >> and
> >> >> someone to drive it forward, then I definitely agree that the effort
> >> >> needs
> >> >> to
> >> >> start upstream with a singular plan for tripleo-ci. From what I
> gather,
> >> >> that
> >> >> would be some sort of alignment and reuse of tripleo-quickstart, and
> >> >> then
> >> >> we
> >> >> could build from there.
> >> >
> >> >
> >> > +1
> >> >
> >> >>
> >> >>
> >> >> That could start as a discussion and plan within that community with
> >> >> some
> >> >> agreed on concensus around that plan. There was an initial thread on
> >> >> openstack-dev related to this topic but it is stalled a bit. It could
> >> >> be
> >> >> continually driven to resolution via specs, the tripleo meeting,
> email
> >> >> or
> >> >> irc
> >> >> discussion until a plan is formed.
> >> >
> >> >
> >> > +1,  I think the first step is to complete the original blueprint and
> >> > move
> >> > on from there.
> >> > I think there has also been interest in having an in person meeting at
> >> > summit.
> >> >
> >> > Thanks!
> >> >
> >> >>
> >> >>
> >> >> --
> >> >> -- James Slagle
> >> >> --
> >> >>
> >> >> _______________________________________________
> >> >> rdo-list mailing list
> >> >> rdo-list at redhat.com
> >> >> https://www.redhat.com/mailman/listinfo/rdo-list
> >> >>
> >> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
> >> >
> >> >
> >> >
> >> > _______________________________________________
> >> > rdo-list mailing list
> >> > rdo-list at redhat.com
> >> > https://www.redhat.com/mailman/listinfo/rdo-list
> >> >
> >> > To unsubscribe: rdo-list-unsubscribe at redhat.com
> >>
> >> I like how the discussion goes though I have some personal (and
> >> probably shared) feeling that I would like to share here, more or less
> >> related.
> >>
> >> As a TripleO core developer, I have some frustration to see that a lot
> >> of people are involved in making TripleO Quickstart better, while we
> >> have a few people actually working on tripleo-ci tool and try to
> >> maintain upstream CI stable.
> >> As a reminder, tripleo-ci tool is currently the ONLY ONE thing that
> >> actually gates TripleO, even if we don't like the tool. It is right
> >> now, testing TripleO upstream, everything that is not tested in there
> >> will probably break one day downstream CIs.
> >> Yes we have this tooling discussion here and that's awesome, but words
> >> are words. I would like to see some real engagement to help TripleO CI
> >> to converge into something better and not only everyone working on
> >> their side.
> >
> >
> > You have a valid point and reason to be frustrated.
> > Is the point here that everyone downstream should use tripleo.sh or that
> > everyone should be focused on ci and testing at the tripleo level?
>
> Not everyone should use tripleo.sh. My point is that we should move
> forward with a common tool, and stop enlarging the gap between tools.
> We have created (and are still doing) a technical dept where we have
> multiple tools with a ton of overlap, the more we wait, more difficult
> it will be to clean this up.
>

+1

>
> >>
> >>
> >> Some examples:
> >> - TripleO Quickstart (downstream) CI has coverage for undercloud &
> >> overcloud upgrades while TripleO CI freshly has a undercloud upgrade
> >> job and used to have a overcloud (minor) upgrade job (disabled now,
> >> for some reasons related to our capacity to run jobs and also some
> >> blockers into code itself).
> >> - TripleO CI has some TripleO Heat templates that could also be re-use
> >> by TripleO Quickstart (I'm working on moving them from tripleo-ci to
> >> THT, WIP here: https://review.openstack.org/350775).
> >> - TripleO CI deploys Ceph Jewel repository, TripleO Quickstart doesn't.
> >> - (...)
> >
> >
> > As others have mentioned, there are at least 5-10 tools in development
> that
> > are used to deploy tripleo in some CI fashion.  Calling out
> > tripleo-quickstart alone is not quite right imho.   There are a number of
> > tripleo devs that burn cycles on their own ci tools and maybe that is
> fine
> > thing to do.
>
> I called quickstart because that's the one I see everyday but my
> frustration is about all our tools in general.
> I'm actually a OOOQ user and I like this tool, really.
> But as you can see, I'm also working on tripleo-ci right now because I
> want TripleO CI better and I haven't seen until now some interest to
> converge.
> James started something cool by trying to deploy an undercloud using
> OOOQ from tripleo-ci. That's a start ! We need things like this,
> prototyping convergence, and see what we can do.
>

+1 working in multiple ci systems is painful, I know your pain!
I've been playing around w/ Jame's patch [1] to see if I can help.
I like Jame's approach, I think I may be missing some setup steps that
nodepool provides.

[1] https://review.openstack.org/#/c/348530/

>
> > TripleO-Quickstart is meant to replace instack-virt-setup which it does
> > quite well.   The only group that was actually running instack-virt-setup
> > was the RDO CI team, upstream had taken it out of the ci system.  I think
> > it's not unfair to say gaps have been left for other teams to fill.
>
> Gotcha. It was just some examples.
>
> >>
> >>
> >> We have been having this discussion for a while now but we're still
> >> not making much progress here, I feel like we're in statu quo.
> >> James mentioned a blueprint, I like it. We need to engage some
> >> upstream discussion about this major CI refactor, like we need with
> >> specs and then we'll decide if whether or not we need to change the
> >> tool, and how.
> >
> >
> > Well, this would take some leadership imho.  We need some people that are
> > familiar with the upstream, midstream and downstream requirements of CI.
> > This was addressed at the production chain meetings initially but then
> > pretty much ignored.   The leaders responsible at the various stages of a
> > build (upstream -> downstream ) failed to take this issue on.  Here we
> are
> > today.
> >
> > Would it be acceptable by anyone.. IF
> >
> > tripleo-quickstart  replaced instack-virt-setup [1] and walked through
> the
> > undercloud install, then handed off to tripleo.sh to deploy, upgrade,
> > update, scale, validate etc???
>
> That's something we can try.
>
> > That these two tools *would* in fact be the the official CI tools of
> tripleo
> > at the upstream, RDO, and at least parts of the downstream?
>
> My opinion on this is that upstream and downstream CI should only differ
> on:
> * the packages (OSP vs RDO)
> * the scenarios (downstream could have customer-specific things)
> And that's it. Tools should remain the same IMHO.
>
> > Would that help to ease the current frustration around CI? Emilien what
> do
> > you think?
>
> I spent the last months working on composable roles and I have now
> more time to work on CI; $topic is definitely something where I would
> like to help.
>

woot, I'm excited that you are freeing up and can be more involved!
Thanks as usual Emilien!

> --
> Emilien Macchi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20160804/fcef3831/attachment.html>