[rdo-dev] Long queue in RDO SF

[Javier Pena]

Hi,

I see no issues in nodepool. Looking at the current Zuul queue, we have a single job stuck for ~90 hours, queued on "gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens".

When that happens, it's usually a configuration issue, and this is the case here: we have no definition for the queens gate job for featureset035 in https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/jobs/tripleo-upstream.yml#L979-L990.

An easy way to troubleshoot this is:

- If we find one or more jobs queued, first check at https://review.rdoproject.org/jenkins/ and see if there are nodes available to jenkins.

- If there are, just check the list of jobs available to Jenkins. If it's not there, we need to double-check the jjb configuration and find what is missing.

My only doubt is why this does not show up as "NOT_REGISTERED" in Zuul as it did before.

I have proposed https://review.rdoproject.org/r/12038 as a fix for this.

Regards,
Javier

----- Original Message -----
> FWIW no alerts during the weekend and I have been able to spawn 10+
> instances without issue.
> 
> Cheers
> David Manchado
> Senior Software Engineer - SysOps Team
> Red Hat
> dmanchad at redhat.com
> 
> 
> On 11 February 2018 at 16:39, Paul Belanger <pabelanger at redhat.com> wrote:
> > On Sun, Feb 11, 2018 at 12:44:52PM +0100, Haïkel Guémar wrote:
> >> On 02/11/2018 12:17 AM, Sagi Shnaidman wrote:
> >> > Hi,
> >> >
> >> > I see openstack-check has 53 hours queue when 1 job only is queued:
> >> > https://review.rdoproject.org/zuul/
> >> >
> >> > Seems like problem with nodepool?
> >> >
> >> > Thanks
> >> >
> >> > --
> >> > Best regards
> >> > Sagi Shnaidman
> >> >
> >> >
> >> > _______________________________________________
> >> > dev mailing list
> >> > dev at lists.rdoproject.org
> >> > http://lists.rdoproject.org/mailman/listinfo/dev
> >> >
> >> > To unsubscribe: dev-unsubscribe at lists.rdoproject.org
> >> >
> >>
> >> Ok, it looks bad enough that a simple nodepool list fails with that error:
> >> os_client_config.exceptions.OpenStackConfigException: Cloud rdo-cloud was
> >> not found.
> >>
> >> Despite RDO Cloud looks up, there might be an outage or incident hence
> >> copying David Manchado.
> >>
> >> Regards,
> >> H.
> >>
> > Okay, I have to run, but this looks like a configuration issue. It is hard
> > to
> > tell without debug logs for nodepool or zuul, but please double check your
> > node
> > is setup properly.
> >
> > I have to run now.
> >
>