FWIW no alerts during the weekend and I have been able to spawn 10+ instances without issue. Cheers David Manchado Senior Software Engineer - SysOps Team Red Hat dmanchad(a)redhat.com On 11 February 2018 at 16:39, Paul Belanger <pabelanger(a)redhat.com&gt; wrote: > On Sun, Feb 11, 2018 at 12:44:52PM +0100, Haïkel Guémar wrote: >> On 02/11/2018 12:17 AM, Sagi Shnaidman wrote: >> > Hi, >> > >> > I see openstack-check has 53 hours queue when 1 job only is queued: >> > https://review.rdoproject.org/zuul/ >> > >> > Seems like problem with nodepool? >> > >> > Thanks >> > >> > -- >> > Best regards >> > Sagi Shnaidman >> > >> > >> > _______________________________________________ >> > dev mailing list >> > dev(a)lists.rdoproject.org >> > http://lists.rdoproject.org/mailman/listinfo/dev >> > >> > To unsubscribe: dev-unsubscribe(a)lists.rdoproject.org >> > >> >> Ok, it looks bad enough that a simple nodepool list fails with that error: >> os_client_config.exceptions.OpenStackConfigException: Cloud rdo-cloud was >> not found. >> >> Despite RDO Cloud looks up, there might be an outage or incident hence >> copying David Manchado. >> >> Regards, >> H. >> > Okay, I have to run, but this looks like a configuration issue. It is hard > to > tell without debug logs for nodepool or zuul, but please double check your > node > is setup properly. > > I have to run now. >

My only doubt is why this does not show up as "NOT_REGISTERED" in Zuul as it did before.

On Mon, Feb 12, 2018 at 5:08 PM, Tristan Cacqueray <tdecacqu(a)redhat.com&gt; wrote: > On February 12, 2018 8:59 am, Javier Pena wrote: > [snip] > >> My only doubt is why this does not show up as "NOT_REGISTERED" in Zuul as >> it did before. >> > > This is because we changed check_job_registration to False in zuul.conf > to make Zuul always queue new job. We did that because during previous > nodepool outage, zuul would fail with NOT_REGISTERED when no slaves > where online (zuul(v2) only register job for available labels). > > Perhaps we could add a check for missing jjb job in zuul.yaml, or revert > that check_job_registration back to true. I was previsously confused by NOT_REGISTERED on wrong configuration too, but it's still better than having the job stuck. That said, I didn't know howto debug this error, someone with experience told me howto fix based on guesswork. So do I understand it correctly that Zuul has no good way of communicating job configuration errors?

Isn't this possibly an issue to be solved in upstream Zuul? Something like returning CONFIG_ERROR that is clickable and leads to a log of config errors.