[rdo-dev] [rhos-dev] [infra][outage] Nodepool outage on review.rdoproject.org, December 2

[Javier Pena]

----- Original Message -----
> On Sat, Dec 02, 2017 at 01:57:08PM +0100, Alfredo Moralejo Alonso wrote:
> > On Sat, Dec 2, 2017 at 11:56 AM, Javier Pena <jpena at redhat.com> wrote:
> > 
> > > Hi all,
> > >
> > > We had another nodepool outage this morning. Around 9:00 UTC, amoralej
> > > noticed that no new jobs were being processed. He restarted nodepool, and
> > > I
> > > helped him later with some stale node cleanup. Nodepool started creating
> > > VMs successfully around 10:00 UTC.
> > >
> > > On a first look at the logs, we see no new messages after 7:30 (not even
> > > DEBUG logs), but I was unable to run more troubleshooting steps because
> > > the
> > > service was already restarted.
> > >
> > >
> > In case it helps, i could run successfully both "nodepool list" and
> > "nodepool delete <id> --now" (for a couple of instances in delete status)
> > before restarting nodepool. However nothing appeared in logs and no
> > instances were created for jobs in queue so i restarted nodepool-launcher
> > (my understanding was that it fixed similar situations in the past) before
> > Javier started working on it.
> > 
> > 
> > > We will go through the logs on Monday to investigate what happened during
> > > the outage.
> > >
> > > Regards,
> > > Javier
> > >
> Please reach out to me the next time you restart it, something is seriously
> wrong is we have to keep restarting nodepool every few days. At this rate, I
> would even leave nodepool-launcher is the bad state until we inspect it.
> 

Hi Paul,

This happened on a Saturday morning, so I did not expect you to be around. Had it been on a working day, of course I would have pinged you.

Leaving nodepool-launcher in bad state for the whole weekend would mean that no jobs would be running at all, including promotion jobs. This is usually not acceptable, but I'll do it if everyone agrees it is ok to wait until Monday.

Regards,
Javier

> Thanks,
> PB
> 
>