[rdo-dev] [infra][outage] Nodepool outage on review.rdoproject.org, December 2

[Javier Pena]

Hi all,

We had another nodepool outage this morning. Around 9:00 UTC, amoralej noticed that no new jobs were being processed. He restarted nodepool, and I helped him later with some stale node cleanup. Nodepool started creating VMs successfully around 10:00 UTC.

On a first look at the logs, we see no new messages after 7:30 (not even DEBUG logs), but I was unable to run more troubleshooting steps because the service was already restarted.

We will go through the logs on Monday to investigate what happened during the outage.

Regards,
Javier