Re: [rdo-dev] [rhos-dev] [infra][outage] Nodepool outage on review.rdoproject.org, December 2
Hello,
it seems like the recurrent deadlock of Nodepool has been fixed with the
upgrade to softwarefactory-2.7 along with centos-7.4. In particular,
this upgraded the python-paramiko and python-requests packages.
Regards,
-Tristan
On December 2, 2017 10:56 am, Javier Pena wrote:
Hi all,
We had another nodepool outage this morning. Around 9:00 UTC, amoralej noticed that no
new jobs were being processed. He restarted nodepool, and I helped him later with some
stale node cleanup. Nodepool started creating VMs successfully around 10:00 UTC.
On a first look at the logs, we see no new messages after 7:30 (not even DEBUG logs), but
I was unable to run more troubleshooting steps because the service was already restarted.
We will go through the logs on Monday to investigate what happened during the outage.
Regards,
Javier
Attachments:
- attachment.sig (application/pgp-signature — 488 bytes)