[rdo-dev] [rhos-dev] [infra][outage] Nodepool outage on review.rdoproject.org, December 2

[Tristan Cacqueray]

Hello,

it seems like the recurrent deadlock of Nodepool has been fixed with the
upgrade to softwarefactory-2.7 along with centos-7.4. In particular,
this upgraded the python-paramiko and python-requests packages.

Regards,
-Tristan

On December 2, 2017 10:56 am, Javier Pena wrote:
> Hi all,
> 
> We had another nodepool outage this morning. Around 9:00 UTC, amoralej noticed that no new jobs were being processed. He restarted nodepool, and I helped him later with some stale node cleanup. Nodepool started creating VMs successfully around 10:00 UTC.
> 
> On a first look at the logs, we see no new messages after 7:30 (not even DEBUG logs), but I was unable to run more troubleshooting steps because the service was already restarted.
> 
> We will go through the logs on Monday to investigate what happened during the outage.
> 
> Regards,
> Javier
> 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20180102/1c931965/attachment.sig>