[rdo-dev] [outage] RDO Container Registry

[David Moreau Simard]

Hi,

While we monitor the disk space utilization of trunk.registry.rdoproject.org,
the alerts for it were silenced due to an ongoing false positive.
Last November 28th, we pruned the metadata of ~5000 image tags [1] >7 days
old after which we were supposed to prune the (now orphaned) blobs.

The blobs were not deleted and this lead to the registry partition running
out of disk space.
Container pushes from approximately the last 48 hours may have failed due
as a result of the issue.

We're currently pruning the orphaned blobs and pushes should work once
enough free space is available.
We'll address the false positive on the monitoring alert ASAP and we hope
to automate the pruning process in the future to prevent this from
re-ocurring.

Let me know if you have any questions,

​[1]: https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw​

David Moreau Simard
Senior Software Engineer | OpenStack RDO

dmsimard = [irc, github, twitter]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20171204/5ebe448d/attachment.html>