[rdo-dev] [outage] RDO Container Registry

Hi, While we monitor the disk space utilization of trunk.registry.rdoproject.org, the alerts for it were silenced due to an ongoing false positive. Last November 28th, we pruned the metadata of ~5000 image tags [1] >7 days old after which we were supposed to prune the (now orphaned) blobs. The blobs were not deleted and this lead to the registry partition running out of disk space. Container pushes from approximately the last 48 hours may have failed due as a result of the issue. We're currently pruning the orphaned blobs and pushes should work once enough free space is available. We'll address the false positive on the monitoring alert ASAP and we hope to automate the pruning process in the future to prevent this from re-ocurring. Let me know if you have any questions, ​[1]: https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw​ David Moreau Simard Senior Software Engineer | OpenStack RDO dmsimard = [irc, github, twitter]