<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">While we monitor the disk space utilization of <a href="http://trunk.registry.rdoproject.org">trunk.registry.rdoproject.org</a>, the alerts for it were silenced due to an ongoing false positive.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Last November 28th, we pruned the metadata of ~5000 image tags [1] >7 days old after which we were supposed to prune the (now orphaned) blobs.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The blobs were not deleted and this lead to the registry partition running out of disk space.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Container pushes from approximately the last 48 hours may have failed due as a result of the issue.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">We're currently pruning the orphaned blobs and pushes should work once enough free space is available.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">We'll address the false positive on the monitoring alert ASAP and we hope to automate the pruning process in the future to prevent this from re-ocurring.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Let me know if you have any questions,<br clear="all"></div><div><div class="gmail_signature"><br></div><div class="gmail_signature"><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">​[1]: <a href="https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw">https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw</a>​</div></div><div class="gmail_signature"><br>David Moreau Simard<br>Senior Software Engineer | OpenStack RDO<br><br>dmsimard = [irc, github, twitter]</div></div>
</div>