<div dir="ltr">Thanks David!</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 4, 2017 at 11:25 PM, David Moreau Simard <span dir="ltr"><<a href="mailto:dms@redhat.com" target="_blank">dms@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">While we monitor the disk space utilization of <a href="http://trunk.registry.rdoproject.org" target="_blank">trunk.registry.rdoproject.org</a>, the alerts for it were silenced due to an ongoing false positive.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Last November 28th, we pruned the metadata of ~5000 image tags [1] >7 days old after which we were supposed to prune the (now orphaned) blobs.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The blobs were not deleted and this lead to the registry partition running out of disk space.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Container pushes from approximately the last 48 hours may have failed due as a result of the issue.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">We're currently pruning the orphaned blobs and pushes should work once enough free space is available.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">We'll address the false positive on the monitoring alert ASAP and we hope to automate the pruning process in the future to prevent this from re-ocurring.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Let me know if you have any questions,<br clear="all"></div><div><div class="m_-6780354228288870603gmail_signature"><br></div><div class="m_-6780354228288870603gmail_signature"><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">​[1]: <a href="https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw" target="_blank">https://paste.fedoraproject.<wbr>org/paste/<wbr>Js2rOwOFWdUqfRrBcZaXHw</a>​</div></div><div class="m_-6780354228288870603gmail_signature"><br>David Moreau Simard<br>Senior Software Engineer | OpenStack RDO<br><br>dmsimard = [irc, github, twitter]</div></div>
</div>
</blockquote></div><br></div>