[rdo-dev] [outage] RDO Container Registry

[Wesley Hayutin]

Thanks David!

On Mon, Dec 4, 2017 at 11:25 PM, David Moreau Simard <dms at redhat.com> wrote:

> Hi,
>
> While we monitor the disk space utilization of
> trunk.registry.rdoproject.org, the alerts for it were silenced due to an
> ongoing false positive.
> Last November 28th, we pruned the metadata of ~5000 image tags [1] >7 days
> old after which we were supposed to prune the (now orphaned) blobs.
>
> The blobs were not deleted and this lead to the registry partition running
> out of disk space.
> Container pushes from approximately the last 48 hours may have failed due
> as a result of the issue.
>
> We're currently pruning the orphaned blobs and pushes should work once
> enough free space is available.
> We'll address the false positive on the monitoring alert ASAP and we hope
> to automate the pruning process in the future to prevent this from
> re-ocurring.
>
> Let me know if you have any questions,
>
> ​[1]: https://paste.fedoraproject.org/paste/Js2rOwOFWdUqfRrBcZaXHw​
>
> David Moreau Simard
> Senior Software Engineer | OpenStack RDO
>
> dmsimard = [irc, github, twitter]
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20171205/daa610b6/attachment.html>