[rdo-list] Problems with the private RDO container registry

[Wesley Hayutin]

On Tue, Oct 10, 2017 at 10:42 PM, David Moreau Simard <dms at redhat.com>
wrote:

> Hi,
>
> TL;DR: All images have been mistakenly deleted by a script [0], sorry
> about that. Images and tags will be repopulated on the next periodic
> job.
>
> As you might already know, the private RDO container registry we use
> for CI purposes is an "OpenShift Standalone Registry" [1].
> This implementation replaced the (now) deprecated Atomic Registry [2][3].
>
> In a nutshell, it is an OpenShift deployment without all the bells and
> whistles of OpenShift: apps.
> It only contains the internal OpenShift registry as well as the
> registry console web interface and this registry is exposed for
> consumption.
>
> OpenShift Standalone Registry was a bit of an uncharted territory, not
> only for us but I feel for upstream as well.
> This has been a learning experience but we have contributed several
> patches and upstream has been very receptive to our feedback which
> resulted in more patches, making the use case better supported in
> general.
>
> For the sake of keeping this short, the latest issue we had been
> looking at was the pruning of older images in order to keep the disk
> usage (and RAM[4]) under control.
> The good news is that in OpenShift trunk, 3.7, they managed to land
> part of the patches [5][6] required to make the whole process easier
> to manage.
>
> However, the bad news is that we're currently running OpenShift 3.5,
> the latest version being 3.6.
> Our last attempt a pruning images deleted legitimate image blobs which
> resulted in an inconsistent state.
> I've forcefully deleted all the images completely in order to start
> from a clean slate.
>
> So, where does that leave us ?
> This is a bit frustrating but not in vain, we've made progress.
>
> In the short term, we'll increase the disk space allocation for the
> registry in order to allow for more retention.
> I also want to test a clean installation of OpenShift 3.7 (ahead of
> release) with our playbooks [7] in order to confirm that our ongoing
> issues have been resolved.
> After confirming the issues have been resolved, we'll move forward to use
> 3.7.
>
> For what it's worth, this work might end up paying off in OpenStack
> upstream infrastructure as well.
> At the last OpenStack PTG in Denver, we agreed that a
> infrastructure-managed image registry would be necessary -- not just
> for TripleO but for other projects such as Kolla.
> Between docker-registry/docker-distribution (which leave much to be
> desired), quay.io (which is not free and open source) and OpenShift
> standalone registry, it's entirely possible that we end up using
> OpenShift upstream.
>
> Thanks, and sorry about that.
>
> [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1408676
> [1]: https://docs.openshift.com/container-platform/latest/
> install_config/install/stand_alone_registry.html
> [2]: http://www.projectatomic.io/registry/
> [3]: https://www.projectatomic.io/blog/2017/05/oo-standalone-registry/
> [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1489501
> [5]: https://github.com/openshift/origin/commit/
> 7783364a6f1fd34cf4833c0be506b8ee90d62691
> [6]: https://github.com/openshift/openshift-docs/commit/
> be0ee4f8a8b7f66fccf77ebbc34c26ba223d794c
> [7]: https://github.com/rdo-infra/rdo-container-registry
>
> David Moreau Simard
> Senior Software Engineer | OpenStack RDO


>
dmsimard = [irc, github, twitter]
>

Thanks for going above the call the duty there David.  It does appear that
we're blazing a path for the upstream.  Well done.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20171010/3ca267db/attachment.html>