[rdo-list] Problems with the private RDO container registry

[David Moreau Simard]

Hi,

TL;DR: All images have been mistakenly deleted by a script [0], sorry
about that. Images and tags will be repopulated on the next periodic
job.

As you might already know, the private RDO container registry we use
for CI purposes is an "OpenShift Standalone Registry" [1].
This implementation replaced the (now) deprecated Atomic Registry [2][3].

In a nutshell, it is an OpenShift deployment without all the bells and
whistles of OpenShift: apps.
It only contains the internal OpenShift registry as well as the
registry console web interface and this registry is exposed for
consumption.

OpenShift Standalone Registry was a bit of an uncharted territory, not
only for us but I feel for upstream as well.
This has been a learning experience but we have contributed several
patches and upstream has been very receptive to our feedback which
resulted in more patches, making the use case better supported in
general.

For the sake of keeping this short, the latest issue we had been
looking at was the pruning of older images in order to keep the disk
usage (and RAM[4]) under control.
The good news is that in OpenShift trunk, 3.7, they managed to land
part of the patches [5][6] required to make the whole process easier
to manage.

However, the bad news is that we're currently running OpenShift 3.5,
the latest version being 3.6.
Our last attempt a pruning images deleted legitimate image blobs which
resulted in an inconsistent state.
I've forcefully deleted all the images completely in order to start
from a clean slate.

So, where does that leave us ?
This is a bit frustrating but not in vain, we've made progress.

In the short term, we'll increase the disk space allocation for the
registry in order to allow for more retention.
I also want to test a clean installation of OpenShift 3.7 (ahead of
release) with our playbooks [7] in order to confirm that our ongoing
issues have been resolved.
After confirming the issues have been resolved, we'll move forward to use 3.7.

For what it's worth, this work might end up paying off in OpenStack
upstream infrastructure as well.
At the last OpenStack PTG in Denver, we agreed that a
infrastructure-managed image registry would be necessary -- not just
for TripleO but for other projects such as Kolla.
Between docker-registry/docker-distribution (which leave much to be
desired), quay.io (which is not free and open source) and OpenShift
standalone registry, it's entirely possible that we end up using
OpenShift upstream.

Thanks, and sorry about that.

[0]: https://bugzilla.redhat.com/show_bug.cgi?id=1408676
[1]: https://docs.openshift.com/container-platform/latest/install_config/install/stand_alone_registry.html
[2]: http://www.projectatomic.io/registry/
[3]: https://www.projectatomic.io/blog/2017/05/oo-standalone-registry/
[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1489501
[5]: https://github.com/openshift/origin/commit/7783364a6f1fd34cf4833c0be506b8ee90d62691
[6]: https://github.com/openshift/openshift-docs/commit/be0ee4f8a8b7f66fccf77ebbc34c26ba223d794c
[7]: https://github.com/rdo-infra/rdo-container-registry

David Moreau Simard
Senior Software Engineer | OpenStack RDO

dmsimard = [irc, github, twitter]