Re: [rdo-list] Problems with the private RDO container registry

Hi, TL;DR: All images have been mistakenly deleted by a script [0], sorry about that. Images and tags will be repopulated on the next periodic job. As you might already know, the private RDO container registry we use for CI purposes is an "OpenShift Standalone Registry" [1]. This implementation replaced the (now) deprecated Atomic Registry [2][3]. In a nutshell, it is an OpenShift deployment without all the bells and whistles of OpenShift: apps. It only contains the internal OpenShift registry as well as the registry console web interface and this registry is exposed for consumption. OpenShift Standalone Registry was a bit of an uncharted territory, not only for us but I feel for upstream as well. This has been a learning experience but we have contributed several patches and upstream has been very receptive to our feedback which resulted in more patches, making the use case better supported in general. For the sake of keeping this short, the latest issue we had been looking at was the pruning of older images in order to keep the disk usage (and RAM[4]) under control. The good news is that in OpenShift trunk, 3.7, they managed to land part of the patches [5][6] required to make the whole process easier to manage. However, the bad news is that we're currently running OpenShift 3.5, the latest version being 3.6. Our last attempt a pruning images deleted legitimate image blobs which resulted in an inconsistent state. I've forcefully deleted all the images completely in order to start from a clean slate. So, where does that leave us ? This is a bit frustrating but not in vain, we've made progress. In the short term, we'll increase the disk space allocation for the registry in order to allow for more retention. I also want to test a clean installation of OpenShift 3.7 (ahead of release) with our playbooks [7] in order to confirm that our ongoing issues have been resolved. After confirming the issues have been resolved, we'll move forward to use 3.7. For what it's worth, this work might end up paying off in OpenStack upstream infrastructure as well. At the last OpenStack PTG in Denver, we agreed that a infrastructure-managed image registry would be necessary -- not just for TripleO but for other projects such as Kolla. Between docker-registry/docker-distribution (which leave much to be desired), quay.io (which is not free and open source) and OpenShift standalone registry, it's entirely possible that we end up using OpenShift upstream. Thanks, and sorry about that. [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1408676 [1]: https://docs.openshift.com/container-platform/latest/ install_config/install/stand_alone_registry.html [2]: http://www.projectatomic.io/registry/ [3]: https://www.projectatomic.io/blog/2017/05/oo-standalone-registry/ [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1489501 [5]: https://github.com/openshift/origin/commit/ 7783364a6f1fd34cf4833c0be506b8ee90d62691 [6]: https://github.com/openshift/openshift-docs/commit/ be0ee4f8a8b7f66fccf77ebbc34c26ba223d794c [7]: https://github.com/rdo-infra/rdo-container-registry David Moreau Simard Senior Software Engineer | OpenStack RDO