Hi,

We incurred a loss of data on the RDO registry today around 4PM UTC during the SSL certificate renewal for the different domains and services used for the registry.

Around 10PM UTC, everything should be back to normal.

Instead of re-building the lost images in-place, we expedited the upgrade of our registry to OpenShift 3.7.

Our implementation of this brand new release contains, amongst other things, a fix to prevent this issue from occurring again.

We built and pushed the following images to recover from what we had:

- master/tripleo-ci-testing: https://trunk.rdoproject.org/centos7-master/df/24/df24d6e51a178263940fa929d52276f3697f75bd_3b7cc045

- master/current-tripleo-rdo: https://trunk.rdoproject.org/centos7-master/3b/85/3b85715c6488a692fcdf4e4c6a589d77c60eb5a6_a3792c8d

- pike/tripleo-ci-testing: https://trunk.rdoproject.org/centos7-pike/16/ee/16ee91b0fdf092037bd8cd788ffcd6476367ff08_940a986d

- pike/current-tripleo-rdo: https://trunk.rdoproject.org/centos7-pike/ca/08/ca0834685d9bf226816e49b195251bd553fe5d9f_c74c332d

The last images builds and tags were pushed around 10PM UTC.

Note that we disabled Jenkins in order to prevent the periodic promotion jobs from interfering or failing outright.

For this reason, if you see errors in your patches with 'NOT_REGISTERED', feel free to do a recheck.

Thanks,

David Moreau Simard
Senior Software Engineer | OpenStack RDO

dmsimard = [irc, github, twitter]