Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org
by Javier Pena
----- Original Message -----
> On Fri, Mar 22, 2019 at 8:35 PM Javier Pena < jpena(a)redhat.com > wrote:
> > ----- Original Message -----
>
> > > I've been working with mjturek and baha on this a bit. I've responded
> > > inline
>
> > > below, but also want to clarify on the desired workflow.
>
> > >
>
> > > TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly
>
> > > integrated and uploaded. This can be done with docker manifest list
> > > images.
>
> > >
>
> > > The following link explains in greater detail:
>
> > > https://docs.docker.com/registry/spec/manifest-v2-2/
>
> > >
>
> > > The process boils down to the following steps:
>
> > >
>
> > > 1) Upload an image of the first architecture (ex: image1:x86_64_01012019)
>
> > > 2) Upload an image of the second architecture (ex:
> > > image1:ppc64le_01012019)
>
> > > 3) Upload manifest list image of the image (ex: image1:01012019)
>
> > >
>
> > This is one of the details where I had my doubts. Currently, the images
> > uploaded to the registry use the following naming convention:
>
> > tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40
>
> > Where:
>
> > - tripleomaster is associated to the release (we have tripleomaster,
> > tripleostein, tripleorocky...)
>
> > - centos is associated to the OS (we have centos and fedora)
>
> > - 42a882962919b867c91a182b83acca6d8004096e_ee467b40 refers to the
> > repository
> > in trunk.rdoproject.org used to build the image (commit hash and short
> > distro hash)
>
> > If we want to go multi-arch, we need to change that tag to include the
> > architecture, is this correct? Otherwise, we could have conflicts between
> > the x86_64 and ppc64le pipelines trying to upload the same image.
>
> Yup. The idea is that the enpoint URL
> (tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40)
> is a container manifest. Where we include the arch would be with an
> additional tag:
> tripleomaster/centos-binary-neutron-l3-agent:42a882962919b867c91a182b83acca6d8004096e_ee467b40_$arch
> but nothing else should change and *explicitly* do not want different orgs
> per architecture. So the publish pipeline would look like:
> * Each architecture builds and publishes all the containers per branch and OS
> [1] all the containers and publishes a container image/layer to:
> 'tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s_%(arch)s'
> * Then checks to see if the manifest exists. manifest =
> 'tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s' if
> exists(manifest):
> add_to_manifest(arch_layer=tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s_%(arch)s')
> else:
> create_manifest(arch_layer=tripleo%(branch)s/%(os)s-%(build_type)s-%(container)s:%(repo)s_%(arch)s')
I have been running some tests to check how this could work, and I've found an issue. It looks like the OpenShift Registry (what we use for our RDO Registry) does not properly support manifest lists, see [2]. It actually failed for me when I tried it, while a plain Docker registry worked (using manifest-tool).
Would the manifest upload be required in the RDO Registry (which is used as an intermediate step), or just in DockerHub (which is used for actual content delivery)? If it's the second case, we're still fine.
Regards,
Javier
[2] - https://trello.com/c/4EcAIJrd/1303-5-quayregistry-add-support-for-manifes...
> This shouldn't break existing consumers as docker and podman both do the
> correct thing when encountering a manifest. and does mean that multi-arch
> consumers can use the same URL scheme. This is how downstream currently
> works
> It's possible and possibly even desirable, due to resource constraints, for
> the ppc64le build to be triggered only when updating current-passed-ci.
> That's exactly what we discussed in Dublin.
> Tony.
> [1] for ppc64le we're starting with centos and master but over time this
> would need to grow out from master to include stein, u etc etc We haven't
> looked at Fedora due to using centos CI but if Fedora is going to stick
> around we can work on that too.
5 years, 8 months
[Meeting] RDO meeting (2019-03-27) minutes
by Javier Pena
==============================
#rdo: RDO meeting - 2019-03-27
==============================
Meeting started by jpena at 15:00:24 UTC. The full logs are available
at
http://eavesdrop.openstack.org/meetings/rdo_meeting___2019_03_27/2019/rdo...
.
Meeting summary
---------------
* roll call (jpena, 15:00:31)
* ppc64le containers build update (jpena, 15:04:11)
* Ceph Nautilus update (jpena, 15:05:10)
* LINK:
https://review.openstack.org/#/c/618320/1/docker/services/ceph-ansible/ce...
(fultonj, 15:27:00)
* ACTION: fmount gets new envs and continues debug of ceph issue. we
pull in ovn team if necessary (fultonj, 15:32:45)
* Decisions on ppc64le arch enablement for containers (jpena, 15:33:33)
* TripleO meeting is 14:00 UTC Tuesday in #tripleo (mjturek,
15:59:33)
* ACTION: mjturek, Vorrtex and the tripleo ci team to agree on
implementation, jpena/amoralej to attend the tripleo-ci community
sync next week (jpena, 15:59:40)
* chair for the next meeting (jpena, 16:01:39)
* ACTION: ykarel to chair next meeting (jpena, 16:02:21)
* open floor (jpena, 16:02:24)
Meeting ended at 16:09:17 UTC.
Action items, by person
-----------------------
* amoralej
* mjturek, Vorrtex and the tripleo ci team to agree on implementation,
jpena/amoralej to attend the tripleo-ci community sync next week
* fmount
* fmount gets new envs and continues debug of ceph issue. we pull in
ovn team if necessary
* jpena
* mjturek, Vorrtex and the tripleo ci team to agree on implementation,
jpena/amoralej to attend the tripleo-ci community sync next week
* mjturek
* mjturek, Vorrtex and the tripleo ci team to agree on implementation,
jpena/amoralej to attend the tripleo-ci community sync next week
* Vorrtex
* mjturek, Vorrtex and the tripleo ci team to agree on implementation,
jpena/amoralej to attend the tripleo-ci community sync next week
* ykarel
* ykarel to chair next meeting
People present (lines said)
---------------------------
* jpena (53)
* amoralej (51)
* fultonj (43)
* fmount (30)
* ykarel (22)
* mjturek (21)
* Vorrtex (19)
* Duck (18)
* openstack (10)
* baha (6)
* rdogerrit (2)
* PagliaccisCloud (1)
* egonzalez (1)
Generated by `MeetBot`_ 0.1.4
5 years, 9 months
[infra] Planned maintenance in the RDO registry, 26 March at 9:00 UTC
by Javier Pena
Hi all,
We need to run a planned maintenance in registry.rdoproject.org on March 26th at 9:00 UTC.
This maintenance will extend the disk currently being used to host the registry images, to cope with current and future requirements. Unfortunately, it cannot be performed online, so a short downtime is required (expected < 1 hour). During the downtime, any operation accessing the registry (including periodic jobs) will fail.
If you have any doubt or concern, please do not hesitate to contact me.
Regards,
Javier
5 years, 9 months
Re: [rdo-dev] [infra][tripleoci] ppc64le container images in registry.rdoproject.org
by Trevor Vardeman
I've been working with mjturek and baha on this a bit. I've responded inline below, but also want to clarify on the desired workflow.
TL;DR: The desired workflow is to have ppc64le and x86_64 seamlessly integrated and uploaded. This can be done with docker manifest list images.
The following link explains in greater detail: https://docs.docker.com/registry/spec/manifest-v2-2/
The process boils down to the following steps:
1) Upload an image of the first architecture (ex: image1:x86_64_01012019)
2) Upload an image of the second architecture (ex: image1:ppc64le_01012019)
3) Upload manifest list image of the image (ex: image1:01012019)
Step 3 is essentially just pushing a JSON body that has descriptors and references to the other two images, such that when someone does a pull request of the manifest list image, it will gather the appropriate architecture for that image based on the host's architecture.
-Trevor
PS. If I've missed something important with the overall concerns here I apologize, but thought it necessary to spell out the goal as I understand it.
> On Mar 21, 2019, at 12:28 PM, Javier Pena <jpena(a)redhat.com> wrote:
>
>
> ----- Original Message -----
>> Hi all,
>>
>> Over the last few weeks, mjturek and baha have been busy working on a set of
>> periodic jobs to build TripleO images for the ppc64le arch [1].
>>
>> The current missing step is publishing those images, and they are proposing
>> to push those to the RDO Registry instance at registry.rdoproject.org, just
>> like we do with our TripleO images. I have tried to understand the
>> requirements, and would like to get input on the following topics:
>>
>> - Which namespace would these images use? Based on some logs [2] it looks
>> like they use tripleomaster-ppc64le, will they also push the images to that
>> namespace?
I have no experience in namespaces inside of a registry or how that differentiates images from one another, but the images should be pushed (in my opinion) to the same location in which the x86 images reside.
>> - Could this create any conflicts with our current promotion pipeline?
This should not cause conflicts in current promotion pipeline, as the process should be an extension to existing functionality.
>> - Is registry.rdo the right place for those images? I'm not familiar with the
>> next steps for ppc64le images after that (will it then go through a
>> promotion pipeline?), so that might affect.
If the x86 images exist in registry.rdo, then the ppc64le (and any other architecture image) should exist there as well. I can't think of a reason to differentiate between architectures when the desired result is parity and support of more architectures.
>>
>> If we decide to upload the images to images.rdo, we'll need to do the
>
> Correction: this should read "registry.rdo"
>
>> following:
>>
>> - Create the tripleomaster-ppc64le namespace in registry.rdo, following a
>> similar pattern to [3].
>> - Schedule a short registry downtime to increase its disk space, since it is
>> currently near its limit.
This is definitely necessary, given the capacity requirement will double, give or take, to support the additional architecture.
>> - Update the job at ci.centos to include the REGISTRY_PASSWORD environment
>> variable with the right token (see [4]). This is missing today, and causing
>> the job failure.
>>
>> Once we get input from all interested parties, we will decide on the next
>> steps.
>>
>> Thanks,
>> Javier
>>
>>
>> [1] -
>> https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/
>> [2] -
>> https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-mast...
>> [3] - https://review.rdoproject.org/r/19063
>> [4] -
>> https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/pla...
>> _______________________________________________
>> dev mailing list
>> dev(a)lists.rdoproject.org
>> http://lists.rdoproject.org/mailman/listinfo/dev
>>
>> To unsubscribe: dev-unsubscribe(a)lists.rdoproject.org
>>
> _______________________________________________
> dev mailing list
> dev(a)lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
>
> To unsubscribe: dev-unsubscribe(a)lists.rdoproject.org
5 years, 9 months
[infra][tripleoci] ppc64le container images in registry.rdoproject.org
by Javier Pena
Hi all,
Over the last few weeks, mjturek and baha have been busy working on a set of periodic jobs to build TripleO images for the ppc64le arch [1].
The current missing step is publishing those images, and they are proposing to push those to the RDO Registry instance at registry.rdoproject.org, just like we do with our TripleO images. I have tried to understand the requirements, and would like to get input on the following topics:
- Which namespace would these images use? Based on some logs [2] it looks like they use tripleomaster-ppc64le, will they also push the images to that namespace?
- Could this create any conflicts with our current promotion pipeline?
- Is registry.rdo the right place for those images? I'm not familiar with the next steps for ppc64le images after that (will it then go through a promotion pipeline?), so that might affect.
If we decide to upload the images to images.rdo, we'll need to do the following:
- Create the tripleomaster-ppc64le namespace in registry.rdo, following a similar pattern to [3].
- Schedule a short registry downtime to increase its disk space, since it is currently near its limit.
- Update the job at ci.centos to include the REGISTRY_PASSWORD environment variable with the right token (see [4]). This is missing today, and causing the job failure.
Once we get input from all interested parties, we will decide on the next steps.
Thanks,
Javier
[1] - https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/
[2] - https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-mast...
[3] - https://review.rdoproject.org/r/19063
[4] - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/pla...
5 years, 9 months
[Meeting] RDO meeting (2019-03-20) minutes
by Alfredo Moralejo Alonso
==============================
#rdo: RDO meeting - 2019-03-20
==============================
Meeting started by amoralej at 15:01:18 UTC. The full logs are
available athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2019_03_20/2019/r...
.
Meeting summary
---------------
* LINK: https://etherpad.openstack.org/p/RDO-Meeting (mjturek,
15:04:09)
* LINK: https://etherpad.openstack.org/p/RDO-Meeting (amoralej,
15:04:12)
* default maintainers are gone in rdoinfo: deps -
https://review.rdoproject.org/r/19389 (amoralej, 15:04:59)
* dependencies without explicit maintainers have been reset to nobody
(amoralej, 15:05:39)
* the processes to manage deps are in
https://www.rdoproject.org/documentation/requirements (amoralej,
15:08:40)
* finally drop *-patches repos (openstack/$PROJECT mirrors) in
review.rdo (amoralej, 15:11:18)
* ACTION: jpena to check how to remove repos using resources in
SoftwareFactory (amoralej, 15:13:39)
* RDO phase1 tripleo jobs running in ci.centos will be moved to some
other system, weshay is looking into it (amoralej, 15:14:57)
* ACTION: weshay to create an epic in taiga to move oooq jobs out of
ci.centos.org (amoralej, 15:27:29)
* ppc64le container builds update (amoralej, 15:28:20)
* logging uploads are working
https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-mast...
(amoralej, 15:28:34)
* LINK: https://review.openstack.org/#/c/644389/ (amoralej, 15:30:37)
* Status of Stein preparation (amoralej, 15:36:31)
* ACTION: maintainers should send required updates to package distgits
for stein asap as new branches stein-rdo are being cut (amoralej,
15:38:14)
* non-OpenStack puppe modules have been frozen for Stein (amoralej,
15:39:09)
* https://trello.com/c/nkadXFWF/702-stein-release-preparation
(amoralej, 15:41:34)
* Nautilus update (amoralej, 15:49:06)
* Ceph Nautilus was released yesterday:
https://ceph.com/releases/v14-2-0-nautilus-released (amoralej,
15:49:27)
* want to get Nautilus into Stein so we can use Ansible 2.7 as per
https://review.rdoproject.org/r/#/c/18721 (amoralej, 15:49:53)
* stein GA should be released with nautilus ceph (amoralej, 15:50:21)
* open floor (amoralej, 16:00:10)
* ACTION: jpena to chair next meeting (amoralej, 16:02:02)
5 years, 9 months
