Re: [rdo-list] Baremetal Tripleo stable version?

Hi Everyone, I just wanted to say I have been following this thread quite closely and can sympathize with some of the pain people are going through to get tripleO to work. Currently it's quite difficult and a bit opaque on how to actually utilise the stable mitaka repos in order to build a functional undercloud and overcloud environment. First I wanted to share the steps I have undergone in order to get a functional overcloud working with RDO Mitaka utilising the RDO stable release built by CentOS, and then I'll talk about some specific steps I think need to be undertaken by the RDO/TripleO team in order to provide a better experience in the future. To get a functional overcloud using RDO Mitaka, you need to do the following 1) Install EPEL on your undercloud 2) Install https://www.rdoproject.org/repos/rdo-release.rpm on your undercloud 3) Following the normal steps to install your undercloud (modifying undercloud.conf, and running openstack undercloud install 4) You will now need to manually patch ironic on the undercloud in order to make sure repeated introspection works. This might not be needed if you don't do any introspection, but I find more often than not you end up having to do it, so it's worthwhile. The bug you need to patch is [1] and I typically run the following commands to apply the patch # sudo su - $ cd /usr/lib/python2.7/site-packages $ curl 'https://review.openstack.org/changes/306421/revisions/abd50d8438e7d371ce24f97d8f8f67052b562007/patch?download' | base64 -d | patch -p1 $ systemctl restart openstack-ironic-inspector $ systemctl restart openstack-ironic-inspector-dnsmasq $ exit #

5) Manually patch the undercloud to build overcloud images using rhos-release rpm only (which utilises the stable Mitaka repo from CentOS, and nothing from RDO Trunk [delorean]). I do this by modifying the file /usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py At around line 467 you will see a reference to epel, I add a new line after that to include the rdo_release DIB element to the build as well. This typically makes the file look something like http://paste.openstack.org/show/508196/ (note like 468). Then I create a directory to store my images and build them specifying the mitaka version of rdo_release. I then upload these images # mkdir ~/images # cd ~/images # export RDO_RELEASE=mitaka # openstack overcloud image build --all # openstack overcloud image upload --update-existing

6) Because of the bug at [2] which affects the ironic agent ramdisk, we need to build a set of images utilising RDO Trunk for the mitaka branch (where the fix is applied), and then upload *only* the new ironic ramdisk. This is done with # mkdir ~/images-mitaka-trunk # cd ~/images-mitaka-trunk # export USE_DELOREAN_TRUNK=1 # export DELOREAN_TRUNK_REPO="http://trunk.rdoproject.org/centos7-mitaka/curr... # export DELOREAN_REPO_FILE="delorean.repo" # openstack overcloud image build --type agent-ramdisk # sudo cp ironic-python-agent.initramfs /httpboot/agent.ramdisk

7) Follow the rest of the documentation to deploy the overcloud normally Please note that obviously your mileage may vary, and this is by all means not an exclusive list of the problems. I have however used these steps to do multiple node deployments (10+ nodes) with HA over different hardware sets with different networking setups (single nic, multiple nic with bonding + vlans). With all the different repos floating around, all which change very rapidly, combined with the documentation defaults targeting developers and CI systems (not end users), it's hard to not only get a stable TripleO install up, but also communicate and discuss clearly with others what is working, what is broken, and how to compare two installations to see if they are experiencing the same issues. To this end I would like to suggest to the RDO and TripleO community that we undertake the following 1) Overhaul all the TripleO documentation so that all the steps default to utilising/deploying using RDO Stable (that is, the releases done by CBS). There should be colored boxes with alt steps for those who wish to use RDO Trunk on the stable branch, and RDO Trunk from master. This basically inverts the current pattern. I think anyone, Operator or developer, who is working through the documentation for the first time, should be given steps that maximise the chance of success, and thus the most stable release we have. Once a user has gone through the process once, they can look at the alternative steps for more aggressive releases

2) Patch python-tripleoclient so that by default, when you run "openstack overcloud image build" it builds the images utilising the rdo_release DIB element, and sets the RDO_RELEASE environment variable to be 'mitaka' or whenever the current stable release is (and we should endevour to update it with new releases). There should be no extra environment variables necessary to build images, and by default it should never touch anything RDO Trunk (delorean) related

3) For bugs like the two I have mentioned above, we need to have some sort of robust process for either backporting those patches to the builds in CBS (I understand we don't do this for various reasons), or we need some kind of tooling or solution that allows operators to apply only the fixes they need from RDO Trunk (delorean). We need to ensure that when an Operator utilises TripleO they have the greatest chance of success, and bugs such as these which severely impact the deployment process harm the adoption of TripleO and RDO.

4) We should curate and keep an up to date page on rdoproject.org that does highlight the outstanding issues related to TripleO on the RDO Stable (CBS) releases. These should have links to relevant bugzillas, clean instructions on how to work around the issue, or cleanly apply a patch to avoid the issue, and as new releases make it out, we should update the page to drop off workarounds that are no longer needed.

The goal is to push Operators/Users to working with our most stable code as much as possible, and track/curate issues around that. This way everyone should be on the same page, issues are easier to discuss and diagnose, and overall peoples experiences should be better. I'm interested in thoughts, feedback, and concerns, both from the RDO and TripleO community, and from the Operator/User community. Regards, Graeme [1] https://bugs.launchpad.net/ironic-inspector/+bug/1570447 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1322892 On 05/06/16 02:04, Pedro Sousa wrote: > Thanks Marius, > > I can confirm that it installs fine with 3 controllers + 3 computes > after recreating the stack > > Regards > > On Sat, Jun 4, 2016 at 4:14 PM, Marius Cornea <marius(a)remote-lab.net > <mailto:marius@remote-lab.net>> wrote: > > Hi Pedro, > > Scaling out controller nodes is not supported at this moment: > https://bugzilla.redhat.com/show_bug.cgi?id=1243312 > > On Sat, Jun 4, 2016 at 5:05 PM, Pedro Sousa <pgsousa(a)gmail.com > <mailto:pgsousa@gmail.com>> wrote: > > Hi, > > > > some update on scaling the cloud: > > > > 1 controller + 1 compute -> 1 controller + 3 computes OK > > > > 1 controller + 3 computes -> 3 controllers + 3 compute FAILS > > > > Problem: The new controller nodes are "stuck" in "pscd start", so > it seems > > to be a problem joining the pacemaker cluster... Did anyone had this > > problem? > > > > Regards > > > > > > > > > > > > > > On Sat, Jun 4, 2016 at 1:50 AM, Pedro Sousa <pgsousa(a)gmail.com > <mailto:pgsousa@gmail.com>> wrote: > >> > >> Hi, > >> > >> I finally managed to install a baremetal in mitaka with 1 > controller + 1 > >> compute with network isolation. Thank god :) > >> > >> All I did was: > >> > >> #yum install centos-release-openstack-mitaka > >> #sudo yum install python-tripleoclient > >> > >> without epel repos. > >> > >> Then followed instructions from Redhat Site. > >> > >> I downloaded the overcloud images from: > >> > http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/mitaka/d... > >> > >> I do have an issue that forces me to delete a json file and run > >> os-refresh-config inside my overcloud nodes other than that it > installs > >> fine. > >> > >> Now I'll test with more 2 controllers + 2 computes to have a full HA > >> deployment. > >> > >> If anyone needs help to document this I'll be happy to help. > >> > >> Regards, > >> Pedro Sousa > >> > >> > >> On Fri, Jun 3, 2016 at 8:26 PM, Ronelle Landy <rlandy(a)redhat.com > <mailto:rlandy@redhat.com>> wrote: > >>> > >>> The report says: "Fix Released" as of 2016-05-24. > >>> Are you installing on a clean system with the latest repositories? > >>> > >>> Might also want to check your version of rabbitmq: I have > >>> rabbitmq-server-3.6.2-3.el7.noarch on CentOS 7. > >>> > >>> ----- Original Message ----- > >>> > From: "Pedro Sousa" <pgsousa(a)gmail.com <mailto:pgsousa@gmail.com>> > >>> > To: "Ronelle Landy" <rlandy(a)redhat.com <mailto:rlandy@redhat.com>> > >>> > Cc: "Christopher Brown" <cbrown2(a)ocf.co.uk > <mailto:cbrown2@ocf.co.uk>>, "Ignacio Bravo" > >>> > <ibravo(a)ltgfederal.com <mailto:ibravo@ltgfederal.com>>, "rdo-list" > >>> > <rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>> > >>> > Sent: Friday, June 3, 2016 1:20:43 PM > >>> > Subject: Re: [rdo-list] Baremetal Tripleo stable version? > >>> > > >>> > Anyway to workaround this? Maybe downgrade hiera? > >>> > > >>> > On Fri, Jun 3, 2016 at 5:55 PM, Ronelle Landy > <rlandy(a)redhat.com <mailto:rlandy@redhat.com>> > >>> > wrote: > >>> > > >>> > > I am not sure exactly where you installed from, and when you > did your > >>> > > installation, but any chance, you've hit: > >>> > > https://bugs.launchpad.net/tripleo/+bug/1584892? > >>> > > There is a link bugzilla record. > >>> > > > >>> > > ----- Original Message ----- > >>> > > > From: "Pedro Sousa" <pgsousa(a)gmail.com > <mailto:pgsousa@gmail.com>> > >>> > > > To: "Ronelle Landy" <rlandy(a)redhat.com > <mailto:rlandy@redhat.com>> > >>> > > > Cc: "Christopher Brown" <cbrown2(a)ocf.co.uk > <mailto:cbrown2@ocf.co.uk>>, "Ignacio Bravo" < > >>> > > ibravo(a)ltgfederal.com <mailto:ibravo@ltgfederal.com>>, > "rdo-list" > >>> > > > <rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>> > >>> > > > Sent: Friday, June 3, 2016 12:26:58 PM > >>> > > > Subject: Re: [rdo-list] Baremetal Tripleo stable version? > >>> > > > > >>> > > > Thanks Ronelle, > >>> > > > > >>> > > > do you think this kind of errors can be related with network > >>> > > > settings? > >>> > > > > >>> > > > "Could not retrieve fact='rabbitmq_nodename', > >>> > > > resolution='<anonymous>': > >>> > > > undefined method `[]' for nil:NilClass Could not retrieve > >>> > > > fact='rabbitmq_nodename', resolution='<anonymous>': undefined > >>> > > > method `[]' > >>> > > > for nil:NilClass" > >>> > > > > >>> > > > On Fri, Jun 3, 2016 at 4:56 PM, Ronelle Landy > <rlandy(a)redhat.com <mailto:rlandy@redhat.com>> > >>> > > > wrote: > >>> > > > > >>> > > > > Hi Pedro, > >>> > > > > > >>> > > > > You could use the docs you referred to. > >>> > > > > Alternatively, if you want to use a vm for the > undercloud and > >>> > > > > baremetal > >>> > > > > machines for the overcloud, it is possible to use Tripleo > >>> > > > > Qucikstart > >>> > > with a > >>> > > > > few modifications. > >>> > > > > https://bugs.launchpad.net/tripleo-quickstart/+bug/1571028. > >>> > > > > > >>> > > > > ----- Original Message ----- > >>> > > > > > From: "Pedro Sousa" <pgsousa(a)gmail.com > <mailto:pgsousa@gmail.com>> > >>> > > > > > To: "Ronelle Landy" <rlandy(a)redhat.com > <mailto:rlandy@redhat.com>> > >>> > > > > > Cc: "Christopher Brown" <cbrown2(a)ocf.co.uk > <mailto:cbrown2@ocf.co.uk>>, "Ignacio Bravo" < > >>> > > > > ibravo(a)ltgfederal.com <mailto:ibravo@ltgfederal.com>>, > "rdo-list" > >>> > > > > > <rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>> > >>> > > > > > Sent: Friday, June 3, 2016 11:48:38 AM > >>> > > > > > Subject: Re: [rdo-list] Baremetal Tripleo stable version? > >>> > > > > > > >>> > > > > > Hi Ronelle, > >>> > > > > > > >>> > > > > > maybe I understand it wrong but I thought that Tripleo > >>> > > > > > Quickstart > >>> > > was for > >>> > > > > > deploying virtual environments? > >>> > > > > > > >>> > > > > > And for baremetal we should use > >>> > > > > > > >>> > > > > > >>> > > > >>> > > > http://docs.openstack.org/developer/tripleo-docs/installation/installatio... > >>> > > > > > ? > >>> > > > > > > >>> > > > > > Thanks > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > On Fri, Jun 3, 2016 at 4:43 PM, Ronelle Landy > >>> > > > > > <rlandy(a)redhat.com <mailto:rlandy@redhat.com>> > >>> > > wrote: > >>> > > > > > > >>> > > > > > > Hello, > >>> > > > > > > > >>> > > > > > > We have had success deploying RDO (Mitaka) on baremetal > >>> > > > > > > systems - > >>> > > using > >>> > > > > > > Tripleo Quickstart with both single-nic-vlans and > >>> > > > > > > bond-with-vlans > >>> > > > > network > >>> > > > > > > isolation configurations. > >>> > > > > > > > >>> > > > > > > Baremetal can have some complicated networking > issues but, > >>> > > > > > > from > >>> > > > > previous > >>> > > > > > > experiences, if a single-controller deployment > worked but a > >>> > > > > > > HA > >>> > > > > deployment > >>> > > > > > > did not, I would check: > >>> > > > > > > - does the HA deployment command include: -e > >>> > > > > > > > >>> > > > > > >>> > > > >>> > > > /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml > >>> > > > > > > - are there possible MTU issues? > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > ----- Original Message ----- > >>> > > > > > > > From: "Christopher Brown" <cbrown2(a)ocf.co.uk > <mailto:cbrown2@ocf.co.uk>> > >>> > > > > > > > To: pgsousa(a)gmail.com <mailto:pgsousa@gmail.com>, > ibravo(a)ltgfederal.com <mailto:ibravo@ltgfederal.com> > >>> > > > > > > > Cc: rdo-list(a)redhat.com <mailto:rdo-list@redhat.com> > >>> > > > > > > > Sent: Friday, June 3, 2016 10:29:39 AM > >>> > > > > > > > Subject: Re: [rdo-list] Baremetal Tripleo stable > version? > >>> > > > > > > > > >>> > > > > > > > Hello Ignacio, > >>> > > > > > > > > >>> > > > > > > > Thanks for your response and good to know it isn't > just me! > >>> > > > > > > > > >>> > > > > > > > I would be more than happy to provide developers with > >>> > > > > > > > access to > >>> > > our > >>> > > > > > > > bare metal environments. I'll also file some bugzilla > >>> > > > > > > > reports to > >>> > > see > >>> > > > > if > >>> > > > > > > > this generates any interest. > >>> > > > > > > > > >>> > > > > > > > Please do let me know if you make any progress - I am > >>> > > > > > > > trying to > >>> > > > > deploy > >>> > > > > > > > HA with network isolation, multiple nics and vlans. > >>> > > > > > > > > >>> > > > > > > > The RDO web page states: > >>> > > > > > > > > >>> > > > > > > > "If you want to create a production-ready cloud, > you'll > >>> > > > > > > > want to > >>> > > use > >>> > > > > the > >>> > > > > > > > TripleO quickstart guide." > >>> > > > > > > > > >>> > > > > > > > which is a contradiction in terms really. > >>> > > > > > > > > >>> > > > > > > > Cheers > >>> > > > > > > > > >>> > > > > > > > On Fri, 2016-06-03 at 14:30 +0100, Ignacio Bravo > wrote: > >>> > > > > > > > > Pedro / Christopher, > >>> > > > > > > > > > >>> > > > > > > > > Just wanted to share with you that I also had > plenty of > >>> > > > > > > > > issues > >>> > > > > > > > > deploying on bare metal HA servers, and have > paused the > >>> > > deployment > >>> > > > > > > > > using TripleO until better winds start to flow > here. I > >>> > > > > > > > > was > >>> > > able to > >>> > > > > > > > > deploy the QuickStart, but on bare metal the > history was > >>> > > different. > >>> > > > > > > > > Couldn't even deploy a two server configuration. > >>> > > > > > > > > > >>> > > > > > > > > I was thinking that it would be good to have the > >>> > > > > > > > > developers > >>> > > have > >>> > > > > > > > > access to one of our environments and go through > a full > >>> > > > > > > > > install > >>> > > > > with > >>> > > > > > > > > us to better see where things fail. We can do this > >>> > > > > > > > > handholding > >>> > > > > > > > > deployment once every week/month based on > developers time > >>> > > > > > > > > availability. That way we can get a working > install, and > >>> > > > > > > > > we can > >>> > > > > > > > > troubleshoot real life environment problems. > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > IB > >>> > > > > > > > > > >>> > > > > > > > > On Jun 3, 2016, at 6:15 AM, Pedro Sousa > >>> > > > > > > > > <pgsousa(a)gmail.com <mailto:pgsousa@gmail.com>> > >>> > > wrote: > >>> > > > > > > > > > >>> > > > > > > > > > Yes. I've used this, but I'll try again as there's > >>> > > > > > > > > > seems to > >>> > > be > >>> > > > > new > >>> > > > > > > > > > updates. > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > Stable Branch Skip all repos mentioned above, > other > >>> > > > > > > > > > than > >>> > > epel- > >>> > > > > > > > > > release which is still required. > >>> > > > > > > > > > Enable latest RDO Stable Delorean repository > for all > >>> > > > > > > > > > packages > >>> > > > > > > > > > sudo curl -o > /etc/yum.repos.d/delorean-liberty.repo > >>> > > > > https://trunk.r > >>> > > > > > > > > > > doproject.org/centos7-liberty/current/delorean.repo > <http://doproject.org/centos7-liberty/current/delorean.repo> > >>> > > > > > > > > > Enable the Delorean Deps repository > >>> > > > > > > > > > sudo curl -o > >>> > > > > > > > > > /etc/yum.repos.d/delorean-deps-liberty.repo > >>> > > > > http://tru > >>> > > > > > > > > > > nk.rdoproject.org/centos7-liberty/delorean-deps.repo > <http://nk.rdoproject.org/centos7-liberty/delorean-deps.repo> > >>> > > > > > > > > > > >>> > > > > > > > > > On Fri, Jun 3, 2016 at 11:10 AM, Christopher > Brown < > >>> > > > > cbrown2(a)ocf.co <mailto:cbrown2@ocf.co>. > >>> > > > > > > > > > uk> wrote: > >>> > > > > > > > > > > No, Liberty deployed ok for us. > >>> > > > > > > > > > > > >>> > > > > > > > > > > It suggests to me a package mismatch. Have you > >>> > > > > > > > > > > completely > >>> > > > > rebuilt > >>> > > > > > > > > > > the > >>> > > > > > > > > > > undercloud and then the images using Liberty? > >>> > > > > > > > > > > > >>> > > > > > > > > > > On Fri, 2016-06-03 at 11:04 +0100, Pedro > Sousa wrote: > >>> > > > > > > > > > > > AttributeError: 'module' object has no > attribute > >>> > > 'PortOpt' > >>> > > > > > > > > > > -- > >>> > > > > > > > > > > Regards, > >>> > > > > > > > > > > > >>> > > > > > > > > > > Christopher Brown > >>> > > > > > > > > > > OpenStack Engineer > >>> > > > > > > > > > > OCF plc > >>> > > > > > > > > > > > >>> > > > > > > > > > > Tel: +44 (0)114 257 2200 > <tel:%2B44%20%280%29114%20257%202200> > >>> > > > > > > > > > > Web: www.ocf.co.uk <http://www.ocf.co.uk> > >>> > > > > > > > > > > Blog: blog.ocf.co.uk <http://blog.ocf.co.uk> > >>> > > > > > > > > > > Twitter: @ocfplc > >>> > > > > > > > > > > > >>> > > > > > > > > > > Please note, any emails relating to an OCF > Support > >>> > > > > > > > > > > request > >>> > > must > >>> > > > > > > > > > > always > >>> > > > > > > > > > > be sent to support(a)ocf.co.uk > <mailto:support@ocf.co.uk> for a ticket number to > >>> > > > > > > > > > > be > >>> > > > > generated > >>> > > > > > > > > > > or > >>> > > > > > > > > > > existing support ticket to be updated. > Should this > >>> > > > > > > > > > > not be > >>> > > done > >>> > > > > > > > > > > then OCF > >>> > > > > > > > > > > > >>> > > > > > > > > > > cannot be held responsible for requests not > dealt > >>> > > > > > > > > > > with in a > >>> > > > > > > > > > > timely > >>> > > > > > > > > > > manner. > >>> > > > > > > > > > > > >>> > > > > > > > > > > OCF plc is a company registered in England > and Wales. > >>> > > > > Registered > >>> > > > > > > > > > > number > >>> > > > > > > > > > > > >>> > > > > > > > > > > 4132533, VAT number GB 780 6803 14. > Registered office > >>> > > address: > >>> > > > > > > > > > > OCF plc, > >>> > > > > > > > > > > > >>> > > > > > > > > > > 5 Rotunda Business Centre, Thorncliffe Park, > >>> > > > > > > > > > > Chapeltown, > >>> > > > > > > > > > > Sheffield S35 > >>> > > > > > > > > > > 2PG. > >>> > > > > > > > > > > > >>> > > > > > > > > > > If you have received this message in error, > please > >>> > > > > > > > > > > notify > >>> > > us > >>> > > > > > > > > > > immediately and remove it from your system. > >>> > > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > _______________________________________________ > >>> > > > > > > > > rdo-list mailing list > >>> > > > > > > > > rdo-list(a)redhat.com <mailto:rdo-list@redhat.com> > >>> > > > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list > >>> > > > > > > > > > >>> > > > > > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > <mailto:rdo-list-unsubscribe@redhat.com> > >>> > > > > > > > -- > >>> > > > > > > > Regards, > >>> > > > > > > > > >>> > > > > > > > Christopher Brown > >>> > > > > > > > OpenStack Engineer > >>> > > > > > > > OCF plc > >>> > > > > > > > > >>> > > > > > > > Tel: +44 (0)114 257 2200 > <tel:%2B44%20%280%29114%20257%202200> > >>> > > > > > > > Web: www.ocf.co.uk <http://www.ocf.co.uk> > >>> > > > > > > > Blog: blog.ocf.co.uk <http://blog.ocf.co.uk> > >>> > > > > > > > Twitter: @ocfplc > >>> > > > > > > > > >>> > > > > > > > Please note, any emails relating to an OCF Support > request > >>> > > > > > > > must > >>> > > > > always > >>> > > > > > > > be sent to support(a)ocf.co.uk > <mailto:support@ocf.co.uk> for a ticket number to be > >>> > > generated or > >>> > > > > > > > existing support ticket to be updated. Should this > not be > >>> > > > > > > > done > >>> > > then > >>> > > > > OCF > >>> > > > > > > > > >>> > > > > > > > cannot be held responsible for requests not dealt > with in a > >>> > > timely > >>> > > > > > > > manner. > >>> > > > > > > > > >>> > > > > > > > OCF plc is a company registered in England and Wales. > >>> > > > > > > > Registered > >>> > > > > number > >>> > > > > > > > > >>> > > > > > > > 4132533, VAT number GB 780 6803 14. Registered office > >>> > > > > > > > address: > >>> > > OCF > >>> > > > > plc, > >>> > > > > > > > > >>> > > > > > > > 5 Rotunda Business Centre, Thorncliffe Park, > Chapeltown, > >>> > > Sheffield > >>> > > > > S35 > >>> > > > > > > > 2PG. > >>> > > > > > > > > >>> > > > > > > > If you have received this message in error, please > notify > >>> > > > > > > > us > >>> > > > > > > > immediately and remove it from your system. > >>> > > > > > > > > >>> > > > > > > > _______________________________________________ > >>> > > > > > > > rdo-list mailing list > >>> > > > > > > > rdo-list(a)redhat.com <mailto:rdo-list@redhat.com> > >>> > > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list > >>> > > > > > > > > >>> > > > > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > <mailto:rdo-list-unsubscribe@redhat.com> > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >> > >> > > > > > > _______________________________________________ > > rdo-list mailing list > > rdo-list(a)redhat.com <mailto:rdo-list@redhat.com> > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > <mailto:rdo-list-unsubscribe@redhat.com> > > > > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >