Hi Everyone,
I just wanted to say I have been following this thread quite closely and
can sympathize with some of the pain people are going through to get
tripleO to work.
Currently it's quite difficult and a bit opaque on how to actually
utilise the stable mitaka repos in order to build a functional
undercloud and overcloud environment.
First I wanted to share the steps I have undergone in order to get a
functional overcloud working with RDO Mitaka utilising the RDO stable
release built by CentOS, and then I'll talk about some specific steps I
think need to be undertaken by the RDO/TripleO team in order to provide
a better experience in the future.
To get a functional overcloud using RDO Mitaka, you need to do the following
1) Install EPEL on your undercloud
2) Install
on your
undercloud
3) Following the normal steps to install your undercloud (modifying
undercloud.conf, and running openstack undercloud install
4) You will now need to manually patch ironic on the undercloud in order
to make sure repeated introspection works. This might not be needed if
you don't do any introspection, but I find more often than not you end
up having to do it, so it's worthwhile. The bug you need to patch is [1]
and I typically run the following commands to apply the patch
# sudo su -
$ cd /usr/lib/python2.7/site-packages
$ curl
'https://review.openstack.org/changes/306421/revisions/abd50d8438e7d371ce24f97d8f8f67052b562007/patch?download'
| base64 -d | patch -p1
$ systemctl restart openstack-ironic-inspector
$ systemctl restart openstack-ironic-inspector-dnsmasq
$ exit
#
5) Manually patch the undercloud to build overcloud images using
rhos-release rpm only (which utilises the stable Mitaka repo from
CentOS, and nothing from RDO Trunk [delorean]). I do this by modifying
the file
/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py
At around line 467 you will see a reference to epel, I add a new line
after that to include the rdo_release DIB element to the build as well.
This typically makes the file look something like
(note like 468). Then I create a directory to store my images and build
them specifying the mitaka version of rdo_release. I then upload these
images
# mkdir ~/images
# cd ~/images
# export RDO_RELEASE=mitaka
# openstack overcloud image build --all
# openstack overcloud image upload --update-existing
6) Because of the bug at [2] which affects the ironic agent ramdisk, we
need to build a set of images utilising RDO Trunk for the mitaka branch
(where the fix is applied), and then upload *only* the new ironic
ramdisk. This is done with
# mkdir ~/images-mitaka-trunk
# cd ~/images-mitaka-trunk
# export USE_DELOREAN_TRUNK=1
# export
# export DELOREAN_REPO_FILE="delorean.repo"
# openstack overcloud image build --type agent-ramdisk
# sudo cp ironic-python-agent.initramfs /httpboot/agent.ramdisk
7) Follow the rest of the documentation to deploy the overcloud normally
Please note that obviously your mileage may vary, and this is by all
means not an exclusive list of the problems. I have however used these
steps to do multiple node deployments (10+ nodes) with HA over different
hardware sets with different networking setups (single nic, multiple nic
with bonding + vlans).
With all the different repos floating around, all which change very
rapidly, combined with the documentation defaults targeting developers
and CI systems (not end users), it's hard to not only get a stable
TripleO install up, but also communicate and discuss clearly with others
what is working, what is broken, and how to compare two installations to
see if they are experiencing the same issues.
To this end I would like to suggest to the RDO and TripleO community
that we undertake the following
1) Overhaul all the TripleO documentation so that all the steps default
to utilising/deploying using RDO Stable (that is, the releases done by
CBS). There should be colored boxes with alt steps for those who wish to
use RDO Trunk on the stable branch, and RDO Trunk from master. This
basically inverts the current pattern. I think anyone, Operator or
developer, who is working through the documentation for the first time,
should be given steps that maximise the chance of success, and thus the
most stable release we have. Once a user has gone through the process
once, they can look at the alternative steps for more aggressive releases
2) Patch python-tripleoclient so that by default, when you run
"openstack overcloud image build" it builds the images utilising the
rdo_release DIB element, and sets the RDO_RELEASE environment variable
to be 'mitaka' or whenever the current stable release is (and we should
endevour to update it with new releases). There should be no extra
environment variables necessary to build images, and by default it
should never touch anything RDO Trunk (delorean) related
3) For bugs like the two I have mentioned above, we need to have some
sort of robust process for either backporting those patches to the
builds in CBS (I understand we don't do this for various reasons), or we
need some kind of tooling or solution that allows operators to apply
only the fixes they need from RDO Trunk (delorean). We need to ensure
that when an Operator utilises TripleO they have the greatest chance of
success, and bugs such as these which severely impact the deployment
process harm the adoption of TripleO and RDO.
4) We should curate and keep an up to date page on
that
does highlight the outstanding issues related to TripleO on the RDO
Stable (CBS) releases. These should have links to relevant bugzillas,
clean instructions on how to work around the issue, or cleanly apply a
patch to avoid the issue, and as new releases make it out, we should
update the page to drop off workarounds that are no longer needed.
The goal is to push Operators/Users to working with our most stable code
as much as possible, and track/curate issues around that. This way
everyone should be on the same page, issues are easier to discuss and
diagnose, and overall peoples experiences should be better.
I'm interested in thoughts, feedback, and concerns, both from the RDO
and TripleO community, and from the Operator/User community.
Regards,
Graeme
[1]
Thanks Marius,
I can confirm that it installs fine with 3 controllers + 3 computes
after recreating the stack
Regards
On Sat, Jun 4, 2016 at 4:14 PM, Marius Cornea <marius(a)remote-lab.net
<mailto:marius@remote-lab.net>> wrote:
Hi Pedro,
Scaling out controller nodes is not supported at this moment:
https://bugzilla.redhat.com/show_bug.cgi?id=1243312
On Sat, Jun 4, 2016 at 5:05 PM, Pedro Sousa <pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>> wrote:
> Hi,
>
> some update on scaling the cloud:
>
> 1 controller + 1 compute -> 1 controller + 3 computes OK
>
> 1 controller + 3 computes -> 3 controllers + 3 compute FAILS
>
> Problem: The new controller nodes are "stuck" in "pscd
start", so
it seems
> to be a problem joining the pacemaker cluster... Did anyone had this
> problem?
>
> Regards
>
>
>
>
>
>
> On Sat, Jun 4, 2016 at 1:50 AM, Pedro Sousa <pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>> wrote:
>>
>> Hi,
>>
>> I finally managed to install a baremetal in mitaka with 1
controller + 1
>> compute with network isolation. Thank god :)
>>
>> All I did was:
>>
>> #yum install centos-release-openstack-mitaka
>> #sudo yum install python-tripleoclient
>>
>> without epel repos.
>>
>> Then followed instructions from Redhat Site.
>>
>> I downloaded the overcloud images from:
>>
http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/mitaka/d...
>>
>> I do have an issue that forces me to delete a json file and run
>> os-refresh-config inside my overcloud nodes other than that it
installs
>> fine.
>>
>> Now I'll test with more 2 controllers + 2 computes to have a full HA
>> deployment.
>>
>> If anyone needs help to document this I'll be happy to help.
>>
>> Regards,
>> Pedro Sousa
>>
>>
>> On Fri, Jun 3, 2016 at 8:26 PM, Ronelle Landy <rlandy(a)redhat.com
<mailto:rlandy@redhat.com>> wrote:
>>>
>>> The report says: "Fix Released" as of 2016-05-24.
>>> Are you installing on a clean system with the latest repositories?
>>>
>>> Might also want to check your version of rabbitmq: I have
>>> rabbitmq-server-3.6.2-3.el7.noarch on CentOS 7.
>>>
>>> ----- Original Message -----
>>> > From: "Pedro Sousa" <pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>>
>>> > To: "Ronelle Landy" <rlandy(a)redhat.com
<mailto:rlandy@redhat.com>>
>>> > Cc: "Christopher Brown" <cbrown2(a)ocf.co.uk
<mailto:cbrown2@ocf.co.uk>>, "Ignacio Bravo"
>>> > <ibravo(a)ltgfederal.com <mailto:ibravo@ltgfederal.com>>,
"rdo-list"
>>> > <rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>>
>>> > Sent: Friday, June 3, 2016 1:20:43 PM
>>> > Subject: Re: [rdo-list] Baremetal Tripleo stable version?
>>> >
>>> > Anyway to workaround this? Maybe downgrade hiera?
>>> >
>>> > On Fri, Jun 3, 2016 at 5:55 PM, Ronelle Landy
<rlandy(a)redhat.com <mailto:rlandy@redhat.com>>
>>> > wrote:
>>> >
>>> > > I am not sure exactly where you installed from, and when you
did your
>>> > > installation, but any chance, you've hit:
>>> > >
https://bugs.launchpad.net/tripleo/+bug/1584892?
>>> > > There is a link bugzilla record.
>>> > >
>>> > > ----- Original Message -----
>>> > > > From: "Pedro Sousa" <pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>>
>>> > > > To: "Ronelle Landy" <rlandy(a)redhat.com
<mailto:rlandy@redhat.com>>
>>> > > > Cc: "Christopher Brown" <cbrown2(a)ocf.co.uk
<mailto:cbrown2@ocf.co.uk>>, "Ignacio Bravo" <
>>> > > ibravo(a)ltgfederal.com
<mailto:ibravo@ltgfederal.com>>,
"rdo-list"
>>> > > > <rdo-list(a)redhat.com
<mailto:rdo-list@redhat.com>>
>>> > > > Sent: Friday, June 3, 2016 12:26:58 PM
>>> > > > Subject: Re: [rdo-list] Baremetal Tripleo stable
version?
>>> > > >
>>> > > > Thanks Ronelle,
>>> > > >
>>> > > > do you think this kind of errors can be related with
network
>>> > > > settings?
>>> > > >
>>> > > > "Could not retrieve
fact='rabbitmq_nodename',
>>> > > > resolution='<anonymous>':
>>> > > > undefined method `[]' for nil:NilClass Could not
retrieve
>>> > > > fact='rabbitmq_nodename',
resolution='<anonymous>': undefined
>>> > > > method `[]'
>>> > > > for nil:NilClass"
>>> > > >
>>> > > > On Fri, Jun 3, 2016 at 4:56 PM, Ronelle Landy
<rlandy(a)redhat.com <mailto:rlandy@redhat.com>>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi Pedro,
>>> > > > >
>>> > > > > You could use the docs you referred to.
>>> > > > > Alternatively, if you want to use a vm for the
undercloud and
>>> > > > > baremetal
>>> > > > > machines for the overcloud, it is possible to use
Tripleo
>>> > > > > Qucikstart
>>> > > with a
>>> > > > > few modifications.
>>> > > > >
https://bugs.launchpad.net/tripleo-quickstart/+bug/1571028.
>>> > > > >
>>> > > > > ----- Original Message -----
>>> > > > > > From: "Pedro Sousa"
<pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>>
>>> > > > > > To: "Ronelle Landy"
<rlandy(a)redhat.com
<mailto:rlandy@redhat.com>>
>>> > > > > > Cc: "Christopher Brown"
<cbrown2(a)ocf.co.uk
<mailto:cbrown2@ocf.co.uk>>, "Ignacio Bravo" <
>>> > > > > ibravo(a)ltgfederal.com
<mailto:ibravo@ltgfederal.com>>,
"rdo-list"
>>> > > > > > <rdo-list(a)redhat.com
<mailto:rdo-list@redhat.com>>
>>> > > > > > Sent: Friday, June 3, 2016 11:48:38 AM
>>> > > > > > Subject: Re: [rdo-list] Baremetal Tripleo
stable version?
>>> > > > > >
>>> > > > > > Hi Ronelle,
>>> > > > > >
>>> > > > > > maybe I understand it wrong but I thought that
Tripleo
>>> > > > > > Quickstart
>>> > > was for
>>> > > > > > deploying virtual environments?
>>> > > > > >
>>> > > > > > And for baremetal we should use
>>> > > > > >
>>> > > > >
>>> > >
>>> > >
http://docs.openstack.org/developer/tripleo-docs/installation/installatio...
>>> > > > > > ?
>>> > > > > >
>>> > > > > > Thanks
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > On Fri, Jun 3, 2016 at 4:43 PM, Ronelle Landy
>>> > > > > > <rlandy(a)redhat.com
<mailto:rlandy@redhat.com>>
>>> > > wrote:
>>> > > > > >
>>> > > > > > > Hello,
>>> > > > > > >
>>> > > > > > > We have had success deploying RDO (Mitaka)
on baremetal
>>> > > > > > > systems -
>>> > > using
>>> > > > > > > Tripleo Quickstart with both
single-nic-vlans and
>>> > > > > > > bond-with-vlans
>>> > > > > network
>>> > > > > > > isolation configurations.
>>> > > > > > >
>>> > > > > > > Baremetal can have some complicated
networking
issues but,
>>> > > > > > > from
>>> > > > > previous
>>> > > > > > > experiences, if a single-controller
deployment
worked but a
>>> > > > > > > HA
>>> > > > > deployment
>>> > > > > > > did not, I would check:
>>> > > > > > > - does the HA deployment command include:
-e
>>> > > > > > >
>>> > > > >
>>> > >
>>> > >
/usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
>>> > > > > > > - are there possible MTU issues?
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > ----- Original Message -----
>>> > > > > > > > From: "Christopher Brown"
<cbrown2(a)ocf.co.uk
<mailto:cbrown2@ocf.co.uk>>
>>> > > > > > > > To: pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>,
ibravo(a)ltgfederal.com <mailto:ibravo@ltgfederal.com>
>>> > > > > > > > Cc: rdo-list(a)redhat.com
<mailto:rdo-list@redhat.com>
>>> > > > > > > > Sent: Friday, June 3, 2016 10:29:39
AM
>>> > > > > > > > Subject: Re: [rdo-list] Baremetal
Tripleo stable
version?
>>> > > > > > > >
>>> > > > > > > > Hello Ignacio,
>>> > > > > > > >
>>> > > > > > > > Thanks for your response and good to
know it isn't
just me!
>>> > > > > > > >
>>> > > > > > > > I would be more than happy to provide
developers with
>>> > > > > > > > access to
>>> > > our
>>> > > > > > > > bare metal environments. I'll
also file some bugzilla
>>> > > > > > > > reports to
>>> > > see
>>> > > > > if
>>> > > > > > > > this generates any interest.
>>> > > > > > > >
>>> > > > > > > > Please do let me know if you make any
progress - I am
>>> > > > > > > > trying to
>>> > > > > deploy
>>> > > > > > > > HA with network isolation, multiple
nics and vlans.
>>> > > > > > > >
>>> > > > > > > > The RDO web page states:
>>> > > > > > > >
>>> > > > > > > > "If you want to create a
production-ready cloud,
you'll
>>> > > > > > > > want to
>>> > > use
>>> > > > > the
>>> > > > > > > > TripleO quickstart guide."
>>> > > > > > > >
>>> > > > > > > > which is a contradiction in terms
really.
>>> > > > > > > >
>>> > > > > > > > Cheers
>>> > > > > > > >
>>> > > > > > > > On Fri, 2016-06-03 at 14:30 +0100,
Ignacio Bravo
wrote:
>>> > > > > > > > > Pedro / Christopher,
>>> > > > > > > > >
>>> > > > > > > > > Just wanted to share with you
that I also had
plenty of
>>> > > > > > > > > issues
>>> > > > > > > > > deploying on bare metal HA
servers, and have
paused the
>>> > > deployment
>>> > > > > > > > > using TripleO until better winds
start to flow
here. I
>>> > > > > > > > > was
>>> > > able to
>>> > > > > > > > > deploy the QuickStart, but on
bare metal the
history was
>>> > > different.
>>> > > > > > > > > Couldn't even deploy a two
server configuration.
>>> > > > > > > > >
>>> > > > > > > > > I was thinking that it would be
good to have the
>>> > > > > > > > > developers
>>> > > have
>>> > > > > > > > > access to one of our
environments and go through
a full
>>> > > > > > > > > install
>>> > > > > with
>>> > > > > > > > > us to better see where things
fail. We can do this
>>> > > > > > > > > handholding
>>> > > > > > > > > deployment once every week/month
based on
developers time
>>> > > > > > > > > availability. That way we can
get a working
install, and
>>> > > > > > > > > we can
>>> > > > > > > > > troubleshoot real life
environment problems.
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > IB
>>> > > > > > > > >
>>> > > > > > > > > On Jun 3, 2016, at 6:15 AM,
Pedro Sousa
>>> > > > > > > > > <pgsousa(a)gmail.com
<mailto:pgsousa@gmail.com>>
>>> > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Yes. I've used this,
but I'll try again as there's
>>> > > > > > > > > > seems to
>>> > > be
>>> > > > > new
>>> > > > > > > > > > updates.
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > Stable Branch Skip all
repos mentioned above,
other
>>> > > > > > > > > > than
>>> > > epel-
>>> > > > > > > > > > release which is still
required.
>>> > > > > > > > > > Enable latest RDO Stable
Delorean repository
for all
>>> > > > > > > > > > packages
>>> > > > > > > > > > sudo curl -o
/etc/yum.repos.d/delorean-liberty.repo
>>> > > > >
https://trunk.r
>>> > > > > > > > > >
doproject.org/centos7-liberty/current/delorean.repo
<
http://doproject.org/centos7-liberty/current/delorean.repo>
>>> > > > > > > > > > Enable the Delorean Deps
repository
>>> > > > > > > > > > sudo curl -o
>>> > > > > > > > > >
/etc/yum.repos.d/delorean-deps-liberty.repo
>>> > > > >
http://tru
>>> > > > > > > > > >
nk.rdoproject.org/centos7-liberty/delorean-deps.repo
<
http://nk.rdoproject.org/centos7-liberty/delorean-deps.repo>
>>> > > > > > > > > >
>>> > > > > > > > > > On Fri, Jun 3, 2016 at
11:10 AM, Christopher
Brown <
>>> > > > > cbrown2(a)ocf.co <mailto:cbrown2@ocf.co>.
>>> > > > > > > > > > uk> wrote:
>>> > > > > > > > > > > No, Liberty deployed
ok for us.
>>> > > > > > > > > > >
>>> > > > > > > > > > > It suggests to me a
package mismatch. Have you
>>> > > > > > > > > > > completely
>>> > > > > rebuilt
>>> > > > > > > > > > > the
>>> > > > > > > > > > > undercloud and then
the images using Liberty?
>>> > > > > > > > > > >
>>> > > > > > > > > > > On Fri, 2016-06-03 at
11:04 +0100, Pedro
Sousa wrote:
>>> > > > > > > > > > > > AttributeError:
'module' object has no
attribute
>>> > > 'PortOpt'
>>> > > > > > > > > > > --
>>> > > > > > > > > > > Regards,
>>> > > > > > > > > > >
>>> > > > > > > > > > > Christopher Brown
>>> > > > > > > > > > > OpenStack Engineer
>>> > > > > > > > > > > OCF plc
>>> > > > > > > > > > >
>>> > > > > > > > > > > Tel: +44 (0)114 257
2200
<tel:%2B44%20%280%29114%20257%202200>
>>> > > > > > > > > > > Web:
www.ocf.co.uk
<
http://www.ocf.co.uk>
>>> > > > > > > > > > > Blog: blog.ocf.co.uk
<
http://blog.ocf.co.uk>
>>> > > > > > > > > > > Twitter: @ocfplc
>>> > > > > > > > > > >
>>> > > > > > > > > > > Please note, any
emails relating to an OCF
Support
>>> > > > > > > > > > > request
>>> > > must
>>> > > > > > > > > > > always
>>> > > > > > > > > > > be sent to
support(a)ocf.co.uk
<mailto:support@ocf.co.uk> for a ticket number to
>>> > > > > > > > > > > be
>>> > > > > generated
>>> > > > > > > > > > > or
>>> > > > > > > > > > > existing support
ticket to be updated.
Should this
>>> > > > > > > > > > > not be
>>> > > done
>>> > > > > > > > > > > then OCF
>>> > > > > > > > > > >
>>> > > > > > > > > > > cannot be held
responsible for requests not
dealt
>>> > > > > > > > > > > with in a
>>> > > > > > > > > > > timely
>>> > > > > > > > > > > manner.
>>> > > > > > > > > > >
>>> > > > > > > > > > > OCF plc is a company
registered in England
and Wales.
>>> > > > > Registered
>>> > > > > > > > > > > number
>>> > > > > > > > > > >
>>> > > > > > > > > > > 4132533, VAT number GB
780 6803 14.
Registered office
>>> > > address:
>>> > > > > > > > > > > OCF plc,
>>> > > > > > > > > > >
>>> > > > > > > > > > > 5 Rotunda Business
Centre, Thorncliffe Park,
>>> > > > > > > > > > > Chapeltown,
>>> > > > > > > > > > > Sheffield S35
>>> > > > > > > > > > > 2PG.
>>> > > > > > > > > > >
>>> > > > > > > > > > > If you have received
this message in error,
please
>>> > > > > > > > > > > notify
>>> > > us
>>> > > > > > > > > > > immediately and remove
it from your system.
>>> > > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
_______________________________________________
>>> > > > > > > > > rdo-list mailing list
>>> > > > > > > > > rdo-list(a)redhat.com
<mailto:rdo-list@redhat.com>
>>> > > > > > > > >
https://www.redhat.com/mailman/listinfo/rdo-list
>>> > > > > > > > >
>>> > > > > > > > > To unsubscribe:
rdo-list-unsubscribe(a)redhat.com
<mailto:rdo-list-unsubscribe@redhat.com>
>>> > > > > > > > --
>>> > > > > > > > Regards,
>>> > > > > > > >
>>> > > > > > > > Christopher Brown
>>> > > > > > > > OpenStack Engineer
>>> > > > > > > > OCF plc
>>> > > > > > > >
>>> > > > > > > > Tel: +44 (0)114 257 2200
<tel:%2B44%20%280%29114%20257%202200>
>>> > > > > > > > Web:
www.ocf.co.uk
<
http://www.ocf.co.uk>
>>> > > > > > > > Blog: blog.ocf.co.uk
<
http://blog.ocf.co.uk>
>>> > > > > > > > Twitter: @ocfplc
>>> > > > > > > >
>>> > > > > > > > Please note, any emails relating to
an OCF Support
request
>>> > > > > > > > must
>>> > > > > always
>>> > > > > > > > be sent to support(a)ocf.co.uk
<mailto:support@ocf.co.uk> for a ticket number to be
>>> > > generated or
>>> > > > > > > > existing support ticket to be
updated. Should this
not be
>>> > > > > > > > done
>>> > > then
>>> > > > > OCF
>>> > > > > > > >
>>> > > > > > > > cannot be held responsible for
requests not dealt
with in a
>>> > > timely
>>> > > > > > > > manner.
>>> > > > > > > >
>>> > > > > > > > OCF plc is a company registered in
England and Wales.
>>> > > > > > > > Registered
>>> > > > > number
>>> > > > > > > >
>>> > > > > > > > 4132533, VAT number GB 780 6803 14.
Registered office
>>> > > > > > > > address:
>>> > > OCF
>>> > > > > plc,
>>> > > > > > > >
>>> > > > > > > > 5 Rotunda Business Centre,
Thorncliffe Park,
Chapeltown,
>>> > > Sheffield
>>> > > > > S35
>>> > > > > > > > 2PG.
>>> > > > > > > >
>>> > > > > > > > If you have received this message in
error, please
notify
>>> > > > > > > > us
>>> > > > > > > > immediately and remove it from your
system.
>>> > > > > > > >
>>> > > > > > > >
_______________________________________________
>>> > > > > > > > rdo-list mailing list
>>> > > > > > > > rdo-list(a)redhat.com
<mailto:rdo-list@redhat.com>
>>> > > > > > > >
https://www.redhat.com/mailman/listinfo/rdo-list
>>> > > > > > > >
>>> > > > > > > > To unsubscribe:
rdo-list-unsubscribe(a)redhat.com
<mailto:rdo-list-unsubscribe@redhat.com>
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>
>>
>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>
https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
<mailto:rdo-list-unsubscribe@redhat.com>
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
Graeme Gillies
Principal Systems Administrator
Openstack Infrastructure
Red Hat Australia