[rdo-list] Baremetal Tripleo stable version?
Graeme Gillies
ggillies at redhat.com
Sun Jun 5 23:37:53 UTC 2016
Hi Everyone,
I just wanted to say I have been following this thread quite closely and
can sympathize with some of the pain people are going through to get
tripleO to work.
Currently it's quite difficult and a bit opaque on how to actually
utilise the stable mitaka repos in order to build a functional
undercloud and overcloud environment.
First I wanted to share the steps I have undergone in order to get a
functional overcloud working with RDO Mitaka utilising the RDO stable
release built by CentOS, and then I'll talk about some specific steps I
think need to be undertaken by the RDO/TripleO team in order to provide
a better experience in the future.
To get a functional overcloud using RDO Mitaka, you need to do the following
1) Install EPEL on your undercloud
2) Install https://www.rdoproject.org/repos/rdo-release.rpm on your
undercloud
3) Following the normal steps to install your undercloud (modifying
undercloud.conf, and running openstack undercloud install
4) You will now need to manually patch ironic on the undercloud in order
to make sure repeated introspection works. This might not be needed if
you don't do any introspection, but I find more often than not you end
up having to do it, so it's worthwhile. The bug you need to patch is [1]
and I typically run the following commands to apply the patch
# sudo su -
$ cd /usr/lib/python2.7/site-packages
$ curl
'https://review.openstack.org/changes/306421/revisions/abd50d8438e7d371ce24f97d8f8f67052b562007/patch?download'
| base64 -d | patch -p1
$ systemctl restart openstack-ironic-inspector
$ systemctl restart openstack-ironic-inspector-dnsmasq
$ exit
#
5) Manually patch the undercloud to build overcloud images using
rhos-release rpm only (which utilises the stable Mitaka repo from
CentOS, and nothing from RDO Trunk [delorean]). I do this by modifying
the file
/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py
At around line 467 you will see a reference to epel, I add a new line
after that to include the rdo_release DIB element to the build as well.
This typically makes the file look something like
http://paste.openstack.org/show/508196/
(note like 468). Then I create a directory to store my images and build
them specifying the mitaka version of rdo_release. I then upload these
images
# mkdir ~/images
# cd ~/images
# export RDO_RELEASE=mitaka
# openstack overcloud image build --all
# openstack overcloud image upload --update-existing
6) Because of the bug at [2] which affects the ironic agent ramdisk, we
need to build a set of images utilising RDO Trunk for the mitaka branch
(where the fix is applied), and then upload *only* the new ironic
ramdisk. This is done with
# mkdir ~/images-mitaka-trunk
# cd ~/images-mitaka-trunk
# export USE_DELOREAN_TRUNK=1
# export
DELOREAN_TRUNK_REPO="http://trunk.rdoproject.org/centos7-mitaka/current/"
# export DELOREAN_REPO_FILE="delorean.repo"
# openstack overcloud image build --type agent-ramdisk
# sudo cp ironic-python-agent.initramfs /httpboot/agent.ramdisk
7) Follow the rest of the documentation to deploy the overcloud normally
Please note that obviously your mileage may vary, and this is by all
means not an exclusive list of the problems. I have however used these
steps to do multiple node deployments (10+ nodes) with HA over different
hardware sets with different networking setups (single nic, multiple nic
with bonding + vlans).
With all the different repos floating around, all which change very
rapidly, combined with the documentation defaults targeting developers
and CI systems (not end users), it's hard to not only get a stable
TripleO install up, but also communicate and discuss clearly with others
what is working, what is broken, and how to compare two installations to
see if they are experiencing the same issues.
To this end I would like to suggest to the RDO and TripleO community
that we undertake the following
1) Overhaul all the TripleO documentation so that all the steps default
to utilising/deploying using RDO Stable (that is, the releases done by
CBS). There should be colored boxes with alt steps for those who wish to
use RDO Trunk on the stable branch, and RDO Trunk from master. This
basically inverts the current pattern. I think anyone, Operator or
developer, who is working through the documentation for the first time,
should be given steps that maximise the chance of success, and thus the
most stable release we have. Once a user has gone through the process
once, they can look at the alternative steps for more aggressive releases
2) Patch python-tripleoclient so that by default, when you run
"openstack overcloud image build" it builds the images utilising the
rdo_release DIB element, and sets the RDO_RELEASE environment variable
to be 'mitaka' or whenever the current stable release is (and we should
endevour to update it with new releases). There should be no extra
environment variables necessary to build images, and by default it
should never touch anything RDO Trunk (delorean) related
3) For bugs like the two I have mentioned above, we need to have some
sort of robust process for either backporting those patches to the
builds in CBS (I understand we don't do this for various reasons), or we
need some kind of tooling or solution that allows operators to apply
only the fixes they need from RDO Trunk (delorean). We need to ensure
that when an Operator utilises TripleO they have the greatest chance of
success, and bugs such as these which severely impact the deployment
process harm the adoption of TripleO and RDO.
4) We should curate and keep an up to date page on rdoproject.org that
does highlight the outstanding issues related to TripleO on the RDO
Stable (CBS) releases. These should have links to relevant bugzillas,
clean instructions on how to work around the issue, or cleanly apply a
patch to avoid the issue, and as new releases make it out, we should
update the page to drop off workarounds that are no longer needed.
The goal is to push Operators/Users to working with our most stable code
as much as possible, and track/curate issues around that. This way
everyone should be on the same page, issues are easier to discuss and
diagnose, and overall peoples experiences should be better.
I'm interested in thoughts, feedback, and concerns, both from the RDO
and TripleO community, and from the Operator/User community.
Regards,
Graeme
[1] https://bugs.launchpad.net/ironic-inspector/+bug/1570447
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1322892
On 05/06/16 02:04, Pedro Sousa wrote:
> Thanks Marius,
>
> I can confirm that it installs fine with 3 controllers + 3 computes
> after recreating the stack
>
> Regards
>
> On Sat, Jun 4, 2016 at 4:14 PM, Marius Cornea <marius at remote-lab.net
> <mailto:marius at remote-lab.net>> wrote:
>
> Hi Pedro,
>
> Scaling out controller nodes is not supported at this moment:
> https://bugzilla.redhat.com/show_bug.cgi?id=1243312
>
> On Sat, Jun 4, 2016 at 5:05 PM, Pedro Sousa <pgsousa at gmail.com
> <mailto:pgsousa at gmail.com>> wrote:
> > Hi,
> >
> > some update on scaling the cloud:
> >
> > 1 controller + 1 compute -> 1 controller + 3 computes OK
> >
> > 1 controller + 3 computes -> 3 controllers + 3 compute FAILS
> >
> > Problem: The new controller nodes are "stuck" in "pscd start", so
> it seems
> > to be a problem joining the pacemaker cluster... Did anyone had this
> > problem?
> >
> > Regards
> >
> >
> >
> >
> >
> >
> > On Sat, Jun 4, 2016 at 1:50 AM, Pedro Sousa <pgsousa at gmail.com
> <mailto:pgsousa at gmail.com>> wrote:
> >>
> >> Hi,
> >>
> >> I finally managed to install a baremetal in mitaka with 1
> controller + 1
> >> compute with network isolation. Thank god :)
> >>
> >> All I did was:
> >>
> >> #yum install centos-release-openstack-mitaka
> >> #sudo yum install python-tripleoclient
> >>
> >> without epel repos.
> >>
> >> Then followed instructions from Redhat Site.
> >>
> >> I downloaded the overcloud images from:
> >>
> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/mitaka/delorean/
> >>
> >> I do have an issue that forces me to delete a json file and run
> >> os-refresh-config inside my overcloud nodes other than that it
> installs
> >> fine.
> >>
> >> Now I'll test with more 2 controllers + 2 computes to have a full HA
> >> deployment.
> >>
> >> If anyone needs help to document this I'll be happy to help.
> >>
> >> Regards,
> >> Pedro Sousa
> >>
> >>
> >> On Fri, Jun 3, 2016 at 8:26 PM, Ronelle Landy <rlandy at redhat.com
> <mailto:rlandy at redhat.com>> wrote:
> >>>
> >>> The report says: "Fix Released" as of 2016-05-24.
> >>> Are you installing on a clean system with the latest repositories?
> >>>
> >>> Might also want to check your version of rabbitmq: I have
> >>> rabbitmq-server-3.6.2-3.el7.noarch on CentOS 7.
> >>>
> >>> ----- Original Message -----
> >>> > From: "Pedro Sousa" <pgsousa at gmail.com <mailto:pgsousa at gmail.com>>
> >>> > To: "Ronelle Landy" <rlandy at redhat.com <mailto:rlandy at redhat.com>>
> >>> > Cc: "Christopher Brown" <cbrown2 at ocf.co.uk
> <mailto:cbrown2 at ocf.co.uk>>, "Ignacio Bravo"
> >>> > <ibravo at ltgfederal.com <mailto:ibravo at ltgfederal.com>>, "rdo-list"
> >>> > <rdo-list at redhat.com <mailto:rdo-list at redhat.com>>
> >>> > Sent: Friday, June 3, 2016 1:20:43 PM
> >>> > Subject: Re: [rdo-list] Baremetal Tripleo stable version?
> >>> >
> >>> > Anyway to workaround this? Maybe downgrade hiera?
> >>> >
> >>> > On Fri, Jun 3, 2016 at 5:55 PM, Ronelle Landy
> <rlandy at redhat.com <mailto:rlandy at redhat.com>>
> >>> > wrote:
> >>> >
> >>> > > I am not sure exactly where you installed from, and when you
> did your
> >>> > > installation, but any chance, you've hit:
> >>> > > https://bugs.launchpad.net/tripleo/+bug/1584892?
> >>> > > There is a link bugzilla record.
> >>> > >
> >>> > > ----- Original Message -----
> >>> > > > From: "Pedro Sousa" <pgsousa at gmail.com
> <mailto:pgsousa at gmail.com>>
> >>> > > > To: "Ronelle Landy" <rlandy at redhat.com
> <mailto:rlandy at redhat.com>>
> >>> > > > Cc: "Christopher Brown" <cbrown2 at ocf.co.uk
> <mailto:cbrown2 at ocf.co.uk>>, "Ignacio Bravo" <
> >>> > > ibravo at ltgfederal.com <mailto:ibravo at ltgfederal.com>>,
> "rdo-list"
> >>> > > > <rdo-list at redhat.com <mailto:rdo-list at redhat.com>>
> >>> > > > Sent: Friday, June 3, 2016 12:26:58 PM
> >>> > > > Subject: Re: [rdo-list] Baremetal Tripleo stable version?
> >>> > > >
> >>> > > > Thanks Ronelle,
> >>> > > >
> >>> > > > do you think this kind of errors can be related with network
> >>> > > > settings?
> >>> > > >
> >>> > > > "Could not retrieve fact='rabbitmq_nodename',
> >>> > > > resolution='<anonymous>':
> >>> > > > undefined method `[]' for nil:NilClass Could not retrieve
> >>> > > > fact='rabbitmq_nodename', resolution='<anonymous>': undefined
> >>> > > > method `[]'
> >>> > > > for nil:NilClass"
> >>> > > >
> >>> > > > On Fri, Jun 3, 2016 at 4:56 PM, Ronelle Landy
> <rlandy at redhat.com <mailto:rlandy at redhat.com>>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Hi Pedro,
> >>> > > > >
> >>> > > > > You could use the docs you referred to.
> >>> > > > > Alternatively, if you want to use a vm for the
> undercloud and
> >>> > > > > baremetal
> >>> > > > > machines for the overcloud, it is possible to use Tripleo
> >>> > > > > Qucikstart
> >>> > > with a
> >>> > > > > few modifications.
> >>> > > > > https://bugs.launchpad.net/tripleo-quickstart/+bug/1571028.
> >>> > > > >
> >>> > > > > ----- Original Message -----
> >>> > > > > > From: "Pedro Sousa" <pgsousa at gmail.com
> <mailto:pgsousa at gmail.com>>
> >>> > > > > > To: "Ronelle Landy" <rlandy at redhat.com
> <mailto:rlandy at redhat.com>>
> >>> > > > > > Cc: "Christopher Brown" <cbrown2 at ocf.co.uk
> <mailto:cbrown2 at ocf.co.uk>>, "Ignacio Bravo" <
> >>> > > > > ibravo at ltgfederal.com <mailto:ibravo at ltgfederal.com>>,
> "rdo-list"
> >>> > > > > > <rdo-list at redhat.com <mailto:rdo-list at redhat.com>>
> >>> > > > > > Sent: Friday, June 3, 2016 11:48:38 AM
> >>> > > > > > Subject: Re: [rdo-list] Baremetal Tripleo stable version?
> >>> > > > > >
> >>> > > > > > Hi Ronelle,
> >>> > > > > >
> >>> > > > > > maybe I understand it wrong but I thought that Tripleo
> >>> > > > > > Quickstart
> >>> > > was for
> >>> > > > > > deploying virtual environments?
> >>> > > > > >
> >>> > > > > > And for baremetal we should use
> >>> > > > > >
> >>> > > > >
> >>> > >
> >>> > >
> http://docs.openstack.org/developer/tripleo-docs/installation/installation.html
> >>> > > > > > ?
> >>> > > > > >
> >>> > > > > > Thanks
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > On Fri, Jun 3, 2016 at 4:43 PM, Ronelle Landy
> >>> > > > > > <rlandy at redhat.com <mailto:rlandy at redhat.com>>
> >>> > > wrote:
> >>> > > > > >
> >>> > > > > > > Hello,
> >>> > > > > > >
> >>> > > > > > > We have had success deploying RDO (Mitaka) on baremetal
> >>> > > > > > > systems -
> >>> > > using
> >>> > > > > > > Tripleo Quickstart with both single-nic-vlans and
> >>> > > > > > > bond-with-vlans
> >>> > > > > network
> >>> > > > > > > isolation configurations.
> >>> > > > > > >
> >>> > > > > > > Baremetal can have some complicated networking
> issues but,
> >>> > > > > > > from
> >>> > > > > previous
> >>> > > > > > > experiences, if a single-controller deployment
> worked but a
> >>> > > > > > > HA
> >>> > > > > deployment
> >>> > > > > > > did not, I would check:
> >>> > > > > > > - does the HA deployment command include: -e
> >>> > > > > > >
> >>> > > > >
> >>> > >
> >>> > >
> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
> >>> > > > > > > - are there possible MTU issues?
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > ----- Original Message -----
> >>> > > > > > > > From: "Christopher Brown" <cbrown2 at ocf.co.uk
> <mailto:cbrown2 at ocf.co.uk>>
> >>> > > > > > > > To: pgsousa at gmail.com <mailto:pgsousa at gmail.com>,
> ibravo at ltgfederal.com <mailto:ibravo at ltgfederal.com>
> >>> > > > > > > > Cc: rdo-list at redhat.com <mailto:rdo-list at redhat.com>
> >>> > > > > > > > Sent: Friday, June 3, 2016 10:29:39 AM
> >>> > > > > > > > Subject: Re: [rdo-list] Baremetal Tripleo stable
> version?
> >>> > > > > > > >
> >>> > > > > > > > Hello Ignacio,
> >>> > > > > > > >
> >>> > > > > > > > Thanks for your response and good to know it isn't
> just me!
> >>> > > > > > > >
> >>> > > > > > > > I would be more than happy to provide developers with
> >>> > > > > > > > access to
> >>> > > our
> >>> > > > > > > > bare metal environments. I'll also file some bugzilla
> >>> > > > > > > > reports to
> >>> > > see
> >>> > > > > if
> >>> > > > > > > > this generates any interest.
> >>> > > > > > > >
> >>> > > > > > > > Please do let me know if you make any progress - I am
> >>> > > > > > > > trying to
> >>> > > > > deploy
> >>> > > > > > > > HA with network isolation, multiple nics and vlans.
> >>> > > > > > > >
> >>> > > > > > > > The RDO web page states:
> >>> > > > > > > >
> >>> > > > > > > > "If you want to create a production-ready cloud,
> you'll
> >>> > > > > > > > want to
> >>> > > use
> >>> > > > > the
> >>> > > > > > > > TripleO quickstart guide."
> >>> > > > > > > >
> >>> > > > > > > > which is a contradiction in terms really.
> >>> > > > > > > >
> >>> > > > > > > > Cheers
> >>> > > > > > > >
> >>> > > > > > > > On Fri, 2016-06-03 at 14:30 +0100, Ignacio Bravo
> wrote:
> >>> > > > > > > > > Pedro / Christopher,
> >>> > > > > > > > >
> >>> > > > > > > > > Just wanted to share with you that I also had
> plenty of
> >>> > > > > > > > > issues
> >>> > > > > > > > > deploying on bare metal HA servers, and have
> paused the
> >>> > > deployment
> >>> > > > > > > > > using TripleO until better winds start to flow
> here. I
> >>> > > > > > > > > was
> >>> > > able to
> >>> > > > > > > > > deploy the QuickStart, but on bare metal the
> history was
> >>> > > different.
> >>> > > > > > > > > Couldn't even deploy a two server configuration.
> >>> > > > > > > > >
> >>> > > > > > > > > I was thinking that it would be good to have the
> >>> > > > > > > > > developers
> >>> > > have
> >>> > > > > > > > > access to one of our environments and go through
> a full
> >>> > > > > > > > > install
> >>> > > > > with
> >>> > > > > > > > > us to better see where things fail. We can do this
> >>> > > > > > > > > handholding
> >>> > > > > > > > > deployment once every week/month based on
> developers time
> >>> > > > > > > > > availability. That way we can get a working
> install, and
> >>> > > > > > > > > we can
> >>> > > > > > > > > troubleshoot real life environment problems.
> >>> > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > IB
> >>> > > > > > > > >
> >>> > > > > > > > > On Jun 3, 2016, at 6:15 AM, Pedro Sousa
> >>> > > > > > > > > <pgsousa at gmail.com <mailto:pgsousa at gmail.com>>
> >>> > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Yes. I've used this, but I'll try again as there's
> >>> > > > > > > > > > seems to
> >>> > > be
> >>> > > > > new
> >>> > > > > > > > > > updates.
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > Stable Branch Skip all repos mentioned above,
> other
> >>> > > > > > > > > > than
> >>> > > epel-
> >>> > > > > > > > > > release which is still required.
> >>> > > > > > > > > > Enable latest RDO Stable Delorean repository
> for all
> >>> > > > > > > > > > packages
> >>> > > > > > > > > > sudo curl -o
> /etc/yum.repos.d/delorean-liberty.repo
> >>> > > > > https://trunk.r
> >>> > > > > > > > > >
> doproject.org/centos7-liberty/current/delorean.repo
> <http://doproject.org/centos7-liberty/current/delorean.repo>
> >>> > > > > > > > > > Enable the Delorean Deps repository
> >>> > > > > > > > > > sudo curl -o
> >>> > > > > > > > > > /etc/yum.repos.d/delorean-deps-liberty.repo
> >>> > > > > http://tru
> >>> > > > > > > > > >
> nk.rdoproject.org/centos7-liberty/delorean-deps.repo
> <http://nk.rdoproject.org/centos7-liberty/delorean-deps.repo>
> >>> > > > > > > > > >
> >>> > > > > > > > > > On Fri, Jun 3, 2016 at 11:10 AM, Christopher
> Brown <
> >>> > > > > cbrown2 at ocf.co <mailto:cbrown2 at ocf.co>.
> >>> > > > > > > > > > uk> wrote:
> >>> > > > > > > > > > > No, Liberty deployed ok for us.
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > It suggests to me a package mismatch. Have you
> >>> > > > > > > > > > > completely
> >>> > > > > rebuilt
> >>> > > > > > > > > > > the
> >>> > > > > > > > > > > undercloud and then the images using Liberty?
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > On Fri, 2016-06-03 at 11:04 +0100, Pedro
> Sousa wrote:
> >>> > > > > > > > > > > > AttributeError: 'module' object has no
> attribute
> >>> > > 'PortOpt'
> >>> > > > > > > > > > > --
> >>> > > > > > > > > > > Regards,
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Christopher Brown
> >>> > > > > > > > > > > OpenStack Engineer
> >>> > > > > > > > > > > OCF plc
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Tel: +44 (0)114 257 2200
> <tel:%2B44%20%280%29114%20257%202200>
> >>> > > > > > > > > > > Web: www.ocf.co.uk <http://www.ocf.co.uk>
> >>> > > > > > > > > > > Blog: blog.ocf.co.uk <http://blog.ocf.co.uk>
> >>> > > > > > > > > > > Twitter: @ocfplc
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Please note, any emails relating to an OCF
> Support
> >>> > > > > > > > > > > request
> >>> > > must
> >>> > > > > > > > > > > always
> >>> > > > > > > > > > > be sent to support at ocf.co.uk
> <mailto:support at ocf.co.uk> for a ticket number to
> >>> > > > > > > > > > > be
> >>> > > > > generated
> >>> > > > > > > > > > > or
> >>> > > > > > > > > > > existing support ticket to be updated.
> Should this
> >>> > > > > > > > > > > not be
> >>> > > done
> >>> > > > > > > > > > > then OCF
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > cannot be held responsible for requests not
> dealt
> >>> > > > > > > > > > > with in a
> >>> > > > > > > > > > > timely
> >>> > > > > > > > > > > manner.
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > OCF plc is a company registered in England
> and Wales.
> >>> > > > > Registered
> >>> > > > > > > > > > > number
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > 4132533, VAT number GB 780 6803 14.
> Registered office
> >>> > > address:
> >>> > > > > > > > > > > OCF plc,
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > 5 Rotunda Business Centre, Thorncliffe Park,
> >>> > > > > > > > > > > Chapeltown,
> >>> > > > > > > > > > > Sheffield S35
> >>> > > > > > > > > > > 2PG.
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > If you have received this message in error,
> please
> >>> > > > > > > > > > > notify
> >>> > > us
> >>> > > > > > > > > > > immediately and remove it from your system.
> >>> > > > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > > > _______________________________________________
> >>> > > > > > > > > rdo-list mailing list
> >>> > > > > > > > > rdo-list at redhat.com <mailto:rdo-list at redhat.com>
> >>> > > > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list
> >>> > > > > > > > >
> >>> > > > > > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com
> <mailto:rdo-list-unsubscribe at redhat.com>
> >>> > > > > > > > --
> >>> > > > > > > > Regards,
> >>> > > > > > > >
> >>> > > > > > > > Christopher Brown
> >>> > > > > > > > OpenStack Engineer
> >>> > > > > > > > OCF plc
> >>> > > > > > > >
> >>> > > > > > > > Tel: +44 (0)114 257 2200
> <tel:%2B44%20%280%29114%20257%202200>
> >>> > > > > > > > Web: www.ocf.co.uk <http://www.ocf.co.uk>
> >>> > > > > > > > Blog: blog.ocf.co.uk <http://blog.ocf.co.uk>
> >>> > > > > > > > Twitter: @ocfplc
> >>> > > > > > > >
> >>> > > > > > > > Please note, any emails relating to an OCF Support
> request
> >>> > > > > > > > must
> >>> > > > > always
> >>> > > > > > > > be sent to support at ocf.co.uk
> <mailto:support at ocf.co.uk> for a ticket number to be
> >>> > > generated or
> >>> > > > > > > > existing support ticket to be updated. Should this
> not be
> >>> > > > > > > > done
> >>> > > then
> >>> > > > > OCF
> >>> > > > > > > >
> >>> > > > > > > > cannot be held responsible for requests not dealt
> with in a
> >>> > > timely
> >>> > > > > > > > manner.
> >>> > > > > > > >
> >>> > > > > > > > OCF plc is a company registered in England and Wales.
> >>> > > > > > > > Registered
> >>> > > > > number
> >>> > > > > > > >
> >>> > > > > > > > 4132533, VAT number GB 780 6803 14. Registered office
> >>> > > > > > > > address:
> >>> > > OCF
> >>> > > > > plc,
> >>> > > > > > > >
> >>> > > > > > > > 5 Rotunda Business Centre, Thorncliffe Park,
> Chapeltown,
> >>> > > Sheffield
> >>> > > > > S35
> >>> > > > > > > > 2PG.
> >>> > > > > > > >
> >>> > > > > > > > If you have received this message in error, please
> notify
> >>> > > > > > > > us
> >>> > > > > > > > immediately and remove it from your system.
> >>> > > > > > > >
> >>> > > > > > > > _______________________________________________
> >>> > > > > > > > rdo-list mailing list
> >>> > > > > > > > rdo-list at redhat.com <mailto:rdo-list at redhat.com>
> >>> > > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list
> >>> > > > > > > >
> >>> > > > > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com
> <mailto:rdo-list-unsubscribe at redhat.com>
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>
> >>
> >
> >
> > _______________________________________________
> > rdo-list mailing list
> > rdo-list at redhat.com <mailto:rdo-list at redhat.com>
> > https://www.redhat.com/mailman/listinfo/rdo-list
> >
> > To unsubscribe: rdo-list-unsubscribe at redhat.com
> <mailto:rdo-list-unsubscribe at redhat.com>
>
>
>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
--
Graeme Gillies
Principal Systems Administrator
Openstack Infrastructure
Red Hat Australia
More information about the dev
mailing list