4:44 a.m.
Hi,
I am running TripleO Newton stable release and am deploying on baremetal
with CentOS.
I have 64 nodes, and the Undercloud has plenty of resource as it is one
of the nodes with 294 GB Memory and 64 CPUs.
The provisioning network is 1Gbps
I have tried tuning the Undercloud using this tuning section in 10.7 as
a guide
https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/p...
My Undercloud passes validations in Clapper
https://github.com/rthallisey/clapper
I am deploying with Network Isolation and 3 Controllers in HA.
If I create a stack with 3 Controllers and 3 compute nodes this takes
about 1 hour
If I create a stack with 3 Controllers and 15 compute nodes this takes
about 1 hour
Both stacks pass Clapper validations.
During deployment I can see that the first 20 to 30 mins is using all
the bandwidth available for the overcloud image deployment and them uses
hardly any bandwidth whilst the rest of the configuration takes place.
So I try a stack with 40 nodes. This is where I have issues.
I set the timeout to 4 hours and leave it over night to deploy.
It seems to timeout and fail to deploy due to the timeout every time.
During the 40 node deployment the overcloud image is distributed in
about 45 mins to all nodes and the all nodes appear ACTIVE and have an
IP address on the deployment network.
So it would appear that the rest of the low bandwidth configuration is
taking well over 3 hours to complete. This seems excessive
I have configured nova.conf for deployment concurrency (from the tuning
link above) and configured the heat.conf 'num_engine_workers' to be 32
taking in to account this bug
https://bugzilla.redhat.com/show_bug.cgi?id=1370516
So my question is how do I tune my Undercloud to speed up the deployment?
Looking at htop during deployment I can see heat is using many CPUs, but
the work pattern is NOT distributed. What typically happens is all the
CPUs are at 0 to 1 % used apart from one which is at 50 to 100%. This
one CPU id changes regularly, but there is no concurrent distributed
workload across all the CPUs that the heat processes are running on. Is
heat really multi-threaded, or does if have limitations so it can only
really do proper work on one CPU at a time (which I am seeing in htop)?
Thanks
Charles
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
2:30 p.m.
Some more testing of different amounts of nodes vs time taken for
successful deployments -
3 controller 3 compute = 1 hour
3 controller 15 compute = 1 hour
3 controller 25 compute = 1 hour 45 mins
3 controller 35 compute = 4 hours
Charles
On 02/11/2016 09:44, Charles Short wrote:
Hi,
I am running TripleO Newton stable release and am deploying on
baremetal with CentOS.
I have 64 nodes, and the Undercloud has plenty of resource as it is
one of the nodes with 294 GB Memory and 64 CPUs.
The provisioning network is 1Gbps
I have tried tuning the Undercloud using this tuning section in 10.7
as a guide
https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/p...
My Undercloud passes validations in Clapper
https://github.com/rthallisey/clapper
I am deploying with Network Isolation and 3 Controllers in HA.
If I create a stack with 3 Controllers and 3 compute nodes this takes
about 1 hour
If I create a stack with 3 Controllers and 15 compute nodes this takes
about 1 hour
Both stacks pass Clapper validations.
During deployment I can see that the first 20 to 30 mins is using all
the bandwidth available for the overcloud image deployment and them
uses hardly any bandwidth whilst the rest of the configuration takes
place.
So I try a stack with 40 nodes. This is where I have issues.
I set the timeout to 4 hours and leave it over night to deploy.
It seems to timeout and fail to deploy due to the timeout every time.
During the 40 node deployment the overcloud image is distributed in
about 45 mins to all nodes and the all nodes appear ACTIVE and have an
IP address on the deployment network.
So it would appear that the rest of the low bandwidth configuration is
taking well over 3 hours to complete. This seems excessive
I have configured nova.conf for deployment concurrency (from the
tuning link above) and configured the heat.conf 'num_engine_workers'
to be 32 taking in to account this bug
https://bugzilla.redhat.com/show_bug.cgi?id=1370516
So my question is how do I tune my Undercloud to speed up the deployment?
Looking at htop during deployment I can see heat is using many CPUs,
but the work pattern is NOT distributed. What typically happens is all
the CPUs are at 0 to 1 % used apart from one which is at 50 to 100%.
This one CPU id changes regularly, but there is no concurrent
distributed workload across all the CPUs that the heat processes are
running on. Is heat really multi-threaded, or does if have limitations
so it can only really do proper work on one CPU at a time (which I am
seeing in htop)?
Thanks
Charles
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
8:16 a.m.
On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk> wrote:
Some more testing of different amounts of nodes vs time taken for
successful
deployments -
3 controller 3 compute = 1 hour
3 controller 15 compute = 1 hour
3 controller 25 compute = 1 hour 45 mins
3 controller 35 compute = 4 hours
Hello,
i'm now preparing my deployment of 3+2 nodes. I'll check what you
reported and give you some feedback.
Luca
--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
<lorenzetto.luca(a)gmail.com>
8:30 a.m.
Hey Charles,
If you want to deploy a large number of machines, I suggest you deploy a
small configuration (maybe 3 controllers 1 compute) and then run the
overcloud deploy command again with 2 computes, so on and so forth until
you reach your full allocation
Realistically you can probably do a stride of 5 computes each time,
experiment with it a bit, as you get up to the full allocation of nodes you
might run into a race condition bug with assigning computes to nodes and
need to pin nodes (pinning is adding as an ironic property that
overcloud-novacompute-0 goes here, 1 here, so on and so forth).
As for actually solving the deployment issues at scale (instead of this
horrible hack) I'm looking into adding some robustness at the ironic or
tripleo level to these operations. It sounds like you're running more into
node assignment issues rather than pxe issues though.
2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto <
lorenzetto.luca(a)gmail.com>:
On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk>
wrote:
> Some more testing of different amounts of nodes vs time taken for
successful
> deployments -
>
> 3 controller 3 compute = 1 hour
> 3 controller 15 compute = 1 hour
> 3 controller 25 compute = 1 hour 45 mins
> 3 controller 35 compute = 4 hours
Hello,
i'm now preparing my deployment of 3+2 nodes. I'll check what you
reported and give you some feedback.
Luca
--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <
lorenzetto.luca(a)gmail.com>
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
9:09 a.m.
On 03/11/2016 13:30, Justin Kilpatrick wrote:
It sounds like you're running more into node assignment issues
rather
than pxe issues though
Hi,
Thanks for the reply. This is an interesting thought. I already assign
the controllers to specific nodes, but let the compute nodes get
assigned at random.
I will try pinning the compute nodes and see if that helps.
A 50 node deployment is about to complete. It looks like it will take in
the region of 6-7 hours to complete!
I will then assign the remaining compute nodes explicitly and run a
stack update with the extra nodes.
I had previously tried the approach of lots of updates of smaller
numbers of nodes, but I found that for example adding 10 nodes to a
deployment of say 30 nodes would also take an age (hours and hours).
I will report back after more testing
Charles
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
9:45 a.m.
Hi,
So you are implying that tripleO is not really currently able to roll
out large deployments easily as it is is prone to scaling delays/errors?
Is the same true for RH OSP9 (out of the box) as this also uses
tripleO? I would expect exactly the same scaling issues. But surely
OSP9 is designed for large enterprise Openstack installations?
So if OSP9 does work well with large deployments, what are the tripleO
tweaks that make this work (if any)?
Many Thanks
Charles
On 03/11/2016 13:30, Justin Kilpatrick wrote:
Hey Charles,
If you want to deploy a large number of machines, I suggest you deploy
a small configuration (maybe 3 controllers 1 compute) and then run the
overcloud deploy command again with 2 computes, so on and so forth
until you reach your full allocation
Realistically you can probably do a stride of 5 computes each time,
experiment with it a bit, as you get up to the full allocation of
nodes you might run into a race condition bug with assigning computes
to nodes and need to pin nodes (pinning is adding as an ironic
property that overcloud-novacompute-0 goes here, 1 here, so on and so
forth).
As for actually solving the deployment issues at scale (instead of
this horrible hack) I'm looking into adding some robustness at the
ironic or tripleo level to these operations. It sounds like you're
running more into node assignment issues rather than pxe issues though.
2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
<lorenzetto.luca(a)gmail.com <mailto:lorenzetto.luca@gmail.com>>:
On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk
<mailto:cems@ebi.ac.uk>> wrote:
> Some more testing of different amounts of nodes vs time taken
for successful
> deployments -
>
> 3 controller 3 compute = 1 hour
> 3 controller 15 compute = 1 hour
> 3 controller 25 compute = 1 hour 45 mins
> 3 controller 35 compute = 4 hours
Hello,
i'm now preparing my deployment of 3+2 nodes. I'll check what you
reported and give you some feedback.
Luca
--
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)
Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
<lorenzetto.luca(a)gmail.com <mailto:lorenzetto.luca@gmail.com>>
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
https://www.redhat.com/mailman/listinfo/rdo-list
<https://www.redhat.com/mailman/listinfo/rdo-list>
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
<mailto:rdo-list-unsubscribe@redhat.com>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
10:17 a.m.
Hey Charles,
What sort of issues are you seeing now? How did node pinning work out and
did a slow scale up present any more problems?
Deployments tend to be disk and network limited, you don't mention what
sort of disks your machines have but you do note 1g nics, which are doable
but might require some timeout adjustments or other considerations to give
everything time to complete.
On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk> wrote:
Hi,
So you are implying that tripleO is not really currently able to roll out
large deployments easily as it is is prone to scaling delays/errors?
Is the same true for RH OSP9 (out of the box) as this also uses tripleO?
I would expect exactly the same scaling issues. But surely OSP9 is designed
for large enterprise Openstack installations?
So if OSP9 does work well with large deployments, what are the tripleO
tweaks that make this work (if any)?
Many Thanks
Charles
On 03/11/2016 13:30, Justin Kilpatrick wrote:
Hey Charles,
If you want to deploy a large number of machines, I suggest you deploy a
small configuration (maybe 3 controllers 1 compute) and then run the
overcloud deploy command again with 2 computes, so on and so forth until
you reach your full allocation
Realistically you can probably do a stride of 5 computes each time,
experiment with it a bit, as you get up to the full allocation of nodes you
might run into a race condition bug with assigning computes to nodes and
need to pin nodes (pinning is adding as an ironic property that
overcloud-novacompute-0 goes here, 1 here, so on and so forth).
As for actually solving the deployment issues at scale (instead of this
horrible hack) I'm looking into adding some robustness at the ironic or
tripleo level to these operations. It sounds like you're running more into
node assignment issues rather than pxe issues though.
2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto <
lorenzetto.luca(a)gmail.com>:
> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk> wrote:
> > Some more testing of different amounts of nodes vs time taken for
> successful
> > deployments -
> >
> > 3 controller 3 compute = 1 hour
> > 3 controller 15 compute = 1 hour
> > 3 controller 25 compute = 1 hour 45 mins
> > 3 controller 35 compute = 4 hours
>
> Hello,
>
> i'm now preparing my deployment of 3+2 nodes. I'll check what you
> reported and give you some feedback.
>
> Luca
>
>
> --
> "E' assurdo impiegare gli uomini di intelligenza eccellente per fare
> calcoli che potrebbero essere affidati a chiunque se si usassero delle
> macchine"
> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
>
> "Internet è la più grande biblioteca del mondo.
> Ma il problema è che i libri sono tutti sparsi sul pavimento"
> John Allen Paulos, Matematico (1945-vivente)
>
> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <
> lorenzetto.luca(a)gmail.com>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
10:31 a.m.
Hi,
Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
The 1Gb deployment NIC is not really causing the delay. It is very busy
for the time the overcloud image is rolled out (the first 30 to 45 mins
of deployment), but after that (once all the nodes are up and active
with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on
average for the rest of the deployment. For info the NICS in the nodes
for the Overcloud networks are dual bonded 10Gbit.
The deployment I mentioned before (50 nodes) actually completed in 8
hours (which is double the time it took for 35 nodes!)
I am in the process of a new 3 controller 59 compute node deployment
pinning all the nodes as you suggested. The initial overcloud image roll
out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5
hours in and all is running (slowly). It is currently on Step2 (of 5
Steps). I would expect this deployment to take 10 hours on current speed.
Regards
Charles
On 04/11/2016 15:17, Justin Kilpatrick wrote:
Hey Charles,
What sort of issues are you seeing now? How did node pinning work out
and did a slow scale up present any more problems?
Deployments tend to be disk and network limited, you don't mention
what sort of disks your machines have but you do note 1g nics, which
are doable but might require some timeout adjustments or other
considerations to give everything time to complete.
On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
<mailto:cems@ebi.ac.uk>> wrote:
Hi,
So you are implying that tripleO is not really currently able to
roll out large deployments easily as it is is prone to scaling
delays/errors?
Is the same true for RH OSP9 (out of the box) as this also uses
tripleO? I would expect exactly the same scaling issues. But
surely OSP9 is designed for large enterprise Openstack installations?
So if OSP9 does work well with large deployments, what are the
tripleO tweaks that make this work (if any)?
Many Thanks
Charles
On 03/11/2016 13:30, Justin Kilpatrick wrote:
> Hey Charles,
>
> If you want to deploy a large number of machines, I suggest you
> deploy a small configuration (maybe 3 controllers 1 compute) and
> then run the overcloud deploy command again with 2 computes, so
> on and so forth until you reach your full allocation
>
> Realistically you can probably do a stride of 5 computes each
> time, experiment with it a bit, as you get up to the full
> allocation of nodes you might run into a race condition bug with
> assigning computes to nodes and need to pin nodes (pinning is
> adding as an ironic property that overcloud-novacompute-0 goes
> here, 1 here, so on and so forth).
>
> As for actually solving the deployment issues at scale (instead
> of this horrible hack) I'm looking into adding some robustness at
> the ironic or tripleo level to these operations. It sounds like
> you're running more into node assignment issues rather than pxe
> issues though.
>
> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
> <lorenzetto.luca(a)gmail.com <mailto:lorenzetto.luca@gmail.com>>:
>
> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk
> <mailto:cems@ebi.ac.uk>> wrote:
> > Some more testing of different amounts of nodes vs time
> taken for successful
> > deployments -
> >
> > 3 controller 3 compute = 1 hour
> > 3 controller 15 compute = 1 hour
> > 3 controller 25 compute = 1 hour 45 mins
> > 3 controller 35 compute = 4 hours
>
> Hello,
>
> i'm now preparing my deployment of 3+2 nodes. I'll check what you
> reported and give you some feedback.
>
> Luca
>
>
> --
> "E' assurdo impiegare gli uomini di intelligenza eccellente
> per fare
> calcoli che potrebbero essere affidati a chiunque se si
> usassero delle
> macchine"
> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
>
> "Internet è la più grande biblioteca del mondo.
> Ma il problema è che i libri sono tutti sparsi sul pavimento"
> John Allen Paulos, Matematico (1945-vivente)
>
> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
> <lorenzetto.luca(a)gmail.com <mailto:lorenzetto.luca@gmail.com>>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
> https://www.redhat.com/mailman/listinfo/rdo-list
> <https://www.redhat.com/mailman/listinfo/rdo-list>
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
> <mailto:rdo-list-unsubscribe@redhat.com>
>
>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel:+44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
5:25 p.m.
Hi Charles,
This definitely looks a bit strange to me, as we do deploys around 42
nodes and it takes around 2 hours to do so, similar to your setup (1G
link for provisoning, bonded 10G for everything else).
Would it be possible for you to run an sosreport on your undercloud and
provide it somewhere (if you are comfortable doing so). Also, can you
show us the output of
openstack stack list --nested
And most importantly, if we can get a fully copy of the output of the
overcloud deploy command, that has timestamps against when ever stack is
created/finished, so we can try and narrow down where all the time is
being spent.
You note that you have quite a powerful undercloud (294GB of Memory and
64 cpus), and we have had issues in the past with very powerful
underclouds, because the Openstack components try and tune themselves
around the hardware they are running on and get it wrong for bigger servers.
Are we able to get an output from "sar" or some other tool you are using
to track cpu and memory usage during the deployment? I'd like to check
those values look sane.
Thanks in advance,
Graeme
On 05/11/16 01:31, Charles Short wrote:
Hi,
Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
The 1Gb deployment NIC is not really causing the delay. It is very busy
for the time the overcloud image is rolled out (the first 30 to 45 mins
of deployment), but after that (once all the nodes are up and active
with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on
average for the rest of the deployment. For info the NICS in the nodes
for the Overcloud networks are dual bonded 10Gbit.
The deployment I mentioned before (50 nodes) actually completed in 8
hours (which is double the time it took for 35 nodes!)
I am in the process of a new 3 controller 59 compute node deployment
pinning all the nodes as you suggested. The initial overcloud image roll
out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5
hours in and all is running (slowly). It is currently on Step2 (of 5
Steps). I would expect this deployment to take 10 hours on current speed.
Regards
Charles
On 04/11/2016 15:17, Justin Kilpatrick wrote:
> Hey Charles,
>
> What sort of issues are you seeing now? How did node pinning work out
> and did a slow scale up present any more problems?
>
> Deployments tend to be disk and network limited, you don't mention
> what sort of disks your machines have but you do note 1g nics, which
> are doable but might require some timeout adjustments or other
> considerations to give everything time to complete.
>
> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
> <mailto:cems@ebi.ac.uk>> wrote:
>
> Hi,
>
> So you are implying that tripleO is not really currently able to
> roll out large deployments easily as it is is prone to scaling
> delays/errors?
> Is the same true for RH OSP9 (out of the box) as this also uses
> tripleO? I would expect exactly the same scaling issues. But
> surely OSP9 is designed for large enterprise Openstack installations?
> So if OSP9 does work well with large deployments, what are the
> tripleO tweaks that make this work (if any)?
>
> Many Thanks
>
> Charles
>
> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>> Hey Charles,
>>
>> If you want to deploy a large number of machines, I suggest you
>> deploy a small configuration (maybe 3 controllers 1 compute) and
>> then run the overcloud deploy command again with 2 computes, so
>> on and so forth until you reach your full allocation
>>
>> Realistically you can probably do a stride of 5 computes each
>> time, experiment with it a bit, as you get up to the full
>> allocation of nodes you might run into a race condition bug with
>> assigning computes to nodes and need to pin nodes (pinning is
>> adding as an ironic property that overcloud-novacompute-0 goes
>> here, 1 here, so on and so forth).
>>
>> As for actually solving the deployment issues at scale (instead
>> of this horrible hack) I'm looking into adding some robustness at
>> the ironic or tripleo level to these operations. It sounds like
>> you're running more into node assignment issues rather than pxe
>> issues though.
>>
>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
>> <lorenzetto.luca(a)gmail.com <mailto:lorenzetto.luca@gmail.com>>:
>>
>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk
>> <mailto:cems@ebi.ac.uk>> wrote:
>> > Some more testing of different amounts of nodes vs time
>> taken for successful
>> > deployments -
>> >
>> > 3 controller 3 compute = 1 hour
>> > 3 controller 15 compute = 1 hour
>> > 3 controller 25 compute = 1 hour 45 mins
>> > 3 controller 35 compute = 4 hours
>>
>> Hello,
>>
>> i'm now preparing my deployment of 3+2 nodes. I'll check what
you
>> reported and give you some feedback.
>>
>> Luca
>>
>>
>> --
>> "E' assurdo impiegare gli uomini di intelligenza eccellente
>> per fare
>> calcoli che potrebbero essere affidati a chiunque se si
>> usassero delle
>> macchine"
>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
>>
>> "Internet è la più grande biblioteca del mondo.
>> Ma il problema è che i libri sono tutti sparsi sul pavimento"
>> John Allen Paulos, Matematico (1945-vivente)
>>
>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
>> <lorenzetto.luca(a)gmail.com
<mailto:lorenzetto.luca@gmail.com>>
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>> https://www.redhat.com/mailman/listinfo/rdo-list
>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>
>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>> <mailto:rdo-list-unsubscribe@redhat.com>
>>
>>
>
> --
> Charles Short
> Cloud Engineer
> Virtualization and Cloud Team
> European Bioinformatics Institute (EMBL-EBI)
> Tel: +44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205>
>
>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
--
Graeme Gillies
Principal Systems Administrator
Openstack Infrastructure
Red Hat Australia
10:18 a.m.
Hi,
Just some feedback on this thread.
I have redeployed several times and have begun to suspect DNS as being
the cause for delays (just a guess as the deployment always competes
with no obvious errors)
I had a look at the local hosts files on the nodes during deployment and
can see that lots of them (not all) are incorrectly formatted as they
contain '\n'.
For example a small part of one hosts file -
<<
\n10.0.7.30 overcloud-novacompute-32.localdomain overcloud-novacompute-32
192.168.0.39 overcloud-novacompute-32.external.localdomain
overcloud-novacompute-32.external
10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
overcloud-novacompute-32.internalapi
10.35.5.67 overcloud-novacompute-32.storage.localdomain
overcloud-novacompute-32.storage
192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
overcloud-novacompute-32.storagemgmt
10.0.8.39 overcloud-novacompute-32.tenant.localdomain
overcloud-novacompute-32.tenant
192.168.0.39 overcloud-novacompute-32.management.localdomain
overcloud-novacompute-32.management
192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
overcloud-novacompute-32.ctlplane
\n10.0.7.21 overcloud-novacompute-33.localdomain overcloud-novacompute-33
>
I wondered if maybe the image I was using was the issue so I tried the
RH OSP9 official image - Same hosts file formatting issues in deployment.
Maybe a workaround would be to change nsswitch.conf in the image to look
up from DNS first - my Undercloud dnsmasq server - and have this
populated with the correct entries from a node (once all nodes are
pingable).
Charles
On 06/11/2016 23:25, Graeme Gillies wrote:
Hi Charles,
This definitely looks a bit strange to me, as we do deploys around 42
nodes and it takes around 2 hours to do so, similar to your setup (1G
link for provisoning, bonded 10G for everything else).
Would it be possible for you to run an sosreport on your undercloud and
provide it somewhere (if you are comfortable doing so). Also, can you
show us the output of
openstack stack list --nested
And most importantly, if we can get a fully copy of the output of the
overcloud deploy command, that has timestamps against when ever stack is
created/finished, so we can try and narrow down where all the time is
being spent.
You note that you have quite a powerful undercloud (294GB of Memory and
64 cpus), and we have had issues in the past with very powerful
underclouds, because the Openstack components try and tune themselves
around the hardware they are running on and get it wrong for bigger servers.
Are we able to get an output from "sar" or some other tool you are using
to track cpu and memory usage during the deployment? I'd like to check
those values look sane.
Thanks in advance,
Graeme
On 05/11/16 01:31, Charles Short wrote:
> Hi,
>
> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
> The 1Gb deployment NIC is not really causing the delay. It is very busy
> for the time the overcloud image is rolled out (the first 30 to 45 mins
> of deployment), but after that (once all the nodes are up and active
> with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on
> average for the rest of the deployment. For info the NICS in the nodes
> for the Overcloud networks are dual bonded 10Gbit.
>
> The deployment I mentioned before (50 nodes) actually completed in 8
> hours (which is double the time it took for 35 nodes!)
>
> I am in the process of a new 3 controller 59 compute node deployment
> pinning all the nodes as you suggested. The initial overcloud image roll
> out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5
> hours in and all is running (slowly). It is currently on Step2 (of 5
> Steps). I would expect this deployment to take 10 hours on current speed.
>
> Regards
>
> Charles
>
> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>> Hey Charles,
>>
>> What sort of issues are you seeing now? How did node pinning work out
>> and did a slow scale up present any more problems?
>>
>> Deployments tend to be disk and network limited, you don't mention
>> what sort of disks your machines have but you do note 1g nics, which
>> are doable but might require some timeout adjustments or other
>> considerations to give everything time to complete.
>>
>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
>> <mailto:cems@ebi.ac.uk>> wrote:
>>
>> Hi,
>>
>> So you are implying that tripleO is not really currently able to
>> roll out large deployments easily as it is is prone to scaling
>> delays/errors?
>> Is the same true for RH OSP9 (out of the box) as this also uses
>> tripleO? I would expect exactly the same scaling issues. But
>> surely OSP9 is designed for large enterprise Openstack installations?
>> So if OSP9 does work well with large deployments, what are the
>> tripleO tweaks that make this work (if any)?
>>
>> Many Thanks
>>
>> Charles
>>
>> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>> Hey Charles,
>>>
>>> If you want to deploy a large number of machines, I suggest you
>>> deploy a small configuration (maybe 3 controllers 1 compute) and
>>> then run the overcloud deploy command again with 2 computes, so
>>> on and so forth until you reach your full allocation
>>>
>>> Realistically you can probably do a stride of 5 computes each
>>> time, experiment with it a bit, as you get up to the full
>>> allocation of nodes you might run into a race condition bug with
>>> assigning computes to nodes and need to pin nodes (pinning is
>>> adding as an ironic property that overcloud-novacompute-0 goes
>>> here, 1 here, so on and so forth).
>>>
>>> As for actually solving the deployment issues at scale (instead
>>> of this horrible hack) I'm looking into adding some robustness at
>>> the ironic or tripleo level to these operations. It sounds like
>>> you're running more into node assignment issues rather than pxe
>>> issues though.
>>>
>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
>>> <lorenzetto.luca(a)gmail.com
<mailto:lorenzetto.luca@gmail.com>>:
>>>
>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <cems(a)ebi.ac.uk
>>> <mailto:cems@ebi.ac.uk>> wrote:
>>> > Some more testing of different amounts of nodes vs time
>>> taken for successful
>>> > deployments -
>>> >
>>> > 3 controller 3 compute = 1 hour
>>> > 3 controller 15 compute = 1 hour
>>> > 3 controller 25 compute = 1 hour 45 mins
>>> > 3 controller 35 compute = 4 hours
>>>
>>> Hello,
>>>
>>> i'm now preparing my deployment of 3+2 nodes. I'll check
what you
>>> reported and give you some feedback.
>>>
>>> Luca
>>>
>>>
>>> --
>>> "E' assurdo impiegare gli uomini di intelligenza
eccellente
>>> per fare
>>> calcoli che potrebbero essere affidati a chiunque se si
>>> usassero delle
>>> macchine"
>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
>>>
>>> "Internet è la più grande biblioteca del mondo.
>>> Ma il problema è che i libri sono tutti sparsi sul pavimento"
>>> John Allen Paulos, Matematico (1945-vivente)
>>>
>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
>>> <lorenzetto.luca(a)gmail.com
<mailto:lorenzetto.luca@gmail.com>>
>>>
>>> _______________________________________________
>>> rdo-list mailing list
>>> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>
>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>> <mailto:rdo-list-unsubscribe@redhat.com>
>>>
>>>
>> --
>> Charles Short
>> Cloud Engineer
>> Virtualization and Cloud Team
>> European Bioinformatics Institute (EMBL-EBI)
>> Tel: +44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205>
>>
>>
> --
> Charles Short
> Cloud Engineer
> Virtualization and Cloud Team
> European Bioinformatics Institute (EMBL-EBI)
> Tel: +44 (0)1223 494205
>
>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
9:13 p.m.
On 10/11/16 02:18, Charles Short wrote:
Hi,
Just some feedback on this thread.
I have redeployed several times and have begun to suspect DNS as being
the cause for delays (just a guess as the deployment always competes
with no obvious errors)
I had a look at the local hosts files on the nodes during deployment and
can see that lots of them (not all) are incorrectly formatted as they
contain '\n'.
For example a small part of one hosts file -
<<
\n10.0.7.30 overcloud-novacompute-32.localdomain overcloud-novacompute-32
192.168.0.39 overcloud-novacompute-32.external.localdomain
overcloud-novacompute-32.external
10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
overcloud-novacompute-32.internalapi
10.35.5.67 overcloud-novacompute-32.storage.localdomain
overcloud-novacompute-32.storage
192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
overcloud-novacompute-32.storagemgmt
10.0.8.39 overcloud-novacompute-32.tenant.localdomain
overcloud-novacompute-32.tenant
192.168.0.39 overcloud-novacompute-32.management.localdomain
overcloud-novacompute-32.management
192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
overcloud-novacompute-32.ctlplane
\n10.0.7.21 overcloud-novacompute-33.localdomain overcloud-novacompute-33
>>
I wondered if maybe the image I was using was the issue so I tried the
RH OSP9 official image - Same hosts file formatting issues in deployment.
Maybe a workaround would be to change nsswitch.conf in the image to look
up from DNS first - my Undercloud dnsmasq server - and have this
populated with the correct entries from a node (once all nodes are
pingable).
Charles
Hi Charles,
If you are getting formatting issues in /etc/hosts, it's possible that
the templates directory you are using might have problems, especially if
it's been edited on windows machines. Are you using unmodified templates
from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9
images will not match RDO Newton templates, as RHOS 9 is mitaka, and
overcloud images contain puppet modules which must sync with the
templates used on the undercloud.
If you are using the templates in
/usr/share/openstack-tripleo-heat-templates, can you give the output (if
any) from
rpm -V openstack-tripleo-heat-templates
Also perhaps getting a copy of your full overcloud deploy command will
help shed some light as well.
Thanks in advance,
Graeme
On 06/11/2016 23:25, Graeme Gillies wrote:
> Hi Charles,
>
> This definitely looks a bit strange to me, as we do deploys around 42
> nodes and it takes around 2 hours to do so, similar to your setup (1G
> link for provisoning, bonded 10G for everything else).
>
> Would it be possible for you to run an sosreport on your undercloud and
> provide it somewhere (if you are comfortable doing so). Also, can you
> show us the output of
>
> openstack stack list --nested
>
> And most importantly, if we can get a fully copy of the output of the
> overcloud deploy command, that has timestamps against when ever stack is
> created/finished, so we can try and narrow down where all the time is
> being spent.
>
> You note that you have quite a powerful undercloud (294GB of Memory and
> 64 cpus), and we have had issues in the past with very powerful
> underclouds, because the Openstack components try and tune themselves
> around the hardware they are running on and get it wrong for bigger
> servers.
>
> Are we able to get an output from "sar" or some other tool you are using
> to track cpu and memory usage during the deployment? I'd like to check
> those values look sane.
>
> Thanks in advance,
>
> Graeme
>
> On 05/11/16 01:31, Charles Short wrote:
>> Hi,
>>
>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
>> The 1Gb deployment NIC is not really causing the delay. It is very busy
>> for the time the overcloud image is rolled out (the first 30 to 45 mins
>> of deployment), but after that (once all the nodes are up and active
>> with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on
>> average for the rest of the deployment. For info the NICS in the nodes
>> for the Overcloud networks are dual bonded 10Gbit.
>>
>> The deployment I mentioned before (50 nodes) actually completed in 8
>> hours (which is double the time it took for 35 nodes!)
>>
>> I am in the process of a new 3 controller 59 compute node deployment
>> pinning all the nodes as you suggested. The initial overcloud image roll
>> out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5
>> hours in and all is running (slowly). It is currently on Step2 (of 5
>> Steps). I would expect this deployment to take 10 hours on current
>> speed.
>>
>> Regards
>>
>> Charles
>>
>> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>>> Hey Charles,
>>>
>>> What sort of issues are you seeing now? How did node pinning work out
>>> and did a slow scale up present any more problems?
>>>
>>> Deployments tend to be disk and network limited, you don't mention
>>> what sort of disks your machines have but you do note 1g nics, which
>>> are doable but might require some timeout adjustments or other
>>> considerations to give everything time to complete.
>>>
>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>
>>> Hi,
>>>
>>> So you are implying that tripleO is not really currently able to
>>> roll out large deployments easily as it is is prone to scaling
>>> delays/errors?
>>> Is the same true for RH OSP9 (out of the box) as this also uses
>>> tripleO? I would expect exactly the same scaling issues. But
>>> surely OSP9 is designed for large enterprise Openstack
>>> installations?
>>> So if OSP9 does work well with large deployments, what are the
>>> tripleO tweaks that make this work (if any)?
>>>
>>> Many Thanks
>>>
>>> Charles
>>>
>>> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>>> Hey Charles,
>>>>
>>>> If you want to deploy a large number of machines, I suggest you
>>>> deploy a small configuration (maybe 3 controllers 1 compute) and
>>>> then run the overcloud deploy command again with 2 computes, so
>>>> on and so forth until you reach your full allocation
>>>>
>>>> Realistically you can probably do a stride of 5 computes each
>>>> time, experiment with it a bit, as you get up to the full
>>>> allocation of nodes you might run into a race condition bug with
>>>> assigning computes to nodes and need to pin nodes (pinning is
>>>> adding as an ironic property that overcloud-novacompute-0 goes
>>>> here, 1 here, so on and so forth).
>>>>
>>>> As for actually solving the deployment issues at scale (instead
>>>> of this horrible hack) I'm looking into adding some robustness
at
>>>> the ironic or tripleo level to these operations. It sounds like
>>>> you're running more into node assignment issues rather than pxe
>>>> issues though.
>>>>
>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
>>>> <lorenzetto.luca(a)gmail.com
<mailto:lorenzetto.luca@gmail.com>>:
>>>>
>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short
<cems(a)ebi.ac.uk
>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>> > Some more testing of different amounts of nodes vs time
>>>> taken for successful
>>>> > deployments -
>>>> >
>>>> > 3 controller 3 compute = 1 hour
>>>> > 3 controller 15 compute = 1 hour
>>>> > 3 controller 25 compute = 1 hour 45 mins
>>>> > 3 controller 35 compute = 4 hours
>>>>
>>>> Hello,
>>>>
>>>> i'm now preparing my deployment of 3+2 nodes. I'll
check
>>>> what you
>>>> reported and give you some feedback.
>>>>
>>>> Luca
>>>>
>>>>
>>>> --
>>>> "E' assurdo impiegare gli uomini di intelligenza
eccellente
>>>> per fare
>>>> calcoli che potrebbero essere affidati a chiunque se si
>>>> usassero delle
>>>> macchine"
>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico
>>>> (1646-1716)
>>>>
>>>> "Internet è la più grande biblioteca del mondo.
>>>> Ma il problema è che i libri sono tutti sparsi sul
pavimento"
>>>> John Allen Paulos, Matematico (1945-vivente)
>>>>
>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
>>>> <lorenzetto.luca(a)gmail.com
>>>> <mailto:lorenzetto.luca@gmail.com>>
>>>>
>>>> _______________________________________________
>>>> rdo-list mailing list
>>>> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>>
>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>> <mailto:rdo-list-unsubscribe@redhat.com>
>>>>
>>>>
>>> --
>>> Charles Short
>>> Cloud Engineer
>>> Virtualization and Cloud Team
>>> European Bioinformatics Institute (EMBL-EBI)
>>> Tel: +44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205>
>>>
>>>
>> --
>> Charles Short
>> Cloud Engineer
>> Virtualization and Cloud Team
>> European Bioinformatics Institute (EMBL-EBI)
>> Tel: +44 (0)1223 494205
>>
>>
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>
>
--
Graeme Gillies
Principal Systems Administrator
Openstack Infrastructure
Red Hat Australia
2:20 a.m.
Hi,
Deploy command here
http://pastebin.com/xNZXTWPE
no output from rpm command.
Yes re OSP9 images I was just interested how they behaved early on in
the deployment before any puppet errors (cloud init etc).
Not a good test, just morbid fascination out of desperation.
No Windows involved, and I have not altered the main puppet template
directory at all.
I am going to try and update the Undercloud to the latest stable, use
the provided images and see how that goes.
If all else fails I will install OSP9 and consider myself exhausted from
all the swimming upstream ;)
Charles
On 10/11/2016 03:13, Graeme Gillies wrote:
On 10/11/16 02:18, Charles Short wrote:
> Hi,
>
> Just some feedback on this thread.
>
> I have redeployed several times and have begun to suspect DNS as being
> the cause for delays (just a guess as the deployment always competes
> with no obvious errors)
> I had a look at the local hosts files on the nodes during deployment and
> can see that lots of them (not all) are incorrectly formatted as they
> contain '\n'.
>
> For example a small part of one hosts file -
> <<
> \n10.0.7.30 overcloud-novacompute-32.localdomain overcloud-novacompute-32
> 192.168.0.39 overcloud-novacompute-32.external.localdomain
> overcloud-novacompute-32.external
> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
> overcloud-novacompute-32.internalapi
> 10.35.5.67 overcloud-novacompute-32.storage.localdomain
> overcloud-novacompute-32.storage
> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
> overcloud-novacompute-32.storagemgmt
> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain
> overcloud-novacompute-32.tenant
> 192.168.0.39 overcloud-novacompute-32.management.localdomain
> overcloud-novacompute-32.management
> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
> overcloud-novacompute-32.ctlplane
> \n10.0.7.21 overcloud-novacompute-33.localdomain overcloud-novacompute-33
> I wondered if maybe the image I was using was the issue so I tried the
> RH OSP9 official image - Same hosts file formatting issues in deployment.
> Maybe a workaround would be to change nsswitch.conf in the image to look
> up from DNS first - my Undercloud dnsmasq server - and have this
> populated with the correct entries from a node (once all nodes are
> pingable).
>
> Charles
Hi Charles,
If you are getting formatting issues in /etc/hosts, it's possible that
the templates directory you are using might have problems, especially if
it's been edited on windows machines. Are you using unmodified templates
from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9
images will not match RDO Newton templates, as RHOS 9 is mitaka, and
overcloud images contain puppet modules which must sync with the
templates used on the undercloud.
If you are using the templates in
/usr/share/openstack-tripleo-heat-templates, can you give the output (if
any) from
rpm -V openstack-tripleo-heat-templates
Also perhaps getting a copy of your full overcloud deploy command will
help shed some light as well.
Thanks in advance,
Graeme
> On 06/11/2016 23:25, Graeme Gillies wrote:
>> Hi Charles,
>>
>> This definitely looks a bit strange to me, as we do deploys around 42
>> nodes and it takes around 2 hours to do so, similar to your setup (1G
>> link for provisoning, bonded 10G for everything else).
>>
>> Would it be possible for you to run an sosreport on your undercloud and
>> provide it somewhere (if you are comfortable doing so). Also, can you
>> show us the output of
>>
>> openstack stack list --nested
>>
>> And most importantly, if we can get a fully copy of the output of the
>> overcloud deploy command, that has timestamps against when ever stack is
>> created/finished, so we can try and narrow down where all the time is
>> being spent.
>>
>> You note that you have quite a powerful undercloud (294GB of Memory and
>> 64 cpus), and we have had issues in the past with very powerful
>> underclouds, because the Openstack components try and tune themselves
>> around the hardware they are running on and get it wrong for bigger
>> servers.
>>
>> Are we able to get an output from "sar" or some other tool you are
using
>> to track cpu and memory usage during the deployment? I'd like to check
>> those values look sane.
>>
>> Thanks in advance,
>>
>> Graeme
>>
>> On 05/11/16 01:31, Charles Short wrote:
>>> Hi,
>>>
>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
>>> The 1Gb deployment NIC is not really causing the delay. It is very busy
>>> for the time the overcloud image is rolled out (the first 30 to 45 mins
>>> of deployment), but after that (once all the nodes are up and active
>>> with an ip address (pingable)) ,the bandwidth is a fraction of 1Gbps on
>>> average for the rest of the deployment. For info the NICS in the nodes
>>> for the Overcloud networks are dual bonded 10Gbit.
>>>
>>> The deployment I mentioned before (50 nodes) actually completed in 8
>>> hours (which is double the time it took for 35 nodes!)
>>>
>>> I am in the process of a new 3 controller 59 compute node deployment
>>> pinning all the nodes as you suggested. The initial overcloud image roll
>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am now 4.5
>>> hours in and all is running (slowly). It is currently on Step2 (of 5
>>> Steps). I would expect this deployment to take 10 hours on current
>>> speed.
>>>
>>> Regards
>>>
>>> Charles
>>>
>>> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>>>> Hey Charles,
>>>>
>>>> What sort of issues are you seeing now? How did node pinning work out
>>>> and did a slow scale up present any more problems?
>>>>
>>>> Deployments tend to be disk and network limited, you don't mention
>>>> what sort of disks your machines have but you do note 1g nics, which
>>>> are doable but might require some timeout adjustments or other
>>>> considerations to give everything time to complete.
>>>>
>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> So you are implying that tripleO is not really currently able to
>>>> roll out large deployments easily as it is is prone to scaling
>>>> delays/errors?
>>>> Is the same true for RH OSP9 (out of the box) as this also uses
>>>> tripleO? I would expect exactly the same scaling issues. But
>>>> surely OSP9 is designed for large enterprise Openstack
>>>> installations?
>>>> So if OSP9 does work well with large deployments, what are the
>>>> tripleO tweaks that make this work (if any)?
>>>>
>>>> Many Thanks
>>>>
>>>> Charles
>>>>
>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>>>> Hey Charles,
>>>>>
>>>>> If you want to deploy a large number of machines, I suggest
you
>>>>> deploy a small configuration (maybe 3 controllers 1 compute)
and
>>>>> then run the overcloud deploy command again with 2 computes,
so
>>>>> on and so forth until you reach your full allocation
>>>>>
>>>>> Realistically you can probably do a stride of 5 computes each
>>>>> time, experiment with it a bit, as you get up to the full
>>>>> allocation of nodes you might run into a race condition bug
with
>>>>> assigning computes to nodes and need to pin nodes (pinning is
>>>>> adding as an ironic property that overcloud-novacompute-0 goes
>>>>> here, 1 here, so on and so forth).
>>>>>
>>>>> As for actually solving the deployment issues at scale
(instead
>>>>> of this horrible hack) I'm looking into adding some
robustness at
>>>>> the ironic or tripleo level to these operations. It sounds
like
>>>>> you're running more into node assignment issues rather than
pxe
>>>>> issues though.
>>>>>
>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
>>>>> <lorenzetto.luca(a)gmail.com
<mailto:lorenzetto.luca@gmail.com>>:
>>>>>
>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short
<cems(a)ebi.ac.uk
>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>> > Some more testing of different amounts of nodes vs
time
>>>>> taken for successful
>>>>> > deployments -
>>>>> >
>>>>> > 3 controller 3 compute = 1 hour
>>>>> > 3 controller 15 compute = 1 hour
>>>>> > 3 controller 25 compute = 1 hour 45 mins
>>>>> > 3 controller 35 compute = 4 hours
>>>>>
>>>>> Hello,
>>>>>
>>>>> i'm now preparing my deployment of 3+2 nodes. I'll
check
>>>>> what you
>>>>> reported and give you some feedback.
>>>>>
>>>>> Luca
>>>>>
>>>>>
>>>>> --
>>>>> "E' assurdo impiegare gli uomini di intelligenza
eccellente
>>>>> per fare
>>>>> calcoli che potrebbero essere affidati a chiunque se si
>>>>> usassero delle
>>>>> macchine"
>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico
>>>>> (1646-1716)
>>>>>
>>>>> "Internet è la più grande biblioteca del mondo.
>>>>> Ma il problema è che i libri sono tutti sparsi sul
pavimento"
>>>>> John Allen Paulos, Matematico (1945-vivente)
>>>>>
>>>>> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net
,
>>>>> <lorenzetto.luca(a)gmail.com
>>>>> <mailto:lorenzetto.luca@gmail.com>>
>>>>>
>>>>> _______________________________________________
>>>>> rdo-list mailing list
>>>>> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>>>
>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>> <mailto:rdo-list-unsubscribe@redhat.com>
>>>>>
>>>>>
>>>> --
>>>> Charles Short
>>>> Cloud Engineer
>>>> Virtualization and Cloud Team
>>>> European Bioinformatics Institute (EMBL-EBI)
>>>> Tel: +44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205>
>>>>
>>>>
>>> --
>>> Charles Short
>>> Cloud Engineer
>>> Virtualization and Cloud Team
>>> European Bioinformatics Institute (EMBL-EBI)
>>> Tel: +44 (0)1223 494205
>>>
>>>
>>>
>>> _______________________________________________
>>> rdo-list mailing list
>>> rdo-list(a)redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
10:48 a.m.
Update -
I updated Undercloud to latest stable Newton release, and used the
provided CentOS Overcloud images. I first completed a small test
deployment no problem (3 controller 3 compute) .
I then deployed again with a larger environment (40 compute, 3 controllers).
When the nodes were up and ACTIVE/pingable early in deployment I checked
the hosts files. This time no formatting errors.
However during deployment there were lots of long pauses and I noticed
plenty of these sorts of messages in the nova logs during the pauses -
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance:
f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise
exceptions.ConnectTimeout(msg)
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed]
ConnectTimeout: Request to
http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f0...
timed out
While this was happening I could not use nova from the Undercloud at all-
source stackrc
nova list
ERROR (ClientException): The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5)
After 2 hours of deployment and only on Step 1 of 5 the deployment fails
with -
ERROR: Timed out waiting for a reply to message ID
971157a211e549998bb7a6f6e494688b
note - I have a timeout value way over 2 hours in the deployment
commands (2000)
Post failed deployment I still cannot use nova. Looks like the
Undercloud is very unhappy (same error as above)
The only way I can get the Undercloud working again is to restart all
services (restarting nova alone does not work)
sudo systemctl restart neutron*
sudo systemctl restart openstack*
I think I may try OSP9 as I am running out of ideas. Either that or
giving Openstack-Ansible a try.....
Charles
On 10/11/2016 08:20, Charles Short wrote:
Hi,
Deploy command here
http://pastebin.com/xNZXTWPE
no output from rpm command.
Yes re OSP9 images I was just interested how they behaved early on in
the deployment before any puppet errors (cloud init etc).
Not a good test, just morbid fascination out of desperation.
No Windows involved, and I have not altered the main puppet template
directory at all.
I am going to try and update the Undercloud to the latest stable, use
the provided images and see how that goes.
If all else fails I will install OSP9 and consider myself exhausted
from all the swimming upstream ;)
Charles
On 10/11/2016 03:13, Graeme Gillies wrote:
> On 10/11/16 02:18, Charles Short wrote:
>> Hi,
>>
>> Just some feedback on this thread.
>>
>> I have redeployed several times and have begun to suspect DNS as being
>> the cause for delays (just a guess as the deployment always competes
>> with no obvious errors)
>> I had a look at the local hosts files on the nodes during deployment
>> and
>> can see that lots of them (not all) are incorrectly formatted as they
>> contain '\n'.
>>
>> For example a small part of one hosts file -
>> <<
>> \n10.0.7.30 overcloud-novacompute-32.localdomain
>> overcloud-novacompute-32
>> 192.168.0.39 overcloud-novacompute-32.external.localdomain
>> overcloud-novacompute-32.external
>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
>> overcloud-novacompute-32.internalapi
>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain
>> overcloud-novacompute-32.storage
>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
>> overcloud-novacompute-32.storagemgmt
>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain
>> overcloud-novacompute-32.tenant
>> 192.168.0.39 overcloud-novacompute-32.management.localdomain
>> overcloud-novacompute-32.management
>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
>> overcloud-novacompute-32.ctlplane
>> \n10.0.7.21 overcloud-novacompute-33.localdomain
>> overcloud-novacompute-33
>> I wondered if maybe the image I was using was the issue so I tried the
>> RH OSP9 official image - Same hosts file formatting issues in
>> deployment.
>> Maybe a workaround would be to change nsswitch.conf in the image to
>> look
>> up from DNS first - my Undercloud dnsmasq server - and have this
>> populated with the correct entries from a node (once all nodes are
>> pingable).
>>
>> Charles
> Hi Charles,
>
> If you are getting formatting issues in /etc/hosts, it's possible that
> the templates directory you are using might have problems, especially if
> it's been edited on windows machines. Are you using unmodified templates
> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9
> images will not match RDO Newton templates, as RHOS 9 is mitaka, and
> overcloud images contain puppet modules which must sync with the
> templates used on the undercloud.
>
> If you are using the templates in
> /usr/share/openstack-tripleo-heat-templates, can you give the output (if
> any) from
>
> rpm -V openstack-tripleo-heat-templates
>
> Also perhaps getting a copy of your full overcloud deploy command will
> help shed some light as well.
>
> Thanks in advance,
>
> Graeme
>
>> On 06/11/2016 23:25, Graeme Gillies wrote:
>>> Hi Charles,
>>>
>>> This definitely looks a bit strange to me, as we do deploys around 42
>>> nodes and it takes around 2 hours to do so, similar to your setup (1G
>>> link for provisoning, bonded 10G for everything else).
>>>
>>> Would it be possible for you to run an sosreport on your undercloud
>>> and
>>> provide it somewhere (if you are comfortable doing so). Also, can you
>>> show us the output of
>>>
>>> openstack stack list --nested
>>>
>>> And most importantly, if we can get a fully copy of the output of the
>>> overcloud deploy command, that has timestamps against when ever
>>> stack is
>>> created/finished, so we can try and narrow down where all the time is
>>> being spent.
>>>
>>> You note that you have quite a powerful undercloud (294GB of Memory
>>> and
>>> 64 cpus), and we have had issues in the past with very powerful
>>> underclouds, because the Openstack components try and tune themselves
>>> around the hardware they are running on and get it wrong for bigger
>>> servers.
>>>
>>> Are we able to get an output from "sar" or some other tool you are
>>> using
>>> to track cpu and memory usage during the deployment? I'd like to check
>>> those values look sane.
>>>
>>> Thanks in advance,
>>>
>>> Graeme
>>>
>>> On 05/11/16 01:31, Charles Short wrote:
>>>> Hi,
>>>>
>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
>>>> The 1Gb deployment NIC is not really causing the delay. It is very
>>>> busy
>>>> for the time the overcloud image is rolled out (the first 30 to 45
>>>> mins
>>>> of deployment), but after that (once all the nodes are up and active
>>>> with an ip address (pingable)) ,the bandwidth is a fraction of
>>>> 1Gbps on
>>>> average for the rest of the deployment. For info the NICS in the
>>>> nodes
>>>> for the Overcloud networks are dual bonded 10Gbit.
>>>>
>>>> The deployment I mentioned before (50 nodes) actually completed in 8
>>>> hours (which is double the time it took for 35 nodes!)
>>>>
>>>> I am in the process of a new 3 controller 59 compute node deployment
>>>> pinning all the nodes as you suggested. The initial overcloud
>>>> image roll
>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am
>>>> now 45
>>>> hours in and all is running (slowly). It is currently on Step2 (of 5
>>>> Steps). I would expect this deployment to take 10 hours on current
>>>> speed.
>>>>
>>>> Regards
>>>>
>>>> Charles
>>>>
>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>>>>> Hey Charles,
>>>>>
>>>>> What sort of issues are you seeing now? How did node pinning work
>>>>> out
>>>>> and did a slow scale up present any more problems?
>>>>>
>>>>> Deployments tend to be disk and network limited, you don't
mention
>>>>> what sort of disks your machines have but you do note 1g nics, which
>>>>> are doable but might require some timeout adjustments or other
>>>>> considerations to give everything time to complete.
>>>>>
>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> So you are implying that tripleO is not really currently
>>>>> able to
>>>>> roll out large deployments easily as it is is prone to scaling
>>>>> delays/errors?
>>>>> Is the same true for RH OSP9 (out of the box) as this also
>>>>> uses
>>>>> tripleO? I would expect exactly the same scaling issues. But
>>>>> surely OSP9 is designed for large enterprise Openstack
>>>>> installations?
>>>>> So if OSP9 does work well with large deployments, what are the
>>>>> tripleO tweaks that make this work (if any)?
>>>>>
>>>>> Many Thanks
>>>>>
>>>>> Charles
>>>>>
>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>>>>> Hey Charles,
>>>>>>
>>>>>> If you want to deploy a large number of machines, I
>>>>>> suggest you
>>>>>> deploy a small configuration (maybe 3 controllers 1
>>>>>> compute) and
>>>>>> then run the overcloud deploy command again with 2
>>>>>> computes, so
>>>>>> on and so forth until you reach your full allocation
>>>>>>
>>>>>> Realistically you can probably do a stride of 5 computes
each
>>>>>> time, experiment with it a bit, as you get up to the full
>>>>>> allocation of nodes you might run into a race condition
>>>>>> bug with
>>>>>> assigning computes to nodes and need to pin nodes (pinning
is
>>>>>> adding as an ironic property that overcloud-novacompute-0
>>>>>> goes
>>>>>> here, 1 here, so on and so forth).
>>>>>>
>>>>>> As for actually solving the deployment issues at scale
>>>>>> (instead
>>>>>> of this horrible hack) I'm looking into adding some
>>>>>> robustness at
>>>>>> the ironic or tripleo level to these operations. It sounds
>>>>>> like
>>>>>> you're running more into node assignment issues rather
>>>>>> than pxe
>>>>>> issues though.
>>>>>>
>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj'
Lorenzetto
>>>>>> <lorenzetto.luca(a)gmail.com
>>>>>> <mailto:lorenzetto.luca@gmail.com>>:
>>>>>>
>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short
>>>>>> <cems(a)ebi.acuk
>>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>>> > Some more testing of different amounts of nodes vs
time
>>>>>> taken for successful
>>>>>> > deployments -
>>>>>> >
>>>>>> > 3 controller 3 compute = 1 hour
>>>>>> > 3 controller 15 compute = 1 hour
>>>>>> > 3 controller 25 compute = 1 hour 45 mins
>>>>>> > 3 controller 35 compute = 4 hours
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> i'm now preparing my deployment of 3+2 nodes.
I'll check
>>>>>> what you
>>>>>> reported and give you some feedback.
>>>>>>
>>>>>> Luca
>>>>>>
>>>>>>
>>>>>> --
>>>>>> "E' assurdo impiegare gli uomini di
intelligenza
>>>>>> eccellente
>>>>>> per fare
>>>>>> calcoli che potrebbero essere affidati a chiunque se
si
>>>>>> usassero delle
>>>>>> macchine"
>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico
>>>>>> (1646-1716)
>>>>>>
>>>>>> "Internet è la più grande biblioteca del mondo.
>>>>>> Ma il problema è che i libri sono tutti sparsi sul
>>>>>> pavimento"
>>>>>> John Allen Paulos, Matematico (1945-vivente)
>>>>>>
>>>>>> Luca 'remix_tj' Lorenzetto,
http://www.remixtj.net ,
>>>>>> <lorenzetto.luca(a)gmail.com
>>>>>> <mailto:lorenzetto.luca@gmail.com>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> rdo-list mailing list
>>>>>> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>>>>
>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>>> <mailto:rdo-list-unsubscribe@redhat.com>
>>>>>>
>>>>>>
>>>>> --
>>>>> Charles Short
>>>>> Cloud Engineer
>>>>> Virtualization and Cloud Team
>>>>> European Bioinformatics Institute (EMBL-EBI)
>>>>> Tel: +44 (0)1223 494205
<tel:%2B44%20%280%291223%20494205>
>>>>>
>>>>>
>>>> --
>>>> Charles Short
>>>> Cloud Engineer
>>>> Virtualization and Cloud Team
>>>> European Bioinformatics Institute (EMBL-EBI)
>>>> Tel: +44 (0)1223 494205
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rdo-list mailing list
>>>> rdo-list(a)redhat.com
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>
>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>
>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
12:44 p.m.
________________________________
From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com> on behalf of Charles
Short <cems(a)ebi.ac.uk>
Sent: Friday, November 11, 2016 7:48 PM
To: rdo-list(a)redhat.com
Subject: Re: [rdo-list] [TripleO] Newton large baremetal deployment issues
Update -
I updated Undercloud to latest stable Newton release, and used the
provided CentOS Overcloud images. I first completed a small test
deployment no problem (3 controller 3 compute) .
I then deployed again with a larger environment (40 compute, 3 controllers).
When the nodes were up and ACTIVE/pingable early in deployment I checked
the hosts files. This time no formatting errors.
However during deployment there were lots of long pauses and I noticed
plenty of these sorts of messages in the nova logs during the pauses -
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance:
f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise
exceptions.ConnectTimeout(msg)
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed]
ConnectTimeout: Request to
http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f0...
timed out
While this was happening I could not use nova from the Undercloud at all-
source stackrc
nova list
ERROR (ClientException): The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5)
After 2 hours of deployment and only on Step 1 of 5 the deployment fails
with -
ERROR: Timed out waiting for a reply to message ID
971157a211e549998bb7a6f6e494688b
note - I have a timeout value way over 2 hours in the deployment
commands (2000)
Post failed deployment I still cannot use nova. Looks like the
Undercloud is very unhappy (same error as above)
The only way I can get the Undercloud working again is to restart all
services (restarting nova alone does not work)
sudo systemctl restart neutron*
sudo systemctl restart openstack*
What would happen ?
# openstack stack delete overcloud
Stack deleted cleanly
# shutdown -r now
Boris.
I think I may try OSP9 as I am running out of ideas. Either that or
giving Openstack-Ansible a try.....
Charles
On 10/11/2016 08:20, Charles Short wrote:
Hi,
Deploy command here
http://pastebin.com/xNZXTWPE
[http://pastebin.com/i/facebook.png]<http://pastebin.com/xNZXTWPE>
stack1 - Pastebin.com<http://pastebin.com/xNZXTWPE>
pastebin.com
![http://pastebin.com/i/facebook.png]<http://pastebin.com/xNZXTWPE>](http://pastebin.com/i/facebook.png]%3Chttp://pastebin.com/xNZXTWPE%3E){kind=link}
no output from rpm command.
Yes re OSP9 images I was just interested how they behaved early on in
the deployment before any puppet errors (cloud init etc).
Not a good test, just morbid fascination out of desperation.
No Windows involved, and I have not altered the main puppet template
directory at all.
I am going to try and update the Undercloud to the latest stable, use
the provided images and see how that goes.
If all else fails I will install OSP9 and consider myself exhausted
from all the swimming upstream ;)
Charles
On 10/11/2016 03:13, Graeme Gillies wrote:
> On 10/11/16 02:18, Charles Short wrote:
>> Hi,
>>
>> Just some feedback on this thread.
>>
>> I have redeployed several times and have begun to suspect DNS as being
>> the cause for delays (just a guess as the deployment always competes
>> with no obvious errors)
>> I had a look at the local hosts files on the nodes during deployment
>> and
>> can see that lots of them (not all) are incorrectly formatted as they
>> contain '\n'.
>>
>> For example a small part of one hosts file -
>> <<
>> \n10.0.7.30 overcloud-novacompute-32.localdomain
>> overcloud-novacompute-32
>> 192.168.0.39 overcloud-novacompute-32.external.localdomain
>> overcloud-novacompute-32.external
>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
>> overcloud-novacompute-32.internalapi
>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain
>> overcloud-novacompute-32.storage
>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
>> overcloud-novacompute-32.storagemgmt
>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain
>> overcloud-novacompute-32.tenant
>> 192.168.0.39 overcloud-novacompute-32.management.localdomain
>> overcloud-novacompute-32.management
>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
>> overcloud-novacompute-32.ctlplane
>> \n10.0.7.21 overcloud-novacompute-33.localdomain
>> overcloud-novacompute-33
>> I wondered if maybe the image I was using was the issue so I tried the
>> RH OSP9 official image - Same hosts file formatting issues in
>> deployment.
>> Maybe a workaround would be to change nsswitch.conf in the image to
>> look
>> up from DNS first - my Undercloud dnsmasq server - and have this
>> populated with the correct entries from a node (once all nodes are
>> pingable).
>>
>> Charles
> Hi Charles,
>
> If you are getting formatting issues in /etc/hosts, it's possible that
> the templates directory you are using might have problems, especially if
> it's been edited on windows machines. Are you using unmodified templates
> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9
> images will not match RDO Newton templates, as RHOS 9 is mitaka, and
> overcloud images contain puppet modules which must sync with the
> templates used on the undercloud.
>
> If you are using the templates in
> /usr/share/openstack-tripleo-heat-templates, can you give the output (if
> any) from
>
> rpm -V openstack-tripleo-heat-templates
>
> Also perhaps getting a copy of your full overcloud deploy command will
> help shed some light as well.
>
> Thanks in advance,
>
> Graeme
>
>> On 06/11/2016 23:25, Graeme Gillies wrote:
>>> Hi Charles,
>>>
>>> This definitely looks a bit strange to me, as we do deploys around 42
>>> nodes and it takes around 2 hours to do so, similar to your setup (1G
>>> link for provisoning, bonded 10G for everything else).
>>>
>>> Would it be possible for you to run an sosreport on your undercloud
>>> and
>>> provide it somewhere (if you are comfortable doing so). Also, can you
>>> show us the output of
>>>
>>> openstack stack list --nested
>>>
>>> And most importantly, if we can get a fully copy of the output of the
>>> overcloud deploy command, that has timestamps against when ever
>>> stack is
>>> created/finished, so we can try and narrow down where all the time is
>>> being spent.
>>>
>>> You note that you have quite a powerful undercloud (294GB of Memory
>>> and
>>> 64 cpus), and we have had issues in the past with very powerful
>>> underclouds, because the Openstack components try and tune themselves
>>> around the hardware they are running on and get it wrong for bigger
>>> servers.
>>>
>>> Are we able to get an output from "sar" or some other tool you are
>>> using
>>> to track cpu and memory usage during the deployment? I'd like to check
>>> those values look sane.
>>>
>>> Thanks in advance,
>>>
>>> Graeme
>>>
>>> On 05/11/16 01:31, Charles Short wrote:
>>>> Hi,
>>>>
>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
>>>> The 1Gb deployment NIC is not really causing the delay. It is very
>>>> busy
>>>> for the time the overcloud image is rolled out (the first 30 to 45
>>>> mins
>>>> of deployment), but after that (once all the nodes are up and active
>>>> with an ip address (pingable)) ,the bandwidth is a fraction of
>>>> 1Gbps on
>>>> average for the rest of the deployment. For info the NICS in the
>>>> nodes
>>>> for the Overcloud networks are dual bonded 10Gbit.
>>>>
>>>> The deployment I mentioned before (50 nodes) actually completed in 8
>>>> hours (which is double the time it took for 35 nodes!)
>>>>
>>>> I am in the process of a new 3 controller 59 compute node deployment
>>>> pinning all the nodes as you suggested. The initial overcloud
>>>> image roll
>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am
>>>> now 45
>>>> hours in and all is running (slowly). It is currently on Step2 (of 5
>>>> Steps). I would expect this deployment to take 10 hours on current
>>>> speed.
>>>>
>>>> Regards
>>>>
>>>> Charles
>>>>
>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>>>>> Hey Charles,
>>>>>
>>>>> What sort of issues are you seeing now? How did node pinning work
>>>>> out
>>>>> and did a slow scale up present any more problems?
>>>>>
>>>>> Deployments tend to be disk and network limited, you don't
mention
>>>>> what sort of disks your machines have but you do note 1g nics, which
>>>>> are doable but might require some timeout adjustments or other
>>>>> considerations to give everything time to complete.
>>>>>
>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems(a)ebi.ac.uk
>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> So you are implying that tripleO is not really currently
>>>>> able to
>>>>> roll out large deployments easily as it is is prone to scaling
>>>>> delays/errors?
>>>>> Is the same true for RH OSP9 (out of the box) as this also
>>>>> uses
>>>>> tripleO? I would expect exactly the same scaling issues. But
>>>>> surely OSP9 is designed for large enterprise Openstack
>>>>> installations?
>>>>> So if OSP9 does work well with large deployments, what are the
>>>>> tripleO tweaks that make this work (if any)?
>>>>>
>>>>> Many Thanks
>>>>>
>>>>> Charles
>>>>>
>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>>>>> Hey Charles,
>>>>>>
>>>>>> If you want to deploy a large number of machines, I
>>>>>> suggest you
>>>>>> deploy a small configuration (maybe 3 controllers 1
>>>>>> compute) and
>>>>>> then run the overcloud deploy command again with 2
>>>>>> computes, so
>>>>>> on and so forth until you reach your full allocation
>>>>>>
>>>>>> Realistically you can probably do a stride of 5 computes
each
>>>>>> time, experiment with it a bit, as you get up to the full
>>>>>> allocation of nodes you might run into a race condition
>>>>>> bug with
>>>>>> assigning computes to nodes and need to pin nodes (pinning
is
>>>>>> adding as an ironic property that overcloud-novacompute-0
>>>>>> goes
>>>>>> here, 1 here, so on and so forth).
>>>>>>
>>>>>> As for actually solving the deployment issues at scale
>>>>>> (instead
>>>>>> of this horrible hack) I'm looking into adding some
>>>>>> robustness at
>>>>>> the ironic or tripleo level to these operations. It sounds
>>>>>> like
>>>>>> you're running more into node assignment issues rather
>>>>>> than pxe
>>>>>> issues though.
>>>>>>
>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj'
Lorenzetto
>>>>>> <lorenzetto.luca(a)gmail.com
>>>>>> <mailto:lorenzetto.luca@gmail.com>>:
>>>>>>
>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short
>>>>>> <cems(a)ebi.acuk
>>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>>> > Some more testing of different amounts of nodes vs
time
>>>>>> taken for successful
>>>>>> > deployments -
>>>>>> >
>>>>>> > 3 controller 3 compute = 1 hour
>>>>>> > 3 controller 15 compute = 1 hour
>>>>>> > 3 controller 25 compute = 1 hour 45 mins
>>>>>> > 3 controller 35 compute = 4 hours
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> i'm now preparing my deployment of 3+2 nodes.
I'll check
>>>>>> what you
>>>>>> reported and give you some feedback.
>>>>>>
>>>>>> Luca
>>>>>>
>>>>>>
>>>>>> --
>>>>>> "E' assurdo impiegare gli uomini di
intelligenza
>>>>>> eccellente
>>>>>> per fare
>>>>>> calcoli che potrebbero essere affidati a chiunque se
si
>>>>>> usassero delle
>>>>>> macchine"
>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico
>>>>>> (1646-1716)
>>>>>>
>>>>>> "Internet è la più grande biblioteca del mondo.
>>>>>> Ma il problema è che i libri sono tutti sparsi sul
>>>>>> pavimento"
>>>>>> John Allen Paulos, Matematico (1945-vivente)
>>>>>>
>>>>>> Luca 'remix_tj' Lorenzetto,
http://www.remixtj.net ,
>>>>>> <lorenzetto.luca(a)gmail.com
>>>>>> <mailto:lorenzetto.luca@gmail.com>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> rdo-list mailing list
>>>>>> rdo-list(a)redhat.com <mailto:rdo-list@redhat.com>
>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>>>>
>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>>> <mailto:rdo-list-unsubscribe@redhat.com>
>>>>>>
>>>>>>
>>>>> --
>>>>> Charles Short
>>>>> Cloud Engineer
>>>>> Virtualization and Cloud Team
>>>>> European Bioinformatics Institute (EMBL-EBI)
>>>>> Tel: +44 (0)1223 494205
<tel:%2B44%20%280%291223%20494205>
>>>>>
>>>>>
>>>> --
>>>> Charles Short
>>>> Cloud Engineer
>>>> Virtualization and Cloud Team
>>>> European Bioinformatics Institute (EMBL-EBI)
>>>> Tel: +44 (0)1223 494205
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> rdo-list mailing list
>>>> rdo-list(a)redhat.com
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>
>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>
>
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
2:06 a.m.
On 11/11/2016 18:44, Boris Derzhavets wrote:
What would happen ?
# openstack stack delete overcloud
This only works after you restart all
services.
Then if I delete the stack restart and redeploy I get exactly the same
deployment issues on larger deployments (small deployments fine)
Charles
--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205
6:20 p.m.
On 12/11/16 02:48, Charles Short wrote:
Update -
I updated Undercloud to latest stable Newton release, and used the
provided CentOS Overcloud images. I first completed a small test
deployment no problem (3 controller 3 compute) .
I then deployed again with a larger environment (40 compute, 3
controllers).
When the nodes were up and ACTIVE/pingable early in deployment I checked
the hosts files. This time no formatting errors.
However during deployment there were lots of long pauses and I noticed
plenty of these sorts of messages in the nova logs during the pauses -
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance:
f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise
exceptions.ConnectTimeout(msg)
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed]
ConnectTimeout: Request to
http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f0...
timed out
While this was happening I could not use nova from the Undercloud at all-
source stackrc
nova list
ERROR (ClientException): The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5)
After 2 hours of deployment and only on Step 1 of 5 the deployment fails
with -
ERROR: Timed out waiting for a reply to message ID
971157a211e549998bb7a6f6e494688b
note - I have a timeout value way over 2 hours in the deployment
commands (2000)
Post failed deployment I still cannot use nova. Looks like the
Undercloud is very unhappy (same error as above)
The only way I can get the Undercloud working again is to restart all
services (restarting nova alone does not work)
sudo systemctl restart neutron*
sudo systemctl restart openstack*
I think I may try OSP9 as I am running out of ideas. Either that or
giving Openstack-Ansible a try.....
Charles
So the symptoms you are showing me above almost definitely leads me to
believe that neutron-server failed on the undercloud, which would
explain why the deploy and nova failed to work. It could have failed
before or during the deploy. We regularly see instances where
neutron-server times out upon system boot (takes slightly longer to
start than systemd expects), so we need to start it manually.
To be clear, The undercloud has been installed using this repo
http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/
Which overcloud images are you using? I'm not seeing any provided in
that repo, and I just want to make sure the undercloud and overcloud
packages match (as the tripleo-heat-templates package on the undercloud
has to align with the openstack-puppet-modules package on the overcloud
iamges).
Also, is it possible to get a copy of all the neutron-server log from
the undercloud? If we can understand why neutron-server failed, that is
the first step towards getting a working deployment.
It would be great if we could get a full sosreport with all the system
logs, to check for other errors. I'm assuming there were no problems
with the 'openstack undercloud install' process?
Regards,
Graeme
On 10/11/2016 08:20, Charles Short wrote:
> Hi,
>
> Deploy command here
>
> http://pastebin.com/xNZXTWPE
>
> no output from rpm command.
>
> Yes re OSP9 images I was just interested how they behaved early on in
> the deployment before any puppet errors (cloud init etc).
> Not a good test, just morbid fascination out of desperation.
>
> No Windows involved, and I have not altered the main puppet template
> directory at all.
>
> I am going to try and update the Undercloud to the latest stable, use
> the provided images and see how that goes.
>
> If all else fails I will install OSP9 and consider myself exhausted
> from all the swimming upstream ;)
>
> Charles
>
> On 10/11/2016 03:13, Graeme Gillies wrote:
>> On 10/11/16 02:18, Charles Short wrote:
>>> Hi,
>>>
>>> Just some feedback on this thread.
>>>
>>> I have redeployed several times and have begun to suspect DNS as being
>>> the cause for delays (just a guess as the deployment always competes
>>> with no obvious errors)
>>> I had a look at the local hosts files on the nodes during deployment
>>> and
>>> can see that lots of them (not all) are incorrectly formatted as they
>>> contain '\n'.
>>>
>>> For example a small part of one hosts file -
>>> <<
>>> \n10.0.7.30 overcloud-novacompute-32.localdomain
>>> overcloud-novacompute-32
>>> 192.168.0.39 overcloud-novacompute-32.external.localdomain
>>> overcloud-novacompute-32.external
>>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
>>> overcloud-novacompute-32.internalapi
>>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain
>>> overcloud-novacompute-32.storage
>>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
>>> overcloud-novacompute-32.storagemgmt
>>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain
>>> overcloud-novacompute-32.tenant
>>> 192.168.0.39 overcloud-novacompute-32.management.localdomain
>>> overcloud-novacompute-32.management
>>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
>>> overcloud-novacompute-32.ctlplane
>>> \n10.0.7.21 overcloud-novacompute-33.localdomain
>>> overcloud-novacompute-33
>>> I wondered if maybe the image I was using was the issue so I tried the
>>> RH OSP9 official image - Same hosts file formatting issues in
>>> deployment.
>>> Maybe a workaround would be to change nsswitch.conf in the image to
>>> look
>>> up from DNS first - my Undercloud dnsmasq server - and have this
>>> populated with the correct entries from a node (once all nodes are
>>> pingable).
>>>
>>> Charles
>> Hi Charles,
>>
>> If you are getting formatting issues in /etc/hosts, it's possible that
>> the templates directory you are using might have problems, especially if
>> it's been edited on windows machines. Are you using unmodified templates
>> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9
>> images will not match RDO Newton templates, as RHOS 9 is mitaka, and
>> overcloud images contain puppet modules which must sync with the
>> templates used on the undercloud.
>>
>> If you are using the templates in
>> /usr/share/openstack-tripleo-heat-templates, can you give the output (if
>> any) from
>>
>> rpm -V openstack-tripleo-heat-templates
>>
>> Also perhaps getting a copy of your full overcloud deploy command will
>> help shed some light as well.
>>
>> Thanks in advance,
>>
>> Graeme
>>
>>> On 06/11/2016 23:25, Graeme Gillies wrote:
>>>> Hi Charles,
>>>>
>>>> This definitely looks a bit strange to me, as we do deploys around 42
>>>> nodes and it takes around 2 hours to do so, similar to your setup (1G
>>>> link for provisoning, bonded 10G for everything else).
>>>>
>>>> Would it be possible for you to run an sosreport on your undercloud
>>>> and
>>>> provide it somewhere (if you are comfortable doing so). Also, can you
>>>> show us the output of
>>>>
>>>> openstack stack list --nested
>>>>
>>>> And most importantly, if we can get a fully copy of the output of the
>>>> overcloud deploy command, that has timestamps against when ever
>>>> stack is
>>>> created/finished, so we can try and narrow down where all the time is
>>>> being spent.
>>>>
>>>> You note that you have quite a powerful undercloud (294GB of Memory
>>>> and
>>>> 64 cpus), and we have had issues in the past with very powerful
>>>> underclouds, because the Openstack components try and tune themselves
>>>> around the hardware they are running on and get it wrong for bigger
>>>> servers.
>>>>
>>>> Are we able to get an output from "sar" or some other tool you
are
>>>> using
>>>> to track cpu and memory usage during the deployment? I'd like to
check
>>>> those values look sane.
>>>>
>>>> Thanks in advance,
>>>>
>>>> Graeme
>>>>
>>>> On 05/11/16 01:31, Charles Short wrote:
>>>>> Hi,
>>>>>
>>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
>>>>> The 1Gb deployment NIC is not really causing the delay. It is very
>>>>> busy
>>>>> for the time the overcloud image is rolled out (the first 30 to 45
>>>>> mins
>>>>> of deployment), but after that (once all the nodes are up and
active
>>>>> with an ip address (pingable)) ,the bandwidth is a fraction of
>>>>> 1Gbps on
>>>>> average for the rest of the deployment. For info the NICS in the
>>>>> nodes
>>>>> for the Overcloud networks are dual bonded 10Gbit.
>>>>>
>>>>> The deployment I mentioned before (50 nodes) actually completed in 8
>>>>> hours (which is double the time it took for 35 nodes!)
>>>>>
>>>>> I am in the process of a new 3 controller 59 compute node
deployment
>>>>> pinning all the nodes as you suggested. The initial overcloud
>>>>> image roll
>>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am
>>>>> now 45
>>>>> hours in and all is running (slowly). It is currently on Step2 (of
5
>>>>> Steps). I would expect this deployment to take 10 hours on current
>>>>> speed.
>>>>>
>>>>> Regards
>>>>>
>>>>> Charles
>>>>>
>>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>>>>>> Hey Charles,
>>>>>>
>>>>>> What sort of issues are you seeing now? How did node pinning
work
>>>>>> out
>>>>>> and did a slow scale up present any more problems?
>>>>>>
>>>>>> Deployments tend to be disk and network limited, you don't
mention
>>>>>> what sort of disks your machines have but you do note 1g nics,
which
>>>>>> are doable but might require some timeout adjustments or other
>>>>>> considerations to give everything time to complete.
>>>>>>
>>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short
<cems(a)ebi.ac.uk
>>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> So you are implying that tripleO is not really currently
>>>>>> able to
>>>>>> roll out large deployments easily as it is is prone to
scaling
>>>>>> delays/errors?
>>>>>> Is the same true for RH OSP9 (out of the box) as this also
>>>>>> uses
>>>>>> tripleO? I would expect exactly the same scaling issues.
But
>>>>>> surely OSP9 is designed for large enterprise Openstack
>>>>>> installations?
>>>>>> So if OSP9 does work well with large deployments, what are
the
>>>>>> tripleO tweaks that make this work (if any)?
>>>>>>
>>>>>> Many Thanks
>>>>>>
>>>>>> Charles
>>>>>>
>>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>>>>>> Hey Charles,
>>>>>>>
>>>>>>> If you want to deploy a large number of machines, I
>>>>>>> suggest you
>>>>>>> deploy a small configuration (maybe 3 controllers 1
>>>>>>> compute) and
>>>>>>> then run the overcloud deploy command again with 2
>>>>>>> computes, so
>>>>>>> on and so forth until you reach your full allocation
>>>>>>>
>>>>>>> Realistically you can probably do a stride of 5
computes each
>>>>>>> time, experiment with it a bit, as you get up to the
full
>>>>>>> allocation of nodes you might run into a race
condition
>>>>>>> bug with
>>>>>>> assigning computes to nodes and need to pin nodes
(pinning is
>>>>>>> adding as an ironic property that
overcloud-novacompute-0
>>>>>>> goes
>>>>>>> here, 1 here, so on and so forth).
>>>>>>>
>>>>>>> As for actually solving the deployment issues at scale
>>>>>>> (instead
>>>>>>> of this horrible hack) I'm looking into adding
some
>>>>>>> robustness at
>>>>>>> the ironic or tripleo level to these operations. It
sounds
>>>>>>> like
>>>>>>> you're running more into node assignment issues
rather
>>>>>>> than pxe
>>>>>>> issues though.
>>>>>>>
>>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj'
Lorenzetto
>>>>>>> <lorenzetto.luca(a)gmail.com
>>>>>>> <mailto:lorenzetto.luca@gmail.com>>:
>>>>>>>
>>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short
>>>>>>> <cems(a)ebi.acuk
>>>>>>> <mailto:cems@ebi.ac.uk>> wrote:
>>>>>>> > Some more testing of different amounts of
nodes vs time
>>>>>>> taken for successful
>>>>>>> > deployments -
>>>>>>> >
>>>>>>> > 3 controller 3 compute = 1 hour
>>>>>>> > 3 controller 15 compute = 1 hour
>>>>>>> > 3 controller 25 compute = 1 hour 45 mins
>>>>>>> > 3 controller 35 compute = 4 hours
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> i'm now preparing my deployment of 3+2 nodes.
I'll check
>>>>>>> what you
>>>>>>> reported and give you some feedback.
>>>>>>>
>>>>>>> Luca
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> "E' assurdo impiegare gli uomini di
intelligenza
>>>>>>> eccellente
>>>>>>> per fare
>>>>>>> calcoli che potrebbero essere affidati a chiunque
se si
>>>>>>> usassero delle
>>>>>>> macchine"
>>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e
Matematico
>>>>>>> (1646-1716)
>>>>>>>
>>>>>>> "Internet è la più grande biblioteca del
mondo.
>>>>>>> Ma il problema è che i libri sono tutti sparsi sul
>>>>>>> pavimento"
>>>>>>> John Allen Paulos, Matematico (1945-vivente)
>>>>>>>
>>>>>>> Luca 'remix_tj' Lorenzetto,
http://www.remixtj.net ,
>>>>>>> <lorenzetto.luca(a)gmail.com
>>>>>>> <mailto:lorenzetto.luca@gmail.com>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> rdo-list mailing list
>>>>>>> rdo-list(a)redhat.com
<mailto:rdo-list@redhat.com>
>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>>>>>
>>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>>>> <mailto:rdo-list-unsubscribe@redhat.com>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Charles Short
>>>>>> Cloud Engineer
>>>>>> Virtualization and Cloud Team
>>>>>> European Bioinformatics Institute (EMBL-EBI)
>>>>>> Tel: +44 (0)1223 494205
<tel:%2B44%20%280%291223%20494205>
>>>>>>
>>>>>>
>>>>> --
>>>>> Charles Short
>>>>> Cloud Engineer
>>>>> Virtualization and Cloud Team
>>>>> European Bioinformatics Institute (EMBL-EBI)
>>>>> Tel: +44 (0)1223 494205
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rdo-list mailing list
>>>>> rdo-list(a)redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>
>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>>>>>
>>
>
--
Graeme Gillies
Principal Systems Administrator
Openstack Infrastructure
Red Hat Australia
2742
days inactive
2754
days old
15 comments
5 participants
participants (5)
-
Boris Derzhavets -
Charles Short -
Graeme Gillies -
Justin Kilpatrick -
Luca 'remix_tj' Lorenzetto