<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.<br>
The 1Gb deployment NIC is not really causing the delay. It is very
busy for the time the overcloud image is rolled out (the first 30 to
45 mins of deployment), but after that (once all the nodes are up
and active with an ip address (pingable)) ,the bandwidth is a
fraction of 1Gbps on average for the rest of the deployment. For
info the NICS in the nodes for the Overcloud networks are dual
bonded 10Gbit.<br>
<br>
The deployment I mentioned before (50 nodes) actually completed in 8
hours (which is double the time it took for 35 nodes!)<br>
<br>
I am in the process of a new 3 controller 59 compute node
deployment pinning all the nodes as you suggested. The initial
overcloud image roll out took just under 1 hour (all nodes ACTIVE
and pingable). I am now 4.5 hours in and all is running (slowly). It
is currently on Step2 (of 5 Steps). I would expect this deployment
to take 10 hours on current speed.<br>
<br>
Regards<br>
<br>
Charles<br>
<br>
<div class="moz-cite-prefix">On 04/11/2016 15:17, Justin Kilpatrick
wrote:<br>
</div>
<blockquote
cite="mid:CANhjow9TTAS=VHb8SdhhA+ozG2de76oaj2OS5AmnqoS75yHWPg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Hey Charles, <br>
<br>
</div>
<div>What sort of issues are you seeing now? How did node
pinning work out and did a slow scale up present any more
problems? <br>
<br>
</div>
<div>Deployments tend to be disk and network limited, you don't
mention what sort of disks your machines have but you do note
1g nics, which are doable but might require some timeout
adjustments or other considerations to give everything time to
complete. <br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Nov 4, 2016 at 10:45 AM,
Charles Short <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:cems@ebi.ac.uk" target="_blank">cems@ebi.ac.uk</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Hi,<br>
<br>
So you are implying that tripleO is not really currently
able to roll out large deployments easily as it is is
prone to scaling delays/errors?<br>
Is the same true for RH OSP9 (out of the box) as this also
uses tripleO? I would expect exactly the same scaling
issues. But surely OSP9 is designed for large enterprise
Openstack installations?<br>
So if OSP9 does work well with large deployments, what are
the tripleO tweaks that make this work (if any)?<br>
<br>
Many Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Charles <br>
</font></span><span class=""> <br>
<div class="m_3742238084525782011moz-cite-prefix">On
03/11/2016 13:30, Justin Kilpatrick wrote:<br>
</div>
</span>
<div>
<div class="h5">
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Hey Charles, <br>
<br>
</div>
If you want to deploy a large number of
machines, I suggest you deploy a small
configuration (maybe 3 controllers 1 compute)
and then run the overcloud deploy command
again with 2 computes, so on and so forth
until you reach your full allocation <br>
<br>
</div>
Realistically you can probably do a stride of 5
computes each time, experiment with it a bit, as
you get up to the full allocation of nodes you
might run into a race condition bug with
assigning computes to nodes and need to pin
nodes (pinning is adding as an ironic property
that overcloud-novacompute-0 goes here, 1 here,
so on and so forth). <br>
<br>
</div>
As for actually solving the deployment issues at
scale (instead of this horrible hack) I'm looking
into adding some robustness at the ironic or
tripleo level to these operations. It sounds like
you're running more into node assignment issues
rather than pxe issues though. <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-11-03 9:16 GMT-04:00
Luca 'remix_tj' Lorenzetto <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:lorenzetto.luca@gmail.com"
target="_blank">lorenzetto.luca@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"><span>On Wed, Nov 2,
2016 at 8:30 PM, Charles Short <<a
moz-do-not-send="true"
href="mailto:cems@ebi.ac.uk"
target="_blank">cems@ebi.ac.uk</a>>
wrote:<br>
> Some more testing of different amounts
of nodes vs time taken for successful<br>
> deployments -<br>
><br>
> 3 controller 3 compute = 1 hour<br>
> 3 controller 15 compute = 1 hour<br>
> 3 controller 25 compute = 1 hour 45
mins<br>
> 3 controller 35 compute = 4 hours<br>
<br>
</span>Hello,<br>
<br>
i'm now preparing my deployment of 3+2 nodes.
I'll check what you<br>
reported and give you some feedback.<br>
<span class="m_3742238084525782011HOEnZb"><font
color="#888888"><br>
Luca<br>
<br>
<br>
--<br>
"E' assurdo impiegare gli uomini di
intelligenza eccellente per fare<br>
calcoli che potrebbero essere affidati a
chiunque se si usassero delle<br>
macchine"<br>
Gottfried Wilhelm von Leibnitz, Filosofo e
Matematico (1646-1716)<br>
<br>
"Internet è la più grande biblioteca del
mondo.<br>
Ma il problema è che i libri sono tutti
sparsi sul pavimento"<br>
John Allen Paulos, Matematico
(1945-vivente)<br>
<br>
Luca 'remix_tj' Lorenzetto, <a
moz-do-not-send="true"
href="http://www.remixtj.net"
rel="noreferrer" target="_blank">http://www.remixtj.net</a>
, <<a moz-do-not-send="true"
href="mailto:lorenzetto.luca@gmail.com"
target="_blank">lorenzetto.luca@gmail.com</a>><br>
</font></span>
<div class="m_3742238084525782011HOEnZb">
<div class="m_3742238084525782011h5"><br>
______________________________<wbr>_________________<br>
rdo-list mailing list<br>
<a moz-do-not-send="true"
href="mailto:rdo-list@redhat.com"
target="_blank">rdo-list@redhat.com</a><br>
<a moz-do-not-send="true"
href="https://www.redhat.com/mailman/listinfo/rdo-list"
rel="noreferrer" target="_blank">https://www.redhat.com/mailman<wbr>/listinfo/rdo-list</a><br>
<br>
To unsubscribe: <a moz-do-not-send="true"
href="mailto:rdo-list-unsubscribe@redhat.com" target="_blank">rdo-list-unsubscribe@redhat.co<wbr>m</a></div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
<span class="">
<pre class="m_3742238084525782011moz-signature" cols="72">--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: <a moz-do-not-send="true" href="tel:%2B44%20%280%291223%20494205" value="+441223494205" target="_blank">+44 (0)1223 494205</a> </pre>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205 </pre>
</body>
</html>