<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p><br>
</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> rdo-list-bounces@redhat.com <rdo-list-bounces@redhat.com> on behalf of Charles Short <cems@ebi.ac.uk><br>
<b>Sent:</b> Friday, November 11, 2016 7:48 PM<br>
<b>To:</b> rdo-list@redhat.com<br>
<b>Subject:</b> Re: [rdo-list] [TripleO] Newton large baremetal deployment issues</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Update -<br>
<br>
I updated Undercloud to latest stable Newton release, and used the <br>
provided CentOS Overcloud images. I first completed a small test <br>
deployment no problem (3 controller 3 compute) .<br>
I then deployed again with a larger environment (40 compute, 3 controllers).<br>
When the nodes were up and ACTIVE/pingable early in deployment I checked <br>
the hosts files. This time no formatting errors.<br>
<br>
However during deployment there were lots of long pauses and I noticed <br>
plenty of these sorts of messages in the nova logs during the pauses -<br>
<br>
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR <br>
nova.compute.manager [instance: <br>
f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] raise <br>
exceptions.ConnectTimeout(msg)<br>
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR <br>
nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed] <br>
ConnectTimeout: Request to <br>
<a href="http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f028412977f&device_id=f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed" id="LPlnk185166" previewremoved="true">http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f028412977f&device_id=f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed</a>
<br>
timed out<br>
<br>
While this was happening I could not use nova from the Undercloud at all-<br>
<br>
source stackrc<br>
nova list<br>
ERROR (ClientException): The server has either erred or is incapable of <br>
performing the requested operation. (HTTP 500) (Request-ID: <br>
req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5)<br>
<br>
After 2 hours of deployment and only on Step 1 of 5 the deployment fails <br>
with -<br>
<br>
ERROR: Timed out waiting for a reply to message ID <br>
971157a211e549998bb7a6f6e494688b<br>
<br>
note - I have a timeout value way over 2 hours in the deployment <br>
commands (2000)<br>
<br>
Post failed deployment I still cannot use nova. Looks like the <br>
Undercloud is very unhappy (same error as above)<br>
<br>
The only way I can get the Undercloud working again is to restart all <br>
services (restarting nova alone does not work)<br>
sudo systemctl restart neutron*<br>
sudo systemctl restart openstack*<br>
<br>
What would happen ?<br>
# openstack stack delete overcloud<br>
Stack deleted cleanly<br>
# shutdown -r now<br>
<br>
Boris.<br>
<br>
I think I may try OSP9 as I am running out of ideas. Either that or <br>
giving Openstack-Ansible a try.....<br>
<br>
<br>
<br>
Charles<br>
<br>
<br>
On 10/11/2016 08:20, Charles Short wrote:<br>
> Hi,<br>
><br>
> Deploy command here<br>
><br>
> <a href="http://pastebin.com/xNZXTWPE" id="LPlnk419007" previewremoved="true">http://pastebin.com/xNZXTWPE</a>
<div id="LPBorder_GT_14788897592710.43032376798013483" style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;">
<table id="LPContainer_14788897592630.4899215716268188" style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);" cellspacing="0">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td id="ImageCell_14788897592640.3833245509053458" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;" colspan="1">
<div id="LPImageContainer_14788897592640.8834649001579573" style="background-color: rgb(255, 255, 255); height: 250px; position: relative; margin: auto; display: table; width: 250px;">
<a id="LPImageAnchor_14788897592670.8574507432835123" style="display: table-cell; text-align: center;" href="http://pastebin.com/xNZXTWPE" target="_blank"><img style="display: inline-block; max-width: 250px; max-height: 250px; height: 250px; width: 250px; border-width: 0px; vertical-align: bottom;" aria-label="Preview image with link selected. Double-tap to open the link." id="LPThumbnailImageID_14788897592670.24469905918272639" width="250" height="250" src="http://pastebin.com/i/facebook.png"></a></div>
</td>
<td id="TextCell_14788897592680.6771846326433241" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;" colspan="2">
<div id="LPRemovePreviewContainer_14788897592680.9251933746792493"></div>
<div id="LPTitle_14788897592680.6894707818163407" style="top: 0px; color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: "wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; line-height: 21px;">
<a id="LPUrlAnchor_14788897592690.3478064960779984" style="text-decoration: none;" href="http://pastebin.com/xNZXTWPE" target="_blank">stack1 - Pastebin.com</a></div>
<div id="LPMetadata_14788897592700.05600952336495468" style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 14px;">
pastebin.com</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
><br>
> no output from rpm command.<br>
><br>
> Yes re OSP9 images I was just interested how they behaved early on in <br>
> the deployment before any puppet errors (cloud init etc).<br>
> Not a good test, just morbid fascination out of desperation.<br>
><br>
> No Windows involved, and I have not altered the main puppet template <br>
> directory at all.<br>
><br>
> I am going to try and update the Undercloud to the latest stable, use <br>
> the provided images and see how that goes.<br>
><br>
> If all else fails I will install OSP9 and consider myself exhausted <br>
> from all the swimming upstream ;)<br>
><br>
> Charles<br>
><br>
> On 10/11/2016 03:13, Graeme Gillies wrote:<br>
>> On 10/11/16 02:18, Charles Short wrote:<br>
>>> Hi,<br>
>>><br>
>>> Just some feedback on this thread.<br>
>>><br>
>>> I have redeployed several times and have begun to suspect DNS as being<br>
>>> the cause for delays (just a guess as the deployment always competes<br>
>>> with no obvious errors)<br>
>>> I had a look at the local hosts files on the nodes during deployment <br>
>>> and<br>
>>> can see that lots of them (not all) are incorrectly formatted as they<br>
>>> contain '\n'.<br>
>>><br>
>>> For example a small part of one hosts file -<br>
>>> <<<br>
>>> \n10.0.7.30 overcloud-novacompute-32.localdomain <br>
>>> overcloud-novacompute-32<br>
>>> 192.168.0.39 overcloud-novacompute-32.external.localdomain<br>
>>> overcloud-novacompute-32.external<br>
>>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain<br>
>>> overcloud-novacompute-32.internalapi<br>
>>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain<br>
>>> overcloud-novacompute-32.storage<br>
>>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain<br>
>>> overcloud-novacompute-32.storagemgmt<br>
>>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain<br>
>>> overcloud-novacompute-32.tenant<br>
>>> 192.168.0.39 overcloud-novacompute-32.management.localdomain<br>
>>> overcloud-novacompute-32.management<br>
>>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain<br>
>>> overcloud-novacompute-32.ctlplane<br>
>>> \n10.0.7.21 overcloud-novacompute-33.localdomain <br>
>>> overcloud-novacompute-33<br>
>>> I wondered if maybe the image I was using was the issue so I tried the<br>
>>> RH OSP9 official image - Same hosts file formatting issues in <br>
>>> deployment.<br>
>>> Maybe a workaround would be to change nsswitch.conf in the image to <br>
>>> look<br>
>>> up from DNS first - my Undercloud dnsmasq server - and have this<br>
>>> populated with the correct entries from a node (once all nodes are<br>
>>> pingable).<br>
>>><br>
>>> Charles<br>
>> Hi Charles,<br>
>><br>
>> If you are getting formatting issues in /etc/hosts, it's possible that<br>
>> the templates directory you are using might have problems, especially if<br>
>> it's been edited on windows machines. Are you using unmodified templates<br>
>> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9<br>
>> images will not match RDO Newton templates, as RHOS 9 is mitaka, and<br>
>> overcloud images contain puppet modules which must sync with the<br>
>> templates used on the undercloud.<br>
>><br>
>> If you are using the templates in<br>
>> /usr/share/openstack-tripleo-heat-templates, can you give the output (if<br>
>> any) from<br>
>><br>
>> rpm -V openstack-tripleo-heat-templates<br>
>><br>
>> Also perhaps getting a copy of your full overcloud deploy command will<br>
>> help shed some light as well.<br>
>><br>
>> Thanks in advance,<br>
>><br>
>> Graeme<br>
>><br>
>>> On 06/11/2016 23:25, Graeme Gillies wrote:<br>
>>>> Hi Charles,<br>
>>>><br>
>>>> This definitely looks a bit strange to me, as we do deploys around 42<br>
>>>> nodes and it takes around 2 hours to do so, similar to your setup (1G<br>
>>>> link for provisoning, bonded 10G for everything else).<br>
>>>><br>
>>>> Would it be possible for you to run an sosreport on your undercloud <br>
>>>> and<br>
>>>> provide it somewhere (if you are comfortable doing so). Also, can you<br>
>>>> show us the output of<br>
>>>><br>
>>>> openstack stack list --nested<br>
>>>><br>
>>>> And most importantly, if we can get a fully copy of the output of the<br>
>>>> overcloud deploy command, that has timestamps against when ever <br>
>>>> stack is<br>
>>>> created/finished, so we can try and narrow down where all the time is<br>
>>>> being spent.<br>
>>>><br>
>>>> You note that you have quite a powerful undercloud (294GB of Memory <br>
>>>> and<br>
>>>> 64 cpus), and we have had issues in the past with very powerful<br>
>>>> underclouds, because the Openstack components try and tune themselves<br>
>>>> around the hardware they are running on and get it wrong for bigger<br>
>>>> servers.<br>
>>>><br>
>>>> Are we able to get an output from "sar" or some other tool you are <br>
>>>> using<br>
>>>> to track cpu and memory usage during the deployment? I'd like to check<br>
>>>> those values look sane.<br>
>>>><br>
>>>> Thanks in advance,<br>
>>>><br>
>>>> Graeme<br>
>>>><br>
>>>> On 05/11/16 01:31, Charles Short wrote:<br>
>>>>> Hi,<br>
>>>>><br>
>>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.<br>
>>>>> The 1Gb deployment NIC is not really causing the delay. It is very <br>
>>>>> busy<br>
>>>>> for the time the overcloud image is rolled out (the first 30 to 45 <br>
>>>>> mins<br>
>>>>> of deployment), but after that (once all the nodes are up and active<br>
>>>>> with an ip address (pingable)) ,the bandwidth is a fraction of <br>
>>>>> 1Gbps on<br>
>>>>> average for the rest of the deployment. For info the NICS in the <br>
>>>>> nodes<br>
>>>>> for the Overcloud networks are dual bonded 10Gbit.<br>
>>>>><br>
>>>>> The deployment I mentioned before (50 nodes) actually completed in 8<br>
>>>>> hours (which is double the time it took for 35 nodes!)<br>
>>>>><br>
>>>>> I am in the process of a new 3 controller 59 compute node deployment<br>
>>>>> pinning all the nodes as you suggested. The initial overcloud <br>
>>>>> image roll<br>
>>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am <br>
>>>>> now 45<br>
>>>>> hours in and all is running (slowly). It is currently on Step2 (of 5<br>
>>>>> Steps). I would expect this deployment to take 10 hours on current<br>
>>>>> speed.<br>
>>>>><br>
>>>>> Regards<br>
>>>>><br>
>>>>> Charles<br>
>>>>><br>
>>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote:<br>
>>>>>> Hey Charles,<br>
>>>>>><br>
>>>>>> What sort of issues are you seeing now? How did node pinning work <br>
>>>>>> out<br>
>>>>>> and did a slow scale up present any more problems?<br>
>>>>>><br>
>>>>>> Deployments tend to be disk and network limited, you don't mention<br>
>>>>>> what sort of disks your machines have but you do note 1g nics, which<br>
>>>>>> are doable but might require some timeout adjustments or other<br>
>>>>>> considerations to give everything time to complete.<br>
>>>>>><br>
>>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems@ebi.ac.uk<br>
>>>>>> <<a href="mailto:cems@ebi.ac.uk">mailto:cems@ebi.ac.uk</a>>> wrote:<br>
>>>>>><br>
>>>>>> Hi,<br>
>>>>>><br>
>>>>>> So you are implying that tripleO is not really currently <br>
>>>>>> able to<br>
>>>>>> roll out large deployments easily as it is is prone to scaling<br>
>>>>>> delays/errors?<br>
>>>>>> Is the same true for RH OSP9 (out of the box) as this also <br>
>>>>>> uses<br>
>>>>>> tripleO? I would expect exactly the same scaling issues. But<br>
>>>>>> surely OSP9 is designed for large enterprise Openstack<br>
>>>>>> installations?<br>
>>>>>> So if OSP9 does work well with large deployments, what are the<br>
>>>>>> tripleO tweaks that make this work (if any)?<br>
>>>>>><br>
>>>>>> Many Thanks<br>
>>>>>><br>
>>>>>> Charles<br>
>>>>>><br>
>>>>>> On 03/11/2016 13:30, Justin Kilpatrick wrote:<br>
>>>>>>> Hey Charles,<br>
>>>>>>><br>
>>>>>>> If you want to deploy a large number of machines, I <br>
>>>>>>> suggest you<br>
>>>>>>> deploy a small configuration (maybe 3 controllers 1 <br>
>>>>>>> compute) and<br>
>>>>>>> then run the overcloud deploy command again with 2 <br>
>>>>>>> computes, so<br>
>>>>>>> on and so forth until you reach your full allocation<br>
>>>>>>><br>
>>>>>>> Realistically you can probably do a stride of 5 computes each<br>
>>>>>>> time, experiment with it a bit, as you get up to the full<br>
>>>>>>> allocation of nodes you might run into a race condition <br>
>>>>>>> bug with<br>
>>>>>>> assigning computes to nodes and need to pin nodes (pinning is<br>
>>>>>>> adding as an ironic property that overcloud-novacompute-0 <br>
>>>>>>> goes<br>
>>>>>>> here, 1 here, so on and so forth).<br>
>>>>>>><br>
>>>>>>> As for actually solving the deployment issues at scale <br>
>>>>>>> (instead<br>
>>>>>>> of this horrible hack) I'm looking into adding some <br>
>>>>>>> robustness at<br>
>>>>>>> the ironic or tripleo level to these operations. It sounds <br>
>>>>>>> like<br>
>>>>>>> you're running more into node assignment issues rather <br>
>>>>>>> than pxe<br>
>>>>>>> issues though.<br>
>>>>>>><br>
>>>>>>> 2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto<br>
>>>>>>> <lorenzetto.luca@gmail.com <br>
>>>>>>> <<a href="mailto:lorenzetto.luca@gmail.com">mailto:lorenzetto.luca@gmail.com</a>>>:<br>
>>>>>>><br>
>>>>>>> On Wed, Nov 2, 2016 at 8:30 PM, Charles Short <br>
>>>>>>> <cems@ebi.acuk<br>
>>>>>>> <<a href="mailto:cems@ebi.ac.uk">mailto:cems@ebi.ac.uk</a>>> wrote:<br>
>>>>>>> > Some more testing of different amounts of nodes vs time<br>
>>>>>>> taken for successful<br>
>>>>>>> > deployments -<br>
>>>>>>> ><br>
>>>>>>> > 3 controller 3 compute = 1 hour<br>
>>>>>>> > 3 controller 15 compute = 1 hour<br>
>>>>>>> > 3 controller 25 compute = 1 hour 45 mins<br>
>>>>>>> > 3 controller 35 compute = 4 hours<br>
>>>>>>><br>
>>>>>>> Hello,<br>
>>>>>>><br>
>>>>>>> i'm now preparing my deployment of 3+2 nodes. I'll check<br>
>>>>>>> what you<br>
>>>>>>> reported and give you some feedback.<br>
>>>>>>><br>
>>>>>>> Luca<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> "E' assurdo impiegare gli uomini di intelligenza <br>
>>>>>>> eccellente<br>
>>>>>>> per fare<br>
>>>>>>> calcoli che potrebbero essere affidati a chiunque se si<br>
>>>>>>> usassero delle<br>
>>>>>>> macchine"<br>
>>>>>>> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico<br>
>>>>>>> (1646-1716)<br>
>>>>>>><br>
>>>>>>> "Internet è la più grande biblioteca del mondo.<br>
>>>>>>> Ma il problema è che i libri sono tutti sparsi sul <br>
>>>>>>> pavimento"<br>
>>>>>>> John Allen Paulos, Matematico (1945-vivente)<br>
>>>>>>><br>
>>>>>>> Luca 'remix_tj' Lorenzetto, <a href="http://www.remixtj.net">http://www.remixtj.net</a> ,<br>
>>>>>>> <lorenzetto.luca@gmail.com<br>
>>>>>>> <<a href="mailto:lorenzetto.luca@gmail.com">mailto:lorenzetto.luca@gmail.com</a>>><br>
>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> rdo-list mailing list<br>
>>>>>>> rdo-list@redhat.com <<a href="mailto:rdo-list@redhat.com">mailto:rdo-list@redhat.com</a>><br>
>>>>>>> <a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
>>>>>>> <<a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a>><br>
>>>>>>><br>
>>>>>>> To unsubscribe: rdo-list-unsubscribe@redhat.com<br>
>>>>>>> <<a href="mailto:rdo-list-unsubscribe@redhat.com">mailto:rdo-list-unsubscribe@redhat.com</a>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>> --<br>
>>>>>> Charles Short<br>
>>>>>> Cloud Engineer<br>
>>>>>> Virtualization and Cloud Team<br>
>>>>>> European Bioinformatics Institute (EMBL-EBI)<br>
>>>>>> Tel: +44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205><br>
>>>>>><br>
>>>>>><br>
>>>>> -- <br>
>>>>> Charles Short<br>
>>>>> Cloud Engineer<br>
>>>>> Virtualization and Cloud Team<br>
>>>>> European Bioinformatics Institute (EMBL-EBI)<br>
>>>>> Tel: +44 (0)1223 494205<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> rdo-list mailing list<br>
>>>>> rdo-list@redhat.com<br>
>>>>> <a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
>>>>><br>
>>>>> To unsubscribe: rdo-list-unsubscribe@redhat.com<br>
>>>>><br>
>><br>
><br>
<br>
-- <br>
Charles Short<br>
Cloud Engineer<br>
Virtualization and Cloud Team<br>
European Bioinformatics Institute (EMBL-EBI)<br>
Tel: +44 (0)1223 494205<br>
<br>
_______________________________________________<br>
rdo-list mailing list<br>
rdo-list@redhat.com<br>
<a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
<br>
To unsubscribe: rdo-list-unsubscribe@redhat.com<br>
</div>
</span></font></div>
</div>
</body>
</html>