[rdo-list] [TripleO] Newton large baremetal deployment issues

Boris Derzhavets bderzhavets at hotmail.com
Fri Nov 11 18:44:19 UTC 2016




________________________________
From: rdo-list-bounces at redhat.com <rdo-list-bounces at redhat.com> on behalf of Charles Short <cems at ebi.ac.uk>
Sent: Friday, November 11, 2016 7:48 PM
To: rdo-list at redhat.com
Subject: Re: [rdo-list] [TripleO] Newton large baremetal deployment issues

Update -

I updated Undercloud to latest stable Newton release, and used the
provided CentOS Overcloud images. I first completed a small test
deployment no problem (3 controller 3 compute) .
I then deployed again with a larger environment (40 compute, 3 controllers).
When the nodes were up and ACTIVE/pingable early in deployment I checked
the hosts files. This time no formatting errors.

However during deployment there were lots of long pauses and I noticed
plenty of these sorts of messages in the nova logs during the pauses -

/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance:
f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed]     raise
exceptions.ConnectTimeout(msg)
/var/log/nova/nova-compute.log:2016-11-11 19:56:07.322 7840 ERROR
nova.compute.manager [instance: f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed]
ConnectTimeout: Request to
http://192.168.0.1:9696/v2.0/ports.json?tenant_id=30401f505075414fbd700f028412977f&device_id=f6fd4127-fc94-4a36-9b3b-5e4f21bd08ed
timed out

While this was happening I could not use nova from the Undercloud at all-

source stackrc
nova list
ERROR (ClientException): The server has either erred or is incapable of
performing the requested operation. (HTTP 500) (Request-ID:
req-367c9ac2-6f27-4e71-a451-681c8c3d2ce5)

After 2 hours of deployment and only on Step 1 of 5 the deployment fails
with -

ERROR: Timed out waiting for a reply to message ID
971157a211e549998bb7a6f6e494688b

note - I have a timeout value way over 2 hours in the deployment
commands (2000)

Post failed deployment I still cannot use nova. Looks like the
Undercloud is very unhappy (same error as above)

The only way I can get the Undercloud working again is to restart all
services (restarting nova alone does not work)
sudo systemctl restart neutron*
sudo systemctl restart openstack*

What would happen ?
 # openstack stack delete overcloud
   Stack deleted cleanly
 # shutdown -r now

Boris.

I think I may try OSP9 as I am running out of ideas. Either that or
giving Openstack-Ansible a try.....



Charles


On 10/11/2016 08:20, Charles Short wrote:
> Hi,
>
> Deploy command here
>
> http://pastebin.com/xNZXTWPE
[http://pastebin.com/i/facebook.png]<http://pastebin.com/xNZXTWPE>

stack1 - Pastebin.com<http://pastebin.com/xNZXTWPE>
pastebin.com


>
> no output from rpm command.
>
> Yes re OSP9 images I was just interested how they behaved early on in
> the deployment before any puppet errors (cloud init etc).
> Not a good test, just morbid fascination out of desperation.
>
> No Windows involved, and I have not altered the main puppet template
> directory at all.
>
> I am going to try and update the Undercloud to the latest stable, use
> the provided images and see how that goes.
>
> If all else fails I will install OSP9 and consider myself exhausted
> from all the swimming upstream ;)
>
> Charles
>
> On 10/11/2016 03:13, Graeme Gillies wrote:
>> On 10/11/16 02:18, Charles Short wrote:
>>> Hi,
>>>
>>> Just some feedback on this thread.
>>>
>>> I have redeployed several times and have begun to suspect DNS as being
>>> the cause for delays (just a guess as the deployment always competes
>>> with no obvious errors)
>>> I had a look at the local hosts files on the nodes during deployment
>>> and
>>> can see that lots of them (not all) are incorrectly formatted as they
>>> contain '\n'.
>>>
>>> For example a small part of one hosts file -
>>> <<
>>> \n10.0.7.30 overcloud-novacompute-32.localdomain
>>> overcloud-novacompute-32
>>> 192.168.0.39 overcloud-novacompute-32.external.localdomain
>>> overcloud-novacompute-32.external
>>> 10.0.7.30 overcloud-novacompute-32.internalapi.localdomain
>>> overcloud-novacompute-32.internalapi
>>> 10.35.5.67 overcloud-novacompute-32.storage.localdomain
>>> overcloud-novacompute-32.storage
>>> 192.168.0.39 overcloud-novacompute-32.storagemgmt.localdomain
>>> overcloud-novacompute-32.storagemgmt
>>> 10.0.8.39 overcloud-novacompute-32.tenant.localdomain
>>> overcloud-novacompute-32.tenant
>>> 192.168.0.39 overcloud-novacompute-32.management.localdomain
>>> overcloud-novacompute-32.management
>>> 192.168.0.39 overcloud-novacompute-32.ctlplane.localdomain
>>> overcloud-novacompute-32.ctlplane
>>> \n10.0.7.21 overcloud-novacompute-33.localdomain
>>> overcloud-novacompute-33
>>> I wondered if maybe the image I was using was the issue so I tried the
>>> RH OSP9 official image -  Same hosts file formatting issues in
>>> deployment.
>>> Maybe a workaround would be to change nsswitch.conf in the image to
>>> look
>>> up from DNS first  -  my Undercloud dnsmasq server - and have this
>>> populated with the correct entries from a node (once all nodes are
>>> pingable).
>>>
>>> Charles
>> Hi Charles,
>>
>> If you are getting formatting issues in /etc/hosts, it's possible that
>> the templates directory you are using might have problems, especially if
>> it's been edited on windows machines. Are you using unmodified templates
>> from /usr/share/openstack-tripleo-heat-templates? Also note that RHOS 9
>> images will not match RDO Newton templates, as RHOS 9 is mitaka, and
>> overcloud images contain puppet modules which must sync with the
>> templates used on the undercloud.
>>
>> If you are using the templates in
>> /usr/share/openstack-tripleo-heat-templates, can you give the output (if
>> any) from
>>
>> rpm -V openstack-tripleo-heat-templates
>>
>> Also perhaps getting a copy of your full overcloud deploy command will
>> help shed some light as well.
>>
>> Thanks in advance,
>>
>> Graeme
>>
>>> On 06/11/2016 23:25, Graeme Gillies wrote:
>>>> Hi Charles,
>>>>
>>>> This definitely looks a bit strange to me, as we do deploys around 42
>>>> nodes and it takes around 2 hours to do so, similar to your setup (1G
>>>> link for provisoning, bonded 10G for everything else).
>>>>
>>>> Would it be possible for you to run an sosreport on your undercloud
>>>> and
>>>> provide it somewhere (if you are comfortable doing so). Also, can you
>>>> show us the output of
>>>>
>>>> openstack stack list --nested
>>>>
>>>> And most importantly, if we can get a fully copy of the output of the
>>>> overcloud deploy command, that has timestamps against when ever
>>>> stack is
>>>> created/finished, so we can try and narrow down where all the time is
>>>> being spent.
>>>>
>>>> You note that you have quite a powerful undercloud (294GB of Memory
>>>> and
>>>> 64 cpus), and we have had issues in the past with very powerful
>>>> underclouds, because the Openstack components try and tune themselves
>>>> around the hardware they are running on and get it wrong for bigger
>>>> servers.
>>>>
>>>> Are we able to get an output from "sar" or some other tool you are
>>>> using
>>>> to track cpu and memory usage during the deployment? I'd like to check
>>>> those values look sane.
>>>>
>>>> Thanks in advance,
>>>>
>>>> Graeme
>>>>
>>>> On 05/11/16 01:31, Charles Short wrote:
>>>>> Hi,
>>>>>
>>>>> Each node has 2X HP 900GB 12G SAS 10K 2.5in SC ENT HDD.
>>>>> The 1Gb deployment NIC is not really causing the delay. It is very
>>>>> busy
>>>>> for the time the overcloud image is rolled out (the first 30 to 45
>>>>> mins
>>>>> of deployment), but after that  (once all the nodes are up and active
>>>>> with an ip address (pingable)) ,the bandwidth is a fraction of
>>>>> 1Gbps on
>>>>> average for the rest of the deployment. For info the NICS in the
>>>>> nodes
>>>>> for the Overcloud networks are dual bonded 10Gbit.
>>>>>
>>>>> The deployment I mentioned before (50 nodes) actually completed in 8
>>>>> hours (which is double the time it took for 35 nodes!)
>>>>>
>>>>> I am in the process of a new  3 controller 59 compute node deployment
>>>>> pinning all the nodes as you suggested. The initial overcloud
>>>>> image roll
>>>>> out took just under 1 hour (all nodes ACTIVE and pingable). I am
>>>>> now 45
>>>>> hours in and all is running (slowly). It is currently on Step2  (of 5
>>>>> Steps). I would expect this deployment to take 10 hours on current
>>>>> speed.
>>>>>
>>>>> Regards
>>>>>
>>>>> Charles
>>>>>
>>>>> On 04/11/2016 15:17, Justin Kilpatrick wrote:
>>>>>> Hey Charles,
>>>>>>
>>>>>> What sort of issues are you seeing now? How did node pinning work
>>>>>> out
>>>>>> and did a slow scale up present any more problems?
>>>>>>
>>>>>> Deployments tend to be disk and network limited, you don't mention
>>>>>> what sort of disks your machines have but you do note 1g nics, which
>>>>>> are doable but might require some timeout adjustments or other
>>>>>> considerations to give everything time to complete.
>>>>>>
>>>>>> On Fri, Nov 4, 2016 at 10:45 AM, Charles Short <cems at ebi.ac.uk
>>>>>> <mailto:cems at ebi.ac.uk>> wrote:
>>>>>>
>>>>>>       Hi,
>>>>>>
>>>>>>       So you are implying that tripleO is not really currently
>>>>>> able to
>>>>>>       roll out large deployments easily as it is is prone to scaling
>>>>>>       delays/errors?
>>>>>>       Is the same true for RH OSP9 (out of the box) as this also
>>>>>> uses
>>>>>>       tripleO?  I would expect exactly the same scaling issues. But
>>>>>>       surely OSP9 is designed for large enterprise Openstack
>>>>>> installations?
>>>>>>       So if OSP9 does work well with large deployments, what are the
>>>>>>       tripleO tweaks that make this work (if any)?
>>>>>>
>>>>>>       Many Thanks
>>>>>>
>>>>>>       Charles
>>>>>>
>>>>>>       On 03/11/2016 13:30, Justin Kilpatrick wrote:
>>>>>>>       Hey Charles,
>>>>>>>
>>>>>>>       If you want to deploy a large number of machines, I
>>>>>>> suggest you
>>>>>>>       deploy a small configuration (maybe 3 controllers 1
>>>>>>> compute) and
>>>>>>>       then run the overcloud deploy command again with 2
>>>>>>> computes, so
>>>>>>>       on and so forth until you reach your full allocation
>>>>>>>
>>>>>>>       Realistically you can probably do a stride of 5 computes each
>>>>>>>       time, experiment with it a bit, as you get up to the full
>>>>>>>       allocation of nodes you might run into a race condition
>>>>>>> bug with
>>>>>>>       assigning computes to nodes and need to pin nodes (pinning is
>>>>>>>       adding as an ironic property that overcloud-novacompute-0
>>>>>>> goes
>>>>>>>       here, 1 here, so on and so forth).
>>>>>>>
>>>>>>>       As for actually solving the deployment issues at scale
>>>>>>> (instead
>>>>>>>       of this horrible hack) I'm looking into adding some
>>>>>>> robustness at
>>>>>>>       the ironic or tripleo level to these operations. It sounds
>>>>>>> like
>>>>>>>       you're running more into node assignment issues rather
>>>>>>> than pxe
>>>>>>>       issues though.
>>>>>>>
>>>>>>>       2016-11-03 9:16 GMT-04:00 Luca 'remix_tj' Lorenzetto
>>>>>>>       <lorenzetto.luca at gmail.com
>>>>>>> <mailto:lorenzetto.luca at gmail.com>>:
>>>>>>>
>>>>>>>           On Wed, Nov 2, 2016 at 8:30 PM, Charles Short
>>>>>>> <cems at ebi.acuk
>>>>>>>           <mailto:cems at ebi.ac.uk>> wrote:
>>>>>>>           > Some more testing of different amounts of nodes vs time
>>>>>>>           taken for successful
>>>>>>>           > deployments -
>>>>>>>           >
>>>>>>>           > 3 controller 3 compute = 1 hour
>>>>>>>           > 3 controller 15 compute = 1 hour
>>>>>>>           > 3 controller 25 compute  = 1 hour 45 mins
>>>>>>>           > 3 controller 35 compute  = 4 hours
>>>>>>>
>>>>>>>           Hello,
>>>>>>>
>>>>>>>           i'm now preparing my deployment of 3+2 nodes. I'll check
>>>>>>> what you
>>>>>>>           reported and give you some feedback.
>>>>>>>
>>>>>>>           Luca
>>>>>>>
>>>>>>>
>>>>>>>           --
>>>>>>>           "E' assurdo impiegare gli uomini di intelligenza
>>>>>>> eccellente
>>>>>>>           per fare
>>>>>>>           calcoli che potrebbero essere affidati a chiunque se si
>>>>>>>           usassero delle
>>>>>>>           macchine"
>>>>>>>           Gottfried Wilhelm von Leibnitz, Filosofo e Matematico
>>>>>>> (1646-1716)
>>>>>>>
>>>>>>>           "Internet è la più grande biblioteca del mondo.
>>>>>>>           Ma il problema è che i libri sono tutti sparsi sul
>>>>>>> pavimento"
>>>>>>>           John Allen Paulos, Matematico (1945-vivente)
>>>>>>>
>>>>>>>           Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
>>>>>>>           <lorenzetto.luca at gmail.com
>>>>>>> <mailto:lorenzetto.luca at gmail.com>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>>           rdo-list mailing list
>>>>>>>           rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>> <https://www.redhat.com/mailman/listinfo/rdo-list>
>>>>>>>
>>>>>>>           To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>>>>> <mailto:rdo-list-unsubscribe at redhat.com>
>>>>>>>
>>>>>>>
>>>>>>       --
>>>>>>       Charles Short
>>>>>>       Cloud Engineer
>>>>>>       Virtualization and Cloud Team
>>>>>>       European Bioinformatics Institute (EMBL-EBI)
>>>>>>       Tel: +44 (0)1223 494205 <tel:%2B44%20%280%291223%20494205>
>>>>>>
>>>>>>
>>>>> --
>>>>> Charles Short
>>>>> Cloud Engineer
>>>>> Virtualization and Cloud Team
>>>>> European Bioinformatics Institute (EMBL-EBI)
>>>>> Tel: +44 (0)1223 494205
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rdo-list mailing list
>>>>> rdo-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>
>>>>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>>>
>>
>

--
Charles Short
Cloud Engineer
Virtualization and Cloud Team
European Bioinformatics Institute (EMBL-EBI)
Tel: +44 (0)1223 494205

_______________________________________________
rdo-list mailing list
rdo-list at redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

To unsubscribe: rdo-list-unsubscribe at redhat.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20161111/fb7cca9a/attachment.html>


More information about the dev mailing list