Hi, So I have finally tried OSP9 and here are the results - 3 Controllers 40 compute - 1 hours 20 mins to deploy. This is much more the sort of deployment time I was expecting :) I then tried TripleO Newton Stable again with 3 Controllers 40 Compute - 4 hours and counting..... The two deployment scripts (for OSP9 and TripleO Newton) were pretty much identical (allowing for any changes between releases) During the OSP9 deployment I could use nova list to list the nodes. The Undercloud API access was in general very responsive. During the TripleO Newton deployment 'nova list' hangs - ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) Undercloud API access was very sluggish. I noticed Keystone was stuck at 140% for most of the deployment (albeit multi threaded) which is not the case for OSP9. I know it is hard to compare two releases, but the difference is enormous. I will stick with OSP9 for now as this for me works properly out of the box for large deployments. Charles

On 14/11/2016 09:01, Charles Short wrote: > Hi Graeme, > > Thanks for the reply. > > I used these images - > > http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/d... > > > I installed the stable repo following the documentation here - > > http://docs.openstack.org/developer/tripleo-docs/installation/installatio... > > > for example - > > sudo curl -L -o /etc/yum.repos.d/delorean-newton.repo > https://trunk.rdoproject.org/centos7-newton/current/delorean.repo > > sudo curl -L -o /etc/yum.repos.d/delorean-deps-newton.repo > http://trunk.rdoproject.org/centos7-newton/delorean-deps.repo > > > The difficulty I am having is that when I test with a small deployment > all works fine. So you would assume just adding more compute nodes > would not be an issue. > Testing this is painful due to the time it takes for a large > deployment to fail. It seems to be only scale that is the issue. > > I will try and get you some logs > > Regards > > Charles > > > >> So the symptoms you are showing me above almost definitely leads me to >> believe that neutron-server failed on the undercloud, which would >> explain why the deploy and nova failed to work. It could have failed >> before or during the deploy. We regularly see instances where >> neutron-server times out upon system boot (takes slightly longer to >> start than systemd expects), so we need to start it manually. >> >> To be clear, The undercloud has been installed using this repo >> >> http://buildlogs.centos.org/centos/7/cloud/x86_64/rdo-trunk-newton-tested/ >> >> >> Which overcloud images are you using? I'm not seeing any provided in >> that repo, and I just want to make sure the undercloud and overcloud >> packages match (as the tripleo-heat-templates package on the undercloud >> has to align with the openstack-puppet-modules package on the overcloud >> iamges). >> >> Also, is it possible to get a copy of all the neutron-server log from >> the undercloud? If we can understand why neutron-server failed, that is >> the first step towards getting a working deployment. >> >> It would be great if we could get a full sosreport with all the system >> logs, to check for other errors. I'm assuming there were no problems >> with the 'openstack undercloud install' process? >> >> Regards, >> >> Graeme >> >

Hi Charles, On Tue, Nov 15, 2016 at 10:39:19PM +0000, Charles Short wrote: > So I have finally tried OSP9 and here are the results - > > 3 Controllers 40 compute - 1 hours 20 mins to deploy. > > This is much more the sort of deployment time I was expecting :) > > I then tried TripleO Newton Stable again with 3 Controllers 40 Compute - > > 4 hours and counting..... > > The two deployment scripts (for OSP9 and TripleO Newton) were pretty much > identical (allowing for any changes between releases) > > During the OSP9 deployment I could use nova list to list the nodes. The > Undercloud API access was in general very responsive. > > During the TripleO Newton deployment 'nova list' hangs - > ERROR (ClientException): The server has either erred or is incapable of > performing the requested operation. (HTTP 500) > Undercloud API access was very sluggish. > I noticed Keystone was stuck at 140% for most of the deployment (albeit > multi threaded) which is not the case for OSP9. > > I know it is hard to compare two releases, but the difference is enormous. > I will stick with OSP9 for now as this for me works properly out of the box > for large deployments. May I ask what version of python-oslo-messaging you have on Newton? The reason I ask, is that Eck fixed a pretty serious epoll() issue which caused deploys to fail with newton on weaker hardware (which would work in mitaka). The change I mention is this one: https://review.openstack.org/#/c/394963/ It would be great to confirm or not if the python-oslo-messaging package you are using has this change included. Thanks, Michele

> The reason I ask, is that Eck fixed a pretty serious epoll() issue which > caused deploys to fail with newton on weaker hardware (which would work > in mitaka). The change I mention is this one: > https://review.openstack.org/#/c/394963/

This was merged last week and not released upstream yet AFAICT. In order to get this into RDO Newton you need to: * request oslo.messaging minor release on stable/newton * bump it in upstream stable/newton upper-constraints.txt * sync that bump in rdoinfo (should be automatic now)