[rdo-list] Tripleo QuickStart HA deployment attempts constantly crash

Fri Jun 3 20:53:08 UTC 2016

I just did an HA deploy locally on master, and I see the same thing wrt
telemetry services being down due to failed redis import. That could be
a packaging bug (something should depend on python-redis, maybe
python-tooz?). That said, it does not appear fatal in my case. Is there
some issue other than telemetry services being down that you are seeing?
That is certainly something we should fix, but I wouldn't characterize
it as the deployment is constantly crashing.

On 06/03/2016 11:30 AM, Boris Derzhavets wrote:
> 1. Attempting to address your concern ( if I understood you correct )
> 
> First log :-
> 
>  [root at overcloud-controller-0 ceilometer]# cat  central.log | grep ERROR
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service [req-4db5f172-0bf0-4200-9cf4-174859cdc00b admin - - - -] Error starting thread.
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service Traceback (most recent call last):
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 680, in run_service
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     service.start()
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 384, in start
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     self.partition_coordinator.start()
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ceilometer/coordination.py", line 84, in start
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     backend_url, self._my_id)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 539, in get_coordinator
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     invoke_args=(member_id, parsed_url, options)).driver
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 46, in __init__
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements=verify_requirements,
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 171, in _load_plugins
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     self._on_load_failure_callback(self, ep, err)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements,
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements,
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 186, in _load_one_plugin
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     plugin = ep.load(require=verify_requirements)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 27, in <module>
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     import redis
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service ImportError: No module named redis
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service 
> [root at overcloud-controller-0 ceilometer]# clear
> [3;J
> [root at overcloud-controller-0 ceilometer]# cat  central.log | grep ERROR
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service [req-4db5f172-0bf0-4200-9cf4-174859cdc00b admin - - - -] Error starting thread.
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service Traceback (most recent call last):
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 680, in run_service
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     service.start()
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 384, in start
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     self.partition_coordinator.start()
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/ceilometer/coordination.py", line 84, in start
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     backend_url, self._my_id)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 539, in get_coordinator
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     invoke_args=(member_id, parsed_url, options)).driver
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 46, in __init__
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements=verify_requirements,
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 171, in _load_plugins
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     self._on_load_failure_callback(self, ep, err)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements,
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     verify_requirements,
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 186, in _load_one_plugin
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     plugin = ep.load(require=verify_requirements)
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 27, in <module>
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service     import redis
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service ImportError: No module named redis
> 2016-06-03 08:50:04.405 17503 ERROR oslo_service.service 
> 
> Second log :-
> 
> [root at overcloud-controller-0 ceilometer]# cd -
> /var/log/aodh
> [root at overcloud-controller-0 aodh]# cat  evaluator.log | grep ERROR
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service [-] Error starting thread.
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service Traceback (most recent call last):
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 680, in run_service
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     service.start()
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/aodh/evaluator/__init__.py", line 229, in start
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     self.partition_coordinator.start()
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/aodh/coordination.py", line 133, in start
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     self.backend_url, self._my_id)
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 539, in get_coordinator
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     invoke_args=(member_id, parsed_url, options)).driver
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 46, in __init__
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     verify_requirements=verify_requirements,
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     verify_requirements)
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 171, in _load_plugins
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     self._on_load_failure_callback(self, ep, err)
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     verify_requirements,
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     verify_requirements,
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 186, in _load_one_plugin
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     plugin = ep.load(require=verify_requirements)
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     entry = __import__(self.module_name, globals(),globals(), ['__name__'])
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 27, in <module>
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service     import redis
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service ImportError: No module named redis
> 2016-06-03 08:46:20.552 32101 ERROR oslo_service.service 
> 
>  2 . Memory DIMMs DDR3 ( Kingston HyperX 1600 MHZ ) is not a problem
>       My board ASUS Z97-P cannot support more 32 GB.  So ....  
> 
>  3. i7  4790  surprised me on doing deployment on TripleO Quickstart , in particular, Controller+2xComputes ( --compute-scale 2 )
> 
> Thank you
> Boris.
> ________________________________________
> From: John Trowbridge <trown at redhat.com>
> Sent: Friday, June 3, 2016 8:43 AM
> To: Boris Derzhavets; John Trowbridge; Lars Kellogg-Stedman
> Cc: rdo-list
> Subject: Re: [rdo-list] Tripleo QuickStart HA deployment attempts constantly crash
> 
> So this last one looks like telemetry services went down. You could
> check the logs on the controllers to see if it was OOM killed. My bet
> would be this is what is happening.
> 
> The reason that HA is not the default for tripleo-quickstart is exactly
> this type of issue. It is pretty difficult to fit a full HA deployment
> of TripleO on a 32G virthost. I think there is near 100% chance that the
> default HA config will crash when trying to do anything on the
> deployed overcloud, due to running out of memory.
> 
> I have had some success in my local test setup using KSM [1] on the
> virthost, and then changing the HA config to give the controllers more
> memory. This results in overcommiting, but KSM can handle overcommiting
> without going into swap. It might even be possible to try to setup KSM
> in the environment setup part of quickstart. I would certainly accept an
> RFE/patch for this [2,3].
> 
> If you have a larger virthost than 32G, you could similarly bump the
> memory for the controllers, which should lead to a much higher success rate.
> 
> There is also a feature coming in TripleO [4] that will allow choosing
> what services get deployed in each role, which will allow us to tweak
> the tripleo-quickstart HA config to deploy a minimal service layout in
> order to reduce memory requirements.
> 
> Thanks a ton for giving tripleo-quickstart a go!
> 
> [1] https://en.wikipedia.org/wiki/Kernel_same-page_merging
> [2] https://bugs.launchpad.net/tripleo-quickstart
> [3] https://review.openstack.org/#/q/project:openstack/tripleo-quickstart
> [4]
> https://blueprints.launchpad.net/tripleo/+spec/composable-services-within-roles
> 
> On 06/03/2016 06:20 AM, Boris Derzhavets wrote:
>> =====================================
>>
>> Fresh HA deployment attempt
>>
>> =====================================
>>
>> [stack at undercloud ~]$ date
>> Fri Jun  3 10:05:35 UTC 2016
>> [stack at undercloud ~]$ heat stack-list
>> +--------------------------------------+------------+-----------------+---------------------+--------------+
>> | id                                   | stack_name | stack_status    | creation_time       | updated_time |
>> +--------------------------------------+------------+-----------------+---------------------+--------------+
>> | 0c6b8205-be86-4a24-be36-fd4ece956c6d | overcloud  | CREATE_COMPLETE | 2016-06-03T08:14:19 | None         |
>> +--------------------------------------+------------+-----------------+---------------------+--------------+
>> [stack at undercloud ~]$ nova list
>> +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
>> | ID                                   | Name                    | Status | Task State | Power State | Networks            |
>> +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
>> | 6a38b7be-3743-4339-970b-6121e687741d | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
>> | 9222dc1b-5974-495b-8b98-b8176ac742f4 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
>> | 76adbb27-220f-42ef-9691-94729ee28749 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
>> | 8f57f7b6-a2d8-4b7b-b435-1c675e63ea84 | overcloud-novacompute-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.8  |
>> +--------------------------------------+-------------------------+--------+------------+-------------+---------------------+
>> [stack at undercloud ~]$ ssh heat-admin at 192.0.2.10
>> Last login: Fri Jun  3 10:01:44 2016 from gateway
>> [heat-admin at overcloud-controller-0 ~]$ sudo su -
>> Last login: Fri Jun  3 10:01:49 UTC 2016 on pts/0
>> [root at overcloud-controller-0 ~]# .  keystonerc_admin
>>
>> [root at overcloud-controller-0 ~]# pcs status
>> Cluster name: tripleo_cluster
>> Last updated: Fri Jun  3 10:07:22 2016        Last change: Fri Jun  3 08:50:59 2016 by root via cibadmin on overcloud-controller-0
>> Stack: corosync
>> Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
>> 3 nodes and 123 resources configured
>>
>> Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>
>> Full list of resources:
>>
>>  ip-192.0.2.6    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-0
>>  Clone Set: haproxy-clone [haproxy]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  ip-192.0.2.7    (ocf::heartbeat:IPaddr2):    Started overcloud-controller-1
>>  Master/Slave Set: galera-master [galera]
>>      Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: memcached-clone [memcached]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: rabbitmq-clone [rabbitmq]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-core-clone [openstack-core]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Master/Slave Set: redis-master [redis]
>>      Masters: [ overcloud-controller-1 ]
>>      Slaves: [ overcloud-controller-0 overcloud-controller-2 ]
>>  Clone Set: mongod-clone [mongod]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  openstack-cinder-volume    (systemd:openstack-cinder-volume):    Started overcloud-controller-2
>>  Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-heat-api-clone [openstack-heat-api]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-glance-api-clone [openstack-glance-api]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-nova-api-clone [openstack-nova-api]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-sahara-api-clone [openstack-sahara-api]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-sahara-engine-clone [openstack-sahara-engine]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: delay-clone [delay]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: neutron-server-clone [neutron-server]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
>>      Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: httpd-clone [httpd]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>  Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
>>      Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
>>
>> Failed Actions:
>> * openstack-aodh-evaluator_monitor_60000 on overcloud-controller-1 'not running' (7): call=76, status=complete, exitreason='none',
>>     last-rc-change='Fri Jun  3 08:47:22 2016', queued=0ms, exec=0ms
>> * openstack-ceilometer-central_start_0 on overcloud-controller-1 'not running' (7): call=290, status=complete, exitreason='none',
>>     last-rc-change='Fri Jun  3 08:51:18 2016', queued=0ms, exec=2132ms
>> * openstack-aodh-evaluator_monitor_60000 on overcloud-controller-2 'not running' (7): call=76, status=complete, exitreason='none',
>>     last-rc-change='Fri Jun  3 08:47:16 2016', queued=0ms, exec=0ms
>> * openstack-ceilometer-central_start_0 on overcloud-controller-2 'not running' (7): call=292, status=complete, exitreason='none',
>>     last-rc-change='Fri Jun  3 08:51:31 2016', queued=0ms, exec=2102ms
>> * openstack-aodh-evaluator_monitor_60000 on overcloud-controller-0 'not running' (7): call=77, status=complete, exitreason='none',
>>     last-rc-change='Fri Jun  3 08:47:19 2016', queued=0ms, exec=0ms
>> * openstack-ceilometer-central_start_0 on overcloud-controller-0 'not running' (7): call=270, status=complete, exitreason='none',
>>     last-rc-change='Fri Jun  3 08:50:02 2016', queued=0ms, exec=2199ms
>>
>>
>> PCSD Status:
>>   overcloud-controller-0: Online
>>   overcloud-controller-1: Online
>>   overcloud-controller-2: Online
>>
>> Daemon Status:
>>   corosync: active/enabled
>>   pacemaker: active/enabled
>>   pcsd: active/enabled
>>
>>
>> ________________________________
>> From: rdo-list-bounces at redhat.com <rdo-list-bounces at redhat.com> on behalf of Boris Derzhavets <bderzhavets at hotmail.com>
>> Sent: Monday, May 30, 2016 4:56 AM
>> To: John Trowbridge; Lars Kellogg-Stedman
>> Cc: rdo-list
>> Subject: Re: [rdo-list] Tripleo QuickStart HA deployment attempts constantly crash
>>
>>
>> Done one more time :-
>>
>>
>> [stack at undercloud ~]$ heat deployment-show 9cc8087a-6d82-4261-8a13-ee8c46e3a02d
>>
>> Uploaded here :-
>>
>> http://textuploader.com/5bm5v
>> ________________________________
>> From: rdo-list-bounces at redhat.com <rdo-list-bounces at redhat.com> on behalf of Boris Derzhavets <bderzhavets at hotmail.com>
>> Sent: Sunday, May 29, 2016 3:39 AM
>> To: John Trowbridge; Lars Kellogg-Stedman
>> Cc: rdo-list
>> Subject: [rdo-list] Tripleo QuickStart HA deploymemt attempts constantly crash
>>
>>
>> Error every time is the same :-
>>
>>
>> 2016-05-29 07:20:17 [0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
>> 2016-05-29 07:20:18 [0]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:18 [overcloud-ControllerNodesPostDeployment-dzawjmjyaidt-ControllerServicesBaseDeployment_Step2-ufz2ccs5egd7]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
>> 2016-05-29 07:20:18 [0]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:19 [ControllerServicesBaseDeployment_Step2]: CREATE_FAILED Error: resources.ControllerServicesBaseDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
>> 2016-05-29 07:20:19 [0]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:19 [0]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:20 [ControllerDeployment]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:20 [overcloud-ControllerNodesPostDeployment-dzawjmjyaidt]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerServicesBaseDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
>> 2016-05-29 07:20:21 [ControllerNodesPostDeployment]: CREATE_FAILED Error: resources.ControllerNodesPostDeployment.resources.ControllerServicesBaseDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
>> 2016-05-29 07:20:21 [0]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:22 [NetworkDeployment]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:20:22 [0]: SIGNAL_COMPLETE Unknown
>> 2016-05-29 07:24:22 [ComputeNodesPostDeployment]: CREATE_FAILED CREATE aborted
>> 2016-05-29 07:24:22 [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerServicesBaseDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
>> Stack overcloud CREATE_FAILED
>> Deployment failed:  Heat Stack create failed.
>> + heat stack-list
>> + grep -q CREATE_FAILED
>> + deploy_status=1
>> ++ heat resource-list --nested-depth 5 overcloud
>> ++ grep FAILED
>> ++ grep 'StructuredDeployment '
>> ++ cut -d '|' -f3
>> + for failed in '$(heat resource-list         --nested-depth 5 overcloud | grep FAILED |
>>         grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)'
>> + heat deployment-show 66bd3fbe-296b-4f88-87a7-5ceafd05c1d3
>> + exit 1
>>
>>
>> Minimal configuration deployments run with no errors and build completely functional environment.
>>
>>
>> However,   template :-
>>
>>
>> #################################
>> # Test Controller + 2*Compute nodes
>> #################################
>> control_memory: 6144
>> compute_memory: 6144
>>
>> undercloud_memory: 8192
>>
>> # Giving the undercloud additional CPUs can greatly improve heat's
>> # performance (and result in a shorter deploy time).
>> undercloud_vcpu: 4
>>
>> # We set introspection to true and use only the minimal amount of nodes
>> # for this job, but test all defaults otherwise.
>> step_introspect: true
>>
>> # Define a single controller node and a single compute node.
>> overcloud_nodes:
>>   - name: control_0
>>     flavor: control
>>
>>   - name: compute_0
>>     flavor: compute
>>
>>   - name: compute_1
>>     flavor: compute
>>
>> # Tell tripleo how we want things done.
>> extra_args: >-
>>   --neutron-network-type vxlan
>>   --neutron-tunnel-types vxlan
>>   --ntp-server pool.ntp.org
>>
>> network_isolation: true
>>
>>
>> Picks up new memory setting but doesn't create second Compute Node.
>>
>> Every time just Controller && (1)* Compute.
>>
>>
>> HW - i74790 , 32 GB RAM
>>
>>
>> Thanks.
>>
>> Boris
>>
>> ________________________________
>>
>>
>>
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>