[rdo-list] [tripleo] Troubles deploying Libery with HA setup

Luca 'remix_tj' Lorenzetto lorenzetto.luca at gmail.com
Tue Aug 2 15:59:00 UTC 2016 

Hello,

I'm deploying Liberty on a set of server using tripleo. I imported and
correctly introspected 6 nodes. First i did a setup with 1 controller
node and 3 computing nodes. No issues during this deploy, everything
went fine. On this deployment i used custom hostname formats (defined
via yaml environment file) and custom dns_domain (changed in
/etc/nova/nova.conf and /etc/neutron/dhcp_agent.ini and all
/usr/share/openstack-tripleo-heat-templates/*.yaml files that contains
localdomain)

Now I'm working to have a 3 controller setup with HA. I'm running this
command to deploy:

openstack overcloud deploy --templates -e
/usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml
-e ~/templates/network-environment.yaml -e
~/templates/puppet-ceph-external.yaml -e ~/hostname-nostri.yaml
--neutron-bridge-mappings datacentre:br-ex,storage-pub:br-stg-pub -e
/usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml
--ntp-server timesrv1 --control-scale 3 --compute-scale 3
--ceph-storage-scale 0 --control-flavor controlhp --compute-flavor
computehp --neutron-network-type vxlan --neutron-tunnel-types vxlan
--verbose --debug --log-file overcloud_$(date +%T).log

I see that stack creation starts and correctly deploy os on the nodes.
Nodes are named according to ControllerHostnameFormat and
ComputeHostnameFormat that i specified into ~/hostname-nostri.yaml.

Everything goes well until HA configuration starts. I see this stack
creation failing:

overcloud-ControllerNodesPostDeployment-bbp3c47jgau2-ControllerLoadBalancerDeployment_Step1-sla6ce7n2arq

The error message of the deployment command is:

Stack failed with status: Resource CREATE failed:
resources.ControllerLoadBalancerDeployment_Step1:
resources.ControllerNodesPostDeployment.Error: resources[0]:
Deployment to server failed: deploy_status_code: Deployment exited
with non-zero status code: 6
Heat Stack create failed.


If i go in depth with heat deployment-show i see that all resources
report this deploy_stderr:

Error: Could not prefetch mysql_user provider 'mysql': Execution of
'/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM
mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local
MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Error: Could not prefetch mysql_database provider 'mysql': Execution
of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002
(HY000): Can't connect to local MySQL server through socket
'/var/lib/mysql/mysql.sock' (2)\
Error: Command exceeded timeout
Error: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns:
change from notrun to 0 failed: Command exceeded timeout
Warning: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster
tripleo_cluster]: Skipping because of failed dependencies
Warning: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster
tripleo_cluster]: Skipping because of failed dependencies
Warning: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]:
Skipping because of failed dependencies
Warning: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]:
Skipping because of failed dependencies
Warning: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]:
Skipping because of failed dependencies",



I see nodes are stuck on puppet step
"auth-successful-across-all-nodes" (defined in
/etc/puppet/modules/pacemaker/manifests/corosync.pp)

/usr/bin/python2 /usr/sbin/pcs cluster auth opsctrl0 opsctrl1 opsctrl2
-u hacluster -p PASSWORD --force

I suppose that the problem is due to corosync service not yet started.
But as far as i can see corosync will never start because
/etc/corosync/corosync.conf file is missing.

in /etc/hosts opsctrl{0-2} are correctly defined and each host can
talk with the others.



I'm stuck and i don't know what to do. Anyone had similar issues? I'm
missing something that i have to enter in the configuration?


I'm using a in-house mirror of rdo-release made on April 20, and
custom images build from rhel image and this repository. A week ago i
tried with latest repository but had the same error (and also others,
that's why i returned to this older mirror that was working).

This is the list of packages installed from rdo-release repository:

crudini-0.7-1.el7.noarch
dib-utils-0.0.9-1.el7.noarch
dibbler-client-1.0.1-0.RC1.2.el7.x86_64
diskimage-builder-1.4.0-1.el7.noarch
erlang-asn1-R16B-03.16.el7.x86_64
erlang-compiler-R16B-03.16.el7.x86_64
erlang-crypto-R16B-03.16.el7.x86_64
erlang-erts-R16B-03.16.el7.x86_64
erlang-hipe-R16B-03.16.el7.x86_64
erlang-inets-R16B-03.16.el7.x86_64
erlang-kernel-R16B-03.16.el7.x86_64
erlang-mnesia-R16B-03.16.el7.x86_64
erlang-os_mon-R16B-03.16.el7.x86_64
erlang-otp_mibs-R16B-03.16.el7.x86_64
erlang-public_key-R16B-03.16.el7.x86_64
erlang-runtime_tools-R16B-03.16.el7.x86_64
erlang-sasl-R16B-03.16.el7.x86_64
erlang-sd_notify-0.1-1.el7.x86_64
erlang-snmp-R16B-03.16.el7.x86_64
erlang-ssl-R16B-03.16.el7.x86_64
erlang-stdlib-R16B-03.16.el7.x86_64
erlang-syntax_tools-R16B-03.16.el7.x86_64
erlang-tools-R16B-03.16.el7.x86_64
erlang-xmerl-R16B-03.16.el7.x86_64
hiera-1.3.4-1.el7.noarch
instack-0.0.8-1.el7.noarch
instack-undercloud-2.1.3-1.el7.noarch
jq-1.3-2.el7.x86_64
liberasurecode-1.1.1-1.el7.x86_64
libnetfilter_queue-1.0.2-2.el7.x86_64
memcached-1.4.25-1.el7.x86_64
1:openstack-ceilometer-alarm-5.0.2-1.el7.noarch
1:openstack-ceilometer-api-5.0.2-1.el7.noarch
1:openstack-ceilometer-central-5.0.2-1.el7.noarch
1:openstack-ceilometer-collector-5.0.2-1.el7.noarch
1:openstack-ceilometer-common-5.0.2-1.el7.noarch
1:openstack-ceilometer-notification-5.0.2-1.el7.noarch
1:openstack-ceilometer-polling-5.0.2-1.el7.noarch
1:openstack-glance-11.0.1-2.el7.noarch
1:openstack-heat-api-5.0.0-1.el7.noarch
1:openstack-heat-api-cfn-5.0.0-1.el7.noarch
1:openstack-heat-api-cloudwatch-5.0.0-1.el7.noarch
1:openstack-heat-common-5.0.0-1.el7.noarch
1:openstack-heat-engine-5.0.0-1.el7.noarch
openstack-heat-templates-0-0.1.20151019.el7.noarch
1:openstack-ironic-api-4.2.2-1.el7.noarch
1:openstack-ironic-common-4.2.2-1.el7.noarch
1:openstack-ironic-conductor-4.2.2-1.el7.noarch
openstack-ironic-inspector-2.2.2-1.el7.noarch
1:openstack-keystone-8.0.1-1.el7.noarch
1:openstack-neutron-7.0.3-1.el7.noarch
1:openstack-neutron-common-7.0.3-1.el7.noarch
1:openstack-neutron-ml2-7.0.3-1.el7.noarch
1:openstack-neutron-openvswitch-7.0.3-1.el7.noarch
1:openstack-nova-api-12.0.1-1.el7.noarch
1:openstack-nova-cert-12.0.1-1.el7.noarch
1:openstack-nova-common-12.0.1-1.el7.noarch
1:openstack-nova-compute-12.0.1-1.el7.noarch
1:openstack-nova-conductor-12.0.1-1.el7.noarch
1:openstack-nova-scheduler-12.0.1-1.el7.noarch
1:openstack-puppet-modules-7.0.1-1.el7.noarch
openstack-selinux-0.6.41-1.el7.noarch
openstack-swift-2.5.0-1.el7.noarch
openstack-swift-account-2.5.0-1.el7.noarch
openstack-swift-container-2.5.0-1.el7.noarch
openstack-swift-object-2.5.0-1.el7.noarch
openstack-swift-plugin-swift3-1.7-4.el7.noarch
openstack-swift-proxy-2.5.0-1.el7.noarch
openstack-tripleo-0.0.6-1.el7.noarch
openstack-tripleo-heat-templates-0.8.7-1.el7.noarch
openstack-tripleo-image-elements-0.9.7-1.el7.noarch
openstack-tripleo-puppet-elements-0.0.2-1.el7.noarch
openstack-utils-2015.2-1.el7.noarch
openvswitch-2.4.0-1.el7.x86_64
os-apply-config-0.1.32-3.el7.noarch
os-cloud-config-0.2.10-2.el7.noarch
os-collect-config-0.1.36-4.el7.noarch
os-net-config-0.1.5-3.el7.noarch
os-refresh-config-0.1.11-2.el7.noarch
puppet-3.6.2-3.el7.noarch
pyOpenSSL-0.15.1-1.el7.noarch
pyparsing-2.0.3-1.el7.noarch
pysendfile-2.0.0-5.el7.x86_64
pysnmp-4.2.5-2.el7.noarch
pystache-0.5.3-2.el7.noarch
python-alembic-0.8.3-3.el7.noarch
python-amqp-1.4.6-1.el7.noarch
python-anyjson-0.3.3-3.el7.noarch
python-automaton-0.7.0-1.el7.noarch
python-babel-1.3-6.el7.noarch
python-bson-3.0.3-1.el7.x86_64
python-cachetools-1.0.3-2.el7.noarch
1:python-ceilometer-5.0.2-1.el7.noarch
python-ceilometerclient-1.5.0-1.el7.noarch
python-cinderclient-1.4.0-1.el7.noarch
python-cliff-1.15.0-1.el7.noarch
python-cliff-tablib-1.1-3.el7.noarch
python-cmd2-0.6.8-3.el7.noarch
python-contextlib2-0.4.0-1.el7.noarch
python-croniter-0.3.4-2.el7.noarch
python-dogpile-cache-0.5.7-3.el7.noarch
python-dogpile-core-0.4.1-2.el7.noarch
python-ecdsa-0.11-3.el7.noarch
python-editor-0.4-4.el7.noarch
python-elasticsearch-1.4.0-2.el7.noarch
python-extras-0.0.3-2.el7.noarch
python-fixtures-1.4.0-2.el7.noarch
python-futures-3.0.3-1.el7.noarch
1:python-glance-11.0.1-2.el7.noarch
python-glance-store-0.9.1-1.el7.noarch
1:python-glanceclient-1.1.0-1.el7.noarch
python-heatclient-0.8.0-1.el7.noarch
python-httplib2-0.9.2-1.el7.noarch
python-idna-2.0-1.el7.noarch
python-ipaddress-1.0.7-4.el7.noarch
python-ironicclient-0.8.1-1.el7.noarch
python-jsonpatch-1.2-2.el7.noarch
python-jsonpath-rw-1.2.3-2.el7.noarch
python-jsonschema-2.3.0-1.el7.noarch
python-keyring-5.0-4.el7.noarch
1:python-keystone-8.0.1-1.el7.noarch
1:python-keystoneclient-1.7.2-1.el7.noarch
python-keystonemiddleware-2.3.1-1.el7.noarch
1:python-kombu-3.0.32-1.el7.noarch
python-ldappool-1.0-4.el7.noarch
python-linecache2-1.0.0-1.el7.noarch
python-logutils-0.3.3-3.el7.noarch
python-memcached-1.54-3.el7.noarch
python-migrate-0.10.0-1.el7.noarch
python-mimeparse-0.1.4-1.el7.noarch
python-monotonic-0.3-1.el7.noarch
python-ncclient-0.4.2-2.el7.noarch
python-netaddr-0.7.18-1.el7.noarch
python-netifaces-0.10.4-1.el7.x86_64
python-networkx-core-1.10-1.el7.noarch
1:python-neutron-7.0.3-1.el7.noarch
python-neutronclient-3.1.0-1.el7.noarch
python-nose-1.3.7-7.el7.noarch
1:python-nova-12.0.1-1.el7.noarch
1:python-novaclient-2.30.1-1.el7.noarch
python-oauthlib-0.7.2-5.20150520git514cad7.el7.noarch
python-openstackclient-1.7.2-1.el7.noarch
python-openvswitch-2.4.0-1.el7.noarch
python-oslo-cache-0.7.0-1.el7.noarch
python-oslo-concurrency-2.6.0-1.el7.noarch
python-oslo-db-2.6.0-3.el7.noarch
python-oslo-log-1.10.0-1.el7.noarch
python-oslo-messaging-2.5.0-1.el7.noarch
python-oslo-middleware-2.8.0-1.el7.noarch
python-oslo-policy-0.11.0-1.el7.noarch
python-oslo-rootwrap-2.3.0-1.el7.noarch
python-oslo-service-0.9.0-1.el7.noarch
python-oslo-versionedobjects-0.10.0-1.el7.noarch
python-oslo-vmware-1.21.0-1.el7.noarch
python-osprofiler-0.3.0-1.el7.noarch
python-paramiko-1.15.1-1.el7.noarch
python-paste-deploy-1.5.2-6.el7.noarch
python-pbr-1.8.1-2.el7.noarch
python-posix_ipc-0.9.8-1.el7.x86_64
python-prettytable-0.7.2-1.el7.noarch
python-proliantutils-2.1.7-1.el7.noarch
python-psutil-1.2.1-1.el7.x86_64
python-pycadf-1.1.0-1.el7.noarch
python-pyeclib-1.2.0-1.el7.x86_64
python-pyghmi-0.8.0-2.el7.noarch
python-pygments-2.0.2-4.el7.noarch
python-pymongo-3.0.3-1.el7.x86_64
python-pysaml2-3.0.2-1.el7.noarch
python-qpid-0.30-1.el7.noarch
python-qpid-common-0.30-1.el7.noarch
python-repoze-lru-0.4-3.el7.noarch
python-repoze-who-2.1-1.el7.noarch
python-requests-2.9.1-2.el7.noarch
python-retrying-1.2.3-4.el7.noarch
python-rfc3986-0.2.0-1.el7.noarch
python-routes-1.13-2.el7.noarch
python-saharaclient-0.11.1-1.el7.noarch
python-semantic_version-2.4.2-1.el7.noarch
python-simplegeneric-0.8-7.el7.noarch
python-simplejson-3.5.3-5.el7.x86_64
python-sqlalchemy-1.0.11-1.el7.x86_64
python-sqlparse-0.1.18-5.el7.noarch
python-stevedore-1.8.0-1.el7.noarch
python-swiftclient-2.6.0-1.el7.noarch
python-tablib-0.10.0-1.el7.noarch
python-taskflow-1.21.0-1.el7.noarch
python-tempita-0.5.1-8.el7.noarch
python-testtools-1.8.0-2.el7.noarch
python-tooz-1.24.0-1.el7.noarch
python-traceback2-1.4.0-2.el7.noarch
python-tripleoclient-0.0.11-3.el7.noarch
python-troveclient-1.3.0-1.el7.noarch
python-unicodecsv-0.14.1-1.el7.noarch
python-unittest2-1.0.1-1.el7.noarch
python-urllib3-1.13.1-3.el7.noarch
python-warlock-1.0.1-1.el7.noarch
python-webob-1.4.1-2.el7.noarch
python-websockify-0.6.0-2.el7.noarch
python-wrapt-1.10.5-3.el7.x86_64
python2-PyMySQL-0.6.7-2.el7.noarch
python2-appdirs-1.4.0-4.el7.noarch
python2-castellan-0.3.1-1.el7.noarch
python2-cffi-1.5.2-1.el7.x86_64
python2-cryptography-1.2.1-3.el7.x86_64
python2-debtcollector-0.8.0-1.el7.noarch
python2-eventlet-0.17.4-4.el7.noarch
python2-fasteners-0.14.1-4.el7.noarch
python2-funcsigs-0.4-2.el7.noarch
python2-futurist-0.5.0-1.el7.noarch
python2-greenlet-0.4.9-1.el7.x86_64
python2-ironic-inspector-client-1.2.0-2.el7.noarch
python2-iso8601-0.1.11-1.el7.noarch
python2-jsonpath-rw-ext-0.1.7-1.1.el7.noarch
python2-mock-1.3.0-1.el7.noarch
python2-os-brick-0.5.0-1.el7.noarch
python2-os-client-config-1.7.4-1.el7.noarch
2:python2-oslo-config-2.4.0-1.el7.noarch
python2-oslo-context-0.6.0-1.el7.noarch
python2-oslo-i18n-2.6.0-1.el7.noarch
python2-oslo-reports-0.5.0-1.el7.noarch
python2-oslo-serialization-1.9.0-1.el7.noarch
python2-oslo-utils-2.5.0-1.el7.noarch
python2-passlib-1.6.5-1.el7.noarch
python2-pecan-1.0.2-2.el7.noarch
python2-pyasn1-0.1.9-6.el7.1.noarch
python2-rsa-3.3-2.el7.noarch
python2-singledispatch-3.4.0.3-4.el7.noarch
python2-suds-0.7-0.1.94664ddd46a6.el7.noarch
python2-wsme-0.7.0-2.el7.noarch
rabbitmq-server-3.3.5-17.el7.noarch
ruby-augeas-0.5.0-1.el7.x86_64
ruby-shadow-1.4.1-23.el7.x86_64
rubygem-rgen-0.6.6-2.el7.noarch
tripleo-common-0.0.1-2.el7.noarch


Thank you,


Luca





-- 
"E' assurdo impiegare gli uomini di intelligenza eccellente per fare
calcoli che potrebbero essere affidati a chiunque se si usassero delle
macchine"
Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)

"Internet è la più grande biblioteca del mondo.
Ma il problema è che i libri sono tutti sparsi sul pavimento"
John Allen Paulos, Matematico (1945-vivente)

Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca at gmail.com>

More information about the dev mailing list