Hello, I'm deploying Liberty on a set of server using tripleo. I imported and correctly introspected 6 nodes. First i did a setup with 1 controller node and 3 computing nodes. No issues during this deploy, everything went fine. On this deployment i used custom hostname formats (defined via yaml environment file) and custom dns_domain (changed in /etc/nova/nova.conf and /etc/neutron/dhcp_agent.ini and all /usr/share/openstack-tripleo-heat-templates/*.yaml files that contains localdomain) Now I'm working to have a 3 controller setup with HA. I'm running this command to deploy: openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network- isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/puppet-ceph-external.yaml -e ~/hostname-nostri.yaml --neutron-bridge-mappings datacentre:br-ex,storage-pub:br-stg-pub -e /usr/share/openstack-tripleo-heat-templates/environments/puppet- pacemaker.yaml --ntp-server timesrv1 --control-scale 3 --compute-scale 3 --ceph-storage-scale 0 --control-flavor controlhp --compute-flavor computehp --neutron-network-type vxlan --neutron-tunnel-types vxlan --verbose --debug --log-file overcloud_$(date +%T).log I see that stack creation starts and correctly deploy os on the nodes. Nodes are named according to ControllerHostnameFormat and ComputeHostnameFormat that i specified into ~/hostname-nostri.yaml. Everything goes well until HA configuration starts. I see this stack creation failing: overcloud-ControllerNodesPostDeployment-bbp3c47jgau2- ControllerLoadBalancerDeployment_Step1-sla6ce7n2arq The error message of the deployment command is: Stack failed with status: Resource CREATE failed: resources.ControllerLoadBalancerDeployment_Step1: resources.ControllerNodesPostDeployment.Error: resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6 Heat Stack create failed. If i go in depth with heat deployment-show i see that all resources report this deploy_stderr: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\ Error: Command exceeded timeout Error: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across- all-nodes]/returns: change from notrun to 0 failed: Command exceeded timeout Warning: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tripleo_cluster]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]: Skipping because of failed dependencies", I see nodes are stuck on puppet step "auth-successful-across-all-nodes" (defined in /etc/puppet/modules/pacemaker/manifests/corosync.pp) /usr/bin/python2 /usr/sbin/pcs cluster auth opsctrl0 opsctrl1 opsctrl2 -u hacluster -p PASSWORD --force I suppose that the problem is due to corosync service not yet started. But as far as i can see corosync will never start because /etc/corosync/corosync.conf file is missing. in /etc/hosts opsctrl{0-2} are correctly defined and each host can talk with the others. I'm stuck and i don't know what to do. Anyone had similar issues? I'm missing something that i have to enter in the configuration? I'm using a in-house mirror of rdo-release made on April 20, and custom images build from rhel image and this repository. A week ago i tried with latest repository but had the same error (and also others, that's why i returned to this older mirror that was working). This is the list of packages installed from rdo-release repository: crudini-0.7-1.el7.noarch dib-utils-0.0.9-1.el7.noarch dibbler-client-1.0.1-0.RC1.2.el7.x86_64 diskimage-builder-1.4.0-1.el7.noarch erlang-asn1-R16B-03.16.el7.x86_64 erlang-compiler-R16B-03.16.el7.x86_64 erlang-crypto-R16B-03.16.el7.x86_64 erlang-erts-R16B-03.16.el7.x86_64 erlang-hipe-R16B-03.16.el7.x86_64 erlang-inets-R16B-03.16.el7.x86_64 erlang-kernel-R16B-03.16.el7.x86_64 erlang-mnesia-R16B-03.16.el7.x86_64 erlang-os_mon-R16B-03.16.el7.x86_64 erlang-otp_mibs-R16B-03.16.el7.x86_64 erlang-public_key-R16B-03.16.el7.x86_64 erlang-runtime_tools-R16B-03.16.el7.x86_64 erlang-sasl-R16B-03.16.el7.x86_64 erlang-sd_notify-0.1-1.el7.x86_64 erlang-snmp-R16B-03.16.el7.x86_64 erlang-ssl-R16B-03.16.el7.x86_64 erlang-stdlib-R16B-03.16.el7.x86_64 erlang-syntax_tools-R16B-03.16.el7.x86_64 erlang-tools-R16B-03.16.el7.x86_64 erlang-xmerl-R16B-03.16.el7.x86_64 hiera-1.3.4-1.el7.noarch instack-0.0.8-1.el7.noarch instack-undercloud-2.1.3-1.el7.noarch jq-1.3-2.el7.x86_64 liberasurecode-1.1.1-1.el7.x86_64 libnetfilter_queue-1.0.2-2.el7.x86_64 memcached-1.4.25-1.el7.x86_64 1:openstack-ceilometer-alarm-5.0.2-1.el7.noarch 1:openstack-ceilometer-api-5.0.2-1.el7.noarch 1:openstack-ceilometer-central-5.0.2-1.el7.noarch 1:openstack-ceilometer-collector-5.0.2-1.el7.noarch 1:openstack-ceilometer-common-5.0.2-1.el7.noarch 1:openstack-ceilometer-notification-5.0.2-1.el7.noarch 1:openstack-ceilometer-polling-5.0.2-1.el7.noarch 1:openstack-glance-11.0.1-2.el7.noarch 1:openstack-heat-api-5.0.0-1.el7.noarch 1:openstack-heat-api-cfn-5.0.0-1.el7.noarch 1:openstack-heat-api-cloudwatch-5.0.0-1.el7.noarch 1:openstack-heat-common-5.0.0-1.el7.noarch 1:openstack-heat-engine-5.0.0-1.el7.noarch openstack-heat-templates-0-0.1.20151019.el7.noarch 1:openstack-ironic-api-4.2.2-1.el7.noarch 1:openstack-ironic-common-4.2.2-1.el7.noarch 1:openstack-ironic-conductor-4.2.2-1.el7.noarch openstack-ironic-inspector-2.2.2-1.el7.noarch 1:openstack-keystone-8.0.1-1.el7.noarch 1:openstack-neutron-7.0.3-1.el7.noarch 1:openstack-neutron-common-7.0.3-1.el7.noarch 1:openstack-neutron-ml2-7.0.3-1.el7.noarch 1:openstack-neutron-openvswitch-7.0.3-1.el7.noarch 1:openstack-nova-api-12.0.1-1.el7.noarch 1:openstack-nova-cert-12.0.1-1.el7.noarch 1:openstack-nova-common-12.0.1-1.el7.noarch 1:openstack-nova-compute-12.0.1-1.el7.noarch 1:openstack-nova-conductor-12.0.1-1.el7.noarch 1:openstack-nova-scheduler-12.0.1-1.el7.noarch 1:openstack-puppet-modules-7.0.1-1.el7.noarch openstack-selinux-0.6.41-1.el7.noarch openstack-swift-2.5.0-1.el7.noarch openstack-swift-account-2.5.0-1.el7.noarch openstack-swift-container-2.5.0-1.el7.noarch openstack-swift-object-2.5.0-1.el7.noarch openstack-swift-plugin-swift3-1.7-4.el7.noarch openstack-swift-proxy-2.5.0-1.el7.noarch openstack-tripleo-0.0.6-1.el7.noarch openstack-tripleo-heat-templates-0.8.7-1.el7.noarch openstack-tripleo-image-elements-0.9.7-1.el7.noarch openstack-tripleo-puppet-elements-0.0.2-1.el7.noarch openstack-utils-2015.2-1.el7.noarch openvswitch-2.4.0-1.el7.x86_64 os-apply-config-0.1.32-3.el7.noarch os-cloud-config-0.2.10-2.el7.noarch os-collect-config-0.1.36-4.el7.noarch os-net-config-0.1.5-3.el7.noarch os-refresh-config-0.1.11-2.el7.noarch puppet-3.6.2-3.el7.noarch pyOpenSSL-0.15.1-1.el7.noarch pyparsing-2.0.3-1.el7.noarch pysendfile-2.0.0-5.el7.x86_64 pysnmp-4.2.5-2.el7.noarch pystache-0.5.3-2.el7.noarch python-alembic-0.8.3-3.el7.noarch python-amqp-1.4.6-1.el7.noarch python-anyjson-0.3.3-3.el7.noarch python-automaton-0.7.0-1.el7.noarch python-babel-1.3-6.el7.noarch python-bson-3.0.3-1.el7.x86_64 python-cachetools-1.0.3-2.el7.noarch 1:python-ceilometer-5.0.2-1.el7.noarch python-ceilometerclient-1.5.0-1.el7.noarch python-cinderclient-1.4.0-1.el7.noarch python-cliff-1.15.0-1.el7.noarch python-cliff-tablib-1.1-3.el7.noarch python-cmd2-0.6.8-3.el7.noarch python-contextlib2-0.4.0-1.el7.noarch python-croniter-0.3.4-2.el7.noarch python-dogpile-cache-0.5.7-3.el7.noarch python-dogpile-core-0.4.1-2.el7.noarch python-ecdsa-0.11-3.el7.noarch python-editor-0.4-4.el7.noarch python-elasticsearch-1.4.0-2.el7.noarch python-extras-0.0.3-2.el7.noarch python-fixtures-1.4.0-2.el7.noarch python-futures-3.0.3-1.el7.noarch 1:python-glance-11.0.1-2.el7.noarch python-glance-store-0.9.1-1.el7.noarch 1:python-glanceclient-1.1.0-1.el7.noarch python-heatclient-0.8.0-1.el7.noarch python-httplib2-0.9.2-1.el7.noarch python-idna-2.0-1.el7.noarch python-ipaddress-1.0.7-4.el7.noarch python-ironicclient-0.8.1-1.el7.noarch python-jsonpatch-1.2-2.el7.noarch python-jsonpath-rw-1.2.3-2.el7.noarch python-jsonschema-2.3.0-1.el7.noarch python-keyring-5.0-4.el7.noarch 1:python-keystone-8.0.1-1.el7.noarch 1:python-keystoneclient-1.7.2-1.el7.noarch python-keystonemiddleware-2.3.1-1.el7.noarch 1:python-kombu-3.0.32-1.el7.noarch python-ldappool-1.0-4.el7.noarch python-linecache2-1.0.0-1.el7.noarch python-logutils-0.3.3-3.el7.noarch python-memcached-1.54-3.el7.noarch python-migrate-0.10.0-1.el7.noarch python-mimeparse-0.1.4-1.el7.noarch python-monotonic-0.3-1.el7.noarch python-ncclient-0.4.2-2.el7.noarch python-netaddr-0.7.18-1.el7.noarch python-netifaces-0.10.4-1.el7.x86_64 python-networkx-core-1.10-1.el7.noarch 1:python-neutron-7.0.3-1.el7.noarch python-neutronclient-3.1.0-1.el7.noarch python-nose-1.3.7-7.el7.noarch 1:python-nova-12.0.1-1.el7.noarch 1:python-novaclient-2.30.1-1.el7.noarch python-oauthlib-0.7.2-5.20150520git514cad7.el7.noarch python-openstackclient-1.7.2-1.el7.noarch python-openvswitch-2.4.0-1.el7.noarch python-oslo-cache-0.7.0-1.el7.noarch python-oslo-concurrency-2.6.0-1.el7.noarch python-oslo-db-2.6.0-3.el7.noarch python-oslo-log-1.10.0-1.el7.noarch python-oslo-messaging-2.5.0-1.el7.noarch python-oslo-middleware-2.8.0-1.el7.noarch python-oslo-policy-0.11.0-1.el7.noarch python-oslo-rootwrap-2.3.0-1.el7.noarch python-oslo-service-0.9.0-1.el7.noarch python-oslo-versionedobjects-0.10.0-1.el7.noarch python-oslo-vmware-1.21.0-1.el7.noarch python-osprofiler-0.3.0-1.el7.noarch python-paramiko-1.15.1-1.el7.noarch python-paste-deploy-1.5.2-6.el7.noarch python-pbr-1.8.1-2.el7.noarch python-posix_ipc-0.9.8-1.el7.x86_64 python-prettytable-0.7.2-1.el7.noarch python-proliantutils-2.1.7-1.el7.noarch python-psutil-1.2.1-1.el7.x86_64 python-pycadf-1.1.0-1.el7.noarch python-pyeclib-1.2.0-1.el7.x86_64 python-pyghmi-0.8.0-2.el7.noarch python-pygments-2.0.2-4.el7.noarch python-pymongo-3.0.3-1.el7.x86_64 python-pysaml2-3.0.2-1.el7.noarch python-qpid-0.30-1.el7.noarch python-qpid-common-0.30-1.el7.noarch python-repoze-lru-0.4-3.el7.noarch python-repoze-who-2.1-1.el7.noarch python-requests-2.9.1-2.el7.noarch python-retrying-1.2.3-4.el7.noarch python-rfc3986-0.2.0-1.el7.noarch python-routes-1.13-2.el7.noarch python-saharaclient-0.11.1-1.el7.noarch python-semantic_version-2.4.2-1.el7.noarch python-simplegeneric-0.8-7.el7.noarch python-simplejson-3.5.3-5.el7.x86_64 python-sqlalchemy-1.0.11-1.el7.x86_64 python-sqlparse-0.1.18-5.el7.noarch python-stevedore-1.8.0-1.el7.noarch python-swiftclient-2.6.0-1.el7.noarch python-tablib-0.10.0-1.el7.noarch python-taskflow-1.21.0-1.el7.noarch python-tempita-0.5.1-8.el7.noarch python-testtools-1.8.0-2.el7.noarch python-tooz-1.24.0-1.el7.noarch python-traceback2-1.4.0-2.el7.noarch python-tripleoclient-0.0.11-3.el7.noarch python-troveclient-1.3.0-1.el7.noarch python-unicodecsv-0.14.1-1.el7.noarch python-unittest2-1.0.1-1.el7.noarch python-urllib3-1.13.1-3.el7.noarch python-warlock-1.0.1-1.el7.noarch python-webob-1.4.1-2.el7.noarch python-websockify-0.6.0-2.el7.noarch python-wrapt-1.10.5-3.el7.x86_64 python2-PyMySQL-0.6.7-2.el7.noarch python2-appdirs-1.4.0-4.el7.noarch python2-castellan-0.3.1-1.el7.noarch python2-cffi-1.5.2-1.el7.x86_64 python2-cryptography-1.2.1-3.el7.x86_64 python2-debtcollector-0.8.0-1.el7.noarch python2-eventlet-0.17.4-4.el7.noarch python2-fasteners-0.14.1-4.el7.noarch python2-funcsigs-0.4-2.el7.noarch python2-futurist-0.5.0-1.el7.noarch python2-greenlet-0.4.9-1.el7.x86_64 python2-ironic-inspector-client-1.2.0-2.el7.noarch python2-iso8601-0.1.11-1.el7.noarch python2-jsonpath-rw-ext-0.1.7-1.1.el7.noarch python2-mock-1.3.0-1.el7.noarch python2-os-brick-0.5.0-1.el7.noarch python2-os-client-config-1.7.4-1.el7.noarch 2:python2-oslo-config-2.4.0-1.el7.noarch python2-oslo-context-0.6.0-1.el7.noarch python2-oslo-i18n-2.6.0-1.el7.noarch python2-oslo-reports-0.5.0-1.el7.noarch python2-oslo-serialization-1.9.0-1.el7.noarch python2-oslo-utils-2.5.0-1.el7.noarch python2-passlib-1.6.5-1.el7.noarch python2-pecan-1.0.2-2.el7.noarch python2-pyasn1-0.1.9-6.el7.1.noarch python2-rsa-3.3-2.el7.noarch python2-singledispatch-3.4.0.3-4.el7.noarch python2-suds-0.7-0.1.94664ddd46a6.el7.noarch python2-wsme-0.7.0-2.el7.noarch rabbitmq-server-3.3.5-17.el7.noarch ruby-augeas-0.5.0-1.el7.x86_64 ruby-shadow-1.4.1-23.el7.x86_64 rubygem-rgen-0.6.6-2.el7.noarch tripleo-common-0.0.1-2.el7.noarch Thank you, Luca -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca(a)gmail.com&gt; _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On Tue, Aug 2, 2016 at 6:34 PM, Christopher Brown <cbrown2(a)ocf.co.uk&gt; wrote: > > Hi Luca, Hi Christopher, > > > It rings a bell but to be honest I would try the following: > > 1. Check hosts file on controllers - I wonder what are the content > there? # HEAT_HOSTS_START - Do not edit manually within this section! 172.25.122.16 opskvmsvil0.company.it opskvmsvil0 172.25.122.15 opskvmsvil1.company.it opskvmsvil1 172.25.122.17 opskvmsvil2.company.it opskvmsvil2 172.25.122.13 opsctrl0.company.it opsctrl0 overcloud 172.25.122.12 opsctrl1.company.it opsctrl1 overcloud 172.25.122.14 opsctrl2.company.it opsctrl2 overcloud # HEAT_HOSTS_END Seems fine to me. This is exactly what i intended.

> > 2. Revert hostname change and see if it is happy I'll try. Seems a good solution, but can't understand why this will impact if hosts file is correct and hosts can talk each other using the assigned name.

> > 3. Definitely try and compose from latest stable CentOS SIG Liberty > packages (not delorean or DLRN or whatever it is these days) I already tried this. I had a strange issue while deploying computing nodes (puppet errors, don't remember exactly what). That's why i returned to my previous mirror since i had successfully deployed a 1+3 setup without issues. > > 4. Try Mitaka perhaps and follow Graeme's instructions here: > > https://www.redhat.com/archives/rdo-list/2016-June/msg00049.html > > You can miss step 4 and 6 as these have been resolved. I'd like to complete the setup with Liberty. We are playing also with other distributions (mainly commercial) that are proposing Liberty and we'd like to have everything aligned. Additionally we'd try a forward jump in place, to make a funny release-upgrade test and see what happens.

> > > I tend to work back and reduce things down in complexity. I agree, i'll try. > > > You can specify hostnames as per: > > http://docs.openstack.org/developer/tripleo-docs/advanced_deploymen > t/no > de_placement.html > > but perhaps this is what you are doing in your ~/hostname- > nostri.yaml > file? That file has simply this inside: parameter_defaults: ControllerHostnameFormat: 'opsctrl%index%' ComputeHostnameFormat: 'opskvmsvil%index%'

That is simply because overcloud-novacompute-0 was too long name

Luca -- "E' assurdo impiegare gli uomini di intelligenza eccellente per fare calcoli che potrebbero essere affidati a chiunque se si usassero delle macchine" Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716) "Internet è la più grande biblioteca del mondo. Ma il problema è che i libri sono tutti sparsi sul pavimento" John Allen Paulos, Matematico (1945-vivente) Luca 'remix_tj' Lorenzetto, http://www.remixtj.net , <lorenzetto.luca(a)gmail.com&gt;

If i go in depth with heat deployment-show i see that all resources report this deploy_stderr: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\ Error: Command exceeded timeout Error: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: change from notrun to 0 failed: Command exceeded timeout Warning: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster tripleo_cluster]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tripleo_cluster]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]: Skipping because of failed dependencies Warning: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]: Skipping because of failed dependencies", I see nodes are stuck on puppet step "auth-successful-across-all-nodes" (defined in /etc/puppet/modules/pacemaker/manifests/corosync.pp) /usr/bin/python2 /usr/sbin/pcs cluster auth opsctrl0 opsctrl1 opsctrl2 -u hacluster -p PASSWORD --force I suppose that the problem is due to corosync service not yet started. But as far as i can see corosync will never start because /etc/corosync/corosync.conf file is missing.