[rdo-list] [tripleo] Troubles deploying Libery with HA setup

Tue Aug 2 16:34:20 UTC 2016

Hi Luca,

It rings a bell but to be honest I would try the following:

1. Check hosts file on controllers - I wonder what are the content
there?
2. Revert hostname change and see if it is happy
3. Definitely try and compose from latest stable CentOS SIG Liberty
packages (not delorean or DLRN or whatever it is these days)
4. Try Mitaka perhaps and follow Graeme's instructions here:

https://www.redhat.com/archives/rdo-list/2016-June/msg00049.html

You can miss step 4 and 6 as these have been resolved.

I tend to work back and reduce things down in complexity.

You can specify hostnames as per:

http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/no
de_placement.html

but perhaps this is what you are doing in your ~/hostname-nostri.yaml
file?

Hope this helps.

On Tue, 2016-08-02 at 16:59 +0100, Luca 'remix_tj' Lorenzetto wrote:
> Hello,
>
> I'm deploying Liberty on a set of server using tripleo. I imported
> and
> correctly introspected 6 nodes. First i did a setup with 1 controller
> node and 3 computing nodes. No issues during this deploy, everything
> went fine. On this deployment i used custom hostname formats (defined
> via yaml environment file) and custom dns_domain (changed in
> /etc/nova/nova.conf and /etc/neutron/dhcp_agent.ini and all
> /usr/share/openstack-tripleo-heat-templates/*.yaml files that
> contains
> localdomain)
>
> Now I'm working to have a 3 controller setup with HA. I'm running
> this
> command to deploy:
>
> openstack overcloud deploy --templates -e
> /usr/share/openstack-tripleo-heat-templates/environments/network-
> isolation.yaml
> -e ~/templates/network-environment.yaml -e
> ~/templates/puppet-ceph-external.yaml -e ~/hostname-nostri.yaml
> --neutron-bridge-mappings datacentre:br-ex,storage-pub:br-stg-pub -e
> /usr/share/openstack-tripleo-heat-templates/environments/puppet-
> pacemaker.yaml
> --ntp-server timesrv1 --control-scale 3 --compute-scale 3
> --ceph-storage-scale 0 --control-flavor controlhp --compute-flavor
> computehp --neutron-network-type vxlan --neutron-tunnel-types vxlan
> --verbose --debug --log-file overcloud_$(date +%T).log
>
> I see that stack creation starts and correctly deploy os on the
> nodes.
> Nodes are named according to ControllerHostnameFormat and
> ComputeHostnameFormat that i specified into ~/hostname-nostri.yaml.
>
> Everything goes well until HA configuration starts. I see this stack
> creation failing:
>
> overcloud-ControllerNodesPostDeployment-bbp3c47jgau2-
> ControllerLoadBalancerDeployment_Step1-sla6ce7n2arq
>
> The error message of the deployment command is:
>
> Stack failed with status: Resource CREATE failed:
> resources.ControllerLoadBalancerDeployment_Step1:
> resources.ControllerNodesPostDeployment.Error: resources[0]:
> Deployment to server failed: deploy_status_code: Deployment exited
> with non-zero status code: 6
> Heat Stack create failed.
>
>
> If i go in depth with heat deployment-show i see that all resources
> report this deploy_stderr:
>
> Error: Could not prefetch mysql_user provider 'mysql': Execution of
> '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM
> mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local
> MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
> Error: Could not prefetch mysql_database provider 'mysql': Execution
> of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002
> (HY000): Can't connect to local MySQL server through socket
> '/var/lib/mysql/mysql.sock' (2)\
> Error: Command exceeded timeout
> Error: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-
> all-nodes]/returns:
> change from notrun to 0 failed: Command exceeded timeout
> Warning: /Stage[main]/Pacemaker::Corosync/Exec[Create Cluster
> tripleo_cluster]: Skipping because of failed dependencies
> Warning: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster
> tripleo_cluster]: Skipping because of failed dependencies
> Warning: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]:
> Skipping because of failed dependencies
> Warning: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]:
> Skipping because of failed dependencies
> Warning: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]:
> Skipping because of failed dependencies",
>
>
>
> I see nodes are stuck on puppet step
> "auth-successful-across-all-nodes" (defined in
> /etc/puppet/modules/pacemaker/manifests/corosync.pp)
>
> /usr/bin/python2 /usr/sbin/pcs cluster auth opsctrl0 opsctrl1
> opsctrl2
> -u hacluster -p PASSWORD --force
>
> I suppose that the problem is due to corosync service not yet
> started.
> But as far as i can see corosync will never start because
> /etc/corosync/corosync.conf file is missing.
>
> in /etc/hosts opsctrl{0-2} are correctly defined and each host can
> talk with the others.
>
>
>
> I'm stuck and i don't know what to do. Anyone had similar issues? I'm
> missing something that i have to enter in the configuration?
>
>
> I'm using a in-house mirror of rdo-release made on April 20, and
> custom images build from rhel image and this repository. A week ago i
> tried with latest repository but had the same error (and also others,
> that's why i returned to this older mirror that was working).
>
> This is the list of packages installed from rdo-release repository:
>
> crudini-0.7-1.el7.noarch
> dib-utils-0.0.9-1.el7.noarch
> dibbler-client-1.0.1-0.RC1.2.el7.x86_64
> diskimage-builder-1.4.0-1.el7.noarch
> erlang-asn1-R16B-03.16.el7.x86_64
> erlang-compiler-R16B-03.16.el7.x86_64
> erlang-crypto-R16B-03.16.el7.x86_64
> erlang-erts-R16B-03.16.el7.x86_64
> erlang-hipe-R16B-03.16.el7.x86_64
> erlang-inets-R16B-03.16.el7.x86_64
> erlang-kernel-R16B-03.16.el7.x86_64
> erlang-mnesia-R16B-03.16.el7.x86_64
> erlang-os_mon-R16B-03.16.el7.x86_64
> erlang-otp_mibs-R16B-03.16.el7.x86_64
> erlang-public_key-R16B-03.16.el7.x86_64
> erlang-runtime_tools-R16B-03.16.el7.x86_64
> erlang-sasl-R16B-03.16.el7.x86_64
> erlang-sd_notify-0.1-1.el7.x86_64
> erlang-snmp-R16B-03.16.el7.x86_64
> erlang-ssl-R16B-03.16.el7.x86_64
> erlang-stdlib-R16B-03.16.el7.x86_64
> erlang-syntax_tools-R16B-03.16.el7.x86_64
> erlang-tools-R16B-03.16.el7.x86_64
> erlang-xmerl-R16B-03.16.el7.x86_64
> hiera-1.3.4-1.el7.noarch
> instack-0.0.8-1.el7.noarch
> instack-undercloud-2.1.3-1.el7.noarch
> jq-1.3-2.el7.x86_64
> liberasurecode-1.1.1-1.el7.x86_64
> libnetfilter_queue-1.0.2-2.el7.x86_64
> memcached-1.4.25-1.el7.x86_64
> 1:openstack-ceilometer-alarm-5.0.2-1.el7.noarch
> 1:openstack-ceilometer-api-5.0.2-1.el7.noarch
> 1:openstack-ceilometer-central-5.0.2-1.el7.noarch
> 1:openstack-ceilometer-collector-5.0.2-1.el7.noarch
> 1:openstack-ceilometer-common-5.0.2-1.el7.noarch
> 1:openstack-ceilometer-notification-5.0.2-1.el7.noarch
> 1:openstack-ceilometer-polling-5.0.2-1.el7.noarch
> 1:openstack-glance-11.0.1-2.el7.noarch
> 1:openstack-heat-api-5.0.0-1.el7.noarch
> 1:openstack-heat-api-cfn-5.0.0-1.el7.noarch
> 1:openstack-heat-api-cloudwatch-5.0.0-1.el7.noarch
> 1:openstack-heat-common-5.0.0-1.el7.noarch
> 1:openstack-heat-engine-5.0.0-1.el7.noarch
> openstack-heat-templates-0-0.1.20151019.el7.noarch
> 1:openstack-ironic-api-4.2.2-1.el7.noarch
> 1:openstack-ironic-common-4.2.2-1.el7.noarch
> 1:openstack-ironic-conductor-4.2.2-1.el7.noarch
> openstack-ironic-inspector-2.2.2-1.el7.noarch
> 1:openstack-keystone-8.0.1-1.el7.noarch
> 1:openstack-neutron-7.0.3-1.el7.noarch
> 1:openstack-neutron-common-7.0.3-1.el7.noarch
> 1:openstack-neutron-ml2-7.0.3-1.el7.noarch
> 1:openstack-neutron-openvswitch-7.0.3-1.el7.noarch
> 1:openstack-nova-api-12.0.1-1.el7.noarch
> 1:openstack-nova-cert-12.0.1-1.el7.noarch
> 1:openstack-nova-common-12.0.1-1.el7.noarch
> 1:openstack-nova-compute-12.0.1-1.el7.noarch
> 1:openstack-nova-conductor-12.0.1-1.el7.noarch
> 1:openstack-nova-scheduler-12.0.1-1.el7.noarch
> 1:openstack-puppet-modules-7.0.1-1.el7.noarch
> openstack-selinux-0.6.41-1.el7.noarch
> openstack-swift-2.5.0-1.el7.noarch
> openstack-swift-account-2.5.0-1.el7.noarch
> openstack-swift-container-2.5.0-1.el7.noarch
> openstack-swift-object-2.5.0-1.el7.noarch
> openstack-swift-plugin-swift3-1.7-4.el7.noarch
> openstack-swift-proxy-2.5.0-1.el7.noarch
> openstack-tripleo-0.0.6-1.el7.noarch
> openstack-tripleo-heat-templates-0.8.7-1.el7.noarch
> openstack-tripleo-image-elements-0.9.7-1.el7.noarch
> openstack-tripleo-puppet-elements-0.0.2-1.el7.noarch
> openstack-utils-2015.2-1.el7.noarch
> openvswitch-2.4.0-1.el7.x86_64
> os-apply-config-0.1.32-3.el7.noarch
> os-cloud-config-0.2.10-2.el7.noarch
> os-collect-config-0.1.36-4.el7.noarch
> os-net-config-0.1.5-3.el7.noarch
> os-refresh-config-0.1.11-2.el7.noarch
> puppet-3.6.2-3.el7.noarch
> pyOpenSSL-0.15.1-1.el7.noarch
> pyparsing-2.0.3-1.el7.noarch
> pysendfile-2.0.0-5.el7.x86_64
> pysnmp-4.2.5-2.el7.noarch
> pystache-0.5.3-2.el7.noarch
> python-alembic-0.8.3-3.el7.noarch
> python-amqp-1.4.6-1.el7.noarch
> python-anyjson-0.3.3-3.el7.noarch
> python-automaton-0.7.0-1.el7.noarch
> python-babel-1.3-6.el7.noarch
> python-bson-3.0.3-1.el7.x86_64
> python-cachetools-1.0.3-2.el7.noarch
> 1:python-ceilometer-5.0.2-1.el7.noarch
> python-ceilometerclient-1.5.0-1.el7.noarch
> python-cinderclient-1.4.0-1.el7.noarch
> python-cliff-1.15.0-1.el7.noarch
> python-cliff-tablib-1.1-3.el7.noarch
> python-cmd2-0.6.8-3.el7.noarch
> python-contextlib2-0.4.0-1.el7.noarch
> python-croniter-0.3.4-2.el7.noarch
> python-dogpile-cache-0.5.7-3.el7.noarch
> python-dogpile-core-0.4.1-2.el7.noarch
> python-ecdsa-0.11-3.el7.noarch
> python-editor-0.4-4.el7.noarch
> python-elasticsearch-1.4.0-2.el7.noarch
> python-extras-0.0.3-2.el7.noarch
> python-fixtures-1.4.0-2.el7.noarch
> python-futures-3.0.3-1.el7.noarch
> 1:python-glance-11.0.1-2.el7.noarch
> python-glance-store-0.9.1-1.el7.noarch
> 1:python-glanceclient-1.1.0-1.el7.noarch
> python-heatclient-0.8.0-1.el7.noarch
> python-httplib2-0.9.2-1.el7.noarch
> python-idna-2.0-1.el7.noarch
> python-ipaddress-1.0.7-4.el7.noarch
> python-ironicclient-0.8.1-1.el7.noarch
> python-jsonpatch-1.2-2.el7.noarch
> python-jsonpath-rw-1.2.3-2.el7.noarch
> python-jsonschema-2.3.0-1.el7.noarch
> python-keyring-5.0-4.el7.noarch
> 1:python-keystone-8.0.1-1.el7.noarch
> 1:python-keystoneclient-1.7.2-1.el7.noarch
> python-keystonemiddleware-2.3.1-1.el7.noarch
> 1:python-kombu-3.0.32-1.el7.noarch
> python-ldappool-1.0-4.el7.noarch
> python-linecache2-1.0.0-1.el7.noarch
> python-logutils-0.3.3-3.el7.noarch
> python-memcached-1.54-3.el7.noarch
> python-migrate-0.10.0-1.el7.noarch
> python-mimeparse-0.1.4-1.el7.noarch
> python-monotonic-0.3-1.el7.noarch
> python-ncclient-0.4.2-2.el7.noarch
> python-netaddr-0.7.18-1.el7.noarch
> python-netifaces-0.10.4-1.el7.x86_64
> python-networkx-core-1.10-1.el7.noarch
> 1:python-neutron-7.0.3-1.el7.noarch
> python-neutronclient-3.1.0-1.el7.noarch
> python-nose-1.3.7-7.el7.noarch
> 1:python-nova-12.0.1-1.el7.noarch
> 1:python-novaclient-2.30.1-1.el7.noarch
> python-oauthlib-0.7.2-5.20150520git514cad7.el7.noarch
> python-openstackclient-1.7.2-1.el7.noarch
> python-openvswitch-2.4.0-1.el7.noarch
> python-oslo-cache-0.7.0-1.el7.noarch
> python-oslo-concurrency-2.6.0-1.el7.noarch
> python-oslo-db-2.6.0-3.el7.noarch
> python-oslo-log-1.10.0-1.el7.noarch
> python-oslo-messaging-2.5.0-1.el7.noarch
> python-oslo-middleware-2.8.0-1.el7.noarch
> python-oslo-policy-0.11.0-1.el7.noarch
> python-oslo-rootwrap-2.3.0-1.el7.noarch
> python-oslo-service-0.9.0-1.el7.noarch
> python-oslo-versionedobjects-0.10.0-1.el7.noarch
> python-oslo-vmware-1.21.0-1.el7.noarch
> python-osprofiler-0.3.0-1.el7.noarch
> python-paramiko-1.15.1-1.el7.noarch
> python-paste-deploy-1.5.2-6.el7.noarch
> python-pbr-1.8.1-2.el7.noarch
> python-posix_ipc-0.9.8-1.el7.x86_64
> python-prettytable-0.7.2-1.el7.noarch
> python-proliantutils-2.1.7-1.el7.noarch
> python-psutil-1.2.1-1.el7.x86_64
> python-pycadf-1.1.0-1.el7.noarch
> python-pyeclib-1.2.0-1.el7.x86_64
> python-pyghmi-0.8.0-2.el7.noarch
> python-pygments-2.0.2-4.el7.noarch
> python-pymongo-3.0.3-1.el7.x86_64
> python-pysaml2-3.0.2-1.el7.noarch
> python-qpid-0.30-1.el7.noarch
> python-qpid-common-0.30-1.el7.noarch
> python-repoze-lru-0.4-3.el7.noarch
> python-repoze-who-2.1-1.el7.noarch
> python-requests-2.9.1-2.el7.noarch
> python-retrying-1.2.3-4.el7.noarch
> python-rfc3986-0.2.0-1.el7.noarch
> python-routes-1.13-2.el7.noarch
> python-saharaclient-0.11.1-1.el7.noarch
> python-semantic_version-2.4.2-1.el7.noarch
> python-simplegeneric-0.8-7.el7.noarch
> python-simplejson-3.5.3-5.el7.x86_64
> python-sqlalchemy-1.0.11-1.el7.x86_64
> python-sqlparse-0.1.18-5.el7.noarch
> python-stevedore-1.8.0-1.el7.noarch
> python-swiftclient-2.6.0-1.el7.noarch
> python-tablib-0.10.0-1.el7.noarch
> python-taskflow-1.21.0-1.el7.noarch
> python-tempita-0.5.1-8.el7.noarch
> python-testtools-1.8.0-2.el7.noarch
> python-tooz-1.24.0-1.el7.noarch
> python-traceback2-1.4.0-2.el7.noarch
> python-tripleoclient-0.0.11-3.el7.noarch
> python-troveclient-1.3.0-1.el7.noarch
> python-unicodecsv-0.14.1-1.el7.noarch
> python-unittest2-1.0.1-1.el7.noarch
> python-urllib3-1.13.1-3.el7.noarch
> python-warlock-1.0.1-1.el7.noarch
> python-webob-1.4.1-2.el7.noarch
> python-websockify-0.6.0-2.el7.noarch
> python-wrapt-1.10.5-3.el7.x86_64
> python2-PyMySQL-0.6.7-2.el7.noarch
> python2-appdirs-1.4.0-4.el7.noarch
> python2-castellan-0.3.1-1.el7.noarch
> python2-cffi-1.5.2-1.el7.x86_64
> python2-cryptography-1.2.1-3.el7.x86_64
> python2-debtcollector-0.8.0-1.el7.noarch
> python2-eventlet-0.17.4-4.el7.noarch
> python2-fasteners-0.14.1-4.el7.noarch
> python2-funcsigs-0.4-2.el7.noarch
> python2-futurist-0.5.0-1.el7.noarch
> python2-greenlet-0.4.9-1.el7.x86_64
> python2-ironic-inspector-client-1.2.0-2.el7.noarch
> python2-iso8601-0.1.11-1.el7.noarch
> python2-jsonpath-rw-ext-0.1.7-1.1.el7.noarch
> python2-mock-1.3.0-1.el7.noarch
> python2-os-brick-0.5.0-1.el7.noarch
> python2-os-client-config-1.7.4-1.el7.noarch
> 2:python2-oslo-config-2.4.0-1.el7.noarch
> python2-oslo-context-0.6.0-1.el7.noarch
> python2-oslo-i18n-2.6.0-1.el7.noarch
> python2-oslo-reports-0.5.0-1.el7.noarch
> python2-oslo-serialization-1.9.0-1.el7.noarch
> python2-oslo-utils-2.5.0-1.el7.noarch
> python2-passlib-1.6.5-1.el7.noarch
> python2-pecan-1.0.2-2.el7.noarch
> python2-pyasn1-0.1.9-6.el7.1.noarch
> python2-rsa-3.3-2.el7.noarch
> python2-singledispatch-3.4.0.3-4.el7.noarch
> python2-suds-0.7-0.1.94664ddd46a6.el7.noarch
> python2-wsme-0.7.0-2.el7.noarch
> rabbitmq-server-3.3.5-17.el7.noarch
> ruby-augeas-0.5.0-1.el7.x86_64
> ruby-shadow-1.4.1-23.el7.x86_64
> rubygem-rgen-0.6.6-2.el7.noarch
> tripleo-common-0.0.1-2.el7.noarch
>
>
> Thank you,
>
>
> Luca
>
>
>
>
>
> --
> "E' assurdo impiegare gli uomini di intelligenza eccellente per fare
> calcoli che potrebbero essere affidati a chiunque se si usassero
> delle
> macchine"
> Gottfried Wilhelm von Leibnitz, Filosofo e Matematico (1646-1716)
>
> "Internet è la più grande biblioteca del mondo.
> Ma il problema è che i libri sono tutti sparsi sul pavimento"
> John Allen Paulos, Matematico (1945-vivente)
>
> Luca 'remix_tj' Lorenzetto, http://www.remixtj.net ,
> <lorenzetto.luca at gmail.com>
>
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com
--
Regards,

Christopher Brown
OpenStack Engineer
OCF plc

Tel: +44 (0)114 257 2200
Web: www.ocf.co.uk
Blog: blog.ocf.co.uk
Twitter: @ocfplc

Please note, any emails relating to an OCF Support request must always
be sent to support at ocf.co.uk for a ticket number to be generated or
existing support ticket to be updated. Should this not be done then OCF
cannot be held responsible for requests not dealt with in a timely
manner.

OCF plc is a company registered in England and Wales. Registered number
4132533, VAT number GB 780 6803 14. Registered office address: OCF plc,
5 Rotunda Business Centre, Thorncliffe Park, Chapeltown, Sheffield S35
2PG.

This message is private and confidential. If you have received this
message in error, please notify us immediately and remove it from your
system.