Hi Dan, Thanks a lot for your response. Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. To recreate the issue , I am mentioning all the configuration steps: 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. 2. created stack user and provided required permission to stack user . 3. setting hostname sudo hostnamectl set-hostname rdo-undercloud.mydomain sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain [stack@rdo-undercloud etc]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain

4. enable required repositories sudo yum -y install epel-release sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo 5. install repos sudo yum -y install yum-plugin-priorities sudo yum install -y python-tripleoclient 6. update undercloud.conf [stack@rdo-undercloud ~]$ cat undercloud.conf [DEFAULT] local_ip = 192.0.2.1/24 undercloud_public_vip = 192.0.2.2 undercloud_admin_vip = 192.0.2.3 local_interface = enp6s0 masquerade_network = 192.0.2.0/24 dhcp_start = 192.0.2.150 dhcp_end = 192.0.2.199 network_cidr = 192.0.2.0/24 network_gateway = 192.0.2.1 discovery_iprange = 192.0.2.200,192.0.2.230 discovery_runbench = false [auth] 7. install undercloud openstack undercloud install install ends in error: Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events Notice: Finished catalog run in 5259.44 seconds + rc=6 + set -e + echo 'puppet apply exited with exit code 6' puppet apply exited with exit code 6 + '[' 6 '!=' 2 -a 6 '!=' 0 ']' + exit 6 [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install _run_orc(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc _run_live_command(args, instack_env, 'os-refresh-config') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: os-refresh-config failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. Thanks a lot. Best Regards, Milind Best Regards, Milind -----Original Message----- From: Dan Sneddon [mailto:dsneddon@redhat.com] Sent: Monday, June 27, 2016 12:40 PM To: Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com>; rdo-list(a)redhat.com Subject: Re: [rdo-list] Redeploying UnderCloud On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: > Hi All, > > Greeting. > > > > This is my first post and I am fairly new to RDO OpenStack. I am > working on RDO Triple-O deployment on baremetal. Due to incorrect > values in undercloud.conf file , my undercloud deployment failed. I > would like to redeploy undercloud and I am trying to understand what > steps has to be taken before redeploying undercloud. All the > deployment was under stack user . So first step will be to delete > stack user. I am not sure what has to be done regarding the networking > configuration autogenerated by os-net-config during the older install. > > Please advise. > > > > Best Regards, > > Milind No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. that are in ~stack, and generally make your life harder than it needs to be. Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. As the stack user, edit your undercloud.conf, and make sure there are no more typos. If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: $ sudo ifdown br-ctlplane $ sudo ovs-vsctl del-br br-ctlplane $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane Next, run the underclound installation again: $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. -- Dan Sneddon | Principal OpenStack Engineer dsneddon(a)redhat.com | redhat.com/openstack 650.254.4025 | dsneddon:irc @dxs:twitter ________________________________ Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off> for details. ________________________________ This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

Hi All, Thanks a lot for continued support. I would just like to get clarity regarding the last recommendation. My current deployment is failing with following error : Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7 undercloud deployment and it was configured with host gateway used by pxe network. Similarly, we have set the current host gateway to 192.0.2.1 as shown below for rdo manager undercloud installation: [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain.

Thanks again for your time and help. Really appreciate it. Best Regards, milind -----Original Message----- From: Marius Cornea [mailto:marius@remote-lab.net] Sent: Tuesday, June 28, 2016 4:28 AM To: Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com&gt; Cc: rdo-list(a)redhat.com Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com&gt; wrote: > Hi Dan, > Thanks a lot for your response. > > Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. > > To recreate the issue , I am mentioning all the configuration steps: > 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. > 2. created stack user and provided required permission to stack user . > 3. setting hostname > sudo hostnamectl set-hostname rdo-undercloud.mydomain > sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain > > [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain Could you try removing the 192.0.2.1 entry from /etc/hosts and replace it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain then rerun openstack undercloud install > 4. enable required repositories > sudo yum -y install epel-release > sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo > sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo > > 5. install repos > > sudo yum -y install yum-plugin-priorities > sudo yum install -y python-tripleoclient > > 6. update undercloud.conf > > [stack@rdo-undercloud ~]$ cat undercloud.conf > [DEFAULT] > local_ip = 192.0.2.1/24 > undercloud_public_vip = 192.0.2.2 > undercloud_admin_vip = 192.0.2.3 > local_interface = enp6s0 > masquerade_network = 192.0.2.0/24 > dhcp_start = 192.0.2.150 > dhcp_end = 192.0.2.199 > network_cidr = 192.0.2.0/24 > network_gateway = 192.0.2.1 > discovery_iprange = 192.0.2.200,192.0.2.230 > discovery_runbench = false > [auth] > > 7. install undercloud > > openstack undercloud install > > install ends in error: > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events > Notice: Finished catalog run in 5259.44 seconds > + rc=6 > + set -e > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install > _run_orc(instack_env) > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc > _run_live_command(args, instack_env, 'os-refresh-config') > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command > raise RuntimeError('%s failed. See log for details.' % name) > RuntimeError: os-refresh-config failed. See log for details. > Command 'instack-install-undercloud' returned non-zero exit status 1 > > > I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. > > Thanks a lot. > > Best Regards, > Milind > > > Best Regards, > Milind > -----Original Message----- > From: Dan Sneddon [mailto:dsneddon@redhat.com] > Sent: Monday, June 27, 2016 12:40 PM > To: Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com>; rdo-list(a)redhat.com > Subject: Re: [rdo-list] Redeploying UnderCloud > > On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >> Hi All, >> >> Greeting. >> >> >> >> This is my first post and I am fairly new to RDO OpenStack. I am >> working on RDO Triple-O deployment on baremetal. Due to incorrect >> values in undercloud.conf file , my undercloud deployment failed. I >> would like to redeploy undercloud and I am trying to understand what >> steps has to be taken before redeploying undercloud. All the >> deployment was under stack user . So first step will be to delete >> stack user. I am not sure what has to be done regarding the networking >> configuration autogenerated by os-net-config during the older install. >> >> Please advise. >> >> >> >> Best Regards, >> >> Milind > > No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. > that are in ~stack, and generally make your life harder than it needs to be. > > Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. > > As the stack user, edit your undercloud.conf, and make sure there are no more typos. > > If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: > > $ sudo ifdown br-ctlplane > $ sudo ovs-vsctl del-br br-ctlplane > $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane > > Next, run the underclound installation again: > > $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install > > Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon(a)redhat.com | redhat.com/openstack > 650.254.4025 | dsneddon:irc @dxs:twitter > > > > ________________________________ > Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off> for details. > > ________________________________ > > This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com

Hi All, Thanks a lot for continued support. I would just like to get clarity regarding the last recommendation. My current deployment is failing with following error : Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7 undercloud deployment and it was configured with host gateway used by pxe network. Similarly, we have set the current host gateway to 192.0.2.1 as shown below for rdo manager undercloud installation: [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain.

Thanks again for your time and help. Really appreciate it. Best Regards, milind -----Original Message----- From: Marius Cornea [mailto:marius@remote-lab.net<mailto:marius@remote-lab.net>] Sent: Tuesday, June 28, 2016 4:28 AM To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.com> Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> wrote: > Hi Dan, > Thanks a lot for your response. > > Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. > > To recreate the issue , I am mentioning all the configuration steps: > 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. > 2. created stack user and provided required permission to stack user . > 3. setting hostname > sudo hostnamectl set-hostname rdo-undercloud.mydomain > sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain > > [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain Could you try removing the 192.0.2.1 entry from /etc/hosts and replace it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain then rerun openstack undercloud install > 4. enable required repositories > sudo yum -y install epel-release > sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo > sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo > > 5. install repos > > sudo yum -y install yum-plugin-priorities > sudo yum install -y python-tripleoclient > > 6. update undercloud.conf > > [stack@rdo-undercloud ~]$ cat undercloud.conf > [DEFAULT] > local_ip = 192.0.2.1/24<http://192.0.2.1/24> > undercloud_public_vip = 192.0.2.2 > undercloud_admin_vip = 192.0.2.3 > local_interface = enp6s0 > masquerade_network = 192.0.2.0/24<http://192.0.2.0/24> > dhcp_start = 192.0.2.150 > dhcp_end = 192.0.2.199 > network_cidr = 192.0.2.0/24<http://192.0.2.0/24> > network_gateway = 192.0.2.1 > discovery_iprange = 192.0.2.200,192.0.2.230 > discovery_runbench = false > [auth] > > 7. install undercloud > > openstack undercloud install > > install ends in error: > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events > Notice: Finished catalog run in 5259.44 seconds > + rc=6 > + set -e > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install > _run_orc(instack_env) > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc > _run_live_command(args, instack_env, 'os-refresh-config') > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command > raise RuntimeError('%s failed. See log for details.' % name) > RuntimeError: os-refresh-config failed. See log for details. > Command 'instack-install-undercloud' returned non-zero exit status 1 > > > I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. > > Thanks a lot. > > Best Regards, > Milind > > > Best Regards, > Milind > -----Original Message----- > From: Dan Sneddon [mailto:dsneddon@redhat.com<mailto:dsneddon@redhat.com>] > Sent: Monday, June 27, 2016 12:40 PM > To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>>; rdo-list@redhat.com<mailto:rdo-list@redhat.com> > Subject: Re: [rdo-list] Redeploying UnderCloud > > On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >> Hi All, >> >> Greeting. >> >> >> >> This is my first post and I am fairly new to RDO OpenStack. I am >> working on RDO Triple-O deployment on baremetal. Due to incorrect >> values in undercloud.conf file , my undercloud deployment failed. I >> would like to redeploy undercloud and I am trying to understand what >> steps has to be taken before redeploying undercloud. All the >> deployment was under stack user . So first step will be to delete >> stack user. I am not sure what has to be done regarding the networking >> configuration autogenerated by os-net-config during the older install. >> >> Please advise. >> >> >> >> Best Regards, >> >> Milind > > No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. > that are in ~stack, and generally make your life harder than it needs to be. > > Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. > > As the stack user, edit your undercloud.conf, and make sure there are no more typos. > > If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: > > $ sudo ifdown br-ctlplane > $ sudo ovs-vsctl del-br br-ctlplane > $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane > > Next, run the underclound installation again: > > $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install > > Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon@redhat.com<mailto:dsneddon@redhat.com> | redhat.com/openstack<http://redhat.com/openstack> > 650.254.4025<tel:650.254.4025> | dsneddon:irc @dxs:twitter > > > > ________________________________ > Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off><http://sprint.com/50of... for details. > > ________________________________ > > This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. > > _______________________________________________ > rdo-list mailing list > rdo-list@redhat.com<mailto:rdo-list@redhat.com> > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe@redhat.com<mailto:rdo-list-unsubscribe@redhat.com>

Hi All, Thanks a lot for continued support. I would just like to get clarity regarding the last recommendation. My current deployment is failing with following error : Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7 undercloud deployment and it was configured with host gateway used by pxe network. Similarly, we have set the current host gateway to 192.0.2.1 as shown below for rdo manager undercloud installation: [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain.

Thanks again for your time and help. Really appreciate it. Best Regards, milind -----Original Message----- From: Marius Cornea [mailto:marius@remote-lab.net<mailto:marius@remote-lab.net>] Sent: Tuesday, June 28, 2016 4:28 AM To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.com> Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> wrote: > Hi Dan, > Thanks a lot for your response. > > Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. > > To recreate the issue , I am mentioning all the configuration steps: > 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. > 2. created stack user and provided required permission to stack user . > 3. setting hostname > sudo hostnamectl set-hostname rdo-undercloud.mydomain > sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain > > [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain Could you try removing the 192.0.2.1 entry from /etc/hosts and replace it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain then rerun openstack undercloud install > 4. enable required repositories > sudo yum -y install epel-release > sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo > sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo > > 5. install repos > > sudo yum -y install yum-plugin-priorities > sudo yum install -y python-tripleoclient > > 6. update undercloud.conf > > [stack@rdo-undercloud ~]$ cat undercloud.conf > [DEFAULT] > local_ip = 192.0.2.1/24<http://192.0.2.1/24> > undercloud_public_vip = 192.0.2.2 > undercloud_admin_vip = 192.0.2.3 > local_interface = enp6s0 > masquerade_network = 192.0.2.0/24<http://192.0.2.0/24> > dhcp_start = 192.0.2.150 > dhcp_end = 192.0.2.199 > network_cidr = 192.0.2.0/24<http://192.0.2.0/24> > network_gateway = 192.0.2.1 > discovery_iprange = 192.0.2.200,192.0.2.230 > discovery_runbench = false > [auth] > > 7. install undercloud > > openstack undercloud install > > install ends in error: > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events > Notice: Finished catalog run in 5259.44 seconds > + rc=6 > + set -e > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install > _run_orc(instack_env) > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc > _run_live_command(args, instack_env, 'os-refresh-config') > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command > raise RuntimeError('%s failed. See log for details.' % name) > RuntimeError: os-refresh-config failed. See log for details. > Command 'instack-install-undercloud' returned non-zero exit status 1 > > > I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. > > Thanks a lot. > > Best Regards, > Milind > > > Best Regards, > Milind > -----Original Message----- > From: Dan Sneddon [mailto:dsneddon@redhat.com<mailto:dsneddon@redhat.com>] > Sent: Monday, June 27, 2016 12:40 PM > To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>>; rdo-list@redhat.com<mailto:rdo-list@redhat.com> > Subject: Re: [rdo-list] Redeploying UnderCloud > > On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >> Hi All, >> >> Greeting. >> >> >> >> This is my first post and I am fairly new to RDO OpenStack. I am >> working on RDO Triple-O deployment on baremetal. Due to incorrect >> values in undercloud.conf file , my undercloud deployment failed. I >> would like to redeploy undercloud and I am trying to understand what >> steps has to be taken before redeploying undercloud. All the >> deployment was under stack user . So first step will be to delete >> stack user. I am not sure what has to be done regarding the networking >> configuration autogenerated by os-net-config during the older install. >> >> Please advise. >> >> >> >> Best Regards, >> >> Milind > > No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. > that are in ~stack, and generally make your life harder than it needs to be. > > Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. > > As the stack user, edit your undercloud.conf, and make sure there are no more typos. > > If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: > > $ sudo ifdown br-ctlplane > $ sudo ovs-vsctl del-br br-ctlplane > $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane > > Next, run the underclound installation again: > > $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install > > Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon@redhat.com<mailto:dsneddon@redhat.com> | redhat.com/openstack<http://redhat.com/openstack> > 650.254.4025<tel:650.254.4025> | dsneddon:irc @dxs:twitter > > > > ________________________________ > Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off><http://sprint.com/50of... for details. > > ________________________________ > > This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. > > _______________________________________________ > rdo-list mailing list > rdo-list@redhat.com<mailto:rdo-list@redhat.com> > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe@redhat.com<mailto:rdo-list-unsubscribe@redhat.com>

On 06/29/2016 07:03 AM, Boris Derzhavets wrote: > Boris Derzhavets has shared a OneDrive file with you. To view it, click > the link below. > > <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > > HeatCrash2.txt 1.gz <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > [HeatCrash2.txt 1.gz] > > Reattach gzip archive via One Drive > > > > ----------------------------------------------------------------------- > *From:* rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on > behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; > *Sent:* Wednesday, June 29, 2016 9:36 AM > *To:* John Trowbridge; shardy(a)redhat.com > *Cc:* rdo-list(a)redhat.com > *Subject:* [rdo-list] HA overcloud-deploy.sh crashes again ( > ControllerOvercloudServicesDeployment_Step4 ) > > > Attempt to follow steps suggested > in http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html > > > ./deploy-overstack crashes > > > 2016-06-29 12:42:41 > [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk-ControllerOvercloudServicesDeployment_Step4-nzdoizlgrmx2]: > CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment > to server failed: deploy_status_code : Deployment exited with non-zero > status code: 6 > 2016-06-29 12:42:42 [ControllerOvercloudServicesDeployment_Step4]: > CREATE_FAILED Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:43 > [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk]: CREATE_FAILED > Resource CREATE failed: Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:44 [ControllerNodesPostDeployment]: CREATE_FAILED > Error: > resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:44 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:46 [overcloud]: CREATE_FAILED Resource CREATE failed: > Error: > resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:46 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:47 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:47 [ControllerDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:48 [NetworkDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:48 [2]: SIGNAL_COMPLETE Unknown > Stack overcloud CREATE_FAILED > Deployment failed: Heat Stack create failed. > + heat stack-list > + grep -q CREATE_FAILED > + deploy_status=1 > ++ heat resource-list --nested-depth 5 overcloud > ++ grep FAILED > ++ grep 'StructuredDeployment ' > ++ cut -d '|' -f3 > + for failed in '$(heat resource-list --nested-depth 5 > overcloud | grep FAILED | > grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' > + heat deployment-show 655c77fc-6a78-4cca-b4b7-a153a3f4ad52 > + for failed in '$(heat resource-list --nested-depth 5 > overcloud | grep FAILED | > grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' > + heat deployment-show 1fe5153c-e017-4ee5-823a-3d1524430c1d > + for failed in '$(heat resource-list --nested-depth 5 > overcloud | grep FAILED | > grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' > + heat deployment-show bf6f25f4-d812-41e9-a7a8-122de619a624 > + exit 1 > > ***************************** > Troubleshooting steps :- > ***************************** > > [stack@undercloud ~]$ . stackrc > [stack@undercloud ~]$ heat resource-list overcloud | grep > ControllerNodesPost > | ControllerNodesPostDeployment | > f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | > OS::TripleO::ControllerPostDeployment | CREATE_FAILED | > 2016-06-29T12:11:21 | > > > [stack@undercloud ~]$ heat stack-list -n | grep "^| > f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3" > | f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk > | CREATE_FAILED | 2016-06-29T12:31:11 | None | > 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | > > > > [stack@undercloud ~]$ heat event-list -m > f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk > > +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ > | resource_name | > id | > resource_status_reason > | resource_status | event_time | > +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ > | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | > 10ec0cf9-b3c9-4191-9966-3f4d47f27e2a | Stack CREATE started > . . . . . . . . . . . . . . . . . > Step1,2,3 succeeded > . . . . . . . . . . . . . . . . . > > | CREATE_IN_PROGRESS | 2016-06-29T12:31:14 | > | ControllerPuppetConfig | > a2a1df33-5106-425c-b16d-8d2df709b19f | state > changed > | CREATE_COMPLETE | 2016-06-29T12:35:02 | > | ControllerOvercloudServicesDeployment_Step4 | > 1e151333-4de5-4e7b-907c-ea0f42d31a47 | state > changed > | CREATE_IN_PROGRESS | 2016-06-29T12:35:03 | > | ControllerOvercloudServicesDeployment_Step4 | > 7bf36334-3d92-4554-b6c0-41294a072ab6 | Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 | CREATE_FAILED | > 2016-06-29T12:42:42 | > | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk > | e72fb6f4-c2aa-4fe8-9bd1-5f5ad152685c | Resource CREATE failed: > Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 | CREATE_FAILED | 2016-06-29T12:42:43 | > +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ > > [stack@undercloud ~]$ heat stack-show > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | grep > NodeConfigIdentifiers > | | "NodeConfigIdentifiers": > "{u'deployment_identifier': 1467202276, u'controller_config': {u'1': > u'os-apply-config deployment 796df02a-7550-414b-a084-8b591a13e6db > completed,Root CA cert injection not enabled.,TLS not enabled.,None,', > u'0': u'os-apply-config deployment 613ec889-d852-470a-8e4c-6e243e1d2033 > completed,Root CA cert injection not enabled.,TLS not enabled.,None,', > u'2': u'os-apply-config deployment c8b099d0-3af4-4ba0-a056-a0ce60f40e2d > completed,Root CA cert injection not enabled.,TLS not enabled.,None,'}, > u'allnodes_extra': u'none'}" | > > However, when stack creating crashed update wouldn't help. > > [stack@undercloud ~]$ heat stack-update -x > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk -e update_env.yaml > ERROR: PATCH update to non-COMPLETE stack is not supported. > > DUE TO :- > > [stack@undercloud ~]$ heat stack-list > +--------------------------------------+------------+---------------+---------------------+--------------+ > | id | stack_name | stack_status | > creation_time | updated_time | > +--------------------------------------+------------+---------------+---------------------+--------------+ > | 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | overcloud | CREATE_FAILED | > 2016-06-29T12:11:20 | None | > +--------------------------------------+------------+---------------+---------------------+------ > > > Complete error file `heat deployment-show > 655c77fc-6a78-4cca-b4b7-a153a3f4ad52` is attached a gzip archive. > > > Thanks. > > Boris. > > > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > The failure occurred during the post-deployment, which means that the initial deployment succeeded, but then the steps that are done to the completed overcloud failed. This is most commonly attributable to network problems between the Undercloud and the Overcloud Public API. The Undercloud needs to reach the Public API in order to do some of the post-configuration steps. If this API isn't reachable, you end up with the error you saw above. You can test this connectivity by pinging the Public API VIP from the Undercloud. Starting with the failed deployment, run "neutron port-list" against the Underlcloud and look for the IP on the port named "public_virtual_ip". You should be able to ping this address from the Undercloud. If you can't reach that IP, then you need to check the connectivity/routing between the Undercloud and the External network on the Overcloud.

On 06/29/2016 07:03 AM, Boris Derzhavets wrote: > Boris Derzhavets has shared a OneDrive file with you. To view it, click > the link below. > > <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk>

> > HeatCrash2.txt 1.gz <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > [HeatCrash2.txt 1.gz] > > Reattach gzip archive via One Drive > > > > ----------------------------------------------------------------------- > *From:* rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on > behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; > *Sent:* Wednesday, June 29, 2016 9:36 AM > *To:* John Trowbridge; shardy(a)redhat.com > *Cc:* rdo-list(a)redhat.com > *Subject:* [rdo-list] HA overcloud-deploy.sh crashes again ( > ControllerOvercloudServicesDeployment_Step4 ) > > > Attempt to follow steps suggested > in http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html > > > ./deploy-overstack crashes > > > 2016-06-29 12:42:41 > [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk-ControllerOvercloudServicesDeployment_Step4-nzdoizlgrmx2]: > CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment > to server failed: deploy_status_code : Deployment exited with non-zero > status code: 6 > 2016-06-29 12:42:42 [ControllerOvercloudServicesDeployment_Step4]: > CREATE_FAILED Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:43 > [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk]: CREATE_FAILED > Resource CREATE failed: Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:44 [ControllerNodesPostDeployment]: CREATE_FAILED > Error: > resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:44 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:46 [overcloud]: CREATE_FAILED Resource CREATE failed: > Error: > resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 > 2016-06-29 12:42:46 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:47 [2]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:47 [ControllerDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:48 [NetworkDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-29 12:42:48 [2]: SIGNAL_COMPLETE Unknown > Stack overcloud CREATE_FAILED > Deployment failed: Heat Stack create failed. > + heat stack-list > + grep -q CREATE_FAILED > + deploy_status=1 > ++ heat resource-list --nested-depth 5 overcloud > ++ grep FAILED > ++ grep 'StructuredDeployment ' > ++ cut -d '|' -f3 > + for failed in '$(heat resource-list --nested-depth 5 > overcloud | grep FAILED | > grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' > + heat deployment-show 655c77fc-6a78-4cca-b4b7-a153a3f4ad52 > + for failed in '$(heat resource-list --nested-depth 5 > overcloud | grep FAILED | > grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' > + heat deployment-show 1fe5153c-e017-4ee5-823a-3d1524430c1d > + for failed in '$(heat resource-list --nested-depth 5 > overcloud | grep FAILED | > grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' > + heat deployment-show bf6f25f4-d812-41e9-a7a8-122de619a624 > + exit 1 > > ***************************** > Troubleshooting steps :- > ***************************** > > [stack@undercloud ~]$ . stackrc > [stack@undercloud ~]$ heat resource-list overcloud | grep > ControllerNodesPost > | ControllerNodesPostDeployment | > f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | > OS::TripleO::ControllerPostDeployment | CREATE_FAILED | > 2016-06-29T12:11:21 | > > > [stack@undercloud ~]$ heat stack-list -n | grep "^| > f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3" > | f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk > | CREATE_FAILED | 2016-06-29T12:31:11 | None | > 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | > > > > [stack@undercloud ~]$ heat event-list -m > f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk > > +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ > | resource_name | > id | > resource_status_reason > | resource_status | event_time | > +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ > | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | > 10ec0cf9-b3c9-4191-9966-3f4d47f27e2a | Stack CREATE started > . . . . . . . . . . . . . . . . . > Step1,2,3 succeeded > . . . . . . . . . . . . . . . . . > > | CREATE_IN_PROGRESS | 2016-06-29T12:31:14 | > | ControllerPuppetConfig | > a2a1df33-5106-425c-b16d-8d2df709b19f | state > changed > | CREATE_COMPLETE | 2016-06-29T12:35:02 | > | ControllerOvercloudServicesDeployment_Step4 | > 1e151333-4de5-4e7b-907c-ea0f42d31a47 | state > changed > | CREATE_IN_PROGRESS | 2016-06-29T12:35:03 | > | ControllerOvercloudServicesDeployment_Step4 | > 7bf36334-3d92-4554-b6c0-41294a072ab6 | Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 | CREATE_FAILED | > 2016-06-29T12:42:42 | > | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk > | e72fb6f4-c2aa-4fe8-9bd1-5f5ad152685c | Resource CREATE failed: > Error: > resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: > Deployment to server failed: deploy_status_code: Deployment exited with > non-zero status code: 6 | CREATE_FAILED | 2016-06-29T12:42:43 | > +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ > > [stack@undercloud ~]$ heat stack-show > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | grep > NodeConfigIdentifiers > | | "NodeConfigIdentifiers": > "{u'deployment_identifier': 1467202276, u'controller_config': {u'1': > u'os-apply-config deployment 796df02a-7550-414b-a084-8b591a13e6db > completed,Root CA cert injection not enabled.,TLS not enabled.,None,', > u'0': u'os-apply-config deployment 613ec889-d852-470a-8e4c-6e243e1d2033 > completed,Root CA cert injection not enabled.,TLS not enabled.,None,', > u'2': u'os-apply-config deployment c8b099d0-3af4-4ba0-a056-a0ce60f40e2d > completed,Root CA cert injection not enabled.,TLS not enabled.,None,'}, > u'allnodes_extra': u'none'}" | > > However, when stack creating crashed update wouldn't help. > > [stack@undercloud ~]$ heat stack-update -x > overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk -e update_env.yaml > ERROR: PATCH update to non-COMPLETE stack is not supported. > > DUE TO :- > > [stack@undercloud ~]$ heat stack-list > +--------------------------------------+------------+---------------+---------------------+--------------+ > | id | stack_name | stack_status | > creation_time | updated_time | > +--------------------------------------+------------+---------------+---------------------+--------------+ > | 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | overcloud | CREATE_FAILED | > 2016-06-29T12:11:20 | None | > +--------------------------------------+------------+---------------+---------------------+------ > > > Complete error file `heat deployment-show > 655c77fc-6a78-4cca-b4b7-a153a3f4ad52` is attached a gzip archive. > > > Thanks. > > Boris. > > > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > The failure occurred during the post-deployment, which means that the initial deployment succeeded, but then the steps that are done to the completed overcloud failed. This is most commonly attributable to network problems between the Undercloud and the Overcloud Public API. The Undercloud needs to reach the Public API in order to do some of the post-configuration steps. If this API isn't reachable, you end up with the error you saw above. You can test this connectivity by pinging the Public API VIP from the Undercloud. Starting with the failed deployment, run "neutron port-list" against the Underlcloud and look for the IP on the port named "public_virtual_ip". You should be able to ping this address from the Undercloud. If you can't reach that IP, then you need to check the connectivity/routing between the Undercloud and the External network on the Overcloud.

________________________________ From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; Sent: Wednesday, June 29, 2016 5:14 PM To: Dan Sneddon; rdo-list(a)redhat.com Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) Yes , attempt to deploy ######################## # HA +2xCompute ######################## control_memory: 6144 compute_memory: 6144 undercloud_memory: 8192 # Giving the undercloud additional CPUs can greatly improve heat's # performance (and result in a shorter deploy time). undercloud_vcpu: 4

# Create three controller nodes and one compute node. overcloud_nodes: - name: control_0 flavor: control - name: control_1 flavor: control - name: control_2 flavor: control - name: compute_0 flavor: compute - name: compute_1 flavor: compute # We don't need introspection in a virtual environment (because we are # creating all the "hardware" we really know the necessary # information). introspect: false # Tell tripleo about our environment. network_isolation: true extra_args: >- --control-scale 3 --compute-scale 2 --neutron-network-type vxlan --neutron-tunnel-types vxlan -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml --ntp-server pool.ntp.org deploy_timeout: 75 tempest: false pingtest: true Results during overcloud deployment :- 2016-06-30 09:09:31 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" 2016-06-30 09:09:31 [NovaCompute]: DELETE_IN_PROGRESS state changed 2016-06-30 09:09:34 [NovaCompute]: DELETE_COMPLETE state changed 2016-06-30 09:09:44 [NovaCompute]: CREATE_IN_PROGRESS state changed 2016-06-30 09:09:48 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" . . . . . 2016-06-30 09:11:36 [overcloud]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Compute.resources[0].resources.NovaCompute: Went to status ERROR due to "Message: Build of instance bf483c34-7010-48ea-8f58-fe192c91093f aborted: Failed to provision instance bf483c34-7010-48ea-8f58-fe192 2016-06-30 09:11:36 [1]: SIGNAL_COMPLETE Unknown 2016-06-30 09:11:36 [ControllerDeployment]: SIGNAL_COMPLETE Unknown 2016-06-30 09:11:36 [1]: CREATE_COMPLETE state changed 2016-06-30 09:11:36 [overcloud-ControllerCephDeployment-62xh7uhtpjqp]: CREATE_COMPLETE Stack CREATE completed successfully 2016-06-30 09:11:37 [NetworkDeployment]: SIGNAL_COMPLETE Unknown 2016-06-30 09:11:37 [1]: SIGNAL_COMPLETE Unknown Stack overcloud CREATE_FAILED Deployment failed: Heat Stack create failed. + heat stack-list + grep -q CREATE_FAILED + deploy_status=1 ++ heat resource-list --nested-depth 5 overcloud ++ grep FAILED ++ grep 'StructuredDeployment ' ++ cut -d '|' -f3 + exit 1 Thanks. Boris ________________________________ From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Dan Sneddon <dsneddon(a)redhat.com&gt; Sent: Wednesday, June 29, 2016 1:46 PM To: rdo-list(a)redhat.com Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) On 06/29/2016 10:42 AM, Dan Sneddon wrote: > On 06/29/2016 07:03 AM, Boris Derzhavets wrote: >> Boris Derzhavets has shared a OneDrive file with you. To view it, click >> the link below. >> >> <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> [https://p.sfx.ms/icons/v2/Large/Default.png]<https://1drv.ms/u/s!AqjiD... HeatCrash2.txt 1.gz<https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> 1drv.ms GZ File >> >> HeatCrash2.txt 1.gz <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> >> [HeatCrash2.txt 1.gz] >> >> Reattach gzip archive via One Drive >> >> >> >> ----------------------------------------------------------------------- >> *From:* rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on >> behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; >> *Sent:* Wednesday, June 29, 2016 9:36 AM >> *To:* John Trowbridge; shardy(a)redhat.com >> *Cc:* rdo-list(a)redhat.com >> *Subject:* [rdo-list] HA overcloud-deploy.sh crashes again ( >> ControllerOvercloudServicesDeployment_Step4 ) >> >> >> Attempt to follow steps suggested >> in http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html >> >> >> ./deploy-overstack crashes >> >> >> 2016-06-29 12:42:41 >> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk-ControllerOvercloudServicesDeployment_Step4-nzdoizlgrmx2]: >> CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment >> to server failed: deploy_status_code : Deployment exited with non-zero >> status code: 6 >> 2016-06-29 12:42:42 [ControllerOvercloudServicesDeployment_Step4]: >> CREATE_FAILED Error: >> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >> Deployment to server failed: deploy_status_code: Deployment exited with >> non-zero status code: 6 >> 2016-06-29 12:42:43 >> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk]: CREATE_FAILED >> Resource CREATE failed: Error: >> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >> Deployment to server failed: deploy_status_code: Deployment exited with >> non-zero status code: 6 >> 2016-06-29 12:42:44 [ControllerNodesPostDeployment]: CREATE_FAILED >> Error: >> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >> Deployment to server failed: deploy_status_code: Deployment exited with >> non-zero status code: 6 >> 2016-06-29 12:42:44 [2]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:46 [overcloud]: CREATE_FAILED Resource CREATE failed: >> Error: >> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >> Deployment to server failed: deploy_status_code: Deployment exited with >> non-zero status code: 6 >> 2016-06-29 12:42:46 [2]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:47 [2]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:47 [ControllerDeployment]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:48 [NetworkDeployment]: SIGNAL_COMPLETE Unknown >> 2016-06-29 12:42:48 [2]: SIGNAL_COMPLETE Unknown >> Stack overcloud CREATE_FAILED >> Deployment failed: Heat Stack create failed. >> + heat stack-list >> + grep -q CREATE_FAILED >> + deploy_status=1 >> ++ heat resource-list --nested-depth 5 overcloud >> ++ grep FAILED >> ++ grep 'StructuredDeployment ' >> ++ cut -d '|' -f3 >> + for failed in '$(heat resource-list --nested-depth 5 >> overcloud | grep FAILED | >> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >> + heat deployment-show 655c77fc-6a78-4cca-b4b7-a153a3f4ad52 >> + for failed in '$(heat resource-list --nested-depth 5 >> overcloud | grep FAILED | >> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >> + heat deployment-show 1fe5153c-e017-4ee5-823a-3d1524430c1d >> + for failed in '$(heat resource-list --nested-depth 5 >> overcloud | grep FAILED | >> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >> + heat deployment-show bf6f25f4-d812-41e9-a7a8-122de619a624 >> + exit 1 >> >> ***************************** >> Troubleshooting steps :- >> ***************************** >> >> [stack@undercloud ~]$ . stackrc >> [stack@undercloud ~]$ heat resource-list overcloud | grep >> ControllerNodesPost >> | ControllerNodesPostDeployment | >> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | >> OS::TripleO::ControllerPostDeployment | CREATE_FAILED | >> 2016-06-29T12:11:21 | >> >> >> [stack@undercloud ~]$ heat stack-list -n | grep "^| >> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3" >> | f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | >> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >> | CREATE_FAILED | 2016-06-29T12:31:11 | None | >> 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | >> >> >> >> [stack@undercloud ~]$ heat event-list -m >> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 >> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >> >> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >> | resource_name | >> id | >> resource_status_reason >> | resource_status | event_time | >> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | >> 10ec0cf9-b3c9-4191-9966-3f4d47f27e2a | Stack CREATE started >> . . . . . . . . . . . . . . . . . >> Step1,2,3 succeeded >> . . . . . . . . . . . . . . . . . >> >> | CREATE_IN_PROGRESS | 2016-06-29T12:31:14 | >> | ControllerPuppetConfig | >> a2a1df33-5106-425c-b16d-8d2df709b19f | state >> changed >> | CREATE_COMPLETE | 2016-06-29T12:35:02 | >> | ControllerOvercloudServicesDeployment_Step4 | >> 1e151333-4de5-4e7b-907c-ea0f42d31a47 | state >> changed >> | CREATE_IN_PROGRESS | 2016-06-29T12:35:03 | >> | ControllerOvercloudServicesDeployment_Step4 | >> 7bf36334-3d92-4554-b6c0-41294a072ab6 | Error: >> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >> Deployment to server failed: deploy_status_code: Deployment exited with >> non-zero status code: 6 | CREATE_FAILED | >> 2016-06-29T12:42:42 | >> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >> | e72fb6f4-c2aa-4fe8-9bd1-5f5ad152685c | Resource CREATE failed: >> Error: >> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >> Deployment to server failed: deploy_status_code: Deployment exited with >> non-zero status code: 6 | CREATE_FAILED | 2016-06-29T12:42:43 | >> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >> >> [stack@undercloud ~]$ heat stack-show >> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | grep >> NodeConfigIdentifiers >> | | "NodeConfigIdentifiers": >> "{u'deployment_identifier': 1467202276, u'controller_config': {u'1': >> u'os-apply-config deployment 796df02a-7550-414b-a084-8b591a13e6db >> completed,Root CA cert injection not enabled.,TLS not enabled.,None,', >> u'0': u'os-apply-config deployment 613ec889-d852-470a-8e4c-6e243e1d2033 >> completed,Root CA cert injection not enabled.,TLS not enabled.,None,', >> u'2': u'os-apply-config deployment c8b099d0-3af4-4ba0-a056-a0ce60f40e2d >> completed,Root CA cert injection not enabled.,TLS not enabled.,None,'}, >> u'allnodes_extra': u'none'}" | >> >> However, when stack creating crashed update wouldn't help. >> >> [stack@undercloud ~]$ heat stack-update -x >> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk -e update_env.yaml >> ERROR: PATCH update to non-COMPLETE stack is not supported. >> >> DUE TO :- >> >> [stack@undercloud ~]$ heat stack-list >> +--------------------------------------+------------+---------------+---------------------+--------------+ >> | id | stack_name | stack_status | >> creation_time | updated_time | >> +--------------------------------------+------------+---------------+---------------------+--------------+ >> | 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | overcloud | CREATE_FAILED | >> 2016-06-29T12:11:20 | None | >> +--------------------------------------+------------+---------------+---------------------+------ >> >> >> Complete error file `heat deployment-show >> 655c77fc-6a78-4cca-b4b7-a153a3f4ad52` is attached a gzip archive. >> >> >> Thanks. >> >> Boris. >> >> >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> > > The failure occurred during the post-deployment, which means that the > initial deployment succeeded, but then the steps that are done to the > completed overcloud failed. > > This is most commonly attributable to network problems between the > Undercloud and the Overcloud Public API. The Undercloud needs to reach > the Public API in order to do some of the post-configuration steps. If > this API isn't reachable, you end up with the error you saw above. > > You can test this connectivity by pinging the Public API VIP from the > Undercloud. Starting with the failed deployment, run "neutron > port-list" against the Underlcloud and look for the IP on the port > named "public_virtual_ip". You should be able to ping this address from > the Undercloud. If you can't reach that IP, then you need to check the > connectivity/routing between the Undercloud and the External network on > the Overcloud. > I should also mention common causes of this problem: * Incorrect value for ExternalInterfaceDefaultRoute in the network environment file. * Controllers do not have the default route on the External network in the NIC config templates (required for reachability from remote subnets). * Incorrect subnet mask on the ExternalNetCidr in the network environment. * Incorrect ExternalAllocationPools values in the network environment. * Incorrect Ethernet switch config for the Controllers. Issue has been reproduced with exactly same error 4 times starting since 06/25/16 on daily basis with exactly same error at Step4 of overcloud-ControllerNodesPostDeployment. In meantime I cannot reproduce the error. Config 3xNode HA Controller + 1xCompute works . There was one more issue 3xNode HA Controller + 2xCompute failed immediately when overcloud-deploy.sh started due to only 4 nodes could be introspected. I will test it tomorrow morning. Thanks a lot. Boris. -- Dan Sneddon | Principal OpenStack Engineer dsneddon(a)redhat.com | redhat.com/openstack 650.254.4025 | dsneddon:irc @dxs:twitter _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com This body part will be downloaded on demand.

________________________________ From: John Trowbridge <trown(a)redhat.com&gt; Sent: Thursday, June 30, 2016 10:14 AM To: Boris Derzhavets; Dan Sneddon; rdo-list(a)redhat.com Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) On 06/30/2016 05:19 AM, Boris Derzhavets wrote: > > > > ________________________________ > From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; > Sent: Wednesday, June 29, 2016 5:14 PM > To: Dan Sneddon; rdo-list(a)redhat.com > Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) > > Yes , attempt to deploy > > ######################## > # HA +2xCompute > ######################## > control_memory: 6144 > compute_memory: 6144 > > undercloud_memory: 8192 > > # Giving the undercloud additional CPUs can greatly improve heat's > # performance (and result in a shorter deploy time). > undercloud_vcpu: 4 Increasing this without also increasing the memory on the undercloud will usually end in sadness, because more CPUs means more worker processes means more memory consumption. In general straying from the values in CI, is unlikely to work unless you have significantly better hardware than what runs in CI (32G hosts with decent CPU). It will be verified tomorrow with undercloud_vcpu: 2 This test would be a fair . It will take about 2 hr. But, I still believe that it is not root cause of issue with Configuration - 3xController(HA) + 2xCompute having :- undercloud_memory: 8192 undercloud_vcpu: 4 which was tested many times OK since 06/05 up to 06/24 with no problems.

Thank you very much for feedback Boris. https://github.com/openstack/tripleo-quickstart/blob/master/config/genera... It is not 100% that is the root cause of your issue, as the logs below look like we hit issues either with Ironic deployment to the nodes, or some issue with Nova scheduler. Note, that is definitely a different problem (and possibly transient), than the one reported in the beginning of this thread. > > # Create three controller nodes and one compute node. > overcloud_nodes: > - name: control_0 > flavor: control > - name: control_1 > flavor: control > - name: control_2 > flavor: control > > - name: compute_0 > flavor: compute > - name: compute_1 > flavor: compute > > # We don't need introspection in a virtual environment (because we are > # creating all the "hardware" we really know the necessary > # information). > introspect: false > > # Tell tripleo about our environment. > network_isolation: true > extra_args: >- > --control-scale 3 --compute-scale 2 --neutron-network-type vxlan > --neutron-tunnel-types vxlan > -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml > --ntp-server pool.ntp.org > deploy_timeout: 75 > tempest: false > pingtest: true > > Results during overcloud deployment :- > > 2016-06-30 09:09:31 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" > 2016-06-30 09:09:31 [NovaCompute]: DELETE_IN_PROGRESS state changed > 2016-06-30 09:09:34 [NovaCompute]: DELETE_COMPLETE state changed > 2016-06-30 09:09:44 [NovaCompute]: CREATE_IN_PROGRESS state changed > 2016-06-30 09:09:48 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" > . . . . . > > 2016-06-30 09:11:36 [overcloud]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Compute.resources[0].resources.NovaCompute: Went to status ERROR due to "Message: Build of instance bf483c34-7010-48ea-8f58-fe192c91093f aborted: Failed to provision instance bf483c34-7010-48ea-8f58-fe192 > 2016-06-30 09:11:36 [1]: SIGNAL_COMPLETE Unknown > 2016-06-30 09:11:36 [ControllerDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-30 09:11:36 [1]: CREATE_COMPLETE state changed > 2016-06-30 09:11:36 [overcloud-ControllerCephDeployment-62xh7uhtpjqp]: CREATE_COMPLETE Stack CREATE completed successfully > 2016-06-30 09:11:37 [NetworkDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-30 09:11:37 [1]: SIGNAL_COMPLETE Unknown > Stack overcloud CREATE_FAILED > Deployment failed: Heat Stack create failed. > + heat stack-list > + grep -q CREATE_FAILED > + deploy_status=1 > ++ heat resource-list --nested-depth 5 overcloud > ++ grep FAILED > ++ grep 'StructuredDeployment ' > ++ cut -d '|' -f3 > + exit 1 > > > Thanks. > > Boris > > > ________________________________ > From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Dan Sneddon <dsneddon(a)redhat.com&gt; > Sent: Wednesday, June 29, 2016 1:46 PM > To: rdo-list(a)redhat.com > Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) > > On 06/29/2016 10:42 AM, Dan Sneddon wrote: >> On 06/29/2016 07:03 AM, Boris Derzhavets wrote: >>> Boris Derzhavets has shared a OneDrive file with you. To view it, click >>> the link below. >>> >>> <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > [https://p.sfx.ms/icons/v2/Large/Default.png]<https://1drv.ms/u/s!AqjiD... > > HeatCrash2.txt 1.gz<https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > 1drv.ms > GZ File > > >>> >>> HeatCrash2.txt 1.gz <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> >>> [HeatCrash2.txt 1.gz] >>> >>> Reattach gzip archive via One Drive >>> >>> >>> >>> ----------------------------------------------------------------------- >>> *From:* rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on >>> behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; >>> *Sent:* Wednesday, June 29, 2016 9:36 AM >>> *To:* John Trowbridge; shardy(a)redhat.com >>> *Cc:* rdo-list(a)redhat.com >>> *Subject:* [rdo-list] HA overcloud-deploy.sh crashes again ( >>> ControllerOvercloudServicesDeployment_Step4 ) >>> >>> >>> Attempt to follow steps suggested >>> in http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html >>> >>> >>> ./deploy-overstack crashes >>> >>> >>> 2016-06-29 12:42:41 >>> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk-ControllerOvercloudServicesDeployment_Step4-nzdoizlgrmx2]: >>> CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment >>> to server failed: deploy_status_code : Deployment exited with non-zero >>> status code: 6 >>> 2016-06-29 12:42:42 [ControllerOvercloudServicesDeployment_Step4]: >>> CREATE_FAILED Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:43 >>> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk]: CREATE_FAILED >>> Resource CREATE failed: Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:44 [ControllerNodesPostDeployment]: CREATE_FAILED >>> Error: >>> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:44 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:46 [overcloud]: CREATE_FAILED Resource CREATE failed: >>> Error: >>> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:46 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:47 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:47 [ControllerDeployment]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:48 [NetworkDeployment]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:48 [2]: SIGNAL_COMPLETE Unknown >>> Stack overcloud CREATE_FAILED >>> Deployment failed: Heat Stack create failed. >>> + heat stack-list >>> + grep -q CREATE_FAILED >>> + deploy_status=1 >>> ++ heat resource-list --nested-depth 5 overcloud >>> ++ grep FAILED >>> ++ grep 'StructuredDeployment ' >>> ++ cut -d '|' -f3 >>> + for failed in '$(heat resource-list --nested-depth 5 >>> overcloud | grep FAILED | >>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >>> + heat deployment-show 655c77fc-6a78-4cca-b4b7-a153a3f4ad52 >>> + for failed in '$(heat resource-list --nested-depth 5 >>> overcloud | grep FAILED | >>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >>> + heat deployment-show 1fe5153c-e017-4ee5-823a-3d1524430c1d >>> + for failed in '$(heat resource-list --nested-depth 5 >>> overcloud | grep FAILED | >>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >>> + heat deployment-show bf6f25f4-d812-41e9-a7a8-122de619a624 >>> + exit 1 >>> >>> ***************************** >>> Troubleshooting steps :- >>> ***************************** >>> >>> [stack@undercloud ~]$ . stackrc >>> [stack@undercloud ~]$ heat resource-list overcloud | grep >>> ControllerNodesPost >>> | ControllerNodesPostDeployment | >>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | >>> OS::TripleO::ControllerPostDeployment | CREATE_FAILED | >>> 2016-06-29T12:11:21 | >>> >>> >>> [stack@undercloud ~]$ heat stack-list -n | grep "^| >>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3" >>> | f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >>> | CREATE_FAILED | 2016-06-29T12:31:11 | None | >>> 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | >>> >>> >>> >>> [stack@undercloud ~]$ heat event-list -m >>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >>> >>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >>> | resource_name | >>> id | >>> resource_status_reason >>> | resource_status | event_time | >>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >>> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | >>> 10ec0cf9-b3c9-4191-9966-3f4d47f27e2a | Stack CREATE started >>> . . . . . . . . . . . . . . . . . >>> Step1,2,3 succeeded >>> . . . . . . . . . . . . . . . . . >>> >>> | CREATE_IN_PROGRESS | 2016-06-29T12:31:14 | >>> | ControllerPuppetConfig | >>> a2a1df33-5106-425c-b16d-8d2df709b19f | state >>> changed >>> | CREATE_COMPLETE | 2016-06-29T12:35:02 | >>> | ControllerOvercloudServicesDeployment_Step4 | >>> 1e151333-4de5-4e7b-907c-ea0f42d31a47 | state >>> changed >>> | CREATE_IN_PROGRESS | 2016-06-29T12:35:03 | >>> | ControllerOvercloudServicesDeployment_Step4 | >>> 7bf36334-3d92-4554-b6c0-41294a072ab6 | Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 | CREATE_FAILED | >>> 2016-06-29T12:42:42 | >>> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >>> | e72fb6f4-c2aa-4fe8-9bd1-5f5ad152685c | Resource CREATE failed: >>> Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 | CREATE_FAILED | 2016-06-29T12:42:43 | >>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >>> >>> [stack@undercloud ~]$ heat stack-show >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | grep >>> NodeConfigIdentifiers >>> | | "NodeConfigIdentifiers": >>> "{u'deployment_identifier': 1467202276, u'controller_config': {u'1': >>> u'os-apply-config deployment 796df02a-7550-414b-a084-8b591a13e6db >>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,', >>> u'0': u'os-apply-config deployment 613ec889-d852-470a-8e4c-6e243e1d2033 >>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,', >>> u'2': u'os-apply-config deployment c8b099d0-3af4-4ba0-a056-a0ce60f40e2d >>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,'}, >>> u'allnodes_extra': u'none'}" | >>> >>> However, when stack creating crashed update wouldn't help. >>> >>> [stack@undercloud ~]$ heat stack-update -x >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk -e update_env.yaml >>> ERROR: PATCH update to non-COMPLETE stack is not supported. >>> >>> DUE TO :- >>> >>> [stack@undercloud ~]$ heat stack-list >>> +--------------------------------------+------------+---------------+---------------------+--------------+ >>> | id | stack_name | stack_status | >>> creation_time | updated_time | >>> +--------------------------------------+------------+---------------+---------------------+--------------+ >>> | 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | overcloud | CREATE_FAILED | >>> 2016-06-29T12:11:20 | None | >>> +--------------------------------------+------------+---------------+---------------------+------ >>> >>> >>> Complete error file `heat deployment-show >>> 655c77fc-6a78-4cca-b4b7-a153a3f4ad52` is attached a gzip archive. >>> >>> >>> Thanks. >>> >>> Boris. >>> >>> >>> >>> _______________________________________________ >>> rdo-list mailing list >>> rdo-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> >> The failure occurred during the post-deployment, which means that the >> initial deployment succeeded, but then the steps that are done to the >> completed overcloud failed. >> >> This is most commonly attributable to network problems between the >> Undercloud and the Overcloud Public API. The Undercloud needs to reach >> the Public API in order to do some of the post-configuration steps. If >> this API isn't reachable, you end up with the error you saw above. >> >> You can test this connectivity by pinging the Public API VIP from the >> Undercloud. Starting with the failed deployment, run "neutron >> port-list" against the Underlcloud and look for the IP on the port >> named "public_virtual_ip". You should be able to ping this address from >> the Undercloud. If you can't reach that IP, then you need to check the >> connectivity/routing between the Undercloud and the External network on >> the Overcloud. >> > > I should also mention common causes of this problem: > > * Incorrect value for ExternalInterfaceDefaultRoute in the network > environment file. > * Controllers do not have the default route on the External network in > the NIC config templates (required for reachability from remote subnets). > * Incorrect subnet mask on the ExternalNetCidr in the network environment. > * Incorrect ExternalAllocationPools values in the network environment. > * Incorrect Ethernet switch config for the Controllers. > > Issue has been reproduced with exactly same error 4 times > starting since 06/25/16 on daily basis with exactly same error at Step4 > of overcloud-ControllerNodesPostDeployment. > In meantime I cannot reproduce the error. > Config 3xNode HA Controller + 1xCompute works . > There was one more issue 3xNode HA Controller + 2xCompute > failed immediately when overcloud-deploy.sh started due to > only 4 nodes could be introspected. I will test it tomorrow morning. > > Thanks a lot. > Boris. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon(a)redhat.com | redhat.com/openstack > 650.254.4025 | dsneddon:irc @dxs:twitter > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > This body part will be downloaded on demand. >

________________________________ From: John Trowbridge <trown(a)redhat.com&gt; Sent: Thursday, June 30, 2016 10:14 AM To: Boris Derzhavets; Dan Sneddon; rdo-list(a)redhat.com Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) On 06/30/2016 05:19 AM, Boris Derzhavets wrote: > > > > ________________________________ > From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; > Sent: Wednesday, June 29, 2016 5:14 PM > To: Dan Sneddon; rdo-list(a)redhat.com > Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) > > Yes , attempt to deploy > > ######################## > # HA +2xCompute > ######################## > control_memory: 6144 > compute_memory: 6144 > > undercloud_memory: 8192 > > # Giving the undercloud additional CPUs can greatly improve heat's > # performance (and result in a shorter deploy time). > undercloud_vcpu: 4 Increasing this without also increasing the memory on the undercloud will usually end in sadness, because more CPUs means more worker processes means more memory consumption. In general straying from the values in CI, is unlikely to work unless you have significantly better hardware than what runs in CI (32G hosts with decent CPU). It will be verified tomorrow with undercloud_vcpu: 2

This test would be a fair . It will take about 2 hr. But, I still believe that it is not root cause of issue with Configuration - 3xController(HA) + 2xCompute having :- undercloud_memory: 8192 undercloud_vcpu: 4 which was tested many times OK since 06/05 up to 06/24 with no problems.

Thank you very much for feedback Boris. https://github.com/openstack/tripleo-quickstart/blob/master/config/genera... It is not 100% that is the root cause of your issue, as the logs below look like we hit issues either with Ironic deployment to the nodes, or some issue with Nova scheduler. Note, that is definitely a different problem (and possibly transient), than the one reported in the beginning of this thread. > > # Create three controller nodes and one compute node. > overcloud_nodes: > - name: control_0 > flavor: control > - name: control_1 > flavor: control > - name: control_2 > flavor: control > > - name: compute_0 > flavor: compute > - name: compute_1 > flavor: compute > > # We don't need introspection in a virtual environment (because we are > # creating all the "hardware" we really know the necessary > # information). > introspect: false > > # Tell tripleo about our environment. > network_isolation: true > extra_args: >- > --control-scale 3 --compute-scale 2 --neutron-network-type vxlan > --neutron-tunnel-types vxlan > -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml > --ntp-server pool.ntp.org > deploy_timeout: 75 > tempest: false > pingtest: true > > Results during overcloud deployment :- > > 2016-06-30 09:09:31 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" > 2016-06-30 09:09:31 [NovaCompute]: DELETE_IN_PROGRESS state changed > 2016-06-30 09:09:34 [NovaCompute]: DELETE_COMPLETE state changed > 2016-06-30 09:09:44 [NovaCompute]: CREATE_IN_PROGRESS state changed > 2016-06-30 09:09:48 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" > . . . . . > > 2016-06-30 09:11:36 [overcloud]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Compute.resources[0].resources.NovaCompute: Went to status ERROR due to "Message: Build of instance bf483c34-7010-48ea-8f58-fe192c91093f aborted: Failed to provision instance bf483c34-7010-48ea-8f58-fe192 > 2016-06-30 09:11:36 [1]: SIGNAL_COMPLETE Unknown > 2016-06-30 09:11:36 [ControllerDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-30 09:11:36 [1]: CREATE_COMPLETE state changed > 2016-06-30 09:11:36 [overcloud-ControllerCephDeployment-62xh7uhtpjqp]: CREATE_COMPLETE Stack CREATE completed successfully > 2016-06-30 09:11:37 [NetworkDeployment]: SIGNAL_COMPLETE Unknown > 2016-06-30 09:11:37 [1]: SIGNAL_COMPLETE Unknown > Stack overcloud CREATE_FAILED > Deployment failed: Heat Stack create failed. > + heat stack-list > + grep -q CREATE_FAILED > + deploy_status=1 > ++ heat resource-list --nested-depth 5 overcloud > ++ grep FAILED > ++ grep 'StructuredDeployment ' > ++ cut -d '|' -f3 > + exit 1 > > > Thanks. > > Boris > > > ________________________________ > From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Dan Sneddon <dsneddon(a)redhat.com&gt; > Sent: Wednesday, June 29, 2016 1:46 PM > To: rdo-list(a)redhat.com > Subject: Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 ) > > On 06/29/2016 10:42 AM, Dan Sneddon wrote: >> On 06/29/2016 07:03 AM, Boris Derzhavets wrote: >>> Boris Derzhavets has shared a OneDrive file with you. To view it, click >>> the link below. >>> >>> <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > [https://p.sfx.ms/icons/v2/Large/Default.png]<https://1drv.ms/u/s!AqjiD... > > HeatCrash2.txt 1.gz<https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> > 1drv.ms > GZ File > > >>> >>> HeatCrash2.txt 1.gz <https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk> >>> [HeatCrash2.txt 1.gz] >>> >>> Reattach gzip archive via One Drive >>> >>> >>> >>> ----------------------------------------------------------------------- >>> *From:* rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on >>> behalf of Boris Derzhavets <bderzhavets(a)hotmail.com&gt; >>> *Sent:* Wednesday, June 29, 2016 9:36 AM >>> *To:* John Trowbridge; shardy(a)redhat.com >>> *Cc:* rdo-list(a)redhat.com >>> *Subject:* [rdo-list] HA overcloud-deploy.sh crashes again ( >>> ControllerOvercloudServicesDeployment_Step4 ) >>> >>> >>> Attempt to follow steps suggested >>> in http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html >>> >>> >>> ./deploy-overstack crashes >>> >>> >>> 2016-06-29 12:42:41 >>> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk-ControllerOvercloudServicesDeployment_Step4-nzdoizlgrmx2]: >>> CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment >>> to server failed: deploy_status_code : Deployment exited with non-zero >>> status code: 6 >>> 2016-06-29 12:42:42 [ControllerOvercloudServicesDeployment_Step4]: >>> CREATE_FAILED Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:43 >>> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk]: CREATE_FAILED >>> Resource CREATE failed: Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:44 [ControllerNodesPostDeployment]: CREATE_FAILED >>> Error: >>> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:44 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:46 [overcloud]: CREATE_FAILED Resource CREATE failed: >>> Error: >>> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 >>> 2016-06-29 12:42:46 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:47 [2]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:47 [ControllerDeployment]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:48 [NetworkDeployment]: SIGNAL_COMPLETE Unknown >>> 2016-06-29 12:42:48 [2]: SIGNAL_COMPLETE Unknown >>> Stack overcloud CREATE_FAILED >>> Deployment failed: Heat Stack create failed. >>> + heat stack-list >>> + grep -q CREATE_FAILED >>> + deploy_status=1 >>> ++ heat resource-list --nested-depth 5 overcloud >>> ++ grep FAILED >>> ++ grep 'StructuredDeployment ' >>> ++ cut -d '|' -f3 >>> + for failed in '$(heat resource-list --nested-depth 5 >>> overcloud | grep FAILED | >>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >>> + heat deployment-show 655c77fc-6a78-4cca-b4b7-a153a3f4ad52 >>> + for failed in '$(heat resource-list --nested-depth 5 >>> overcloud | grep FAILED | >>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >>> + heat deployment-show 1fe5153c-e017-4ee5-823a-3d1524430c1d >>> + for failed in '$(heat resource-list --nested-depth 5 >>> overcloud | grep FAILED | >>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)' >>> + heat deployment-show bf6f25f4-d812-41e9-a7a8-122de619a624 >>> + exit 1 >>> >>> ***************************** >>> Troubleshooting steps :- >>> ***************************** >>> >>> [stack@undercloud ~]$ . stackrc >>> [stack@undercloud ~]$ heat resource-list overcloud | grep >>> ControllerNodesPost >>> | ControllerNodesPostDeployment | >>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | >>> OS::TripleO::ControllerPostDeployment | CREATE_FAILED | >>> 2016-06-29T12:11:21 | >>> >>> >>> [stack@undercloud ~]$ heat stack-list -n | grep "^| >>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3" >>> | f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 | >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >>> | CREATE_FAILED | 2016-06-29T12:31:11 | None | >>> 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | >>> >>> >>> >>> [stack@undercloud ~]$ heat event-list -m >>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >>> >>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >>> | resource_name | >>> id | >>> resource_status_reason >>> | resource_status | event_time | >>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >>> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | >>> 10ec0cf9-b3c9-4191-9966-3f4d47f27e2a | Stack CREATE started >>> . . . . . . . . . . . . . . . . . >>> Step1,2,3 succeeded >>> . . . . . . . . . . . . . . . . . >>> >>> | CREATE_IN_PROGRESS | 2016-06-29T12:31:14 | >>> | ControllerPuppetConfig | >>> a2a1df33-5106-425c-b16d-8d2df709b19f | state >>> changed >>> | CREATE_COMPLETE | 2016-06-29T12:35:02 | >>> | ControllerOvercloudServicesDeployment_Step4 | >>> 1e151333-4de5-4e7b-907c-ea0f42d31a47 | state >>> changed >>> | CREATE_IN_PROGRESS | 2016-06-29T12:35:03 | >>> | ControllerOvercloudServicesDeployment_Step4 | >>> 7bf36334-3d92-4554-b6c0-41294a072ab6 | Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 | CREATE_FAILED | >>> 2016-06-29T12:42:42 | >>> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk >>> | e72fb6f4-c2aa-4fe8-9bd1-5f5ad152685c | Resource CREATE failed: >>> Error: >>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]: >>> Deployment to server failed: deploy_status_code: Deployment exited with >>> non-zero status code: 6 | CREATE_FAILED | 2016-06-29T12:42:43 | >>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+ >>> >>> [stack@undercloud ~]$ heat stack-show >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | grep >>> NodeConfigIdentifiers >>> | | "NodeConfigIdentifiers": >>> "{u'deployment_identifier': 1467202276, u'controller_config': {u'1': >>> u'os-apply-config deployment 796df02a-7550-414b-a084-8b591a13e6db >>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,', >>> u'0': u'os-apply-config deployment 613ec889-d852-470a-8e4c-6e243e1d2033 >>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,', >>> u'2': u'os-apply-config deployment c8b099d0-3af4-4ba0-a056-a0ce60f40e2d >>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,'}, >>> u'allnodes_extra': u'none'}" | >>> >>> However, when stack creating crashed update wouldn't help. >>> >>> [stack@undercloud ~]$ heat stack-update -x >>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk -e update_env.yaml >>> ERROR: PATCH update to non-COMPLETE stack is not supported. >>> >>> DUE TO :- >>> >>> [stack@undercloud ~]$ heat stack-list >>> +--------------------------------------+------------+---------------+---------------------+--------------+ >>> | id | stack_name | stack_status | >>> creation_time | updated_time | >>> +--------------------------------------+------------+---------------+---------------------+--------------+ >>> | 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | overcloud | CREATE_FAILED | >>> 2016-06-29T12:11:20 | None | >>> +--------------------------------------+------------+---------------+---------------------+------ >>> >>> >>> Complete error file `heat deployment-show >>> 655c77fc-6a78-4cca-b4b7-a153a3f4ad52` is attached a gzip archive. >>> >>> >>> Thanks. >>> >>> Boris. >>> >>> >>> >>> _______________________________________________ >>> rdo-list mailing list >>> rdo-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> >> >> The failure occurred during the post-deployment, which means that the >> initial deployment succeeded, but then the steps that are done to the >> completed overcloud failed. >> >> This is most commonly attributable to network problems between the >> Undercloud and the Overcloud Public API. The Undercloud needs to reach >> the Public API in order to do some of the post-configuration steps. If >> this API isn't reachable, you end up with the error you saw above. >> >> You can test this connectivity by pinging the Public API VIP from the >> Undercloud. Starting with the failed deployment, run "neutron >> port-list" against the Underlcloud and look for the IP on the port >> named "public_virtual_ip". You should be able to ping this address from >> the Undercloud. If you can't reach that IP, then you need to check the >> connectivity/routing between the Undercloud and the External network on >> the Overcloud. >> > > I should also mention common causes of this problem: > > * Incorrect value for ExternalInterfaceDefaultRoute in the network > environment file. > * Controllers do not have the default route on the External network in > the NIC config templates (required for reachability from remote subnets). > * Incorrect subnet mask on the ExternalNetCidr in the network environment. > * Incorrect ExternalAllocationPools values in the network environment. > * Incorrect Ethernet switch config for the Controllers. > > Issue has been reproduced with exactly same error 4 times > starting since 06/25/16 on daily basis with exactly same error at Step4 > of overcloud-ControllerNodesPostDeployment. > In meantime I cannot reproduce the error. > Config 3xNode HA Controller + 1xCompute works . > There was one more issue 3xNode HA Controller + 2xCompute > failed immediately when overcloud-deploy.sh started due to > only 4 nodes could be introspected. I will test it tomorrow morning. > > Thanks a lot. > Boris. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon(a)redhat.com | redhat.com/openstack > 650.254.4025 | dsneddon:irc @dxs:twitter > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > This body part will be downloaded on demand. >

Hi Guys, I am still having issues while installing the undercloud. I have exhausted all the options and tried to do undercloud install as per the tripleo guidelines. http://tripleo.org/installation/installation.html But, each time I hit the same errors: Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. Please suggest. From: Gunjan, Milind [CTO] Sent: Wednesday, June 29, 2016 7:39 AM To: 'Mohammed Arafa' <mohammed.arafa(a)gmail.com&gt; Cc: rdo-list(a)redhat.com; Marius Cornea <marius(a)remote-lab.net&gt; Subject: RE: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment Okay . Updated the host file with shortname and fqdn. root@undercloud ~]# cat /etc/hosts 127.0.0.1 undercloud undercloud.poc I am looking to install stable release of Openstack kilo / liberty. Can you please point me to the stable repo which I can pull for the install. Best Regards, Milind From: Mohammed Arafa [mailto:mohammed.arafa@gmail.com] Sent: Wednesday, June 29, 2016 7:34 AM To: Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com&gt; Cc: rdo-list(a)redhat.com; Marius Cornea <marius(a)remote-lab.net&gt; Subject: RE: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment You need both short name and fqdn in your hosts file. (Rabbitmq issue. Don't know if that's been fixed) On Jun 29, 2016 1:19 PM, "Gunjan, Milind [CTO]" <Milind.Gunjan(a)sprint.com&gt; wrote: Thanks a lot for pointing out the mistakes. So, as suggested, I updated the hostname in the host file to match in similar way as instructed in the docs: [root@undercloud ~]# cat /etc/hostname undercloud.poc [root@undercloud ~]# cat /etc/hosts 127.0.0.1 undercloud.poc Is there any stable kilo package which can be used for installation ? Best Regards, Milind Gunjan From: Mohammed Arafa [mailto:mohammed.arafa@gmail.com] Sent: Wednesday, June 29, 2016 3:25 AM To: Marius Cornea <marius(a)remote-lab.net&gt; Cc: rdo-list(a)redhat.com; Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com&gt; Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment The hostname in the hosts file don't match. On Jun 29, 2016 8:58 AM, "Marius Cornea" <marius(a)remote-lab.net&gt; wrote: On Tue, Jun 28, 2016 at 10:45 PM, Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com&gt; wrote: > Hi All, > > Thanks a lot for continued support. > > I would just like to get clarity regarding the last recommendation. > My current deployment is failing with following error : > > Error: Could not prefetch keystone_service provider 'openstack': Execution > of '/bin/openstack service list --quiet --format csv --long' returned 1: > Gateway Timeout (HTTP 504) > Error: Could not prefetch keystone_role provider 'openstack': Execution of > '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout > (HTTP 504) > Error: Could not prefetch keystone_endpoint provider 'openstack': > Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: > Gateway Timeout (HTTP 504) > > > I have previously done RHEL OSP7 deployment and I verified the host file > of RHEL OSP7 undercloud deployment and it was configured with host > gateway used by pxe network. > > Similarly, we have set the current host gateway to 192.0.2.1 as shown > below for rdo manager undercloud installation: > [stack@rdo-undercloud etc]$ cat /etc/hosts >> 127.0.0.1 localhost localhost.localdomain localhost4 >> localhost4.localdomain4 >> ::1 localhost localhost.localdomain localhost6 >> localhost6.localdomain6 >> 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain > > I would like to know if it is required during rdo deployment to have it > with the address of the public interface, e.g: > $ip_public_nic rdo-undercloud rdo-undercloud.mydomain. The reason I asked for it is because of: Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") Typically the undercloud's local_ip is used for these operations but in your case it's added to the hosts files and maybe this ip name mapping doesn't allow the installation to proceed. Please note that the docs[1] point that you should have an FQDN entry in the hosts file before the undercloud installation(when 192.0.2.1 isn't yet set on the system), that's why I mentioned the ip address of the public nic. [1] http://docs.openstack.org/developer/tripleo-docs/installation/installatio... > > Thanks again for your time and help. Really appreciate it. > > Best Regards, > milind > > -----Original Message----- > From: Marius Cornea [mailto:marius@remote-lab.net] > Sent: Tuesday, June 28, 2016 4:28 AM > To: Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com&gt; > Cc: rdo-list(a)redhat.com > Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o > deployment > > On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] > <Milind.Gunjan(a)sprint.com&gt; wrote: >> Hi Dan, >> Thanks a lot for your response. >> >> Even after properly updating the undercloud.conf file and checking the >> network configuration, undercloud deployment still fails. >> >> To recreate the issue , I am mentioning all the configuration steps: >> 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. >> 2. created stack user and provided required permission to stack user . >> 3. setting hostname >> sudo hostnamectl set-hostname rdo-undercloud.mydomain >> sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain >> >> [stack@rdo-undercloud etc]$ cat /etc/hosts >> 127.0.0.1 localhost localhost.localdomain localhost4 >> localhost4.localdomain4 >> ::1 localhost localhost.localdomain localhost6 >> localhost6.localdomain6 >> 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain > > Could you try removing the 192.0.2.1 entry from /etc/hosts and replace > it with the address of the public interface, e.g: > $ip_public_nic rdo-undercloud rdo-undercloud.mydomain > > then rerun openstack undercloud install > >> 4. enable required repositories >> sudo yum -y install epel-release >> sudo curl -o /etc/yum.repos.d/delorean-liberty.repo >> https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo >> sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo >> http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo >> >> 5. install repos >> >> sudo yum -y install yum-plugin-priorities >> sudo yum install -y python-tripleoclient >> >> 6. update undercloud.conf >> >> [stack@rdo-undercloud ~]$ cat undercloud.conf >> [DEFAULT] >> local_ip = 192.0.2.1/24 >> undercloud_public_vip = 192.0.2.2 >> undercloud_admin_vip = 192.0.2.3 >> local_interface = enp6s0 >> masquerade_network = 192.0.2.0/24 >> dhcp_start = 192.0.2.150 >> dhcp_end = 192.0.2.199 >> network_cidr = 192.0.2.0/24 >> network_gateway = 192.0.2.1 >> discovery_iprange = 192.0.2.200,192.0.2.230 >> discovery_runbench = false >> [auth] >> >> 7. install undercloud >> >> openstack undercloud install >> >> install ends in error: >> Error: >> /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: >> Could not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: >> /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: >> Could not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: Could not prefetch keystone_service provider 'openstack': >> Execution of '/bin/openstack service list --quiet --format csv --long' >> returned 1: Gateway Timeout (HTTP 504) >> Error: Not managing Keystone_service[glance] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: >> change from absent to present failed: Not managing Keystone_service[glance] >> due to earlier Keystone API failures. >> Error: Could not prefetch keystone_role provider 'openstack': Execution >> of '/bin/openstack role list --quiet --format csv' returned 1: Gateway >> Timeout (HTTP 504) >> Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone >> API failures. >> Error: >> /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: >> change from absent to present failed: Not managing >> Keystone_role[ResellerAdmin] due to earlier Keystone API failures. >> Error: Not managing Keystone_service[ironic] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: >> change from absent to present failed: Not managing Keystone_service[ironic] >> due to earlier Keystone API failures. >> Error: >> /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova >> service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of >> '/bin/openstack domain show --format shell Default' returned 1: Could not >> find resource Default >> Error: >> /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: >> Could not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: Not managing Keystone_service[novav3] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova >> v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change >> from absent to present failed: Not managing Keystone_service[novav3] due to >> earlier Keystone API failures. >> Error: Not managing Keystone_role[heat_stack_user] due to earlier >> Keystone API failures. >> Error: >> /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: >> change from absent to present failed: Not managing >> Keystone_role[heat_stack_user] due to earlier Keystone API failures. >> Error: >> /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: >> Could not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: Not managing Keystone_service[nova] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova >> service, user nova]/Keystone_service[nova::compute]/ensure: change from >> absent to present failed: Not managing Keystone_service[nova] due to earlier >> Keystone API failures. >> Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone >> API failures. >> Error: >> /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: >> change from absent to present failed: Not managing >> Keystone_role[swiftoperator] due to earlier Keystone API failures. >> Error: >> /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: >> Could not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: Not managing Keystone_service[neutron] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: >> change from absent to present failed: Not managing Keystone_service[neutron] >> due to earlier Keystone API failures. >> Error: Not managing Keystone_service[ceilometer] due to earlier Keystone >> API failures. >> Error: >> /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: >> change from absent to present failed: Not managing >> Keystone_service[ceilometer] due to earlier Keystone API failures. >> Error: Not managing Keystone_service[swift] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: >> change from absent to present failed: Not managing Keystone_service[swift] >> due to earlier Keystone API failures. >> Error: Not managing Keystone_service[keystone] due to earlier Keystone >> API failures. >> Error: >> /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: >> change from absent to present failed: Not managing >> Keystone_service[keystone] due to earlier Keystone API failures. >> Error: Not managing Keystone_service[heat] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: >> change from absent to present failed: Not managing Keystone_service[heat] >> due to earlier Keystone API failures. >> Error: Could not prefetch keystone_endpoint provider 'openstack': >> Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: >> Gateway Timeout (HTTP 504) >> Error: >> /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: >> Could not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: Could not prefetch keystone_tenant provider 'openstack': Execution >> of '/bin/openstack project list --quiet --format csv --long' returned 1: >> Gateway Timeout (HTTP 504) >> Error: Not managing Keystone_tenant[service] due to earlier Keystone API >> failures. >> Error: >> /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change >> from absent to present failed: Not managing Keystone_tenant[service] due to >> earlier Keystone API failures. >> Error: Not managing Keystone_tenant[admin] due to earlier Keystone API >> failures. >> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: >> change from absent to present failed: Not managing Keystone_tenant[admin] >> due to earlier Keystone API failures. >> Error: Not managing Keystone_role[admin] due to earlier Keystone API >> failures. >> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: >> change from absent to present failed: Not managing Keystone_role[admin] due >> to earlier Keystone API failures. >> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could >> not evaluate: Execution of '/bin/openstack domain show --format shell >> Default' returned 1: Could not find resource Default >> Error: Could not prefetch keystone_domain provider 'openstack': Execution >> of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway >> Timeout (HTTP 504) >> Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: >> (pymysql.err.OperationalError) (1045, u"Access denied for user >> 'heat'@'rdo-undercloud' (using password: YES)") >> Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call >> refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 >> instead of one of [0] >> Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage >> --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] >> [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure >> phase. [Command '['dib-run-parts', >> '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status >> 1] >> >> Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered >> 'refresh' from 4 events >> Notice: Finished catalog run in 5259.44 seconds >> + rc=6 >> + set -e >> + echo 'puppet apply exited with exit code 6' >> puppet apply exited with exit code 6 >> + '[' 6 '!=' 2 -a 6 '!=' 0 ']' >> + exit 6 >> [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure >> phase. [Command '['dib-run-parts', >> '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status >> 1] >> >> [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... >> Traceback (most recent call last): >> File "<string>", line 1, in <module> >> File >> "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line >> 815, in install >> _run_orc(instack_env) >> File >> "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line >> 699, in _run_orc >> _run_live_command(args, instack_env, 'os-refresh-config') >> File >> "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line >> 370, in _run_live_command >> raise RuntimeError('%s failed. See log for details.' % name) >> RuntimeError: os-refresh-config failed. See log for details. >> Command 'instack-install-undercloud' returned non-zero exit status 1 >> >> >> I am not able to understand the exact cause of undercloud install >> failure. It would be really helpful if you guys can point be in direction to >> understand the exact cause of issue and any possible resolution. >> >> Thanks a lot. >> >> Best Regards, >> Milind >> >> >> Best Regards, >> Milind >> -----Original Message----- >> From: Dan Sneddon [mailto:dsneddon@redhat.com] >> Sent: Monday, June 27, 2016 12:40 PM >> To: Gunjan, Milind [CTO] <Milind.Gunjan(a)sprint.com>; rdo-list(a)redhat.com >> Subject: Re: [rdo-list] Redeploying UnderCloud >> >> On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >>> Hi All, >>> >>> Greeting. >>> >>> >>> >>> This is my first post and I am fairly new to RDO OpenStack. I am >>> working on RDO Triple-O deployment on baremetal. Due to incorrect >>> values in undercloud.conf file , my undercloud deployment failed. I >>> would like to redeploy undercloud and I am trying to understand what >>> steps has to be taken before redeploying undercloud. All the >>> deployment was under stack user . So first step will be to delete >>> stack user. I am not sure what has to be done regarding the networking >>> configuration autogenerated by os-net-config during the older install. >>> >>> Please advise. >>> >>> >>> >>> Best Regards, >>> >>> Milind >> >> No, definitely you don't want to delete the stack user, especially not as >> your first step! That would get rid of the configuration files, etc. >> that are in ~stack, and generally make your life harder than it needs to >> be. >> >> Anyway, it isn't necessary. You can do a procedure very much like what >> you do when upgrading the undercloud, with a couple of extra steps. >> >> As the stack user, edit your undercloud.conf, and make sure there are no >> more typos. >> >> If the typos were in the network configuration, you should delete the >> bridge and remove the ifcfg files: >> >> $ sudo ifdown br-ctlplane >> $ sudo ovs-vsctl del-br br-ctlplane >> $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane >> >> Next, run the underclound installation again: >> >> $ sudo yum update -y # Reboot after if kernel or core packages updated $ >> openstack undercloud install >> >> Then proceed with the rest of the instructions. You may find that if you >> already uploaded disk images or imported nodes that they will still be in >> the database. That's OK, or you can delete and reimport. >> >> -- >> Dan Sneddon | Principal OpenStack Engineer >> dsneddon(a)redhat.com | redhat.com/openstack >> 650.254.4025 | dsneddon:irc @dxs:twitter >> >> >> >> ________________________________ >> Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T >> or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off> for >> details. >> >> ________________________________ >> >> This e-mail may contain Sprint proprietary information intended for the >> sole use of the recipient(s). Any use by others is prohibited. If you are >> not the intended recipient, please contact the sender and delete all copies >> of the message. >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

Hi All, Thanks a lot for continued support. I would just like to get clarity regarding the last recommendation. My current deployment is failing with following error : Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7 undercloud deployment and it was configured with host gateway used by pxe network. Similarly, we have set the current host gateway to 192.0.2.1 as shown below for rdo manager undercloud installation: [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain.

Thanks again for your time and help. Really appreciate it. Best Regards, milind -----Original Message----- From: Marius Cornea [mailto:marius@remote-lab.net<mailto:marius@remote-lab.net>] Sent: Tuesday, June 28, 2016 4:28 AM To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.com> Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> wrote: > Hi Dan, > Thanks a lot for your response. > > Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. > > To recreate the issue , I am mentioning all the configuration steps: > 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. > 2. created stack user and provided required permission to stack user . > 3. setting hostname > sudo hostnamectl set-hostname rdo-undercloud.mydomain > sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain > > [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain Could you try removing the 192.0.2.1 entry from /etc/hosts and replace it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain then rerun openstack undercloud install > 4. enable required repositories > sudo yum -y install epel-release > sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo > sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo > > 5. install repos > > sudo yum -y install yum-plugin-priorities > sudo yum install -y python-tripleoclient > > 6. update undercloud.conf > > [stack@rdo-undercloud ~]$ cat undercloud.conf > [DEFAULT] > local_ip = 192.0.2.1/24<http://192.0.2.1/24> > undercloud_public_vip = 192.0.2.2 > undercloud_admin_vip = 192.0.2.3 > local_interface = enp6s0 > masquerade_network = 192.0.2.0/24<http://192.0.2.0/24> > dhcp_start = 192.0.2.150 > dhcp_end = 192.0.2.199 > network_cidr = 192.0.2.0/24<http://192.0.2.0/24> > network_gateway = 192.0.2.1 > discovery_iprange = 192.0.2.200,192.0.2.230 > discovery_runbench = false > [auth] > > 7. install undercloud > > openstack undercloud install > > install ends in error: > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events > Notice: Finished catalog run in 5259.44 seconds > + rc=6 > + set -e > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install > _run_orc(instack_env) > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc > _run_live_command(args, instack_env, 'os-refresh-config') > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command > raise RuntimeError('%s failed. See log for details.' % name) > RuntimeError: os-refresh-config failed. See log for details. > Command 'instack-install-undercloud' returned non-zero exit status 1 > > > I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. > > Thanks a lot. > > Best Regards, > Milind > > > Best Regards, > Milind > -----Original Message----- > From: Dan Sneddon [mailto:dsneddon@redhat.com<mailto:dsneddon@redhat.com>] > Sent: Monday, June 27, 2016 12:40 PM > To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>>; rdo-list@redhat.com<mailto:rdo-list@redhat.com> > Subject: Re: [rdo-list] Redeploying UnderCloud > > On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >> Hi All, >> >> Greeting. >> >> >> >> This is my first post and I am fairly new to RDO OpenStack. I am >> working on RDO Triple-O deployment on baremetal. Due to incorrect >> values in undercloud.conf file , my undercloud deployment failed. I >> would like to redeploy undercloud and I am trying to understand what >> steps has to be taken before redeploying undercloud. All the >> deployment was under stack user . So first step will be to delete >> stack user. I am not sure what has to be done regarding the networking >> configuration autogenerated by os-net-config during the older install. >> >> Please advise. >> >> >> >> Best Regards, >> >> Milind > > No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. > that are in ~stack, and generally make your life harder than it needs to be. > > Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. > > As the stack user, edit your undercloud.conf, and make sure there are no more typos. > > If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: > > $ sudo ifdown br-ctlplane > $ sudo ovs-vsctl del-br br-ctlplane > $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane > > Next, run the underclound installation again: > > $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install > > Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon@redhat.com<mailto:dsneddon@redhat.com> | redhat.com/openstack<http://redhat.com/openstack> > 650.254.4025<tel:650.254.4025> | dsneddon:irc @dxs:twitter > > > > ________________________________ > Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off><http://sprint.com/50of... for details. > > ________________________________ > > This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. > > _______________________________________________ > rdo-list mailing list > rdo-list@redhat.com<mailto:rdo-list@redhat.com> > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe@redhat.com<mailto:rdo-list-unsubscribe@redhat.com>

Hi All, Thanks a lot for continued support. I would just like to get clarity regarding the last recommendation. My current deployment is failing with following error : Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7 undercloud deployment and it was configured with host gateway used by pxe network. Similarly, we have set the current host gateway to 192.0.2.1 as shown below for rdo manager undercloud installation: [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain.

Thanks again for your time and help. Really appreciate it. Best Regards, milind -----Original Message----- From: Marius Cornea [mailto:marius@remote-lab.net<mailto:marius@remote-lab.net>] Sent: Tuesday, June 28, 2016 4:28 AM To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.com> Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> wrote: > Hi Dan, > Thanks a lot for your response. > > Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. > > To recreate the issue , I am mentioning all the configuration steps: > 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. > 2. created stack user and provided required permission to stack user . > 3. setting hostname > sudo hostnamectl set-hostname rdo-undercloud.mydomain > sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain > > [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain Could you try removing the 192.0.2.1 entry from /etc/hosts and replace it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain then rerun openstack undercloud install > 4. enable required repositories > sudo yum -y install epel-release > sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo > sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo > > 5. install repos > > sudo yum -y install yum-plugin-priorities > sudo yum install -y python-tripleoclient > > 6. update undercloud.conf > > [stack@rdo-undercloud ~]$ cat undercloud.conf > [DEFAULT] > local_ip = 192.0.2.1/24<http://192.0.2.1/24> > undercloud_public_vip = 192.0.2.2 > undercloud_admin_vip = 192.0.2.3 > local_interface = enp6s0 > masquerade_network = 192.0.2.0/24<http://192.0.2.0/24> > dhcp_start = 192.0.2.150 > dhcp_end = 192.0.2.199 > network_cidr = 192.0.2.0/24<http://192.0.2.0/24> > network_gateway = 192.0.2.1 > discovery_iprange = 192.0.2.200,192.0.2.230 > discovery_runbench = false > [auth] > > 7. install undercloud > > openstack undercloud install > > install ends in error: > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events > Notice: Finished catalog run in 5259.44 seconds > + rc=6 > + set -e > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install > _run_orc(instack_env) > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc > _run_live_command(args, instack_env, 'os-refresh-config') > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command > raise RuntimeError('%s failed. See log for details.' % name) > RuntimeError: os-refresh-config failed. See log for details. > Command 'instack-install-undercloud' returned non-zero exit status 1 > > > I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. > > Thanks a lot. > > Best Regards, > Milind > > > Best Regards, > Milind > -----Original Message----- > From: Dan Sneddon [mailto:dsneddon@redhat.com<mailto:dsneddon@redhat.com>] > Sent: Monday, June 27, 2016 12:40 PM > To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>>; rdo-list@redhat.com<mailto:rdo-list@redhat.com> > Subject: Re: [rdo-list] Redeploying UnderCloud > > On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >> Hi All, >> >> Greeting. >> >> >> >> This is my first post and I am fairly new to RDO OpenStack. I am >> working on RDO Triple-O deployment on baremetal. Due to incorrect >> values in undercloud.conf file , my undercloud deployment failed. I >> would like to redeploy undercloud and I am trying to understand what >> steps has to be taken before redeploying undercloud. All the >> deployment was under stack user . So first step will be to delete >> stack user. I am not sure what has to be done regarding the networking >> configuration autogenerated by os-net-config during the older install. >> >> Please advise. >> >> >> >> Best Regards, >> >> Milind > > No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. > that are in ~stack, and generally make your life harder than it needs to be. > > Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. > > As the stack user, edit your undercloud.conf, and make sure there are no more typos. > > If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: > > $ sudo ifdown br-ctlplane > $ sudo ovs-vsctl del-br br-ctlplane > $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane > > Next, run the underclound installation again: > > $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install > > Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon@redhat.com<mailto:dsneddon@redhat.com> | redhat.com/openstack<http://redhat.com/openstack> > 650.254.4025<tel:650.254.4025> | dsneddon:irc @dxs:twitter > > > > ________________________________ > Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off><http://sprint.com/50of... for details. > > ________________________________ > > This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. > > _______________________________________________ > rdo-list mailing list > rdo-list@redhat.com<mailto:rdo-list@redhat.com> > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe@redhat.com<mailto:rdo-list-unsubscribe@redhat.com>

Hi All, Thanks a lot for continued support. I would just like to get clarity regarding the last recommendation. My current deployment is failing with following error : Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7 undercloud deployment and it was configured with host gateway used by pxe network. Similarly, we have set the current host gateway to 192.0.2.1 as shown below for rdo manager undercloud installation: [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain.

Thanks again for your time and help. Really appreciate it. Best Regards, milind -----Original Message----- From: Marius Cornea [mailto:marius@remote-lab.net<mailto:marius@remote-lab.net>] Sent: Tuesday, June 28, 2016 4:28 AM To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.com> Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>> wrote: > Hi Dan, > Thanks a lot for your response. > > Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails. > > To recreate the issue , I am mentioning all the configuration steps: > 1. Installed CentOS Linux release 7.2.1511 (Core) image on baremetal. > 2. created stack user and provided required permission to stack user . > 3. setting hostname > sudo hostnamectl set-hostname rdo-undercloud.mydomain > sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain > > [stack@rdo-undercloud etc]$ cat /etc/hosts > 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 > 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain Could you try removing the 192.0.2.1 entry from /etc/hosts and replace it with the address of the public interface, e.g: $ip_public_nic rdo-undercloud rdo-undercloud.mydomain then rerun openstack undercloud install > 4. enable required repositories > sudo yum -y install epel-release > sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo > sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo > > 5. install repos > > sudo yum -y install yum-plugin-priorities > sudo yum install -y python-tripleoclient > > 6. update undercloud.conf > > [stack@rdo-undercloud ~]$ cat undercloud.conf > [DEFAULT] > local_ip = 192.0.2.1/24<http://192.0.2.1/24> > undercloud_public_vip = 192.0.2.2 > undercloud_admin_vip = 192.0.2.3 > local_interface = enp6s0 > masquerade_network = 192.0.2.0/24<http://192.0.2.0/24> > dhcp_start = 192.0.2.150 > dhcp_end = 192.0.2.199 > network_cidr = 192.0.2.0/24<http://192.0.2.0/24> > network_gateway = 192.0.2.1 > discovery_iprange = 192.0.2.200,192.0.2.230 > discovery_runbench = false > [auth] > > 7. install undercloud > > openstack undercloud install > > install ends in error: > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures. > Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures. > Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures. > Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures. > Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures. > Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures. > Error: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures. > Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures. > Error: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures. > Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504) > Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures. > Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures. > Error: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures. > Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default > Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504) > Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)") > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0] > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events > Notice: Finished catalog run in 5259.44 seconds > + rc=6 > + set -e > + echo 'puppet apply exited with exit code 6' > puppet apply exited with exit code 6 > + '[' 6 '!=' 2 -a 6 '!=' 0 ']' > + exit 6 > [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1] > > [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting... > Traceback (most recent call last): > File "<string>", line 1, in <module> > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install > _run_orc(instack_env) > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc > _run_live_command(args, instack_env, 'os-refresh-config') > File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command > raise RuntimeError('%s failed. See log for details.' % name) > RuntimeError: os-refresh-config failed. See log for details. > Command 'instack-install-undercloud' returned non-zero exit status 1 > > > I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution. > > Thanks a lot. > > Best Regards, > Milind > > > Best Regards, > Milind > -----Original Message----- > From: Dan Sneddon [mailto:dsneddon@redhat.com<mailto:dsneddon@redhat.com>] > Sent: Monday, June 27, 2016 12:40 PM > To: Gunjan, Milind [CTO] <Milind.Gunjan@sprint.com<mailto:Milind.Gunjan@sprint.com>>; rdo-list@redhat.com<mailto:rdo-list@redhat.com> > Subject: Re: [rdo-list] Redeploying UnderCloud > > On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote: >> Hi All, >> >> Greeting. >> >> >> >> This is my first post and I am fairly new to RDO OpenStack. I am >> working on RDO Triple-O deployment on baremetal. Due to incorrect >> values in undercloud.conf file , my undercloud deployment failed. I >> would like to redeploy undercloud and I am trying to understand what >> steps has to be taken before redeploying undercloud. All the >> deployment was under stack user . So first step will be to delete >> stack user. I am not sure what has to be done regarding the networking >> configuration autogenerated by os-net-config during the older install. >> >> Please advise. >> >> >> >> Best Regards, >> >> Milind > > No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc. > that are in ~stack, and generally make your life harder than it needs to be. > > Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps. > > As the stack user, edit your undercloud.conf, and make sure there are no more typos. > > If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files: > > $ sudo ifdown br-ctlplane > $ sudo ovs-vsctl del-br br-ctlplane > $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane > > Next, run the underclound installation again: > > $ sudo yum update -y # Reboot after if kernel or core packages updated $ openstack undercloud install > > Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport. > > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon@redhat.com<mailto:dsneddon@redhat.com> | redhat.com/openstack<http://redhat.com/openstack> > 650.254.4025<tel:650.254.4025> | dsneddon:irc @dxs:twitter > > > > ________________________________ > Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off><http://sprint.com/50of... for details. > > ________________________________ > > This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. > > _______________________________________________ > rdo-list mailing list > rdo-list@redhat.com<mailto:rdo-list@redhat.com> > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe@redhat.com<mailto:rdo-list-unsubscribe@redhat.com>