[rdo-list] Redeploying UnderCloud for baremetal triple-o deployment

Wed Jun 29 06:56:28 UTC 2016

On Tue, Jun 28, 2016 at 10:45 PM, Gunjan, Milind [CTO]
<Milind.Gunjan at sprint.com> wrote:
> Hi All,
>
> Thanks a lot for continued support.
>
> I would just like to get clarity regarding the last recommendation.
> My current deployment is failing with following error :
>
> Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504)
> Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504)
> Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504)
>
>
> I have previously done RHEL OSP7 deployment and I verified the host file of RHEL OSP7  undercloud deployment and   it was configured with host gateway used by pxe network.
>
> Similarly, we have set the current host gateway to 192.0.2.1 as shown below  for rdo manager undercloud installation:
> [stack at rdo-undercloud etc]$ cat /etc/hosts
>> 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
>> ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
>> 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain
>
> I would like to know if it is required during rdo deployment to have it with the address of the public interface, e.g:
> $ip_public_nic    rdo-undercloud rdo-undercloud.mydomain.

The reason I asked for it is because of:

Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR:
(pymysql.err.OperationalError) (1045, u"Access denied for user
'heat'@'rdo-undercloud' (using password: YES)")

Typically the undercloud's local_ip is used for these operations but
in your case it's added to the hosts files and maybe this ip name
mapping doesn't allow the installation to proceed. Please note that
the docs[1] point that you should have an FQDN entry in the hosts file
before the undercloud installation(when 192.0.2.1 isn't yet set on the
system), that's why I mentioned the ip address of the public nic.

[1] http://docs.openstack.org/developer/tripleo-docs/installation/installation.html

>
> Thanks again for your time and help. Really appreciate it.
>
> Best Regards,
> milind
>
> -----Original Message-----
> From: Marius Cornea [mailto:marius at remote-lab.net]
> Sent: Tuesday, June 28, 2016 4:28 AM
> To: Gunjan, Milind [CTO] <Milind.Gunjan at sprint.com>
> Cc: rdo-list at redhat.com
> Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o deployment
>
> On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO]
> <Milind.Gunjan at sprint.com> wrote:
>> Hi Dan,
>> Thanks a lot for your response.
>>
>> Even after properly updating the undercloud.conf file and checking the network configuration, undercloud deployment still fails.
>>
>> To recreate the issue , I am mentioning all the configuration steps:
>> 1. Installed   CentOS Linux release 7.2.1511 (Core) image on baremetal.
>> 2. created stack user and provided required permission to stack user .
>> 3. setting hostname
>> sudo hostnamectl set-hostname rdo-undercloud.mydomain
>> sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain
>>
>> [stack at rdo-undercloud etc]$ cat /etc/hosts
>> 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
>> ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
>> 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain
>
> Could you try removing the 192.0.2.1 entry from /etc/hosts and replace
> it with the address of the public interface, e.g:
> $ip_public_nic    rdo-undercloud rdo-undercloud.mydomain
>
> then rerun openstack undercloud install
>
>> 4. enable required repositories
>> sudo yum -y install epel-release
>> sudo curl -o /etc/yum.repos.d/delorean-liberty.repo https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo
>> sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo
>>
>> 5. install repos
>>
>> sudo yum -y install yum-plugin-priorities
>> sudo yum install -y python-tripleoclient
>>
>> 6. update undercloud.conf
>>
>> [stack at rdo-undercloud ~]$ cat undercloud.conf
>> [DEFAULT]
>> local_ip = 192.0.2.1/24
>> undercloud_public_vip = 192.0.2.2
>> undercloud_admin_vip = 192.0.2.3
>> local_interface = enp6s0
>> masquerade_network = 192.0.2.0/24
>> dhcp_start = 192.0.2.150
>> dhcp_end = 192.0.2.199
>> network_cidr = 192.0.2.0/24
>> network_gateway = 192.0.2.1
>> discovery_iprange = 192.0.2.200,192.0.2.230
>> discovery_runbench = false
>> [auth]
>>
>> 7. install undercloud
>>
>> openstack undercloud install
>>
>> install ends in error:
>> Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: Could not prefetch keystone_service provider 'openstack': Execution of '/bin/openstack service list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504)
>> Error: Not managing Keystone_service[glance] due to earlier Keystone API failures.
>> Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure: change from absent to present failed: Not managing Keystone_service[glance] due to earlier Keystone API failures.
>> Error: Could not prefetch keystone_role provider 'openstack': Execution of '/bin/openstack role list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504)
>> Error: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures.
>> Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure: change from absent to present failed: Not managing Keystone_role[ResellerAdmin] due to earlier Keystone API failures.
>> Error: Not managing Keystone_service[ironic] due to earlier Keystone API failures.
>> Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure: change from absent to present failed: Not managing Keystone_service[ironic] due to earlier Keystone API failures.
>> Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: Not managing Keystone_service[novav3] due to earlier Keystone API failures.
>> Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change from absent to present failed: Not managing Keystone_service[novav3] due to earlier Keystone API failures.
>> Error: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures.
>> Error: /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure: change from absent to present failed: Not managing Keystone_role[heat_stack_user] due to earlier Keystone API failures.
>> Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: Not managing Keystone_service[nova] due to earlier Keystone API failures.
>> Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_service[nova::compute]/ensure: change from absent to present failed: Not managing Keystone_service[nova] due to earlier Keystone API failures.
>> Error: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures.
>> Error: /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure: change from absent to present failed: Not managing Keystone_role[swiftoperator] due to earlier Keystone API failures.
>> Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: Not managing Keystone_service[neutron] due to earlier Keystone API failures.
>> Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure: change from absent to present failed: Not managing Keystone_service[neutron] due to earlier Keystone API failures.
>> Error: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures.
>> Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure: change from absent to present failed: Not managing Keystone_service[ceilometer] due to earlier Keystone API failures.
>> Error: Not managing Keystone_service[swift] due to earlier Keystone API failures.
>> Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure: change from absent to present failed: Not managing Keystone_service[swift] due to earlier Keystone API failures.
>> Error: Not managing Keystone_service[keystone] due to earlier Keystone API failures.
>> Error: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure: change from absent to present failed: Not managing Keystone_service[keystone] due to earlier Keystone API failures.
>> Error: Not managing Keystone_service[heat] due to earlier Keystone API failures.
>> Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure: change from absent to present failed: Not managing Keystone_service[heat] due to earlier Keystone API failures.
>> Error: Could not prefetch keystone_endpoint provider 'openstack': Execution of '/bin/openstack endpoint list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504)
>> Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Gateway Timeout (HTTP 504)
>> Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures.
>> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures.
>> Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
>> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
>> Error: Not managing Keystone_role[admin] due to earlier Keystone API failures.
>> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures.
>> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/bin/openstack domain show --format shell Default' returned 1: Could not find resource Default
>> Error: Could not prefetch keystone_domain provider 'openstack': Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Gateway Timeout (HTTP 504)
>> Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR: (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud' (using password: YES)")
>> Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
>> Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
>> [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
>>
>> Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered 'refresh' from 4 events
>> Notice: Finished catalog run in 5259.44 seconds
>> + rc=6
>> + set -e
>> + echo 'puppet apply exited with exit code 6'
>> puppet apply exited with exit code 6
>> + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
>> + exit 6
>> [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
>>
>> [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting...
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install
>>     _run_orc(instack_env)
>>   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc
>>     _run_live_command(args, instack_env, 'os-refresh-config')
>>   File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command
>>     raise RuntimeError('%s failed. See log for details.' % name)
>> RuntimeError: os-refresh-config failed. See log for details.
>> Command 'instack-install-undercloud' returned non-zero exit status 1
>>
>>
>> I am not able to understand the exact cause of undercloud install failure. It would be really helpful if you guys can point be in direction to understand the exact cause of issue and any possible resolution.
>>
>> Thanks a lot.
>>
>> Best Regards,
>> Milind
>>
>>
>> Best Regards,
>> Milind
>> -----Original Message-----
>> From: Dan Sneddon [mailto:dsneddon at redhat.com]
>> Sent: Monday, June 27, 2016 12:40 PM
>> To: Gunjan, Milind [CTO] <Milind.Gunjan at sprint.com>; rdo-list at redhat.com
>> Subject: Re: [rdo-list] Redeploying UnderCloud
>>
>> On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote:
>>> Hi All,
>>>
>>> Greeting.
>>>
>>>
>>>
>>> This is my first post and I am fairly new to RDO OpenStack. I am
>>> working on RDO Triple-O deployment on baremetal. Due to incorrect
>>> values in undercloud.conf file , my undercloud deployment failed. I
>>> would like to redeploy undercloud and I am trying to understand what
>>> steps has to be taken before redeploying undercloud. All the
>>> deployment was under stack user . So first step will be to delete
>>> stack user. I am not sure what has to be done regarding the networking
>>> configuration autogenerated by os-net-config during the older install.
>>>
>>> Please advise.
>>>
>>>
>>>
>>> Best Regards,
>>>
>>> Milind
>>
>> No, definitely you don't want to delete the stack user, especially not as your first step! That would get rid of the configuration files, etc.
>> that are in ~stack, and generally make your life harder than it needs to be.
>>
>> Anyway, it isn't necessary. You can do a procedure very much like what you do when upgrading the undercloud, with a couple of extra steps.
>>
>> As the stack user, edit your undercloud.conf, and make sure there are no more typos.
>>
>> If the typos were in the network configuration, you should delete the bridge and remove the ifcfg files:
>>
>> $ sudo ifdown br-ctlplane
>> $ sudo ovs-vsctl del-br br-ctlplane
>> $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane
>>
>> Next, run the underclound installation again:
>>
>> $ sudo yum update -y  # Reboot after if kernel or core packages updated $ openstack undercloud install
>>
>> Then proceed with the rest of the instructions. You may find that if you already uploaded disk images or imported nodes that they will still be in the database. That's OK, or you can delete and reimport.
>>
>> --
>> Dan Sneddon         |  Principal OpenStack Engineer
>> dsneddon at redhat.com |  redhat.com/openstack
>> 650.254.4025        |  dsneddon:irc   @dxs:twitter
>>
>>
>>
>> ________________________________
>> Learn more on how to switch to Sprint and save 50% on most Verizon, AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off> for details.
>>
>> ________________________________
>>
>> This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
>>
>> _______________________________________________
>> rdo-list mailing list
>> rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com