[rdo-list] Redeploying UnderCloud for baremetal triple-o deployment

Wed Jun 29 07:25:13 UTC 2016

The hostname in the hosts file don't match.
On Jun 29, 2016 8:58 AM, "Marius Cornea" <marius at remote-lab.net> wrote:

> On Tue, Jun 28, 2016 at 10:45 PM, Gunjan, Milind [CTO]
> <Milind.Gunjan at sprint.com> wrote:
> > Hi All,
> >
> > Thanks a lot for continued support.
> >
> > I would just like to get clarity regarding the last recommendation.
> > My current deployment is failing with following error :
> >
> > Error: Could not prefetch keystone_service provider 'openstack':
> Execution of '/bin/openstack service list --quiet --format csv --long'
> returned 1: Gateway Timeout (HTTP 504)
> > Error: Could not prefetch keystone_role provider 'openstack': Execution
> of '/bin/openstack role list --quiet --format csv' returned 1: Gateway
> Timeout (HTTP 504)
> > Error: Could not prefetch keystone_endpoint provider 'openstack':
> Execution of '/bin/openstack endpoint list --quiet --format csv' returned
> 1: Gateway Timeout (HTTP 504)
> >
> >
> > I have previously done RHEL OSP7 deployment and I verified the host file
> of RHEL OSP7  undercloud deployment and   it was configured with host
> gateway used by pxe network.
> >
> > Similarly, we have set the current host gateway to 192.0.2.1 as shown
> below  for rdo manager undercloud installation:
> > [stack at rdo-undercloud etc]$ cat /etc/hosts
> >> 127.0.0.1   localhost localhost.localdomain localhost4
> localhost4.localdomain4
> >> ::1         localhost localhost.localdomain localhost6
> localhost6.localdomain6
> >> 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain
> >
> > I would like to know if it is required during rdo deployment to have it
> with the address of the public interface, e.g:
> > $ip_public_nic    rdo-undercloud rdo-undercloud.mydomain.
>
> The reason I asked for it is because of:
>
> Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR:
> (pymysql.err.OperationalError) (1045, u"Access denied for user
> 'heat'@'rdo-undercloud' (using password: YES)")
>
> Typically the undercloud's local_ip is used for these operations but
> in your case it's added to the hosts files and maybe this ip name
> mapping doesn't allow the installation to proceed. Please note that
> the docs[1] point that you should have an FQDN entry in the hosts file
> before the undercloud installation(when 192.0.2.1 isn't yet set on the
> system), that's why I mentioned the ip address of the public nic.
>
> [1]
> http://docs.openstack.org/developer/tripleo-docs/installation/installation.html
>
> >
> > Thanks again for your time and help. Really appreciate it.
> >
> > Best Regards,
> > milind
> >
> > -----Original Message-----
> > From: Marius Cornea [mailto:marius at remote-lab.net]
> > Sent: Tuesday, June 28, 2016 4:28 AM
> > To: Gunjan, Milind [CTO] <Milind.Gunjan at sprint.com>
> > Cc: rdo-list at redhat.com
> > Subject: Re: [rdo-list] Redeploying UnderCloud for baremetal triple-o
> deployment
> >
> > On Tue, Jun 28, 2016 at 5:18 AM, Gunjan, Milind [CTO]
> > <Milind.Gunjan at sprint.com> wrote:
> >> Hi Dan,
> >> Thanks a lot for your response.
> >>
> >> Even after properly updating the undercloud.conf file and checking the
> network configuration, undercloud deployment still fails.
> >>
> >> To recreate the issue , I am mentioning all the configuration steps:
> >> 1. Installed   CentOS Linux release 7.2.1511 (Core) image on baremetal.
> >> 2. created stack user and provided required permission to stack user .
> >> 3. setting hostname
> >> sudo hostnamectl set-hostname rdo-undercloud.mydomain
> >> sudo hostnamectl set-hostname --transient rdo-undercloud.mydomain
> >>
> >> [stack at rdo-undercloud etc]$ cat /etc/hosts
> >> 127.0.0.1   localhost localhost.localdomain localhost4
> localhost4.localdomain4
> >> ::1         localhost localhost.localdomain localhost6
> localhost6.localdomain6
> >> 192.0.2.1 rdo-undercloud undercloud-rdo.mydomain
> >
> > Could you try removing the 192.0.2.1 entry from /etc/hosts and replace
> > it with the address of the public interface, e.g:
> > $ip_public_nic    rdo-undercloud rdo-undercloud.mydomain
> >
> > then rerun openstack undercloud install
> >
> >> 4. enable required repositories
> >> sudo yum -y install epel-release
> >> sudo curl -o /etc/yum.repos.d/delorean-liberty.repo
> https://trunk.rdoproject.org/centos7-liberty/current/delorean.repo
> >> sudo curl -o /etc/yum.repos.d/delorean-deps-liberty.repo
> http://trunk.rdoproject.org/centos7-liberty/delorean-deps.repo
> >>
> >> 5. install repos
> >>
> >> sudo yum -y install yum-plugin-priorities
> >> sudo yum install -y python-tripleoclient
> >>
> >> 6. update undercloud.conf
> >>
> >> [stack at rdo-undercloud ~]$ cat undercloud.conf
> >> [DEFAULT]
> >> local_ip = 192.0.2.1/24
> >> undercloud_public_vip = 192.0.2.2
> >> undercloud_admin_vip = 192.0.2.3
> >> local_interface = enp6s0
> >> masquerade_network = 192.0.2.0/24
> >> dhcp_start = 192.0.2.150
> >> dhcp_end = 192.0.2.199
> >> network_cidr = 192.0.2.0/24
> >> network_gateway = 192.0.2.1
> >> discovery_iprange = 192.0.2.200,192.0.2.230
> >> discovery_runbench = false
> >> [auth]
> >>
> >> 7. install undercloud
> >>
> >> openstack undercloud install
> >>
> >> install ends in error:
> >> Error:
> /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]:
> Could not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error:
> /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]:
> Could not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error: Could not prefetch keystone_service provider 'openstack':
> Execution of '/bin/openstack service list --quiet --format csv --long'
> returned 1: Gateway Timeout (HTTP 504)
> >> Error: Not managing Keystone_service[glance] due to earlier Keystone
> API failures.
> >> Error:
> /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_service[glance::image]/ensure:
> change from absent to present failed: Not managing Keystone_service[glance]
> due to earlier Keystone API failures.
> >> Error: Could not prefetch keystone_role provider 'openstack': Execution
> of '/bin/openstack role list --quiet --format csv' returned 1: Gateway
> Timeout (HTTP 504)
> >> Error: Not managing Keystone_role[ResellerAdmin] due to earlier
> Keystone API failures.
> >> Error:
> /Stage[main]/Ceilometer::Keystone::Auth/Keystone_role[ResellerAdmin]/ensure:
> change from absent to present failed: Not managing
> Keystone_role[ResellerAdmin] due to earlier Keystone API failures.
> >> Error: Not managing Keystone_service[ironic] due to earlier Keystone
> API failures.
> >> Error:
> /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_service[ironic::baremetal]/ensure:
> change from absent to present failed: Not managing Keystone_service[ironic]
> due to earlier Keystone API failures.
> >> Error:
> /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova
> service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of
> '/bin/openstack domain show --format shell Default' returned 1: Could not
> find resource Default
> >> Error:
> /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]:
> Could not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error: Not managing Keystone_service[novav3] due to earlier Keystone
> API failures.
> >> Error:
> /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova
> v3 service, user novav3]/Keystone_service[novav3::computev3]/ensure: change
> from absent to present failed: Not managing Keystone_service[novav3] due to
> earlier Keystone API failures.
> >> Error: Not managing Keystone_role[heat_stack_user] due to earlier
> Keystone API failures.
> >> Error:
> /Stage[main]/Heat::Keystone::Auth/Keystone_role[heat_stack_user]/ensure:
> change from absent to present failed: Not managing
> Keystone_role[heat_stack_user] due to earlier Keystone API failures.
> >> Error:
> /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]:
> Could not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error: Not managing Keystone_service[nova] due to earlier Keystone API
> failures.
> >> Error:
> /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova
> service, user nova]/Keystone_service[nova::compute]/ensure: change from
> absent to present failed: Not managing Keystone_service[nova] due to
> earlier Keystone API failures.
> >> Error: Not managing Keystone_role[swiftoperator] due to earlier
> Keystone API failures.
> >> Error:
> /Stage[main]/Swift::Keystone::Auth/Keystone_role[swiftoperator]/ensure:
> change from absent to present failed: Not managing
> Keystone_role[swiftoperator] due to earlier Keystone API failures.
> >> Error:
> /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]:
> Could not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error: Not managing Keystone_service[neutron] due to earlier Keystone
> API failures.
> >> Error:
> /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_service[neutron::network]/ensure:
> change from absent to present failed: Not managing
> Keystone_service[neutron] due to earlier Keystone API failures.
> >> Error: Not managing Keystone_service[ceilometer] due to earlier
> Keystone API failures.
> >> Error:
> /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_service[ceilometer::metering]/ensure:
> change from absent to present failed: Not managing
> Keystone_service[ceilometer] due to earlier Keystone API failures.
> >> Error: Not managing Keystone_service[swift] due to earlier Keystone API
> failures.
> >> Error:
> /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_service[swift::object-store]/ensure:
> change from absent to present failed: Not managing Keystone_service[swift]
> due to earlier Keystone API failures.
> >> Error: Not managing Keystone_service[keystone] due to earlier Keystone
> API failures.
> >> Error:
> /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]/ensure:
> change from absent to present failed: Not managing
> Keystone_service[keystone] due to earlier Keystone API failures.
> >> Error: Not managing Keystone_service[heat] due to earlier Keystone API
> failures.
> >> Error:
> /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_service[heat::orchestration]/ensure:
> change from absent to present failed: Not managing Keystone_service[heat]
> due to earlier Keystone API failures.
> >> Error: Could not prefetch keystone_endpoint provider 'openstack':
> Execution of '/bin/openstack endpoint list --quiet --format csv' returned
> 1: Gateway Timeout (HTTP 504)
> >> Error:
> /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]:
> Could not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error: Could not prefetch keystone_tenant provider 'openstack':
> Execution of '/bin/openstack project list --quiet --format csv --long'
> returned 1: Gateway Timeout (HTTP 504)
> >> Error: Not managing Keystone_tenant[service] due to earlier Keystone
> API failures.
> >> Error:
> /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change
> from absent to present failed: Not managing Keystone_tenant[service] due to
> earlier Keystone API failures.
> >> Error: Not managing Keystone_tenant[admin] due to earlier Keystone API
> failures.
> >> Error:
> /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change
> from absent to present failed: Not managing Keystone_tenant[admin] due to
> earlier Keystone API failures.
> >> Error: Not managing Keystone_role[admin] due to earlier Keystone API
> failures.
> >> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure:
> change from absent to present failed: Not managing Keystone_role[admin] due
> to earlier Keystone API failures.
> >> Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could
> not evaluate: Execution of '/bin/openstack domain show --format shell
> Default' returned 1: Could not find resource Default
> >> Error: Could not prefetch keystone_domain provider 'openstack':
> Execution of '/bin/openstack domain list --quiet --format csv' returned 1:
> Gateway Timeout (HTTP 504)
> >> Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]/returns: ERROR:
> (pymysql.err.OperationalError) (1045, u"Access denied for user 'heat'@'rdo-undercloud'
> (using password: YES)")
> >> Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Failed to call
> refresh: heat-manage --config-file /etc/heat/heat.conf db_sync returned 1
> instead of one of [0]
> >> Error: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: heat-manage
> --config-file /etc/heat/heat.conf db_sync returned 1 instead of one of [0]
> >> [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure
> phase. [Command '['dib-run-parts',
> '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit
> status 1]
> >>
> >> Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Triggered
> 'refresh' from 4 events
> >> Notice: Finished catalog run in 5259.44 seconds
> >> + rc=6
> >> + set -e
> >> + echo 'puppet apply exited with exit code 6'
> >> puppet apply exited with exit code 6
> >> + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
> >> + exit 6
> >> [2016-06-27 18:54:04,092] (os-refresh-config) [ERROR] during configure
> phase. [Command '['dib-run-parts',
> '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit
> status 1]
> >>
> >> [2016-06-27 18:54:04,093] (os-refresh-config) [ERROR] Aborting...
> >> Traceback (most recent call last):
> >>   File "<string>", line 1, in <module>
> >>   File
> "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line
> 815, in install
> >>     _run_orc(instack_env)
> >>   File
> "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line
> 699, in _run_orc
> >>     _run_live_command(args, instack_env, 'os-refresh-config')
> >>   File
> "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line
> 370, in _run_live_command
> >>     raise RuntimeError('%s failed. See log for details.' % name)
> >> RuntimeError: os-refresh-config failed. See log for details.
> >> Command 'instack-install-undercloud' returned non-zero exit status 1
> >>
> >>
> >> I am not able to understand the exact cause of undercloud install
> failure. It would be really helpful if you guys can point be in direction
> to understand the exact cause of issue and any possible resolution.
> >>
> >> Thanks a lot.
> >>
> >> Best Regards,
> >> Milind
> >>
> >>
> >> Best Regards,
> >> Milind
> >> -----Original Message-----
> >> From: Dan Sneddon [mailto:dsneddon at redhat.com]
> >> Sent: Monday, June 27, 2016 12:40 PM
> >> To: Gunjan, Milind [CTO] <Milind.Gunjan at sprint.com>;
> rdo-list at redhat.com
> >> Subject: Re: [rdo-list] Redeploying UnderCloud
> >>
> >> On 06/27/2016 06:41 AM, Gunjan, Milind [CTO] wrote:
> >>> Hi All,
> >>>
> >>> Greeting.
> >>>
> >>>
> >>>
> >>> This is my first post and I am fairly new to RDO OpenStack. I am
> >>> working on RDO Triple-O deployment on baremetal. Due to incorrect
> >>> values in undercloud.conf file , my undercloud deployment failed. I
> >>> would like to redeploy undercloud and I am trying to understand what
> >>> steps has to be taken before redeploying undercloud. All the
> >>> deployment was under stack user . So first step will be to delete
> >>> stack user. I am not sure what has to be done regarding the networking
> >>> configuration autogenerated by os-net-config during the older install.
> >>>
> >>> Please advise.
> >>>
> >>>
> >>>
> >>> Best Regards,
> >>>
> >>> Milind
> >>
> >> No, definitely you don't want to delete the stack user, especially not
> as your first step! That would get rid of the configuration files, etc.
> >> that are in ~stack, and generally make your life harder than it needs
> to be.
> >>
> >> Anyway, it isn't necessary. You can do a procedure very much like what
> you do when upgrading the undercloud, with a couple of extra steps.
> >>
> >> As the stack user, edit your undercloud.conf, and make sure there are
> no more typos.
> >>
> >> If the typos were in the network configuration, you should delete the
> bridge and remove the ifcfg files:
> >>
> >> $ sudo ifdown br-ctlplane
> >> $ sudo ovs-vsctl del-br br-ctlplane
> >> $ sudo rm /etc/sysconfig/network-scripts/*br-ctlplane
> >>
> >> Next, run the underclound installation again:
> >>
> >> $ sudo yum update -y  # Reboot after if kernel or core packages updated
> $ openstack undercloud install
> >>
> >> Then proceed with the rest of the instructions. You may find that if
> you already uploaded disk images or imported nodes that they will still be
> in the database. That's OK, or you can delete and reimport.
> >>
> >> --
> >> Dan Sneddon         |  Principal OpenStack Engineer
> >> dsneddon at redhat.com |  redhat.com/openstack
> >> 650.254.4025        |  dsneddon:irc   @dxs:twitter
> >>
> >>
> >>
> >> ________________________________
> >> Learn more on how to switch to Sprint and save 50% on most Verizon,
> AT&T or T-Mobile rates. See sprint.com/50off<http://sprint.com/50off> for
> details.
> >>
> >> ________________________________
> >>
> >> This e-mail may contain Sprint proprietary information intended for the
> sole use of the recipient(s). Any use by others is prohibited. If you are
> not the intended recipient, please contact the sender and delete all copies
> of the message.
> >>
> >> _______________________________________________
> >> rdo-list mailing list
> >> rdo-list at redhat.com
> >> https://www.redhat.com/mailman/listinfo/rdo-list
> >>
> >> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
> _______________________________________________
> rdo-list mailing list
> rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20160629/21943eeb/attachment.html>