[rdo-users] RDO install OpenStack queens all-in-one get stuck

Javier Pena jpena at redhat.com
Fri Aug 17 10:35:52 UTC 2018 

(oops, I forgot to add the list to my initial reply) 

----- Original Message -----

> Thanks for reply, Javier.

> After adding --debug to packstack, more information are found in
> controller.pp.running.

> Debug: Executing: '/usr/bin/systemctl is-active rabbitmq-server'

> Debug: Executing: '/usr/bin/systemctl is-enabled rabbitmq-server'

> Debug: Prefetching parsed resources for sysctl

> Debug: Prefetching sysctl_runtime resources for sysctl_runtime

> Debug: Executing: '/usr/sbin/sysctl -a'

> From the log, it hangs at ' /usr/sbin/sysctl -a ', by searching online, this
> link( https://access.redhat.com/solutions/2427351 need rhel account login. )
> give some hints:

> "

> · We are finding issues related to /proc/sys/fs/binfmt_misc in RHEV servers.

> · Whenever we run df or sysctl -a , they hang without displaying any output.

> · ls -l /proc/sys/fs/binfmt_misc also hangs.

> "

> The above issues can be checked manually.

> Also we can get some infor via:

> systemctl -a | grep binfmt_misc

> By continue searching the solution, this link(
> https://bugzilla.redhat.com/show_bug.cgi?id=1591418 ) give a workaround.
> This works without reboot the host(I don’t want the reboot).

> systemctl start proc-sys-fs-binfmt_misc.mount # effect is equal to reboot my
> host.

> As to why it happens, I suspect it's lead by 'yum update' during which new
> kernel version are installed, however, I didn’t reboot the host.

> kernel.x86_64 3.10.0-514.el7 @anaconda

> kernel.x86_64 3.10.0-862.11.6.el7 @updates

> Before reboot: 3.10.0-514.el7.x86_64 -=> After reboot:
> 3.10.0-862.11.6.el7.x86_64

> So in summary, I know now what the problem is. But I wonder why it
> happens(during executing puppet controller.pp,) what does rdo puppet do to
> the system that cause this stuck/hang.

The packstack puppet manifests do some kernel parameter manipulation [1] using the sysctl module. That is running "sysctl -a" in the background, triggering the issue you found, it seems. 

Regards, 
Javier 

[1] - https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/amqp.pp#L66-L76 

> Regards,

> Andrew.

> -----------------------------------

> Andrew Zong

> Sr Software Engineer, F5 Networks

> From: Javier Pena <jpena at redhat.com>
> Date: Friday, August 17, 2018 at 4:51 PM
> To: Andrew ZONG <A.ZONG at f5.com>
> Subject: Re: [rdo-users] RDO install OpenStack queens all-in-one get stuck

> EXTERNAL MAIL: jpena at redhat.com

> Hi Andrew,

> It looks like the puppet manifest execution is getting stuck, but the log
> files are not giving us enough information.

> Could you try running packstack like:

> # packstack --debug --answer-file <answer file>

> This will enable additional puppet debug output at /var/tmp/packstack/<uuid>/
> manifests/10.145.115.45_controller.pp .log

> Regards,

> Javier

> > Hi,
> 

> > When I use packstack to install openstack queens allinone on CentOS 7
> > x86_64,
> > it got stuck at:
> 

> > # sudo packstack --answer-file pack-1534411186.txt
> 

> > Welcome to the Packstack setup utility
> 

> > The installation log file is available at:
> > /var/tmp/packstack/20180816-022431-Qun9eG/openstack-setup.log
> 

> > Installing:
> 

> > Clean Up [ DONE ]
> 

> > Discovering ip protocol version [ DONE ]
> 

> > Setting up ssh keys [ DONE ]
> 

> > ...
> 

> > Testing if puppet apply is finished: 10.145.xxx.yyy_controller.pp [ | ]
> 

> > I find the log mention above, the last line seems no help:
> 

> > tar --dereference -cpzf - aodh apache ceilometer certmonger cinder concat
> > firewall glance gnocchi heat horizon inifile ironic keystone magnum manila
> > memcached mysql neutron nova nssdb openstack openstacklib oslo ovn
> > packstack
> > panko rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest
> > trove vcsrepo vswitch xinetd | ssh -o StrictHostKeyChecking=no -o
> > UserKnownHostsFile=/dev/null root at 10.145.115.45 tar -C
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules -xpzf -
> 

> > Also I go to the puppet running file and find some warning there:
> 

> > # ps -ef | grep puppet
> 

> > root 1870 1869 0 02:25 ? 00:00:00 flock
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/ps.lock puppet apply
> > --modulepath /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp
> 

> > ...
> 

> > Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it
> > will only see those that are resolved. Use 'puppet module list --tree' to
> > see information about modules
> 

> > (file & line not available)
> 

> > Warning: This method is deprecated, please use the stdlib validate_legacy
> > function,
> 

> > with Stdlib::Compat::String. There is further documentation for
> > validate_legacy function in the README. at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/db/mysql.pp",
> > 63]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 21]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it
> > will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'glance' has unresolved dependencies - it
> > will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it
> > will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'aodh' has unresolved dependencies - it will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it
> > will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/db/mysql.pp:53:28
> 

> > Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies -
> > it
> > will only see those that are resolved. Use 'puppet module list --tree' to
> > see information about modules
> 

> > (file & line not available)
> 

> > Warning: This method is deprecated, please use the stdlib validate_legacy
> > function,
> 

> > with Stdlib::Compat::Hash. There is further documentation for
> > validate_legacy
> > function in the README. at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/policy.pp",
> > 34]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/init.pp",
> > 787]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: This method is deprecated, please use the stdlib validate_legacy
> > function,
> 

> > with Pattern[]. There is further documentation for validate_legacy function
> > in the README. at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/init.pp",
> > 806]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 27]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: Scope(Class[Cinder::Keystone::Authtoken]): revocation_cache_time
> > parameter is deprecated, has no effect and will be removed in the future.
> 

> > Warning: This method is deprecated, please use the stdlib validate_legacy
> > function,
> 

> > with Stdlib::Compat::Bool. There is further documentation for
> > validate_legacy
> > function in the README. at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/cinder/manifests/api.pp",
> > 220]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 45]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: Scope(Class[Nova::Db]): placement_database_connection has no
> > effect
> > as of pike, and may be removed in a future release
> 

> > Warning: Scope(Class[Nova::Db]): placement_slave_connection has no effect
> > as
> > of pike, and may be removed in a future release
> 

> > Warning: This method is deprecated, please use the stdlib validate_legacy
> > function,
> 

> > with Stdlib::Compat::Array. There is further documentation for
> > validate_legacy function in the README. at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/init.pp",
> > 564]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 74]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: Scope(Class[Nova::Api]): The default_floating_pool parameter is
> > deprecated. Please use nova::network::neutron::default_floating_pool
> > instead.
> 

> > Warning: Scope(Class[Nova::Api]): The pci_alias parameter is deprecated.
> > Please use nova::pci::aliases instead.
> 

> > Warning: Unknown variable: 'until_complete_real'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/cron/archive_deleted_rows.pp:77:82
> 

> > Warning: This method is deprecated, please use match expressions with
> > Stdlib::Compat::Array instead. They are described at
> > https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.
> > at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/scheduler/filter.pp",
> > 140]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 80]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: This method is deprecated, please use match expressions with
> > Stdlib::Compat::Ip_address instead. They are described at
> > https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.
> > at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/compute/neutron.pp",
> > 31]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 85]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: Scope(Class[Neutron::Plugins::Ml2]): supported_pci_vendor_devs is
> > deprecated, has no effect and will be removed in a future release.
> 

> > Warning: ModuleLoader: module 'horizon' has unresolved dependencies - it
> > will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: This method is deprecated, please use the stdlib validate_legacy
> > function,
> 

> > with Stdlib::Compat::Absolute_Path. There is further documentation for
> > validate_legacy function in the README. at
> > ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/horizon/manifests/init.pp",
> > 610]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> > 123]
> 

> > (at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> > `deprecation')
> 

> > Warning: ModuleLoader: module 'swift' has unresolved dependencies - it will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: Unknown variable: 'methods_real'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:100:56
> 

> > Warning: Unknown variable: 'incoming_remove_headers_real'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:101:56
> 

> > Warning: Unknown variable: 'incoming_allow_headers_real'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:102:56
> 

> > Warning: Unknown variable: 'outgoing_remove_headers_real'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:103:56
> 

> > Warning: Unknown variable: 'outgoing_allow_headers_real'. at
> > /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:104:56
> 

> > Warning: Scope(Class[Swift::Storage::All]): The default port for the object
> > storage server has changed from 6000 to 6200 and will be changed in a later
> > release
> 

> > Warning: Scope(Class[Swift::Storage::All]): The default port for the
> > container storage server has changed from 6001 to 6201 and will be changed
> > in a later release
> 

> > Warning: Scope(Class[Swift::Storage::All]): The default port for the
> > account
> > storage server has changed from 6002 to 6202 and will be changed in a later
> > release
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::logging_context_format_string is deprecated and will be
> > removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::logging_default_format_string is deprecated and will be
> > removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::logging_debug_format_suffix is deprecated and will be
> > removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::logging_exception_prefix is deprecated and will be
> > removed
> > in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::log_config_append
> > is deprecated and will be removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::default_log_levels
> > is deprecated and will be removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::publish_errors
> > is
> > deprecated and will be removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::fatal_deprecations
> > is deprecated and will be removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::instance_format
> > is
> > deprecated and will be removed in future
> 

> > Warning: Scope(Class[Gnocchi::Logging]):
> > gnocchi::logging::instance_uuid_format is deprecated and will be removed in
> > future
> 

> > Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::log_date_format
> > is
> > deprecated and will be removed in future
> 

> > Warning: Scope(Class[Gnocchi::Api]): host is deprecated
> 

> > Warning: Scope(Class[Gnocchi::Statsd]): user_id parameter is deprecated and
> > will be removed in the future release.
> 

> > Warning: Scope(Class[Gnocchi::Statsd]): project_id parameter is deprecated
> > and will be removed in the future release.
> 

> > Warning: ModuleLoader: module 'ceilometer' has unresolved dependencies - it
> > will only see those that are resolved. Use 'puppet module list --tree' to
> > see information about modules
> 

> > (file & line not available)
> 

> > Warning: Scope(Class[Ceilometer::Agent::Central]): This class is
> > deprecated.
> > Please use ceilometer::agent::central with central namespace instead.
> 

> > Warning: ModuleLoader: module 'redis' has unresolved dependencies - it will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'rabbitmq' has unresolved dependencies - it
> > will only see those that are resolved. Use 'puppet module list --tree' to
> > see information about modules
> 

> > (file & line not available)
> 

> > Warning: ModuleLoader: module 'oslo' has unresolved dependencies - it will
> > only see those that are resolved. Use 'puppet module list --tree' to see
> > information about modules
> 

> > (file & line not available)
> 

> > What's the problem? My basic setup steps and environment are as following:
> 

> > CentOS-7-x86_64-Minimal-1611: 8 cores and 16GB mem.
> 

> > sudo yum update -y
> 

> > sudo yum install -y centos-release-openstack-queens
> 

> > yum update -y && yum install -y openstack-packstack
> 

> > cat << EOF > /etc/environment
> 

> > LANG=en_US.utf-8
> 

> > LC_ALL=en_US.utf-8
> 

> > EOF
> 

> > packstack --gen-answer-file pack-`date +%s`.txt
> 

> > <configure as following>
> 

> > CONFIG_KEYSTONE_ADMIN_PW=default
> 

> > CONFIG_CONTROLLER_HOST=10.145.xxx.yyy
> 

> > CONFIG_COMPUTE_HOSTS=10.145.xxx.yyy
> 

> > CONFIG_NETWORK_HOSTS=10.145.xxx.yyy
> 

> > CONFIG_LBAAS_INSTALL=y
> 

> > sudo systemctl disable firewalld
> 

> > sudo systemctl stop firewalld
> 

> > sudo systemctl disable NetworkManager
> 

> > sudo systemctl stop NetworkManager
> 

> > sudo systemctl enable network
> 

> > sudo systemctl start network
> 

> > uname -m | grep -q 'x86_64' && echo 'centos' >/etc/yum/vars/contentdir ||
> > echo 'altarch' >/etc/yum/vars/contentdir
> 

> > packstack --answer-file <answer file>
> 

> > Regards,
> 

> > Andrew.
> 

> > -----------------------------------
> 

> > Andrew Zong
> 

> > Sr Software Engineer, F5 Networks
> 

> > _______________________________________________
> 
> > users mailing list
> 
> > users at lists.rdoproject.org
> 
> > http://lists.rdoproject.org/mailman/listinfo/users
> 

> > To unsubscribe: users-unsubscribe at lists.rdoproject.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/users/attachments/20180817/88d01097/attachment-0001.html>

More information about the users mailing list