(oops, I forgot to add the list to my initial reply)

Thanks for reply, Javier.

After adding --debug to packstack, more information are found in controller.pp.running.

Debug: Executing: '/usr/bin/systemctl is-active rabbitmq-server'
Debug: Executing: '/usr/bin/systemctl is-enabled rabbitmq-server'
Debug: Prefetching parsed resources for sysctl
Debug: Prefetching sysctl_runtime resources for sysctl_runtime
Debug: Executing: '/usr/sbin/sysctl -a'

From the log, it hangs at '/usr/sbin/sysctl -a', by searching online, this link(https://access.redhat.com/solutions/2427351 need rhel account login.) give some hints:
"
·         We are finding issues related to /proc/sys/fs/binfmt_misc in RHEV servers.
·         Whenever we run df or sysctl -a, they hang without displaying any output.
·         ls -l /proc/sys/fs/binfmt_misc also hangs.

"

The above issues can be checked manually.
Also we can get some infor via:

systemctl -a | grep binfmt_misc

By continue searching the solution, this link(https://bugzilla.redhat.com/show_bug.cgi?id=1591418) give a workaround. This works without reboot the host(I don’t want the reboot).

systemctl start proc-sys-fs-binfmt_misc.mount   # effect is equal to reboot my host.

As to why it happens, I suspect it's lead by 'yum update' during which new kernel version are installed, however, I didn’t reboot the host.

kernel.x86_64                     3.10.0-514.el7         @anaconda
kernel.x86_64                     3.10.0-862.11.6.el7    @updates

Before reboot: 3.10.0-514.el7.x86_64   -=> After reboot: 3.10.0-862.11.6.el7.x86_64

So in summary, I know now what the problem is. But I wonder why it happens(during executing puppet controller.pp,) what does rdo puppet do to the system that cause this stuck/hang.

The packstack puppet manifests do some kernel parameter manipulation [1] using the sysctl module. That is running "sysctl -a" in the background, triggering the issue you found, it seems.

Regards,

Javier

[1] - https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/amqp.pp#L66-L76

Regards,
Andrew.

-----------------------------------

Andrew Zong
Sr Software Engineer, F5 Networks

From: Javier Pena <jpena@redhat.com>
Date: Friday, August 17, 2018 at 4:51 PM
To: Andrew ZONG <A.ZONG@f5.com>
Subject: Re: [rdo-users] RDO install OpenStack queens all-in-one get stuck

EXTERNAL MAIL: jpena@redhat.com
Hi Andrew,

It looks like the puppet manifest execution is getting stuck, but the log files are not giving us enough information.

Could you try running packstack like:

# packstack --debug --answer-file <answer file>

This will enable additional puppet debug output at /var/tmp/packstack/<uuid>/manifests/10.145.115.45_controller.pp.log

Regards,
Javier

Hi,
When I use packstack to install openstack queens allinone on CentOS 7 x86_64, it got stuck at:
# sudo packstack --answer-file pack-1534411186.txt
Welcome to the Packstack setup utility

The installation log file is available at: /var/tmp/packstack/20180816-022431-Qun9eG/openstack-setup.log

Installing:
Clean Up                                             [ DONE ]
Discovering ip protocol version                      [ DONE ]
Setting up ssh keys                                  [ DONE ]
...

Testing if puppet apply is finished: 10.145.xxx.yyy_controller.pp [ | ]

I find the log mention above, the last line seems no help:
tar --dereference -cpzf - aodh apache ceilometer certmonger cinder concat firewall glance gnocchi heat horizon inifile ironic keystone magnum manila memcached mysql neutron nova nssdb openstack openstacklib oslo ovn packstack panko rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest trove vcsrepo vswitch xinetd | ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@10.145.115.45 tar -C /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules -xpzf -

Also I go to the puppet running file and find some warning there:
# ps -ef | grep puppet
root      1870 1869 0 02:25 ?        00:00:00 flock /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/ps.lock puppet apply --modulepath /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp
...

Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/db/mysql.pp", 63]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 21]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'gnocchi' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'aodh' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: Unknown variable: '::nova::db::mysql_api::setup_cell0'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/db/mysql.pp:53:28
Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/policy.pp", 34]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/init.pp", 787]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Pattern[]. There is further documentation for validate_legacy function in the README. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/init.pp", 806]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 27]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: Scope(Class[Cinder::Keystone::Authtoken]): revocation_cache_time parameter is deprecated, has no effect and will be removed in the future.
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/cinder/manifests/api.pp", 220]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 45]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: Scope(Class[Nova::Db]): placement_database_connection has no effect as of pike, and may be removed in a future release
Warning: Scope(Class[Nova::Db]): placement_slave_connection has no effect as of pike, and may be removed in a future release
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/init.pp", 564]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 74]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: Scope(Class[Nova::Api]): The default_floating_pool parameter is deprecated. Please use nova::network::neutron::default_floating_pool instead.
Warning: Scope(Class[Nova::Api]): The pci_alias parameter is deprecated. Please use nova::pci::aliases instead.
Warning: Unknown variable: 'until_complete_real'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/cron/archive_deleted_rows.pp:77:82
Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/scheduler/filter.pp", 140]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 80]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ip_address instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/compute/neutron.pp", 31]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 85]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: Scope(Class[Neutron::Plugins::Ml2]): supported_pci_vendor_devs is deprecated, has no effect and will be removed in a future release.
Warning: ModuleLoader: module 'horizon' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: This method is deprecated, please use the stdlib validate_legacy function,
                    with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/horizon/manifests/init.pp", 610]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp", 123]
   (at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in `deprecation')
Warning: ModuleLoader: module 'swift' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: Unknown variable: 'methods_real'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:100:56
Warning: Unknown variable: 'incoming_remove_headers_real'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:101:56
Warning: Unknown variable: 'incoming_allow_headers_real'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:102:56
Warning: Unknown variable: 'outgoing_remove_headers_real'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:103:56
Warning: Unknown variable: 'outgoing_allow_headers_real'. at /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:104:56
Warning: Scope(Class[Swift::Storage::All]): The default port for the object storage server has changed from 6000 to 6200 and will be changed in a later release
Warning: Scope(Class[Swift::Storage::All]): The default port for the container storage server has changed from 6001 to 6201 and will be changed in a later release
Warning: Scope(Class[Swift::Storage::All]): The default port for the account storage server has changed from 6002 to 6202 and will be changed in a later release
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::logging_context_format_string is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::logging_default_format_string is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::logging_debug_format_suffix is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::logging_exception_prefix is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::log_config_append is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::default_log_levels is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::publish_errors is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::fatal_deprecations is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::instance_format is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::instance_uuid_format is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Logging]): gnocchi::logging::log_date_format is deprecated and will be removed in future
Warning: Scope(Class[Gnocchi::Api]): host is deprecated
Warning: Scope(Class[Gnocchi::Statsd]): user_id parameter is deprecated and will be removed in the future release.
Warning: Scope(Class[Gnocchi::Statsd]): project_id parameter is deprecated and will be removed in the future release.
Warning: ModuleLoader: module 'ceilometer' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: Scope(Class[Ceilometer::Agent::Central]): This class is deprecated. Please use ceilometer::agent::central with central namespace instead.
Warning: ModuleLoader: module 'redis' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'rabbitmq' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: ModuleLoader: module 'oslo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)

What's the problem? My basic setup steps and environment are as following:
CentOS-7-x86_64-Minimal-1611: 8 cores and 16GB mem.
sudo yum update -y
sudo yum install -y centos-release-openstack-queens
yum update -y && yum install -y openstack-packstack
cat << EOF > /etc/environment

LANG=en_US.utf-8
LC_ALL=en_US.utf-8
EOF
packstack --gen-answer-file pack-`date +%s`.txt
<configure as following>
CONFIG_KEYSTONE_ADMIN_PW=default
CONFIG_CONTROLLER_HOST=10.145.xxx.yyy
CONFIG_COMPUTE_HOSTS=10.145.xxx.yyy
CONFIG_NETWORK_HOSTS=10.145.xxx.yyy
CONFIG_LBAAS_INSTALL=y

sudo systemctl disable firewalld
sudo systemctl stop firewalld
sudo systemctl disable NetworkManager
sudo systemctl stop NetworkManager
sudo systemctl enable network
sudo systemctl start network

uname -m | grep -q 'x86_64' && echo 'centos' >/etc/yum/vars/contentdir || echo 'altarch' >/etc/yum/vars/contentdir

packstack --answer-file <answer file>

Regards,
Andrew.

-----------------------------------

Andrew Zong
Sr Software Engineer, F5 Networks

_______________________________________________
users mailing list
users@lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/users

To unsubscribe: users-unsubscribe@lists.rdoproject.org