5:35 a.m.
(oops, I forgot to add the list to my initial reply)
----- Original Message -----
Thanks for reply, Javier.
After adding --debug to packstack, more information are found in
controller.pp.running.
Debug: Executing: '/usr/bin/systemctl is-active
rabbitmq-server'
Debug: Executing: '/usr/bin/systemctl is-enabled
rabbitmq-server'
Debug: Prefetching parsed resources for sysctl
Debug: Prefetching sysctl_runtime resources for sysctl_runtime
Debug: Executing: '/usr/sbin/sysctl -a'
From the log, it hangs at ' /usr/sbin/sysctl -a ', by
searching online, this
link( https://access.redhat.com/solutions/2427351 need rhel account login. )
give some hints:
"
· We are finding issues related to /proc/sys/fs/binfmt_misc in RHEV
servers.
· Whenever we run df or sysctl -a , they hang without displaying any
output.
· ls -l /proc/sys/fs/binfmt_misc also hangs.
"
The above issues can be checked manually.
Also we can get some infor via:
systemctl -a | grep binfmt_misc
By continue searching the solution, this link(
https://bugzilla.redhat.com/show_bug.cgi?id=1591418 ) give a workaround.
This works without reboot the host(I don’t want the reboot).
systemctl start proc-sys-fs-binfmt_misc.mount # effect is equal to
reboot my
host.
As to why it happens, I suspect it's lead by 'yum update'
during which new
kernel version are installed, however, I didn’t reboot the host.
kernel.x86_64 3.10.0-514.el7 @anaconda
kernel.x86_64 3.10.0-862.11.6.el7 @updates
Before reboot: 3.10.0-514.el7.x86_64 -=> After reboot:
3.10.0-862.11.6.el7.x86_64
So in summary, I know now what the problem is. But I wonder why it
happens(during executing puppet controller.pp,) what does rdo puppet do to
the system that cause this stuck/hang.
The packstack puppet manifests do some kernel parameter manipulation [1] using the sysctl
module. That is running "sysctl -a" in the background, triggering the issue you
found, it seems.
Regards,
Javier
[1] -
https://github.com/openstack/packstack/blob/master/packstack/puppet/modul...
Regards,
Andrew.
-----------------------------------
Andrew Zong
Sr Software Engineer, F5 Networks
From: Javier Pena <jpena(a)redhat.com>
Date: Friday, August 17, 2018 at 4:51 PM
To: Andrew ZONG <A.ZONG(a)f5.com>
Subject: Re: [rdo-users] RDO install OpenStack queens all-in-one get stuck
EXTERNAL MAIL: jpena(a)redhat.com
Hi Andrew,
It looks like the puppet manifest execution is getting stuck, but the
log
files are not giving us enough information.
Could you try running packstack like:
# packstack --debug --answer-file <answer file>
This will enable additional puppet debug output at
/var/tmp/packstack/<uuid>/
manifests/10.145.115.45_controller.pp .log
Regards,
Javier
> Hi,
> When I use packstack to install openstack queens allinone on
CentOS 7
> x86_64,
> it got stuck at:
> # sudo packstack --answer-file pack-1534411186.txt
> Welcome to the Packstack setup utility
> The installation log file is available at:
> /var/tmp/packstack/20180816-022431-Qun9eG/openstack-setup.log
> Installing:
> Clean Up [ DONE ]
> Discovering ip protocol version [ DONE ]
> Setting up ssh keys [ DONE ]
> ...
> Testing if puppet apply is finished:
10.145.xxx.yyy_controller.pp [ | ]
> I find the log mention above, the last line seems no help:
> tar --dereference -cpzf - aodh apache ceilometer certmonger
cinder concat
> firewall glance gnocchi heat horizon inifile ironic keystone magnum manila
> memcached mysql neutron nova nssdb openstack openstacklib oslo ovn
> packstack
> panko rabbitmq redis remote rsync sahara ssh stdlib swift sysctl tempest
> trove vcsrepo vswitch xinetd | ssh -o StrictHostKeyChecking=no -o
> UserKnownHostsFile=/dev/null root(a)10.145.115.45 tar -C
> /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules -xpzf -
> Also I go to the puppet running file and find some warning
there:
> # ps -ef | grep puppet
> root 1870 1869 0 02:25 ? 00:00:00 flock
> /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/ps.lock puppet apply
> --modulepath /var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp
> ...
> Warning: ModuleLoader: module 'mysql' has unresolved
dependencies - it will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'keystone' has unresolved
dependencies - it
> will only see those that are resolved. Use 'puppet module list --tree' to
> see information about modules
> (file & line not available)
> Warning: This method is deprecated, please use the stdlib
validate_legacy
> function,
> with Stdlib::Compat::String. There is further documentation for
> validate_legacy function in the README. at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/db/mysql.pp",
>
63]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 21]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: ModuleLoader: module 'cinder' has unresolved
dependencies - it
> will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'glance' has unresolved
dependencies - it
> will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'gnocchi' has unresolved
dependencies - it
> will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'aodh' has unresolved
dependencies - it will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'neutron' has unresolved
dependencies - it
> will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'nova' has unresolved
dependencies - it will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: Unknown variable:
'::nova::db::mysql_api::setup_cell0'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/db/mysql.pp:53:28
> Warning: ModuleLoader: module 'openstacklib' has
unresolved dependencies -
> it
> will only see those that are resolved. Use 'puppet module list --tree' to
> see information about modules
> (file & line not available)
> Warning: This method is deprecated, please use the stdlib
validate_legacy
> function,
> with Stdlib::Compat::Hash. There is further documentation for
> validate_legacy
> function in the README. at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/policy.pp",
>
34]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/init.pp",
> 787]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: This method is deprecated, please use the stdlib
validate_legacy
> function,
> with Pattern[]. There is further documentation for
validate_legacy function
> in the README. at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/keystone/manifests/init.pp",
>
806]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 27]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: Scope(Class[Cinder::Keystone::Authtoken]):
revocation_cache_time
> parameter is deprecated, has no effect and will be removed in the future.
> Warning: This method is deprecated, please use the stdlib
validate_legacy
> function,
> with Stdlib::Compat::Bool. There is further documentation for
> validate_legacy
> function in the README. at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/cinder/manifests/api.pp",
>
220]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 45]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: Scope(Class[Nova::Db]): placement_database_connection
has no
> effect
> as of pike, and may be removed in a future release
> Warning: Scope(Class[Nova::Db]): placement_slave_connection has
no effect
> as
> of pike, and may be removed in a future release
> Warning: This method is deprecated, please use the stdlib
validate_legacy
> function,
> with Stdlib::Compat::Array. There is further documentation for
> validate_legacy function in the README. at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/init.pp",
>
564]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 74]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: Scope(Class[Nova::Api]): The default_floating_pool
parameter is
> deprecated. Please use nova::network::neutron::default_floating_pool
> instead.
> Warning: Scope(Class[Nova::Api]): The pci_alias parameter is
deprecated.
> Please use nova::pci::aliases instead.
> Warning: Unknown variable: 'until_complete_real'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/cron/archive_deleted_rows.pp:77:82
> Warning: This method is deprecated, please use match expressions
with
> Stdlib::Compat::Array instead. They are described at
>
https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match....
> at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/scheduler/filter.pp",
>
140]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 80]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: This method is deprecated, please use match expressions
with
> Stdlib::Compat::Ip_address instead. They are described at
>
https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match....
> at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/nova/manifests/compute/neutron.pp",
>
31]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 85]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: Scope(Class[Neutron::Plugins::Ml2]):
supported_pci_vendor_devs is
> deprecated, has no effect and will be removed in a future release.
> Warning: ModuleLoader: module 'horizon' has unresolved
dependencies - it
> will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: This method is deprecated, please use the stdlib
validate_legacy
> function,
> with Stdlib::Compat::Absolute_Path. There is further
documentation for
> validate_legacy function in the README. at
>
["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/horizon/manifests/init.pp",
>
610]:["/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/manifests/10.145.115.45_controller.pp",
> 123]
> (at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/stdlib/lib/puppet/functions/deprecation.rb:28:in
> `deprecation')
> Warning: ModuleLoader: module 'swift' has unresolved
dependencies - it will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: Unknown variable: 'methods_real'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:100:56
> Warning: Unknown variable:
'incoming_remove_headers_real'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:101:56
> Warning: Unknown variable:
'incoming_allow_headers_real'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:102:56
> Warning: Unknown variable:
'outgoing_remove_headers_real'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:103:56
> Warning: Unknown variable:
'outgoing_allow_headers_real'. at
>
/var/tmp/packstack/a73923ae3b8f4332b9fa71e5f30902f0/modules/swift/manifests/proxy/tempurl.pp:104:56
> Warning: Scope(Class[Swift::Storage::All]): The default port for
the object
> storage server has changed from 6000 to 6200 and will be changed in a later
> release
> Warning: Scope(Class[Swift::Storage::All]): The default port for
the
> container storage server has changed from 6001 to 6201 and will be changed
> in a later release
> Warning: Scope(Class[Swift::Storage::All]): The default port for
the
> account
> storage server has changed from 6002 to 6202 and will be changed in a later
> release
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::logging_context_format_string is deprecated and will be
> removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::logging_default_format_string is deprecated and will be
> removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::logging_debug_format_suffix is deprecated and will be
> removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::logging_exception_prefix is deprecated and will be
> removed
> in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::log_config_append
> is deprecated and will be removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::default_log_levels
> is deprecated and will be removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
gnocchi::logging::publish_errors
> is
> deprecated and will be removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::fatal_deprecations
> is deprecated and will be removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
gnocchi::logging::instance_format
> is
> deprecated and will be removed in future
> Warning: Scope(Class[Gnocchi::Logging]):
> gnocchi::logging::instance_uuid_format is deprecated and will be removed in
> future
> Warning: Scope(Class[Gnocchi::Logging]):
gnocchi::logging::log_date_format
> is
> deprecated and will be removed in future
> Warning: Scope(Class[Gnocchi::Api]): host is deprecated
> Warning: Scope(Class[Gnocchi::Statsd]): user_id parameter is
deprecated and
> will be removed in the future release.
> Warning: Scope(Class[Gnocchi::Statsd]): project_id parameter is
deprecated
> and will be removed in the future release.
> Warning: ModuleLoader: module 'ceilometer' has
unresolved dependencies - it
> will only see those that are resolved. Use 'puppet module list --tree' to
> see information about modules
> (file & line not available)
> Warning: Scope(Class[Ceilometer::Agent::Central]): This class
is
> deprecated.
> Please use ceilometer::agent::central with central namespace instead.
> Warning: ModuleLoader: module 'redis' has unresolved
dependencies - it will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'rabbitmq' has unresolved
dependencies - it
> will only see those that are resolved. Use 'puppet module list --tree' to
> see information about modules
> (file & line not available)
> Warning: ModuleLoader: module 'oslo' has unresolved
dependencies - it will
> only see those that are resolved. Use 'puppet module list --tree' to see
> information about modules
> (file & line not available)
> What's the problem? My basic setup steps and environment are
as following:
> CentOS-7-x86_64-Minimal-1611: 8 cores and 16GB mem.
> sudo yum update -y
> sudo yum install -y centos-release-openstack-queens
> yum update -y && yum install -y openstack-packstack
> cat << EOF > /etc/environment
> LANG=en_US.utf-8
> LC_ALL=en_US.utf-8
> EOF
> packstack --gen-answer-file pack-`date +%s`.txt
> <configure as following>
> CONFIG_KEYSTONE_ADMIN_PW=default
> CONFIG_CONTROLLER_HOST=10.145.xxx.yyy
> CONFIG_COMPUTE_HOSTS=10.145.xxx.yyy
> CONFIG_NETWORK_HOSTS=10.145.xxx.yyy
> CONFIG_LBAAS_INSTALL=y
> sudo systemctl disable firewalld
> sudo systemctl stop firewalld
> sudo systemctl disable NetworkManager
> sudo systemctl stop NetworkManager
> sudo systemctl enable network
> sudo systemctl start network
> uname -m | grep -q 'x86_64' && echo
'centos' >/etc/yum/vars/contentdir ||
> echo 'altarch' >/etc/yum/vars/contentdir
> packstack --answer-file <answer file>
> Regards,
>
> Andrew.
>
> -----------------------------------
>
> Andrew Zong
>
> Sr Software Engineer, F5 Networks
>
> _______________________________________________
> users mailing list
> users(a)lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/users
> To unsubscribe: users-unsubscribe(a)lists.rdoproject.org