<< Nova log >> 2019-11-30 06:29:27.391 145061 INFO os_vif [-] Loaded VIF plugins: ovs, linux_bridge, noop 2019-11-30 06:29:28.560 145061 INFO nova.virt.driver [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Loading compute driver 'libvirt.LibvirtDriver' 2019-11-30 06:29:29.228 145061 WARNING os_brick.initiator.connectors.remotefs [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Connection details not present. RemoteFsClient may not initialize properly. 2019-11-30 06:29:29.244 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "dhcpbridge" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-11-30 06:29:29.244 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "dhcpbridge_flagfile" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-11-30 06:29:29.246 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "force_dhcp_release" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-11-30 06:29:29.250 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "force_snat_range" from group "DEFAULT" is deprecated for removal ( nova-network is deprecated, as are any related configuration options. ). Its value may be silently ignored in the future. 2019-11-30 06:29:29.272 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "url" from group "neutron" is deprecated for removal (Endpoint lookup uses the service catalog via common keystoneauth1 Adapter configuration options. In the current release, "url" will override this behavior, but will be ignored and/or removed in a future release. To achieve the same result, use the endpoint_override option instead.). Its value may be silently ignored in the future. 2019-11-30 06:29:29.274 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "vncserver_listen" from group "vnc" is deprecated. Use option "server_listen" from group "vnc". 2019-11-30 06:29:29.275 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "vncserver_proxyclient_address" from group "vnc" is deprecated. Use option "server_proxyclient_address" from group "vnc". 2019-11-30 06:29:29.279 145061 WARNING oslo_config.cfg [req-0631d103-d97e-48c7-974d-7fc399a0e0ee - - - - -] Deprecated: Option "live_migration_uri" from group "libvirt" is deprecated for removal ( live_migration_uri is deprecated for removal in favor of two other options that allow to change live migration scheme and target URI: ``live_migration_scheme`` and ``live_migration_inbound_addr`` respectively. ). Its value may be silently ignored in the future. 2019-11-30 06:29:29.296 145061 INFO nova.service [-] Starting compute node (version 19.0.3-1.el7) 2019-11-30 06:29:29.373 145061 INFO nova.virt.libvirt.driver [-] Connection event '1' reason 'None' 2019-11-30 06:29:29.398 145061 INFO nova.virt.libvirt.host [-] Libvirt host capabilities 98761abc-dd6f-450a-8f2f-13db228bd2ba x86_64 Westmere-IBRS Intel tcp rdma 5242332 1310583 0 none 0 dac 0 +107:+107 +107:+107 hvm 32 /usr/libexec/qemu-kvm pc-i440fx-rhel7.6.0 pc pc-i440fx-rhel7.0.0 pc-q35-rhel7.6.0 q35 rhel6.3.0 rhel6.4.0 rhel6.0.0 pc-i440fx-rhel7.5.0 pc-i440fx-rhel7.1.0 pc-i440fx-rhel7.2.0 pc-q35-rhel7.3.0 rhel6.5.0 pc-q35-rhel7.4.0 rhel6.6.0 rhel6.1.0 rhel6.2.0 pc-i440fx-rhel7.3.0 pc-i440fx-rhel7.4.0 pc-q35-rhel7.5.0 hvm 64 /usr/libexec/qemu-kvm pc-i440fx-rhel7.6.0 pc pc-i440fx-rhel7.0.0 pc-q35-rhel7.6.0 q35 rhel6.3.0 rhel6.4.0 rhel6.0.0 pc-i440fx-rhel7.5.0 pc-i440fx-rhel7.1.0 pc-i440fx-rhel7.2.0 pc-q35-rhel7.3.0 rhel6.5.0 pc-q35-rhel7.4.0 rhel6.6.0 rhel6.1.0 rhel6.2.0 pc-i440fx-rhel7.3.0 pc-i440fx-rhel7.4.0 pc-q35-rhel7.5.0 2019-11-30 06:29:30.324 145061 ERROR nova.compute.resource_tracker [req-6a4d7b4f-9ab1-4669-8c7e-99e3adb5cb18 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 27a39914-a509-4261-90f5-8135ad471843: 500 Internal Server Error

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

: ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 27a39914-a509-4261-90f5-8135ad471843: 2019-11-30 06:29:30.388 145061 ERROR nova.scheduler.client.report [req-6a4d7b4f-9ab1-4669-8c7e-99e3adb5cb18 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 27a39914-a509-4261-90f5-8135ad471843. Got 500: 500 Internal Server Error

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

. 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager [req-6a4d7b4f-9ab1-4669-8c7e-99e3adb5cb18 - - - - -] Error updating resources for node openstack.bipul.com.: ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 27a39914-a509-4261-90f5-8135ad471843 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager Traceback (most recent call last): 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8148, in _update_available_resource_for_node 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager startup=startup) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 748, in update_available_resource 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager return f(*args, **kwargs) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 829, in _update_available_resource 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 1036, in _update 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 962, in _update_to_placement 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 873, in get_provider_tree_and_ensure_root 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 655, in _ensure_resource_provider 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager rps_to_refresh = self._get_providers_in_tree(context, uuid) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 71, in wrapper 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager return f(self, *a, **k) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 522, in _get_providers_in_tree 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 27a39914-a509-4261-90f5-8135ad471843 2019-11-30 06:29:30.388 145061 ERROR nova.compute.manager <> Nova service status : [root@openstack ~(keystone_admin)]# systemctl status openstack-nova-compute.service ? openstack-nova-compute.service - OpenStack Nova Compute Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2019-12-01 03:05:02 EST; 4h 46min left Main PID: 1632 (nova-compute) Tasks: 22 CGroup: /system.slice/openstack-nova-compute.service +-1632 /usr/bin/python2 /usr/bin/nova-compute Dec 01 03:04:23 openstack.bipul.com systemd[1]: Starting OpenStack Nova Compute Server... Dec 01 03:05:02 openstack.bipul.com systemd[1]: Started OpenStack Nova Compute Server. [root@openstack ~(keystone_admin)]# systemctl status openstack-nova-conductor.service ? openstack-nova-conductor.service - OpenStack Nova Conductor Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2019-12-01 03:04:50 EST; 4h 46min left Main PID: 1224 (nova-conductor) Tasks: 3 CGroup: /system.slice/openstack-nova-conductor.service +-1224 /usr/bin/python2 /usr/bin/nova-conductor +-2306 /usr/bin/python2 /usr/bin/nova-conductor +-2307 /usr/bin/python2 /usr/bin/nova-conductor Dec 01 03:04:20 openstack.bipul.com systemd[1]: Starting OpenStack Nova Conductor Server... Dec 01 03:04:50 openstack.bipul.com systemd[1]: Started OpenStack Nova Conductor Server. [root@openstack ~(keystone_admin)]# systemctl status openstack-nova-consoleauth.service ? openstack-nova-consoleauth.service - OpenStack Nova VNC console auth Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-consoleauth.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2019-12-01 03:04:46 EST; 4h 46min left Main PID: 1233 (nova-consoleaut) Tasks: 1 CGroup: /system.slice/openstack-nova-consoleauth.service +-1233 /usr/bin/python2 /usr/bin/nova-consoleauth Dec 01 03:04:20 openstack.bipul.com systemd[1]: Starting OpenStack Nova VNC console auth Server... Dec 01 03:04:46 openstack.bipul.com systemd[1]: Started OpenStack Nova VNC console auth Server. [root@openstack ~(keystone_admin)]# systemctl status openstack-nova-scheduler.service ? openstack-nova-scheduler.service - OpenStack Nova Scheduler Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2019-12-01 03:04:52 EST; 4h 45min left Main PID: 1215 (nova-scheduler) Tasks: 3 CGroup: /system.slice/openstack-nova-scheduler.service +-1215 /usr/bin/python2 /usr/bin/nova-scheduler +-2321 /usr/bin/python2 /usr/bin/nova-scheduler +-2322 /usr/bin/python2 /usr/bin/nova-scheduler Dec 01 03:04:20 openstack.bipul.com systemd[1]: Starting OpenStack Nova Scheduler Server... Dec 01 03:04:52 openstack.bipul.com systemd[1]: Started OpenStack Nova Scheduler Server. [root@openstack ~(keystone_admin)]#