Hi, I have been hitting the RDO bug https://bugzilla.redhat.com/show_bug.cgi?id=1087589, and not able to boot a nova instance. I see that the fix https://review.openstack.org/#/c/92387/ was merged on May 29th 2014. Is this fix available in the latest RDO ? Below are the package versions of neutron and nova in my RHEL 7 IceHouse setup. How can I check if the fix is available in my packages below ? Nova: ===== [root@compute1 nova(keystone_admin)]# rpm -qa | grep -i openstack | grep neutron openstack-neutron-openvswitch-2014.1.2-2.el7ost.noarch openstack-neutron-2014.1.2-2.el7ost.noarch [root@compute1 nova(keystone_admin)]# yum info openstack-neutron-openvswitch-2014.1.2-2.el7ost.noarch Name : openstack-neutron-openvswitch Arch : noarch Version : 2014.1.2 Release : 2.el7ost [root@compute1 nova(keystone_admin)]# yum info openstack-neutron-2014.1.2-2.el7ost.noarch Name : openstack-neutron Arch : noarch Version : 2014.1.2 Release : 2.el7ost Neutron: ======== [root@compute1 nova(keystone_admin)]# rpm -qa | grep -i openstack | grep nova openstack-nova-common-2014.1.2-1.el7ost.noarch openstack-nova-compute-2014.1.2-1.el7ost.noarch [root@compute1 nova(keystone_admin)]# yum info openstack-nova-common-2014.1.2-1.el7ost.noarch Name : openstack-nova-common Arch : noarch Version : 2014.1.2 Release : 1.el7ost [root@compute1 nova(keystone_admin)]# yum info openstack-nova-compute-2014.1.2-1.el7ost.noarch Name : openstack-nova-compute Arch : noarch Version : 2014.1.2 Release : 1.el7ost Below are the errors of the above bug I see when I boot an instance. Please let me know the solution/work-around. Thanks a lot! In /var/log/nova/nova-compute.log: ============================== 2015-01-05 11:31:15.870 5275 ERROR nova.compute.manager [req-9dee8033-a261-4a89-9477-772141cb747e ca5d4869d4e2464bb6e1e3c12a82c3a2 4dd830d71afb48d6a5e3970249669970] [instance: 228289f2-d3c2-401d-b779-f359121b4e71] Instance failed to spawn 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] Traceback (most recent call last): 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] block_device_info) 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2289, in spawn 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] block_device_info) 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3710, in _create_domain_and_network 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] raise exception.VirtualInterfaceCreateException() 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] VirtualInterfaceCreateException: Virtual Interface creation failed In /var/log/neutron/server.log: ========================= 2015-01-05 11:11:40.321 3888 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'cc051252-1147-43a7-8cb2-0c9f2aa26c1c', 'name': 'network-vif-unplugged', 'server_uuid': u'13cfb377-4fc4-4fbf-ac53-c0a0d7904b93'}] 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova Traceback (most recent call last): 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 221, in send_events 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova batched_events) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/server_external_events.py", line 39, in create 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova return_raw=True) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 152, in _create 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 312, in post 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova return self._cs_request(url, 'POST', **kwargs) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 286, in _cs_request 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova **kwargs) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 268, in _time_request 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova resp, body = self.request(url, method, **kwargs) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 262, in request 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova raise exceptions.from_response(resp, body, url, method) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova NotFound: No instances found for any event (HTTP 404) (Request-ID: req-240de890-6511-4acc-b873-49deb7c6ae56) 2015-01-05 11:11:40.321 3888 TRACE neutron.notifiers.nova Regards, Vikram Hosakote OpenStack Software Engineer | vhosakot@cisco.com<mailto:vhosakot@cisco.com> Cloud and Virtualization Group | Cisco Systems Boxborough MA | Work : 978-936-8799

[Removed rhos-maint(a)redhat.com as I saw an email sent to it failed and could not be delivered] Hi Kashyap, Thanks a lot for your reply. I do see the fix in my environment. In /usr/lib/python2.7/site-packages/neutron/common/config.py, I see the following code snippet at line numbers 89 and 90. 89 cfg.StrOpt('nova_url', 90 default='http://127.0.0.1:8774/v2', In /etc/neutron/neutron.conf, I do have the Nova v2 API version number in the URL below. But, instead of 127.0.0.1, I have the IP address of the network host which is same as CONFIG_NETWORK_HOSTS in my packstack-answers.txt. # URL for connection to nova (Only supports one nova region currently). nova_url = http://172.29.68.136:8774/v2 I have the below line in my packstack-answers.txt. CONFIG_NETWORK_HOSTS=172.29.68.136 Yes, I can reproduce this issue every single time. Am I seeing a new bug ? Please let me know if there is any work-around, and also if I need to send any logs/debugs. Regards, Vikram Hosakote OpenStack Software Engineer | vhosakot(a)cisco.com Cloud and Virtualization Group | Cisco Systems Boxborough MA | Work : 978-936-8799 From: Kashyap Chamarthy <kchamart@redhat.com<mailto:kchamart@redhat.com>> Date: Tuesday, January 6, 2015 at 5:35 AM To: Vikram Hosakote <vhosakot@cisco.com<mailto:vhosakot@cisco.com>> Cc: "rdo-list@redhat.com<mailto:rdo-list@redhat.com>" <rdo-list@redhat.com<mailto:rdo-list@redhat.com>>, "rhos-maint@redhat.com<mailto:rhos-maint@redhat.com>" <rhos-maint@redhat.com<mailto:rhos-maint@redhat.com>> Subject: Re: [Rdo-list] RDO neutron bug 1087589 On Tue, Jan 06, 2015 at 02:34:14AM +0000, Vikram Hosakote (vhosakot) wrote: Hi, I have been hitting the RDO bug https://bugzilla.redhat.com/show_bug.cgi?id=1087589, and not able to boot a nova instance. As you might have noticed, that bug is fixed upstream more than 6 months ago, so it's reasonable to expect to have it in current RDO stable versions for sure (Juno). I see that the fix https://review.openstack.org/#/c/92387/ was merged on May 29th 2014. Is this fix available in the latest RDO ? I checked the source of python-neutron RPM for for both CentOS (from latest RDO) and your package version -- the above fix is available in both. - python-neutron-2014.2.1-1.el7.centos.noarch.rpm (From here https://repos.fedorapeople.org/repos/openstack/openstack-juno/epel-7/) - python-neutron-2014.1.2-2.el7ost.noarch.rpm Below are the package versions of neutron and nova in my RHEL 7 IceHouse setup. How can I check if the fix is available in my packages below ? That fix is part of this file: usr/lib/python2.7/site-packages/neutron/common/config.py You can try two things: - In this file /usr/lib/python2.7/site-packages/neutron/common/config.py, you can check if you have the relevant code snippet: . . . cfg.StrOpt('nova_url', default='http://127.0.0.1:8774/v2', . . . - In your /etc/neutron.conf, ensure the 'nova_url' config parameter has the Nova v2 API version number in the URL: nova_url = http://127.0.0.1:8774/v2 [. . .] Below are the errors of the above bug I see when I boot an instance. Please let me know the solution/work-around. Thanks a lot! In /var/log/nova/nova-compute.log: ============================== 2015-01-05 11:31:15.870 5275 ERROR nova.compute.manager [req-9dee8033-a261-4a89-9477-772141cb747e ca5d4869d4e2464bb6e1e3c12a82c3a2 4dd830d71afb48d6a5e3970249669970] [instance: 228289f2-d3c2-401d-b779-f359121b4e71] Instance failed to spawn 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] Traceback (most recent call last): 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] block_device_info) 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2289, in spawn 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] block_device_info) 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3710, in _create_domain_and_network 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] raise exception.VirtualInterfaceCreateException() 2015-01-05 11:31:15.870 5275 TRACE nova.compute.manager [instance: 228289f2-d3c2-401d-b779-f359121b4e71] VirtualInterfaceCreateException: Virtual Interface creation failed Are you able to consistently reproduce this issue in a different test environment? -- /kashyap