Re: [Rdo-list] Icehouse multi-node - Centos7 - live migration failed because of "network qbr no such device"

Hello, Thank you for taking time ! Well on the compute node, when i activate "vif_plugging_is_fatal = True", the vm creation stuck in spawning state, and in neutron server log i have : ======================================= 2014-07-15 14:12:52.351 18448 DEBUG neutron.notifiers.nova [-] Sending events: [{'status': 'completed', 'tag': u'ba56921c-4628-4fcc-9dc5-2b324cd91faf', 'name': 'network-vif-plugged', 'server_uuid': u'f8554441-565c-49ec-bc88-3cbc628b0579'}] send_events /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:218 2014-07-15 14:12:52.354 18448 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): localhost 2014-07-15 14:12:52.360 18448 DEBUG urllib3.connectionpool [-] "POST /v2/5c9c186a909e499e9da0dd5cf2c403e0/os-server-external-events HTTP/1.1" 401 23 _make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:295 2014-07-15 14:12:52.362 18448 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): localhost 2014-07-15 14:12:52.452 18448 DEBUG urllib3.connectionpool [-] "POST /v2.0/tokens HTTP/1.1" 401 114 _make_request /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:295 2014-07-15 14:12:52.453 18448 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'ba56921c-4628-4fcc-9dc5-2b324cd91faf', 'name': 'network-vif-plugged', 'server_uuid': u'f8554441-565c-49ec-bc88-3cbc628b0579'}] 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova Traceback (most recent call last): 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 221, in send_events 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova batched_events) 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/server_external_events.py", line 39, in create 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova return_raw=True) 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 152, in _create 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body) 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 312, in post 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova return self._cs_request(url, 'POST', **kwargs) 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in _cs_request 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova raise e 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova Unauthorized: Unauthorized (HTTP 401) 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova 2014-07-15 14:12:58.321 18448 DEBUG neutron.openstack.common.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_context_request_id': u'req-9bf35c42-3477-4ed3-8092-af729c21198c', u'_context_read_deleted': u'no', u'_context_user_name': None, u'_context_project_name': None, u'namespace': None, u'_context_tenant_id': None, u'args': {u'agent_state': {u'agent_state': {u'topic': u'N/A', u'binary': u'neutron-openvswitch-agent', u'host': u'pvidgsh006.pvi', u'agent_type': u'Open vSwitch agent', u'configurations': {u'tunnel_types': [u'vxlan'], u'tunneling_ip': u'192.168.40.5', u'bridge_mappings': {}, u'l2_population': False, u'devices': 1}}}, u'time': u'2014-07-15T12:12:58.313995'}, u'_context_tenant': None, u'_unique_id': u'7c9a4dfcd256494caf6e1327c8051e29', u'_context_is_admin': True, u'version': u'1.0', u'_context_timestamp': u'2014-07-15 12:01:28.190772', u'_context_tenant_name': None, u'_context_user': None, u'_context_user_id': None, u'method': u'report_state', u'_context_project_id': None} _safe_log /usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/common.py:280 ======================================= Well I'm supposed it's related ... Perhaps with those options in neutron.conf : ====================================== notify_nova_on_port_status_changes = True notify_nova_on_port_data_changes = True nova_url = http://localhost:8774/v2 nova_admin_tenant_name = services nova_admin_username = nova nova_admin_password = nova nova_admin_auth_url = http://localhost:35357/v2.0 ====================================== But well didnt see anything wrong ... Thank you in advance ! Regards, 2014-07-11 16:08 GMT+02:00 Vimal Kumar <vimal7370(a)gmail.com&gt;: ----- > File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line > 239, in authenticate\\n content_type="application/json")\\n\', u\' File > "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 163, in > _cs_request\\n raise exceptions.Unauthorized(message=body)\\n\', > u\'Unauthorized: {"error": {"message": "The request you have made requires > authentication.", "code": 401, "title": "Unauthorized"}}\\n\'].\n'] > ----- > > Looks like HTTP connection to neutron server is resulting in 401 error. > > Try enabling debug mode for neutron server and then tail > /var/log/neutron/server.log , hopefully you should get more info. > > > On Fri, Jul 11, 2014 at 7:13 PM, Benoit ML <ben42ml(a)gmail.com&gt; wrote: > >> Hello, >> >> Ok I see. Nova telles neutron/openvswitch to create the bridge qbr prior >> to the migration itself. >> I ve already activate debug and verbose ... But well i'm really stuck, >> dont know how and where to search/look ... >> >> >> >> Regards, >> >> >> >> >> >> 2014-07-11 15:09 GMT+02:00 Miguel Angel <miguelangel(a)ajo.es&gt;: >> >> Hi Benoit, >>> >>> A manual virsh migration should fail, because the >>> network ports are not migrated to the destination host. >>> >>> You must investigate on the authentication problem itself, >>> and let nova handle all the underlying API calls which should happen... >>> >>> May be it's worth setting nova.conf to debug=True >>> >>> >>> >>> --- >>> irc: ajo / mangelajo >>> Miguel Angel Ajo Pelayo >>> +34 636 52 25 69 >>> skype: ajoajoajo >>> >>> >>> 2014-07-11 14:41 GMT+02:00 Benoit ML <ben42ml(a)gmail.com&gt;: >>> >>> Hello, >>>> >>>> cat /etc/redhat-release >>>> CentOS Linux release 7 (Rebuilt from: RHEL 7.0) >>>> >>>> >>>> Regards, >>>> >>>> >>>> 2014-07-11 13:40 GMT+02:00 Boris Derzhavets <bderzhavets(a)hotmail.com&gt;: >>>> >>>> Could you please post /etc/redhat-release. >>>>> >>>>> Boris. >>>>> >>>>> ------------------------------ >>>>> Date: Fri, 11 Jul 2014 11:57:12 +0200 >>>>> From: ben42ml(a)gmail.com >>>>> To: rdo-list(a)redhat.com >>>>> Subject: [Rdo-list] Icehouse multi-node - Centos7 - live migration >>>>> failed because of "network qbr no such device" >>>>> >>>>> >>>>> Hello, >>>>> >>>>> I'm working on a multi-node setup of openstack Icehouse using centos7. >>>>> Well i have : >>>>> - one controllor node with all server services thing stuff >>>>> - one network node with openvswitch agent, l3-agent, dhcp-agent >>>>> - two compute node with nova-compute and neutron-openvswitch >>>>> - one storage nfs node >>>>> >>>>> NetworkManager is deleted on compute nodes and network node. >>>>> >>>>> My network use is configured to use vxlan. I can create VM, >>>>> tenant-network, external-network, routeur, assign floating-ip to VM, push >>>>> ssh-key into VM, create volume from glance image, etc... Evrything is >>>>> conected and reacheable. Pretty cool :) >>>>> >>>>> But when i try to migrate VM things go wrong ... I have configured >>>>> nova, libvirtd and qemu to use migration through libvirt-tcp. >>>>> I have create and exchanged ssh-key for nova user on all node. I >>>>> have verified userid and groupid of nova. >>>>> >>>>> Well nova-compute log, on the target compute node, : >>>>> 2014-07-11 11:45:02.749 6984 TRACE nova.compute.manager [instance: >>>>> a5326fe1-4faa-4347-ba29-159fce26a85c] RemoteError: Remote error: >>>>> Unauthorized {"error": {"m >>>>> essage": "The request you have made requires authentication.", >>>>> "code": 401, "title": "Unauthorized"}} >>>>> >>>>> >>>>> So well after searching a lots in all logs, i have fount that i cant >>>>> simply migration VM between compute node with a simple virsh : >>>>> virsh migrate instance-00000084 qemu+tcp://<dest>/system >>>>> >>>>> The error is : >>>>> erreur :Cannot get interface MTU on 'qbr3ca65809-05': No such device >>>>> >>>>> Well when i look on the source hyperviseur the bridge "qbr3ca65809" >>>>> exists and have a network tap device. And moreover i manually create >>>>> qbr3ca65809 on the target hypervisor, virsh migrate succed ! >>>>> >>>>> Can you help me plz ? >>>>> What can i do wrong ? Perhpas neutron must create the bridge before >>>>> migration but didnt for a mis configuration ? >>>>> >>>>> Plz ask anything you need ! >>>>> >>>>> Thank you in advance. >>>>> >>>>> >>>>> The full nova-compute log attached. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> -- >>>>> -- >>>>> Benoit >>>>> >>>>> _______________________________________________ Rdo-list mailing list >>>>> Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list >>>>> >>>> >>>> >>>> >>>> -- >>>> -- >>>> Benoit >>>> >>>> _______________________________________________ >>>> Rdo-list mailing list >>>> Rdo-list(a)redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>> >>>> >>> >> >> >> -- >> -- >> Benoit >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> > -- -- Benoit