[Rdo-list] Icehouse multi-node - Centos7 - live migration failed because of "network qbr no such device"
Benoit ML
ben42ml at gmail.com
Tue Jul 15 13:13:07 UTC 2014
Hello again,
Ok on controller node I modify the neutron server configuration with
nova_admin_tenant_id = f23ed5be5f534fdba31d23f60621347d
where id is "services" in keystone and now it's working with
"vif_plugging_is_fatal
= True". Good thing.
Well by the way the migrate doesnt working ...
2014-07-15 14:20 GMT+02:00 Benoit ML <ben42ml at gmail.com>:
> Hello,
>
> Thank you for taking time !
>
> Well on the compute node, when i activate "vif_plugging_is_fatal = True",
> the vm creation stuck in spawning state, and in neutron server log i have :
>
> =======================================
> 2014-07-15 14:12:52.351 18448 DEBUG neutron.notifiers.nova [-] Sending
> events: [{'status': 'completed', 'tag':
> u'ba56921c-4628-4fcc-9dc5-2b324cd91faf', 'name': 'network-vif-plugged',
> 'server_uuid': u'f8554441-565c-49ec-bc88-3cbc628b0579'}] send_events
> /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:218
> 2014-07-15 14:12:52.354 18448 INFO urllib3.connectionpool [-] Starting new
> HTTP connection (1): localhost
> 2014-07-15 14:12:52.360 18448 DEBUG urllib3.connectionpool [-] "POST
> /v2/5c9c186a909e499e9da0dd5cf2c403e0/os-server-external-events HTTP/1.1"
> 401 23 _make_request
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:295
> 2014-07-15 14:12:52.362 18448 INFO urllib3.connectionpool [-] Starting new
> HTTP connection (1): localhost
> 2014-07-15 14:12:52.452 18448 DEBUG urllib3.connectionpool [-] "POST
> /v2.0/tokens HTTP/1.1" 401 114 _make_request
> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:295
> 2014-07-15 14:12:52.453 18448 ERROR neutron.notifiers.nova [-] Failed to
> notify nova on events: [{'status': 'completed', 'tag':
> u'ba56921c-4628-4fcc-9dc5-2b324cd91faf', 'name': 'network-vif-plugged',
> 'server_uuid': u'f8554441-565c-49ec-bc88-3cbc628b0579'}]
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova Traceback (most
> recent call last):
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
> "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 221, in
> send_events
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova
> batched_events)
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
> "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/server_external_events.py",
> line 39, in create
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova
> return_raw=True)
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
> "/usr/lib/python2.7/site-packages/novaclient/base.py", line 152, in _create
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova _resp, body
> = self.api.client.post(url, body=body)
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 312, in post
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova return
> self._cs_request(url, 'POST', **kwargs)
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in
> _cs_request
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova raise e
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova Unauthorized:
> Unauthorized (HTTP 401)
> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova
> 2014-07-15 14:12:58.321 18448 DEBUG neutron.openstack.common.rpc.amqp [-]
> received {u'_context_roles': [u'admin'], u'_context_request_id':
> u'req-9bf35c42-3477-4ed3-8092-af729c21198c', u'_context_read_deleted':
> u'no', u'_context_user_name': None, u'_context_project_name': None,
> u'namespace': None, u'_context_tenant_id': None, u'args': {u'agent_state':
> {u'agent_state': {u'topic': u'N/A', u'binary':
> u'neutron-openvswitch-agent', u'host': u'pvidgsh006.pvi', u'agent_type':
> u'Open vSwitch agent', u'configurations': {u'tunnel_types': [u'vxlan'],
> u'tunneling_ip': u'192.168.40.5', u'bridge_mappings': {}, u'l2_population':
> False, u'devices': 1}}}, u'time': u'2014-07-15T12:12:58.313995'},
> u'_context_tenant': None, u'_unique_id':
> u'7c9a4dfcd256494caf6e1327c8051e29', u'_context_is_admin': True,
> u'version': u'1.0', u'_context_timestamp': u'2014-07-15 12:01:28.190772',
> u'_context_tenant_name': None, u'_context_user': None, u'_context_user_id':
> None, u'method': u'report_state', u'_context_project_id': None} _safe_log
> /usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/common.py:280
> =======================================
>
> Well I'm supposed it's related ... Perhaps with those options in
> neutron.conf :
> ======================================
> notify_nova_on_port_status_changes = True
> notify_nova_on_port_data_changes = True
> nova_url = http://localhost:8774/v2
> nova_admin_tenant_name = services
> nova_admin_username = nova
> nova_admin_password = nova
> nova_admin_auth_url = http://localhost:35357/v2.0
> ======================================
>
> But well didnt see anything wrong ...
>
> Thank you in advance !
>
> Regards,
>
>
>
> 2014-07-11 16:08 GMT+02:00 Vimal Kumar <vimal7370 at gmail.com>:
>
> -----
>> File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line
>> 239, in authenticate\\n content_type="application/json")\\n\', u\' File
>> "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 163, in
>> _cs_request\\n raise exceptions.Unauthorized(message=body)\\n\',
>> u\'Unauthorized: {"error": {"message": "The request you have made requires
>> authentication.", "code": 401, "title": "Unauthorized"}}\\n\'].\n']
>> -----
>>
>> Looks like HTTP connection to neutron server is resulting in 401 error.
>>
>> Try enabling debug mode for neutron server and then tail
>> /var/log/neutron/server.log , hopefully you should get more info.
>>
>>
>> On Fri, Jul 11, 2014 at 7:13 PM, Benoit ML <ben42ml at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Ok I see. Nova telles neutron/openvswitch to create the bridge qbr prior
>>> to the migration itself.
>>> I ve already activate debug and verbose ... But well i'm really stuck,
>>> dont know how and where to search/look ...
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>>
>>>
>>> 2014-07-11 15:09 GMT+02:00 Miguel Angel <miguelangel at ajo.es>:
>>>
>>> Hi Benoit,
>>>>
>>>> A manual virsh migration should fail, because the
>>>> network ports are not migrated to the destination host.
>>>>
>>>> You must investigate on the authentication problem itself,
>>>> and let nova handle all the underlying API calls which should happen...
>>>>
>>>> May be it's worth setting nova.conf to debug=True
>>>>
>>>>
>>>>
>>>> ---
>>>> irc: ajo / mangelajo
>>>> Miguel Angel Ajo Pelayo
>>>> +34 636 52 25 69
>>>> skype: ajoajoajo
>>>>
>>>>
>>>> 2014-07-11 14:41 GMT+02:00 Benoit ML <ben42ml at gmail.com>:
>>>>
>>>> Hello,
>>>>>
>>>>> cat /etc/redhat-release
>>>>> CentOS Linux release 7 (Rebuilt from: RHEL 7.0)
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>
>>>>> 2014-07-11 13:40 GMT+02:00 Boris Derzhavets <bderzhavets at hotmail.com>:
>>>>>
>>>>> Could you please post /etc/redhat-release.
>>>>>>
>>>>>> Boris.
>>>>>>
>>>>>> ------------------------------
>>>>>> Date: Fri, 11 Jul 2014 11:57:12 +0200
>>>>>> From: ben42ml at gmail.com
>>>>>> To: rdo-list at redhat.com
>>>>>> Subject: [Rdo-list] Icehouse multi-node - Centos7 - live migration
>>>>>> failed because of "network qbr no such device"
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm working on a multi-node setup of openstack Icehouse using centos7.
>>>>>> Well i have :
>>>>>> - one controllor node with all server services thing stuff
>>>>>> - one network node with openvswitch agent, l3-agent, dhcp-agent
>>>>>> - two compute node with nova-compute and neutron-openvswitch
>>>>>> - one storage nfs node
>>>>>>
>>>>>> NetworkManager is deleted on compute nodes and network node.
>>>>>>
>>>>>> My network use is configured to use vxlan. I can create VM,
>>>>>> tenant-network, external-network, routeur, assign floating-ip to VM, push
>>>>>> ssh-key into VM, create volume from glance image, etc... Evrything is
>>>>>> conected and reacheable. Pretty cool :)
>>>>>>
>>>>>> But when i try to migrate VM things go wrong ... I have configured
>>>>>> nova, libvirtd and qemu to use migration through libvirt-tcp.
>>>>>> I have create and exchanged ssh-key for nova user on all node. I
>>>>>> have verified userid and groupid of nova.
>>>>>>
>>>>>> Well nova-compute log, on the target compute node, :
>>>>>> 2014-07-11 11:45:02.749 6984 TRACE nova.compute.manager [instance:
>>>>>> a5326fe1-4faa-4347-ba29-159fce26a85c] RemoteError: Remote error:
>>>>>> Unauthorized {"error": {"m
>>>>>> essage": "The request you have made requires authentication.",
>>>>>> "code": 401, "title": "Unauthorized"}}
>>>>>>
>>>>>>
>>>>>> So well after searching a lots in all logs, i have fount that i cant
>>>>>> simply migration VM between compute node with a simple virsh :
>>>>>> virsh migrate instance-00000084 qemu+tcp://<dest>/system
>>>>>>
>>>>>> The error is :
>>>>>> erreur :Cannot get interface MTU on 'qbr3ca65809-05': No such device
>>>>>>
>>>>>> Well when i look on the source hyperviseur the bridge "qbr3ca65809"
>>>>>> exists and have a network tap device. And moreover i manually create
>>>>>> qbr3ca65809 on the target hypervisor, virsh migrate succed !
>>>>>>
>>>>>> Can you help me plz ?
>>>>>> What can i do wrong ? Perhpas neutron must create the bridge before
>>>>>> migration but didnt for a mis configuration ?
>>>>>>
>>>>>> Plz ask anything you need !
>>>>>>
>>>>>> Thank you in advance.
>>>>>>
>>>>>>
>>>>>> The full nova-compute log attached.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Benoit
>>>>>>
>>>>>> _______________________________________________ Rdo-list mailing list
>>>>>> Rdo-list at redhat.com https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --
>>>>> Benoit
>>>>>
>>>>> _______________________________________________
>>>>> Rdo-list mailing list
>>>>> Rdo-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> --
>>> Benoit
>>>
>>> _______________________________________________
>>> Rdo-list mailing list
>>> Rdo-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>>
>>
>
>
> --
> --
> Benoit
>
--
--
Benoit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20140715/a2741d60/attachment.html>
More information about the dev
mailing list