[Rdo-list] Icehouse multi-node - Centos7 - live migration failed because of "network qbr no such device"
Benoit ML
ben42ml at gmail.com
Wed Jul 16 15:28:42 UTC 2014
Hello,
Another mail about the problem.... Well i have enable debug = True in
keystone.conf
And after a nova migrate <VM>, when i nova show <VM> :
==============================================================================
| fault | {"message": "Remote error:
Unauthorized {\"error\": {\"message\": \"User
0b45ccc267e04b59911e88381bb450c0 is unauthorized for tenant services\",
\"code\": 401, \"title\": \"Unauthorized\"}} |
==============================================================================
So well User with id 0b45ccc267e04b59911e88381bb450c0 is neutron :
==============================================================================
keystone user-list
| 0b45ccc267e04b59911e88381bb450c0 | neutron | True | |
==============================================================================
And the role seems good :
==============================================================================
keystone user-role-add --user=neutron --tenant=services --role=admin
Conflict occurred attempting to store role grant. User
0b45ccc267e04b59911e88381bb450c0 already has role
734c2fb6fb444792b5ede1fa1e17fb7e in tenant dea82f7937064b6da1c370280d8bfdad
(HTTP 409)
keystone user-role-list --user neutron --tenant services
+----------------------------------+-------+----------------------------------+----------------------------------+
| id | name |
user_id | tenant_id |
+----------------------------------+-------+----------------------------------+----------------------------------+
| 734c2fb6fb444792b5ede1fa1e17fb7e | admin |
0b45ccc267e04b59911e88381bb450c0 | dea82f7937064b6da1c370280d8bfdad |
+----------------------------------+-------+----------------------------------+----------------------------------+
keystone tenant-list
+----------------------------------+----------+---------+
| id | name | enabled |
+----------------------------------+----------+---------+
| e250f7573010415da6f191e0b53faae5 | admin | True |
| fa30c6bdd56e45dea48dfbe9c3ee8782 | exploit | True |
| dea82f7937064b6da1c370280d8bfdad | services | True |
+----------------------------------+----------+---------+
==============================================================================
Really i didn't see where is my mistake ... can you help me plz ?
Thank you in advance !
Regards,
2014-07-15 15:13 GMT+02:00 Benoit ML <ben42ml at gmail.com>:
> Hello again,
>
> Ok on controller node I modify the neutron server configuration with
> nova_admin_tenant_id = f23ed5be5f534fdba31d23f60621347d
>
> where id is "services" in keystone and now it's working with "vif_plugging_is_fatal
> = True". Good thing.
>
> Well by the way the migrate doesnt working ...
>
>
>
>
> 2014-07-15 14:20 GMT+02:00 Benoit ML <ben42ml at gmail.com>:
>
> Hello,
>>
>> Thank you for taking time !
>>
>> Well on the compute node, when i activate "vif_plugging_is_fatal = True",
>> the vm creation stuck in spawning state, and in neutron server log i have :
>>
>> =======================================
>> 2014-07-15 14:12:52.351 18448 DEBUG neutron.notifiers.nova [-] Sending
>> events: [{'status': 'completed', 'tag':
>> u'ba56921c-4628-4fcc-9dc5-2b324cd91faf', 'name': 'network-vif-plugged',
>> 'server_uuid': u'f8554441-565c-49ec-bc88-3cbc628b0579'}] send_events
>> /usr/lib/python2.7/site-packages/neutron/notifiers/nova.py:218
>> 2014-07-15 14:12:52.354 18448 INFO urllib3.connectionpool [-] Starting
>> new HTTP connection (1): localhost
>> 2014-07-15 14:12:52.360 18448 DEBUG urllib3.connectionpool [-] "POST
>> /v2/5c9c186a909e499e9da0dd5cf2c403e0/os-server-external-events HTTP/1.1"
>> 401 23 _make_request
>> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:295
>> 2014-07-15 14:12:52.362 18448 INFO urllib3.connectionpool [-] Starting
>> new HTTP connection (1): localhost
>> 2014-07-15 14:12:52.452 18448 DEBUG urllib3.connectionpool [-] "POST
>> /v2.0/tokens HTTP/1.1" 401 114 _make_request
>> /usr/lib/python2.7/site-packages/urllib3/connectionpool.py:295
>> 2014-07-15 14:12:52.453 18448 ERROR neutron.notifiers.nova [-] Failed to
>> notify nova on events: [{'status': 'completed', 'tag':
>> u'ba56921c-4628-4fcc-9dc5-2b324cd91faf', 'name': 'network-vif-plugged',
>> 'server_uuid': u'f8554441-565c-49ec-bc88-3cbc628b0579'}]
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova Traceback
>> (most recent call last):
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
>> "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 221, in
>> send_events
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova
>> batched_events)
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
>> "/usr/lib/python2.7/site-packages/novaclient/v1_1/contrib/server_external_events.py",
>> line 39, in create
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova
>> return_raw=True)
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
>> "/usr/lib/python2.7/site-packages/novaclient/base.py", line 152, in _create
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova _resp,
>> body = self.api.client.post(url, body=body)
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
>> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 312, in post
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova return
>> self._cs_request(url, 'POST', **kwargs)
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova File
>> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 301, in
>> _cs_request
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova raise e
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova Unauthorized:
>> Unauthorized (HTTP 401)
>> 2014-07-15 14:12:52.453 18448 TRACE neutron.notifiers.nova
>> 2014-07-15 14:12:58.321 18448 DEBUG neutron.openstack.common.rpc.amqp [-]
>> received {u'_context_roles': [u'admin'], u'_context_request_id':
>> u'req-9bf35c42-3477-4ed3-8092-af729c21198c', u'_context_read_deleted':
>> u'no', u'_context_user_name': None, u'_context_project_name': None,
>> u'namespace': None, u'_context_tenant_id': None, u'args': {u'agent_state':
>> {u'agent_state': {u'topic': u'N/A', u'binary':
>> u'neutron-openvswitch-agent', u'host': u'pvidgsh006.pvi', u'agent_type':
>> u'Open vSwitch agent', u'configurations': {u'tunnel_types': [u'vxlan'],
>> u'tunneling_ip': u'192.168.40.5', u'bridge_mappings': {}, u'l2_population':
>> False, u'devices': 1}}}, u'time': u'2014-07-15T12:12:58.313995'},
>> u'_context_tenant': None, u'_unique_id':
>> u'7c9a4dfcd256494caf6e1327c8051e29', u'_context_is_admin': True,
>> u'version': u'1.0', u'_context_timestamp': u'2014-07-15 12:01:28.190772',
>> u'_context_tenant_name': None, u'_context_user': None, u'_context_user_id':
>> None, u'method': u'report_state', u'_context_project_id': None} _safe_log
>> /usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/common.py:280
>> =======================================
>>
>> Well I'm supposed it's related ... Perhaps with those options in
>> neutron.conf :
>> ======================================
>> notify_nova_on_port_status_changes = True
>> notify_nova_on_port_data_changes = True
>> nova_url = http://localhost:8774/v2
>> nova_admin_tenant_name = services
>> nova_admin_username = nova
>> nova_admin_password = nova
>> nova_admin_auth_url = http://localhost:35357/v2.0
>> ======================================
>>
>> But well didnt see anything wrong ...
>>
>> Thank you in advance !
>>
>> Regards,
>>
>>
>>
>> 2014-07-11 16:08 GMT+02:00 Vimal Kumar <vimal7370 at gmail.com>:
>>
>> -----
>>> File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line
>>> 239, in authenticate\\n content_type="application/json")\\n\', u\' File
>>> "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 163, in
>>> _cs_request\\n raise exceptions.Unauthorized(message=body)\\n\',
>>> u\'Unauthorized: {"error": {"message": "The request you have made requires
>>> authentication.", "code": 401, "title": "Unauthorized"}}\\n\'].\n']
>>> -----
>>>
>>> Looks like HTTP connection to neutron server is resulting in 401 error.
>>>
>>> Try enabling debug mode for neutron server and then tail
>>> /var/log/neutron/server.log , hopefully you should get more info.
>>>
>>>
>>> On Fri, Jul 11, 2014 at 7:13 PM, Benoit ML <ben42ml at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Ok I see. Nova telles neutron/openvswitch to create the bridge qbr
>>>> prior to the migration itself.
>>>> I ve already activate debug and verbose ... But well i'm really stuck,
>>>> dont know how and where to search/look ...
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2014-07-11 15:09 GMT+02:00 Miguel Angel <miguelangel at ajo.es>:
>>>>
>>>> Hi Benoit,
>>>>>
>>>>> A manual virsh migration should fail, because the
>>>>> network ports are not migrated to the destination host.
>>>>>
>>>>> You must investigate on the authentication problem itself,
>>>>> and let nova handle all the underlying API calls which should happen...
>>>>>
>>>>> May be it's worth setting nova.conf to debug=True
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> irc: ajo / mangelajo
>>>>> Miguel Angel Ajo Pelayo
>>>>> +34 636 52 25 69
>>>>> skype: ajoajoajo
>>>>>
>>>>>
>>>>> 2014-07-11 14:41 GMT+02:00 Benoit ML <ben42ml at gmail.com>:
>>>>>
>>>>> Hello,
>>>>>>
>>>>>> cat /etc/redhat-release
>>>>>> CentOS Linux release 7 (Rebuilt from: RHEL 7.0)
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> 2014-07-11 13:40 GMT+02:00 Boris Derzhavets <bderzhavets at hotmail.com>
>>>>>> :
>>>>>>
>>>>>> Could you please post /etc/redhat-release.
>>>>>>>
>>>>>>> Boris.
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> Date: Fri, 11 Jul 2014 11:57:12 +0200
>>>>>>> From: ben42ml at gmail.com
>>>>>>> To: rdo-list at redhat.com
>>>>>>> Subject: [Rdo-list] Icehouse multi-node - Centos7 - live migration
>>>>>>> failed because of "network qbr no such device"
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm working on a multi-node setup of openstack Icehouse using
>>>>>>> centos7.
>>>>>>> Well i have :
>>>>>>> - one controllor node with all server services thing stuff
>>>>>>> - one network node with openvswitch agent, l3-agent, dhcp-agent
>>>>>>> - two compute node with nova-compute and neutron-openvswitch
>>>>>>> - one storage nfs node
>>>>>>>
>>>>>>> NetworkManager is deleted on compute nodes and network node.
>>>>>>>
>>>>>>> My network use is configured to use vxlan. I can create VM,
>>>>>>> tenant-network, external-network, routeur, assign floating-ip to VM, push
>>>>>>> ssh-key into VM, create volume from glance image, etc... Evrything is
>>>>>>> conected and reacheable. Pretty cool :)
>>>>>>>
>>>>>>> But when i try to migrate VM things go wrong ... I have configured
>>>>>>> nova, libvirtd and qemu to use migration through libvirt-tcp.
>>>>>>> I have create and exchanged ssh-key for nova user on all node. I
>>>>>>> have verified userid and groupid of nova.
>>>>>>>
>>>>>>> Well nova-compute log, on the target compute node, :
>>>>>>> 2014-07-11 11:45:02.749 6984 TRACE nova.compute.manager [instance:
>>>>>>> a5326fe1-4faa-4347-ba29-159fce26a85c] RemoteError: Remote error:
>>>>>>> Unauthorized {"error": {"m
>>>>>>> essage": "The request you have made requires authentication.",
>>>>>>> "code": 401, "title": "Unauthorized"}}
>>>>>>>
>>>>>>>
>>>>>>> So well after searching a lots in all logs, i have fount that i cant
>>>>>>> simply migration VM between compute node with a simple virsh :
>>>>>>> virsh migrate instance-00000084 qemu+tcp://<dest>/system
>>>>>>>
>>>>>>> The error is :
>>>>>>> erreur :Cannot get interface MTU on 'qbr3ca65809-05': No such device
>>>>>>>
>>>>>>> Well when i look on the source hyperviseur the bridge "qbr3ca65809"
>>>>>>> exists and have a network tap device. And moreover i manually create
>>>>>>> qbr3ca65809 on the target hypervisor, virsh migrate succed !
>>>>>>>
>>>>>>> Can you help me plz ?
>>>>>>> What can i do wrong ? Perhpas neutron must create the bridge before
>>>>>>> migration but didnt for a mis configuration ?
>>>>>>>
>>>>>>> Plz ask anything you need !
>>>>>>>
>>>>>>> Thank you in advance.
>>>>>>>
>>>>>>>
>>>>>>> The full nova-compute log attached.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> --
>>>>>>> --
>>>>>>> Benoit
>>>>>>>
>>>>>>> _______________________________________________ Rdo-list mailing
>>>>>>> list Rdo-list at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> --
>>>>>> Benoit
>>>>>>
>>>>>> _______________________________________________
>>>>>> Rdo-list mailing list
>>>>>> Rdo-list at redhat.com
>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --
>>>> Benoit
>>>>
>>>> _______________________________________________
>>>> Rdo-list mailing list
>>>> Rdo-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>>
>>>>
>>>
>>
>>
>> --
>> --
>> Benoit
>>
>
>
>
> --
> --
> Benoit
>
--
--
Benoit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20140716/fba175e4/attachment.html>
More information about the dev
mailing list