[Rdo-list] Trying out Neutron Quickstart running into issues with netns (l2 agent and dhcp agent)

Kashyap Chamarthy kchamart at redhat.com
Sun Aug 4 15:47:40 UTC 2013


On 08/04/2013 07:27 PM, Perry Myers wrote:
> Hi,
> 
> I followed the instructions at:
> http://openstack.redhat.com/Neutron-Quickstart
> http://openstack.redhat.com/Running_an_instance_with_Neutron
> 
> I ran this on a RHEL 6.4 VM with latest updates from 6.4.z.  I made sure
> to install the netns enabled kernel from RDO repos and reboot with that
> kernel before running packstack so that I didn't need to reboot the VM
> after the packstack install (and have br-ex disappear)
> 
> The packstack install went without incident.  And I was able to follow
> the launch an instance instructions.
> 
> I noticed that the cirros VM took a long time to get to a login prompt
> on the VNC console.  From looking at the console output it appears that
> the instance was waiting for a dhcp address.

Cirros guests runs a bunch of useful networking commands (there's work to incorporate
similar into Fedora iamges too) for debugging purpose.

You can find the path to your Cirros console log:

   http://kashyapc.wordpress.com/2013/04/06/finding-serial-console-log-of-a-nova-instance/

Cirros images log file gives insight if your guest is recieving DHCP lease requests.

> 
> Once the VNC session got me to a login prompt, I logged in (as the
> cirros user) and confirmed that eth0 did not have an ip address.


In my setup, while debugging with Rhys Oxenham, I noticed we had to explicitly associate
the IP address and route  information (due to [*]). Assuming your private IP network is
30.0.0.x series, many you can try from VNC?


    $ ifconfig eth0 30.0.0.7 netmask 255.255.255.0
    $ route add default gw 30.0.0.1 eth0


[*]  https://bugzilla.redhat.com/show_bug.cgi?id=983672
    - I doubt this will affect the RHEL kernel you're running.

> 
> So, something networking related prevented the instance from getting an
> IP which of course makes ssh'ing into the instance via the floating ip
> later in the instructions not work properly.
> 
> I tried ifup'ing eth0 and dhcp discovers were sent out but not responded to.
> 
> One thing is that on the host running OpenStack services (the VM I ran
> packstack on), I don't see dnsmasq running except for the default
> libvirt network:
> 
>> [admin at rdo-mgmt ~(keystone_demo)]$ ps -ef | grep dnsmas
>> nobody    1968     1  0 08:59 ?        00:00:00 /usr/sbin/dnsmasq --strict-order --local=// --domain-needed --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --bind-interfaces --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override --dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile --addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts
> 
> So... that seems to be a problem :)

Yes, you should have a dnsmasq instance running on the DHCP namespace:

>From my setup, interfaces info inside the DHCP n/w namespace:

    $ ip netns exec qdhcp-4a04382f-03bf-49a9-9d4a-35ab9ffc22ad ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: ns-77ee7ea5-61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
        link/ether fa:16:3e:77:ee:87 brd ff:ff:ff:ff:ff:ff
        inet 30.0.0.3/24 brd 30.0.0.255 scope global ns-77ee7ea5-61
           valid_lft forever preferred_lft forever
        inet6 fe80::f816:3eff:fe77:ee87/64 scope link
           valid_lft forever preferred_lft forever


For reference, dnsmasq instances running on the namespace interface (ns-77ee7ea5-61 in
this case):
=======
  $ ps -ef | grep dnsmasq
    root     26057 30911  0 11:10 pts/0    00:00:00 grep --color=auto dnsmasq
    nobody   29387     1  0 Aug02 ?        00:00:00 dnsmasq --no-hosts --no-resolv
--strict-order --bind-interfaces --interface=ns-77ee7ea5-61 --except-interface=lo
--pid-file=/var/lib/quantum/dhcp/4a04382f-03bf-49a9-9d4a-35ab9ffc22ad/pid
--dhcp-hostsfile=/var/lib/quantum/dhcp/4a04382f-03bf-49a9-9d4a-35ab9ffc22ad/host
--dhcp-optsfile=/var/lib/quantum/dhcp/4a04382f-03bf-49a9-9d4a-35ab9ffc22ad/opts
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro
--dhcp-range=set:tag0,30.0.0.0,static,120s --conf-file= --domain=openstacklocal
    root     29388 29387  0 Aug02 ?        00:00:00 dnsmasq --no-hosts --no-resolv
--strict-order --bind-interfaces --interface=ns-77ee7ea5-61 --except-interface=lo
--pid-file=/var/lib/quantum/dhcp/4a04382f-03bf-49a9-9d4a-35ab9ffc22ad/pid
--dhcp-hostsfile=/var/lib/quantum/dhcp/4a04382f-03bf-49a9-9d4a-35ab9ffc22ad/host
--dhcp-optsfile=/var/lib/quantum/dhcp/4a04382f-03bf-49a9-9d4a-35ab9ffc22ad/opts
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro
--dhcp-range=set:tag0,30.0.0.0,static,120s --conf-file= --domain=openstacklocal
=======



> 
> Just to confirm, I am running the right kernel:
>> [root at rdo-mgmt log(keystone_demo)]# uname -a
>> Linux rdo-mgmt 2.6.32-358.114.1.openstack.el6.x86_64 #1 SMP Wed Jul 3 02:11:25 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
> 
>> [root at rdo-mgmt log(keystone_demo)]# rpm -q iproute kernel
>> iproute-2.6.32-23.el6_4.netns.1.x86_64
>> kernel-2.6.32-358.114.1.openstack.el6.x86_64
> 
>>From quantum server.log:
>> 2013-08-04 09:10:48    ERROR [keystoneclient.common.cms] Verify error: Error opening certificate file /var/lib/quantum/keystone-signing/signing_cert.pem
>> 140222780139336:error:02001002:system library:fopen:No such file or directory:bss_file.c:126:fopen('/var/lib/quantum/keystone-signing/signing_cert.pem','r')
>> 140222780139336:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:129:
>>
>> 2013-08-04 09:10:48    ERROR [keystoneclient.common.cms] Verify error: Error loading file /var/lib/quantum/keystone-signing/cacert.pem
>> 140279285741384:error:02001002:system library:fopen:No such file or directory:bss_file.c:126:fopen('/var/lib/quantum/keystone-signing/cacert.pem','r')
>> 140279285741384:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:129:
>> 140279285741384:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:279:

Do you have the PEM cert file that system call 'fopen' is trying to locate?

   $ file /var/lib/quantum/keystone-signing/cacert.pem

> 
>>From quantum dhcp-agent.log:
> 
>> 2013-08-04 09:08:05    ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response.
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__
>>     data = self._dataqueue.get(timeout=self._timeout)
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get
>>     return waiter.wait()
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait
>>     return get_hub().switch()
>>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
>>     return self.greenlet.switch()
>> Empty
>> 2013-08-04 09:08:05    ERROR [quantum.agent.dhcp_agent] Failed reporting state!
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 702, in _report_state
>>     self.agent_state)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state
>>     topic=self.topic)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
>>     return rpc.call(context, self._get_topic(topic), msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
>>     return _get_impl().call(CONF, context, topic, msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call
>>     rpc_amqp.get_connection_pool(conf, Connection))
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call
>>     rv = list(rv)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__
>>     raise rpc_common.Timeout()
>> Timeout: Timeout while waiting on RPC response.
>> 2013-08-04 09:08:05  WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 56.853869 sec
>> 2013-08-04 09:08:06     INFO [quantum.agent.dhcp_agent] Synchronizing state
>> 2013-08-04 09:32:34    ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp.
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 131, in call_driver
>>     getattr(driver, action)()
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 124, in enable
>>     reuse_existing=True)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 554, in setup
>>     namespace=namespace)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug
>>     ns_dev.link.set_address(mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address
>>     self._as_root('set', self.name, 'address', mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root
>>     kwargs.get('use_root_namespace', False))
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
>>     namespace)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
>>     root_helper=root_helper)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute
>>     raise RuntimeError(m)
>> RuntimeError: 
>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'tap07d8cc77-fc', 'address', 'fa:16:3e:da:66:28']
>> Exit code: 2
>> Stdout: ''
>> Stderr: 'RTNETLINK answers: Device or resource busy\n'
>> 2013-08-04 09:32:36     INFO [quantum.agent.dhcp_agent] Synchronizing state
>> 2013-08-04 09:32:41    ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp.
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 131, in call_driver
>>     getattr(driver, action)()
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 124, in enable
>>     reuse_existing=True)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 554, in setup
>>     namespace=namespace)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug
>>     ns_dev.link.set_address(mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address
>>     self._as_root('set', self.name, 'address', mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root
>>     kwargs.get('use_root_namespace', False))
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
>>     namespace)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
>>     root_helper=root_helper)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute
>>     raise RuntimeError(m)
> 
> The RTNETLINK errors just repeat indefinitely
> 
>>From openvswitch-agent.log:
> 
>> 2013-08-04 09:08:29    ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response.
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__
>>     data = self._dataqueue.get(timeout=self._timeout)
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get
>>     return waiter.wait()
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait
>>     return get_hub().switch()
>>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
>>     return self.greenlet.switch()
>> Empty
>> 2013-08-04 09:08:29    ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed reporting state!
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 201, in _report_state
>>     self.agent_state)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state
>>     topic=self.topic)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
>>     return rpc.call(context, self._get_topic(topic), msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
>>     return _get_impl().call(CONF, context, topic, msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call
>>     rpc_amqp.get_connection_pool(conf, Connection))
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call
>>     rv = list(rv)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__
>>     raise rpc_common.Timeout()
>> Timeout: Timeout while waiting on RPC response.
> 
> Do we have a race condition wrt various Quantum agents connecting to the
> qpid bus that is just generating initial qpid connection error messages
> that can be safely ignored?

Yes - for now I think you can ignore.


Nn my two-node F19 Grizzly w/ Neutron (Quantum) setup (hand-configured), I only see a
couple of occurances of the Timeout message. After

[...]
2013-08-02 08:33:56    ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC
response
.
.
    return _get_impl().call(CONF, context, topic, msg, timeout)
  File "/usr/lib/python2.7/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line
611, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/usr/lib/python2.7/site-packages/quantum/openstack/common/rpc/amqp.py", line 614,
in call
    rv = list(rv)
  File "/usr/lib/python2.7/site-packages/quantum/openstack/common/rpc/amqp.py", line 500,
in __iter__
    raise rpc_common.Timeout()
Timeout: Timeout while waiting on RPC response.
[...]


In my setup, I too noticed a couple of occurances of it. But I made an error, in my
/etc/quantum/quantum.conf on Compute node, it had:

  qpid_hostname=localhost

I changed it to explicit IP address (of the Controller node):

  qpid_hostname=192.168.122.218


> 
> If so, is there any way we can clean this up?
> 
>>From l3-agent.log:
> 
>> 2013-08-04 09:08:06    ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response.
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__
>>     data = self._dataqueue.get(timeout=self._timeout)
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get
>>     return waiter.wait()
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait
>>     return get_hub().switch()
>>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
>>     return self.greenlet.switch()
>> Empty
>> 2013-08-04 09:08:06    ERROR [quantum.agent.l3_agent] Failed reporting state!
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 723, in _report_state
>>     self.agent_state)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state
>>     topic=self.topic)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
>>     return rpc.call(context, self._get_topic(topic), msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
>>     return _get_impl().call(CONF, context, topic, msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call
>>     rpc_amqp.get_connection_pool(conf, Connection))
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call
>>     rv = list(rv)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__
>>     raise rpc_common.Timeout()
>> Timeout: Timeout while waiting on RPC response.
>> 2013-08-04 09:08:06  WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 56.554131 sec
>> 2013-08-04 09:08:10    ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response.
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__
>>     data = self._dataqueue.get(timeout=self._timeout)
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get
>>     return waiter.wait()
>>   File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait
>>     return get_hub().switch()
>>   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
>>     return self.greenlet.switch()
>> Empty
>> 2013-08-04 09:08:10    ERROR [quantum.agent.l3_agent] Failed synchronizing routers
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 637, in _sync_routers_task
>>     context, router_id)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 77, in get_routers
>>     topic=self.topic)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
>>     return rpc.call(context, self._get_topic(topic), msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
>>     return _get_impl().call(CONF, context, topic, msg, timeout)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call
>>     rpc_amqp.get_connection_pool(conf, Connection))
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call
>>     rv = list(rv)
>>   File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__
>>     raise rpc_common.Timeout()
>> Timeout: Timeout while waiting on RPC response.
>> 2013-08-04 09:08:10  WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 20.022704 sec
>> 2013-08-04 09:11:33    ERROR [quantum.agent.l3_agent] Failed synchronizing routers
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 638, in _sync_routers_task
>>     self._process_routers(routers, all_routers=True)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 621, in _process_routers
>>     self.process_router(ri)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 319, in process_router
>>     self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 410, in external_gateway_added
>>     prefix=EXTERNAL_DEV_PREFIX)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug
>>     ns_dev.link.set_address(mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address
>>     self._as_root('set', self.name, 'address', mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root
>>     kwargs.get('use_root_namespace', False))
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
>>     namespace)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
>>     root_helper=root_helper)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute
>>     raise RuntimeError(m)
>> RuntimeError: 
>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-46ed452c-5e', 'address', 'fa:16:3e:e7:d8:30']
>> Exit code: 2
>> Stdout: ''
>> Stderr: 'RTNETLINK answers: Device or resource busy\n'
>> 2013-08-04 09:12:11    ERROR [quantum.agent.l3_agent] Failed synchronizing routers
>> Traceback (most recent call last):
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 638, in _sync_routers_task
>>     self._process_routers(routers, all_routers=True)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 621, in _process_routers
>>     self.process_router(ri)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 319, in process_router
>>     self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 410, in external_gateway_added
>>     prefix=EXTERNAL_DEV_PREFIX)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug
>>     ns_dev.link.set_address(mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address
>>     self._as_root('set', self.name, 'address', mac_address)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root
>>     kwargs.get('use_root_namespace', False))
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
>>     namespace)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
>>     root_helper=root_helper)
>>   File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute
>>     raise RuntimeError(m)
> 
> Same qpid connection issue, which I'm assuming can just be ignored at
> this point.  But also similar device busy errors with creating the
> namespace for the l2 agent

> 
> It appears that the issue with both the l2 agent and the dhcp agent that
> the namespace can't be created, which causes both of them to fail.

I can't pin point the specific issue here, if you prefer, here are my configs


http://kashyapc.fedorapeople.org/virt/openstack/two-node-OpenStack-f19-configs/controller-node-configs/quantum/

And, that's the setup diagram I have (ignore the file name :) )

  http://kashyapc.fedorapeople.org/virt/openstack/namespaces-info-1.txt

(Also, I haven't denoted Compute node in the ascii image. It's just running on a different
VM.)

> 
> Anyone have any thoughts on what to look at next here?
> 
> Perry

-- 
/kashyap




More information about the dev mailing list