Re: [Rdo-list] Trying out Neutron Quickstart running into issues with netns (l2 agent and dhcp agent)

On 08/04/2013 11:27 AM, Perry Myers wrote: > Hi, > > I followed the instructions at: > http://openstack.redhat.com/Neutron-Quickstart > http://openstack.redhat.com/Running_an_instance_with_Neutron > > I ran this on a RHEL 6.4 VM with latest updates from 6.4.z. I made sure > to install the netns enabled kernel from RDO repos and reboot with that > kernel before running packstack so that I didn't need to reboot the VM > after the packstack install (and have br-ex disappear) > > The packstack install went without incident. And I was able to follow > the launch an instance instructions. > > I noticed that the cirros VM took a long time to get to a login prompt > on the VNC console. From looking at the console output it appears that > the instance was waiting for a dhcp address. > > Once the VNC session got me to a login prompt, I logged in (as the > cirros user) and confirmed that eth0 did not have an ip address. > > So, something networking related prevented the instance from getting an > IP which of course makes ssh'ing into the instance via the floating ip > later in the instructions not work properly. > > I tried ifup'ing eth0 and dhcp discovers were sent out but not responded to. > > One thing is that on the host running OpenStack services (the VM I ran > packstack on), I don't see dnsmasq running except for the default > libvirt network: > >> [admin@rdo-mgmt ~(keystone_demo)]$ ps -ef | grep dnsmas >> nobody 1968 1 0 08:59 ? 00:00:00 /usr/sbin/dnsmasq --strict-order --local=// --domain-needed --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --bind-interfaces --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override --dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile --addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts > > So... that seems to be a problem :) > > Just to confirm, I am running the right kernel: >> [root@rdo-mgmt log(keystone_demo)]# uname -a >> Linux rdo-mgmt 2.6.32-358.114.1.openstack.el6.x86_64 #1 SMP Wed Jul 3 02:11:25 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux > >> [root@rdo-mgmt log(keystone_demo)]# rpm -q iproute kernel >> iproute-2.6.32-23.el6_4.netns.1.x86_64 >> kernel-2.6.32-358.114.1.openstack.el6.x86_64 > > From quantum server.log: >> 2013-08-04 09:10:48 ERROR [keystoneclient.common.cms] Verify error: Error opening certificate file /var/lib/quantum/keystone-signing/signing_cert.pem >> 140222780139336:error:02001002:system library:fopen:No such file or directory:bss_file.c:126:fopen('/var/lib/quantum/keystone-signing/signing_cert.pem','r') >> 140222780139336:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:129: >> >> 2013-08-04 09:10:48 ERROR [keystoneclient.common.cms] Verify error: Error loading file /var/lib/quantum/keystone-signing/cacert.pem >> 140279285741384:error:02001002:system library:fopen:No such file or directory:bss_file.c:126:fopen('/var/lib/quantum/keystone-signing/cacert.pem','r') >> 140279285741384:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:129: >> 140279285741384:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:279: > > From quantum dhcp-agent.log: > >> 2013-08-04 09:08:05 ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__ >> data = self._dataqueue.get(timeout=self._timeout) >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get >> return waiter.wait() >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait >> return get_hub().switch() >> File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch >> return self.greenlet.switch() >> Empty >> 2013-08-04 09:08:05 ERROR [quantum.agent.dhcp_agent] Failed reporting state! >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 702, in _report_state >> self.agent_state) >> File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state >> topic=self.topic) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call >> return rpc.call(context, self._get_topic(topic), msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call >> return _get_impl().call(CONF, context, topic, msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call >> rpc_amqp.get_connection_pool(conf, Connection)) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call >> rv = list(rv) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__ >> raise rpc_common.Timeout() >> Timeout: Timeout while waiting on RPC response. >> 2013-08-04 09:08:05 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 56.853869 sec >> 2013-08-04 09:08:06 INFO [quantum.agent.dhcp_agent] Synchronizing state >> 2013-08-04 09:32:34 ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 131, in call_driver >> getattr(driver, action)() >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 124, in enable >> reuse_existing=True) >> File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 554, in setup >> namespace=namespace) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug >> ns_dev.link.set_address(mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address >> self._as_root('set', self.name, 'address', mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root >> kwargs.get('use_root_namespace', False)) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root >> namespace) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute >> root_helper=root_helper) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute >> raise RuntimeError(m) >> RuntimeError: >> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'tap07d8cc77-fc', 'address', 'fa:16:3e:da:66:28'] >> Exit code: 2 >> Stdout: '' >> Stderr: 'RTNETLINK answers: Device or resource busy\n' >> 2013-08-04 09:32:36 INFO [quantum.agent.dhcp_agent] Synchronizing state >> 2013-08-04 09:32:41 ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 131, in call_driver >> getattr(driver, action)() >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 124, in enable >> reuse_existing=True) >> File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 554, in setup >> namespace=namespace) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug >> ns_dev.link.set_address(mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address >> self._as_root('set', self.name, 'address', mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root >> kwargs.get('use_root_namespace', False)) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root >> namespace) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute >> root_helper=root_helper) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute >> raise RuntimeError(m) > > The RTNETLINK errors just repeat indefinitely > > From openvswitch-agent.log: > >> 2013-08-04 09:08:29 ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__ >> data = self._dataqueue.get(timeout=self._timeout) >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get >> return waiter.wait() >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait >> return get_hub().switch() >> File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch >> return self.greenlet.switch() >> Empty >> 2013-08-04 09:08:29 ERROR [quantum.plugins.openvswitch.agent.ovs_quantum_agent] Failed reporting state! >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py", line 201, in _report_state >> self.agent_state) >> File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state >> topic=self.topic) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call >> return rpc.call(context, self._get_topic(topic), msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call >> return _get_impl().call(CONF, context, topic, msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call >> rpc_amqp.get_connection_pool(conf, Connection)) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call >> rv = list(rv) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__ >> raise rpc_common.Timeout() >> Timeout: Timeout while waiting on RPC response. > > Do we have a race condition wrt various Quantum agents connecting to the > qpid bus that is just generating initial qpid connection error messages > that can be safely ignored? > > If so, is there any way we can clean this up? > > From l3-agent.log: > >> 2013-08-04 09:08:06 ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__ >> data = self._dataqueue.get(timeout=self._timeout) >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get >> return waiter.wait() >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait >> return get_hub().switch() >> File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch >> return self.greenlet.switch() >> Empty >> 2013-08-04 09:08:06 ERROR [quantum.agent.l3_agent] Failed reporting state! >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 723, in _report_state >> self.agent_state) >> File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state >> topic=self.topic) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call >> return rpc.call(context, self._get_topic(topic), msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call >> return _get_impl().call(CONF, context, topic, msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call >> rpc_amqp.get_connection_pool(conf, Connection)) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call >> rv = list(rv) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__ >> raise rpc_common.Timeout() >> Timeout: Timeout while waiting on RPC response. >> 2013-08-04 09:08:06 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 56.554131 sec >> 2013-08-04 09:08:10 ERROR [quantum.openstack.common.rpc.amqp] Timed out waiting for RPC response. >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 495, in __iter__ >> data = self._dataqueue.get(timeout=self._timeout) >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 298, in get >> return waiter.wait() >> File "/usr/lib/python2.6/site-packages/eventlet/queue.py", line 129, in wait >> return get_hub().switch() >> File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch >> return self.greenlet.switch() >> Empty >> 2013-08-04 09:08:10 ERROR [quantum.agent.l3_agent] Failed synchronizing routers >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 637, in _sync_routers_task >> context, router_id) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 77, in get_routers >> topic=self.topic) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call >> return rpc.call(context, self._get_topic(topic), msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call >> return _get_impl().call(CONF, context, topic, msg, timeout) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call >> rpc_amqp.get_connection_pool(conf, Connection)) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 614, in call >> rv = list(rv) >> File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 500, in __iter__ >> raise rpc_common.Timeout() >> Timeout: Timeout while waiting on RPC response. >> 2013-08-04 09:08:10 WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 20.022704 sec >> 2013-08-04 09:11:33 ERROR [quantum.agent.l3_agent] Failed synchronizing routers >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 638, in _sync_routers_task >> self._process_routers(routers, all_routers=True) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 621, in _process_routers >> self.process_router(ri) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 319, in process_router >> self.external_gateway_added(ri, ex_gw_port, internal_cidrs) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 410, in external_gateway_added >> prefix=EXTERNAL_DEV_PREFIX) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug >> ns_dev.link.set_address(mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address >> self._as_root('set', self.name, 'address', mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root >> kwargs.get('use_root_namespace', False)) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root >> namespace) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute >> root_helper=root_helper) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute >> raise RuntimeError(m) >> RuntimeError: >> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'set', 'qg-46ed452c-5e', 'address', 'fa:16:3e:e7:d8:30'] >> Exit code: 2 >> Stdout: '' >> Stderr: 'RTNETLINK answers: Device or resource busy\n' >> 2013-08-04 09:12:11 ERROR [quantum.agent.l3_agent] Failed synchronizing routers >> Traceback (most recent call last): >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 638, in _sync_routers_task >> self._process_routers(routers, all_routers=True) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 621, in _process_routers >> self.process_router(ri) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 319, in process_router >> self.external_gateway_added(ri, ex_gw_port, internal_cidrs) >> File "/usr/lib/python2.6/site-packages/quantum/agent/l3_agent.py", line 410, in external_gateway_added >> prefix=EXTERNAL_DEV_PREFIX) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/interface.py", line 181, in plug >> ns_dev.link.set_address(mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 180, in set_address >> self._as_root('set', self.name, 'address', mac_address) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root >> kwargs.get('use_root_namespace', False)) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root >> namespace) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute >> root_helper=root_helper) >> File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute >> raise RuntimeError(m) > > Same qpid connection issue, which I'm assuming can just be ignored at > this point. But also similar device busy errors with creating the > namespace for the l2 agent > > It appears that the issue with both the l2 agent and the dhcp agent that > the namespace can't be created, which causes both of them to fail. > > Anyone have any thoughts on what to look at next here? > > Perry I ran into these issues as well. I noticed that ovs_use_veth was commented out in dhcp_agent.ini and l3_agent.ini. I uncommented them and set them to True and restarted. The vm now has an IP address. I noticed something else peculiar though... the public network.. the one set as the gateway for the router has dhcp enabled. I'm not sure why we would do that.

Cheers, Brent