6:42 a.m.
Setup: A 2-node install (in virtual machines w/ nested virt) with
IceHouse (Neutron w/ ML2+OVS+GRE) on Fedora 20, but OpenStack IceHouse
packages are from Rawhide (Version details below).
Problem
-------
Attempt to launch a Nova instance as a user tenant results in this trace
back saying "Unexpected vif_type". Interesting thing is, the instance
goes into ACTIVE when I launch the Nova instance with admin tenant.
2014-05-13 07:06:32.123 29455 ERROR nova.compute.manager
[req-402f21c1-98ed-4600-96b9-84efdb9c823d cb68d099e78d490ab0adf4030881153b
0a6eb2259ca142e7a80541db10835e71] [instance: 950de10f-4368-4498-b46a-b1595d057e
38] Error: Unexpected vif_type=binding_failed
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] Traceback (most recent call last):
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in
_build_instance
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] set_access_ip=set_access_ip)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in
decorated_function
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] return function(self, context, *args, **kwargs)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in
_spawn
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] LOG.exception(_('Instance failed to
spawn'), instance=instance)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] six.reraise(self.type_, self.value, self.tb)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in
_spawn
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] block_device_info)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2250, in
spawn
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] write_to_disk=True)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3431, in
to_xml
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] disk_info, rescue, block_device_info)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3247, in
get_guest_config
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] flavor)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 384, in
get_config
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] _("Unexpected vif_type=%s") %
vif_type)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] NovaException: Unexpected vif_type=binding_failed
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38]
2014-05-13 07:06:32.846 29455 ERROR oslo.messaging.rpc.dispatcher [-] Exception during
message handling: Unexpected vif_type=binding_failed
Notes/Observations/diagnostics
------------------------------
- I can reach the inter-webs from the router namespace, but not DHCP
namespace.
- In nova.conf, for 'libvirt_vif_driver', I tried (a) both the below
options, separately, , also I tried commenting it out, an upstream
Nova commit[1] from 4APR2014 marks it as deprecated.
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
- Some diagnostics are here[2]
- From some debugging (and from the diagnoistics above), I guess a
br-tun is missing
Related
-------
I see a related Neutron bug[3], but that's not the root cause of this
bug.
Versions
--------
Nova, Neutron, libvirt, QEMU, OpenvSwitch versions:
openstack-nova-compute-2014.1-2.fc21.noarch
openstack-neutron-2014.1-11.fc21.noarch
libvirt-daemon-kvm-1.1.3.5-1.fc20.x86_64
qemu-system-x86-1.6.2-4.fc20.x86_64
openvswitch-2.0.1-1.fc20.x86_64
[1]
https://git.openstack.org/cgit/openstack/nova/commit/?id=9f6070e194504cc2...
[2] https://gist.github.com/kashyapc/0d4869796c7ea79bfb89
[3] https://bugs.launchpad.net/neutron/+bug/1244255
nova.conf and ml2_conf.ini
--------------------------
nova.conf:
$ cat /etc/nova/nova.conf | grep -v ^$ | grep -v ^#
[DEFAULT]
logdir = /var/log/nova
state_path = /var/lib/nova
lock_path = /var/lib/nova/tmp
volumes_dir = /etc/nova/volumes
dhcpbridge = /usr/bin/nova-dhcpbridge
dhcpbridge_flagfile = /etc/nova/nova.conf
force_dhcp_release = True
injected_network_template = /usr/share/nova/interfaces.template
libvirt_nonblocking = True
libvirt_use_virtio_for_bridges=True
libvirt_inject_partition = -1
#libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
#libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
#iscsi_helper = tgtadm
sql_connection = mysql://nova:nova@192.169.142.97/nova
compute_driver = libvirt.LibvirtDriver
libvirt_type=qemu
rootwrap_config = /etc/nova/rootwrap.conf
auth_strategy = keystone
firewall_driver=nova.virt.firewall.NoopFirewallDriver
enabled_apis = ec2,osapi_compute,metadata
my_ip=192.169.142.168
network_api_class = nova.network.neutronv2.api.API
neutron_url = http://192.169.142.97:9696
neutron_auth_strategy = keystone
neutron_admin_tenant_name = services
neutron_admin_username = neutron
neutron_admin_password = fedora
neutron_admin_auth_url = http://192.169.142.97:35357/v2.0
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
security_group_api = neutron
rpc_backend = nova.rpc.impl_kombu
rabbit_host = 192.169.142.97
rabbit_port = 5672
rabbit_userid = guest
rabbit_password = fedora
glance_host = 192.169.142.97
[keystone_authtoken]
auth_uri = http://192.169.142.97:5000
admin_tenant_name = services
admin_user = nova
admin_password = fedora
auth_host = 192.169.142.97
auth_port = 35357
auth_protocol = http
signing_dirname = /tmp/keystone-signing-nova
ml2 plugin:
$ cat /etc/neutron/plugin.ini | grep -v ^$ | grep -v ^#
[ml2]
type_drivers = gre
tenant_network_types = gre
mechanism_drivers = openvswitch
[ml2_type_flat]
[ml2_type_vlan]
[ml2_type_gre]
tunnel_id_ranges = 1:1000
[ml2_type_vxlan]
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
What am I missing?
I'm still investigating by playing with these config settings
enable_tunneling = True
integration_bridge = br-int
tunnel_bridge = br-tun
bridge_mappings = ens2:br-ex
in ml2_conf.ini
--
/kashyap