[Rdo-list] Launching a Nova instance results in "NovaException: Unexpected vif_type=binding_failed"

Tue May 13 11:42:47 UTC 2014

Setup: A 2-node install (in virtual machines w/ nested virt) with
IceHouse (Neutron w/ ML2+OVS+GRE) on Fedora 20, but OpenStack IceHouse
packages are from Rawhide (Version details below).

Problem
-------

Attempt to launch a Nova instance as a user tenant results in this trace
back saying "Unexpected vif_type". Interesting thing is, the instance
goes into ACTIVE when I launch the Nova instance with admin tenant.

  2014-05-13 07:06:32.123 29455 ERROR nova.compute.manager [req-402f21c1-98ed-4600-96b9-84efdb9c823d cb68d099e78d490ab0adf4030881153b 0a6eb2259ca142e7a80541db10835e71] [instance: 950de10f-4368-4498-b46a-b1595d057e
  38] Error: Unexpected vif_type=binding_failed
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38] Traceback (most recent call last):
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     set_access_ip=set_access_ip)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     return function(self, context, *args, **kwargs)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     six.reraise(self.type_, self.value, self.tb)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     block_device_info)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2250, in spawn
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     write_to_disk=True)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3431, in to_xml
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     disk_info, rescue, block_device_info)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3247, in get_guest_config
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     flavor)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 384, in get_config
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38]     _("Unexpected vif_type=%s") % vif_type)
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38] NovaException: Unexpected vif_type=binding_failed
  2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance: 950de10f-4368-4498-b46a-b1595d057e38] 
  2014-05-13 07:06:32.846 29455 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Unexpected vif_type=binding_failed

Notes/Observations/diagnostics
------------------------------

  - I can reach the inter-webs from the router namespace, but not DHCP
    namespace.

  - In nova.conf, for 'libvirt_vif_driver', I tried (a) both the below
    options, separately, , also I tried commenting it out, an upstream
    Nova commit[1] from 4APR2014 marks it as deprecated.

    libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
    libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver

  - Some diagnostics are here[2]

  - From some debugging (and from the diagnoistics above), I guess a
    br-tun is missing

Related
-------

I see a related Neutron bug[3], but that's not the root cause of this
bug.

Versions
--------

Nova, Neutron, libvirt, QEMU, OpenvSwitch versions:

    openstack-nova-compute-2014.1-2.fc21.noarch
    openstack-neutron-2014.1-11.fc21.noarch
    libvirt-daemon-kvm-1.1.3.5-1.fc20.x86_64
    qemu-system-x86-1.6.2-4.fc20.x86_64
    openvswitch-2.0.1-1.fc20.x86_64

[1] https://git.openstack.org/cgit/openstack/nova/commit/?id=9f6070e194504cc2ca2b7f2a2aabbf91c6b81897
[2] https://gist.github.com/kashyapc/0d4869796c7ea79bfb89
[3] https://bugs.launchpad.net/neutron/+bug/1244255

nova.conf and ml2_conf.ini
--------------------------

nova.conf:

  $ cat /etc/nova/nova.conf | grep -v ^$ | grep -v ^#
  [DEFAULT]
  logdir = /var/log/nova
  state_path = /var/lib/nova
  lock_path = /var/lib/nova/tmp
  volumes_dir = /etc/nova/volumes
  dhcpbridge = /usr/bin/nova-dhcpbridge
  dhcpbridge_flagfile = /etc/nova/nova.conf
  force_dhcp_release = True
  injected_network_template = /usr/share/nova/interfaces.template
  libvirt_nonblocking = True
  libvirt_use_virtio_for_bridges=True
  libvirt_inject_partition = -1
  #libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
  #libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
  #iscsi_helper = tgtadm
  sql_connection = mysql://nova:nova@192.169.142.97/nova
  compute_driver = libvirt.LibvirtDriver
  libvirt_type=qemu
  rootwrap_config = /etc/nova/rootwrap.conf
  auth_strategy = keystone
  firewall_driver=nova.virt.firewall.NoopFirewallDriver
  enabled_apis = ec2,osapi_compute,metadata
  my_ip=192.169.142.168
  network_api_class = nova.network.neutronv2.api.API
  neutron_url = http://192.169.142.97:9696
  neutron_auth_strategy = keystone
  neutron_admin_tenant_name = services
  neutron_admin_username = neutron
  neutron_admin_password = fedora
  neutron_admin_auth_url = http://192.169.142.97:35357/v2.0
  linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
  firewall_driver = nova.virt.firewall.NoopFirewallDriver
  security_group_api = neutron
  rpc_backend = nova.rpc.impl_kombu
  rabbit_host = 192.169.142.97
  rabbit_port = 5672
  rabbit_userid = guest
  rabbit_password = fedora
  glance_host = 192.169.142.97
  [keystone_authtoken]
  auth_uri = http://192.169.142.97:5000
  admin_tenant_name = services
  admin_user = nova
  admin_password = fedora
  auth_host = 192.169.142.97
  auth_port = 35357
  auth_protocol = http
  signing_dirname = /tmp/keystone-signing-nova

ml2 plugin:

  $ cat /etc/neutron/plugin.ini | grep -v ^$ | grep -v ^#
  [ml2]
  type_drivers = gre
  tenant_network_types = gre
  mechanism_drivers = openvswitch
  [ml2_type_flat]
  [ml2_type_vlan]
  [ml2_type_gre]
  tunnel_id_ranges = 1:1000
  [ml2_type_vxlan]
  [securitygroup]
  firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
  enable_security_group = True

What am I missing?

I'm still investigating by playing with these config settings

  enable_tunneling = True
  integration_bridge = br-int
  tunnel_bridge = br-tun
  bridge_mappings = ens2:br-ex

in ml2_conf.ini  

-- 
/kashyap