6:42 a.m.
Setup: A 2-node install (in virtual machines w/ nested virt) with
IceHouse (Neutron w/ ML2+OVS+GRE) on Fedora 20, but OpenStack IceHouse
packages are from Rawhide (Version details below).
Problem
-------
Attempt to launch a Nova instance as a user tenant results in this trace
back saying "Unexpected vif_type". Interesting thing is, the instance
goes into ACTIVE when I launch the Nova instance with admin tenant.
2014-05-13 07:06:32.123 29455 ERROR nova.compute.manager
[req-402f21c1-98ed-4600-96b9-84efdb9c823d cb68d099e78d490ab0adf4030881153b
0a6eb2259ca142e7a80541db10835e71] [instance: 950de10f-4368-4498-b46a-b1595d057e
38] Error: Unexpected vif_type=binding_failed
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] Traceback (most recent call last):
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in
_build_instance
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] set_access_ip=set_access_ip)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in
decorated_function
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] return function(self, context, *args, **kwargs)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in
_spawn
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] LOG.exception(_('Instance failed to
spawn'), instance=instance)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] six.reraise(self.type_, self.value, self.tb)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in
_spawn
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] block_device_info)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2250, in
spawn
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] write_to_disk=True)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3431, in
to_xml
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] disk_info, rescue, block_device_info)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3247, in
get_guest_config
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] flavor)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 384, in
get_config
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] _("Unexpected vif_type=%s") %
vif_type)
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38] NovaException: Unexpected vif_type=binding_failed
2014-05-13 07:06:32.123 29455 TRACE nova.compute.manager [instance:
950de10f-4368-4498-b46a-b1595d057e38]
2014-05-13 07:06:32.846 29455 ERROR oslo.messaging.rpc.dispatcher [-] Exception during
message handling: Unexpected vif_type=binding_failed
Notes/Observations/diagnostics
------------------------------
- I can reach the inter-webs from the router namespace, but not DHCP
namespace.
- In nova.conf, for 'libvirt_vif_driver', I tried (a) both the below
options, separately, , also I tried commenting it out, an upstream
Nova commit[1] from 4APR2014 marks it as deprecated.
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
- Some diagnostics are here[2]
- From some debugging (and from the diagnoistics above), I guess a
br-tun is missing
Related
-------
I see a related Neutron bug[3], but that's not the root cause of this
bug.
Versions
--------
Nova, Neutron, libvirt, QEMU, OpenvSwitch versions:
openstack-nova-compute-2014.1-2.fc21.noarch
openstack-neutron-2014.1-11.fc21.noarch
libvirt-daemon-kvm-1.1.3.5-1.fc20.x86_64
qemu-system-x86-1.6.2-4.fc20.x86_64
openvswitch-2.0.1-1.fc20.x86_64
[1]
https://git.openstack.org/cgit/openstack/nova/commit/?id=9f6070e194504cc2...
[2] https://gist.github.com/kashyapc/0d4869796c7ea79bfb89
[3] https://bugs.launchpad.net/neutron/+bug/1244255
nova.conf and ml2_conf.ini
--------------------------
nova.conf:
$ cat /etc/nova/nova.conf | grep -v ^$ | grep -v ^#
[DEFAULT]
logdir = /var/log/nova
state_path = /var/lib/nova
lock_path = /var/lib/nova/tmp
volumes_dir = /etc/nova/volumes
dhcpbridge = /usr/bin/nova-dhcpbridge
dhcpbridge_flagfile = /etc/nova/nova.conf
force_dhcp_release = True
injected_network_template = /usr/share/nova/interfaces.template
libvirt_nonblocking = True
libvirt_use_virtio_for_bridges=True
libvirt_inject_partition = -1
#libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
#libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
#iscsi_helper = tgtadm
sql_connection = mysql://nova:nova@192.169.142.97/nova
compute_driver = libvirt.LibvirtDriver
libvirt_type=qemu
rootwrap_config = /etc/nova/rootwrap.conf
auth_strategy = keystone
firewall_driver=nova.virt.firewall.NoopFirewallDriver
enabled_apis = ec2,osapi_compute,metadata
my_ip=192.169.142.168
network_api_class = nova.network.neutronv2.api.API
neutron_url = http://192.169.142.97:9696
neutron_auth_strategy = keystone
neutron_admin_tenant_name = services
neutron_admin_username = neutron
neutron_admin_password = fedora
neutron_admin_auth_url = http://192.169.142.97:35357/v2.0
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
security_group_api = neutron
rpc_backend = nova.rpc.impl_kombu
rabbit_host = 192.169.142.97
rabbit_port = 5672
rabbit_userid = guest
rabbit_password = fedora
glance_host = 192.169.142.97
[keystone_authtoken]
auth_uri = http://192.169.142.97:5000
admin_tenant_name = services
admin_user = nova
admin_password = fedora
auth_host = 192.169.142.97
auth_port = 35357
auth_protocol = http
signing_dirname = /tmp/keystone-signing-nova
ml2 plugin:
$ cat /etc/neutron/plugin.ini | grep -v ^$ | grep -v ^#
[ml2]
type_drivers = gre
tenant_network_types = gre
mechanism_drivers = openvswitch
[ml2_type_flat]
[ml2_type_vlan]
[ml2_type_gre]
tunnel_id_ranges = 1:1000
[ml2_type_vxlan]
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
What am I missing?
I'm still investigating by playing with these config settings
enable_tunneling = True
integration_bridge = br-int
tunnel_bridge = br-tun
bridge_mappings = ens2:br-ex
in ml2_conf.ini
--
/kashyap
8:26 a.m.
Please be aware of
https://bugs.launchpad.net/neutron/+bug/1303998
Samples of neutron.conf && ml2_conf.ini here
https://bugs.launchpad.net/neutron/+bug/1303998/comments/2
B.
9:30 a.m.
On Tue, May 13, 2014 at 09:26:54AM -0400, Boris Derzhavets wrote:
Please be aware of
https://bugs.launchpad.net/neutron/+bug/1303998
Thanks, I just commented in there[1]. I should note that, I didn't make
the typo error that Phil noted in the bug.
I wonder the issue I see merits a different bug of its own.
[1] https://bugs.launchpad.net/neutron/+bug/1303998/comments/6
Samples of neutron.conf && ml2_conf.ini here
https://bugs.launchpad.net/neutron/+bug/1303998/comments/2
B.
--
/kashyap
12:19 a.m.
To those reading the thread, this is the Nova commit[1] that introduced
this change:
commit 2390857d7ae625dcd18a72b2980f2d862b776623
Author: Dan Smith <dansmith(a)redhat.com>
Date: Wed Feb 19 12:02:45 2014 -0800
Make libvirt wait for neutron to confirm plugging before boot
This makes the libvirt driver use the instance event mechanism to
wait for neutron to confirm that VIF plugging is complete before
actually starting the VM.
Here we introduce a new configuration option of vif_plugging_is_fatal
which allows us to control whether a failure to hear back from
neutron is something that should kill the current operation. In
cases where consoles are not provided for the guest, failing is
the only reasonable action, but if consoles *are* provided, it may
be advantageous to still allow the guest to boot and be accessed
out of band.
In order to properly reflect the neutron failure in the instance's
fault property, this also extends manager to catch and re-raise
the VirtualInterfaceCreateException during spawn(). Also, since
the oslo.messaging merge, we weren't declaring expected exceptions
for run_instance() which this also fixes.
DocImpact: This requires a neutron that is aware of these events
when using nova with neutron, or a configuration that
causes nova not to expect/wait for these events.
Related to blueprint admin-event-callback-api
[1] https://review.openstack.org/#/c/74832/26
--
/kashyap
8:57 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Wed, May 14, 2014 at 10:49:49AM +0530, Kashyap Chamarthy wrote:
To those reading the thread, this is the Nova commit[1] that
introduced
this change:
/me thinks, this is tracked in
https://bugzilla.redhat.com/show_bug.cgi?id=1090605
Matthias
- --
Matthias Runge <mrunge(a)redhat.com>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJTc3YyAAoJEOnz8qQwcaIWSZEIAJxQg/56kKH/Ki0e8EFOBx1b
rdqZNsW03XUw7MWcM0TEy37L9VaIG9GDKTYjQS3P4xhGo0XouRSXLhINvFr9tYlU
pPLb9FTelHKVdo7IMsxNYxVo5q5BWnaF5bCreRIGs5samN1JAyAnDwskMEtuNs1z
IPWhT3ICm8oP9vhm9jkeEPHlSBhd8vA6B0cH345caTutQN8yuog3vIV1hGCdhRMB
AUjClHnvuUQvJztl6tZIO0oy/UT+LAs/jvyZKpuXPn5ixnCVV/68JLIkUXFtgrj+
8s3JISMuc00YO2qy/2PbwFj1yZzfXRmAfbLNDcKpsl3eG0HEnSObZ8C2i+GRm6k=
=/iOJ
-----END PGP SIGNATURE-----
11:37 a.m.
Trace backs in https://www.redhat.com/archives/rdo-list/2014-May/msg00060.html
and mentioned by you ( https://bugzilla.redhat.com/show_bug.cgi?id=1090605 )
seem to me to be different.
Also Kashyap wrote :-
Attempt to launch a Nova instance as a user tenant results in this trace
back saying "Unexpected vif_type". Interesting thing is, the instance
goes into ACTIVE when I launch the Nova instance with admin tenant.
I also handle a thread at ask.openstack.org with person on Multi Node Neutron
ML2&OVS&GRE Setup on Ubuntu 14.04, experiencing same problem as was described by
Kashyap. I just cannot have him to reproduce launching as admin tenant case.
B.
/me thinks, this is tracked in
https://bugzilla.redhat.com/show_bug.cgi?id=1090605
Matthias
- --
Rdo-list mailing list
Rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
12:45 p.m.
New subject: [Rdo-list] [RESOLVED] Re: Launching a Nova instance results in "NovaException: Unexpected vif_type=binding_failed"
Ater a bit (lot) of debugging (thanks to Attila Fazekas), I was able to
boot a Nova instance and get it aquire DHCP lease.
A few things I did that have fixed the non-booting of a guest w/ user
tenant:
- Update ML2 config (specifically, update [agent] and [ovs] sections):
------
$ cat plugins/ml2/ml2_conf.ini | grep -v ^$ | grep -v ^#
[ml2]
type_drivers = gre
tenant_network_types = gre
mechanism_drivers = openvswitch
[ml2_type_flat]
[ml2_type_vlan]
[ml2_type_gre]
tunnel_id_ranges = 1:1000
[ml2_type_vxlan]
[securitygroup]
firewall_driver =
neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
[ovs]
local_ip = 192.169.142.97
[agent]
tunnel_types = gre
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
------
This change brought in the 'br-tun' interface that I was previously
missing.
- Copy contents of `/etc/neutron/plugins/ml2/ml2_conf.ini`
into `/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini`.
(Thanks Attila Fazekas for this tip, he mentioned upstream Devstack
does this, not sure of the rationale).
- Use these config attributes in nova.conf:
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
vif_plugging_is_fatal = False
vif_plugging_timeout = 0
- Add "agent_down_time = 75" to [DEFAULT] section of neutron.conf and
"report_interval = 5" to [agent] section.
Finally, here are my Neutron and Nova configuration details[1] for both
Control and Compute nodes for reference.
Hope this helps for someone who may hit similar issues.
[1]
http://kashyapc.fedorapeople.org/virt/openstack/rdo/IceHouse-Nova-Neutron...
--
/kashyap
3644
days inactive
3645
days old
6 comments
3 participants
participants (3)
-
Boris Derzhavets -
Kashyap Chamarthy -
Matthias Runge