Yeah, I made the changes and then recreated all the networks. For some reason br-int and the individual virtual instance interfaces on the compute node still show 1500 byte frames. Has anyone else configured jumbo frames in a Kilo environment? Or maybe I'm just an outlier... ;) -erich On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote: > Hi Erich, > > did you recreate the neutron networks after the configuration changes? > > Pedro Navarro Pérez > OpenStack product specialist > Red Hat Iberia > Passeig de Gràcia 120, > 08008 Barcelona > Spain > M +34 639 642 379 > E pnavarro(a)redhat.com > > ----- Original Message ----- > From: "Erich Weiler" <weiler(a)soe.ucsc.edu&gt; > To: rdo-list(a)redhat.com > Sent: Wednesday, 7 October, 2015 2:34:28 AM > Subject: [Rdo-list] Jumbo MTU to instances in Kilo? > > Hi Y'all, > > I know someone must have figured this one out, but I can't seem to get > 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes have > MTU=9000 on their interfaces, so does the network node. dnsmasq also is > configured to set MTU=9000 on instances, which works. But I still can't > ping with large packets to my instance: > > [weiler@stacker ~]$ ping 10.50.100.2 > PING 10.50.100.2 (10.50.100.2) 56(84) bytes of data. > 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms > 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms > 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms > > That works fine. This however doesn't work: > > [root@stacker ~]# ping -M do -s 8000 10.50.100.2 > PING 10.50.100.2 (10.50.100.2) 8000(8028) bytes of data. > From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500) > ping: local error: Message too long, mtu=1500 > ping: local error: Message too long, mtu=1500 > ping: local error: Message too long, mtu=1500 > ping: local error: Message too long, mtu=1500 > > It looks like somehow the br-int interface for OVS isn't set at 9000, > but I can't figure out how to do that... > > Here's ifconfig on my compute node: > > br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid > 0x20<link> > ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) > RX packets 2401432 bytes 359276713 (342.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 30 bytes 1572 (1.5 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid > 0x20<link> > ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) > RX packets 69 bytes 6866 (6.7 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 8 bytes 648 (648.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 > inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid > 0x20<link> > ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) > RX packets 130174458 bytes 15334807929 (14.2 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 22919305 bytes 5859090420 (5.4 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 > inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid > 0x20<link> > ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) > RX packets 38429352 bytes 5152853436 (4.7 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 419842 bytes 101161981 (96.4 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > inet 127.0.0.1 netmask 255.0.0.0 > inet6 ::1 prefixlen 128 scopeid 0x10<host> > loop txqueuelen 0 (Local Loopback) > RX packets 22141566 bytes 1185622090 (1.1 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 22141566 bytes 1185622090 (1.1 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid > 0x20<link> > ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet) > RX packets 16 bytes 1472 (1.4 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 8 bytes 648 (648.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid > 0x20<link> > ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) > RX packets 15 bytes 1456 (1.4 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 8 bytes 648 (648.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu 1500 > inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid > 0x20<link> > ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet) > RX packets 247 bytes 28323 (27.6 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 233 bytes 25355 (24.7 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu 1500 > inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid > 0x20<link> > ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) > RX packets 377 bytes 57664 (56.3 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 333 bytes 38765 (37.8 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu 1500 > inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid > 0x20<link> > ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet) > RX packets 233 bytes 25355 (24.7 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 247 bytes 28323 (27.6 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu 1500 > inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid > 0x20<link> > ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) > RX packets 333 bytes 38765 (37.8 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 377 bytes 57664 (56.3 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid > 0x20<link> > ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet) > RX packets 219 bytes 24239 (23.6 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 224 bytes 26661 (26.0 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 > inet 192.168.122.1 netmask 255.255.255.0 broadcast > 192.168.122.255 > ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > This is on RHEL 7.1. Any obvious way I can get all the intermediate > bridges to MTU=9000? I've RTFM'd and googled to no avail... > > Here's the ovs-vsctl outout: > > [root@node-136 ~]# ovs-vsctl show > 6f5a5f00-59e2-4420-aeaf-7ad464ead232 > Bridge br-int > fail_mode: secure > Port br-int > Interface br-int > type: internal > Port "qvo247da3ed-a4" > tag: 1 > Interface "qvo247da3ed-a4" > Port "int-br-eth1" > Interface "int-br-eth1" > Port "int-br-enp3s0f0" > Interface "int-br-enp3s0f0" > type: patch > options: {peer="phy-br-enp3s0f0"} > Bridge "br-enp3s0f0" > Port "enp3s0f0" > Interface "enp3s0f0" > Port "br-enp3s0f0" > Interface "br-enp3s0f0" > type: internal > Port "phy-br-enp3s0f0" > Interface "phy-br-enp3s0f0" > type: patch > options: {peer="int-br-enp3s0f0"} > ovs_version: "2.3.1" > > Many thanks if anyone has any information on this topic! Or can point > me to some documentation I missed... > > Thanks, > erich > > _______________________________________________ > Rdo-list mailing list > Rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >

Actually I think I'm closer - on the compute nodes, I set this in nova.conf: network_device_mtu=9000 even though there was a big note above it that said not to use it because this option was deprecated. But after setting that option, and restarting nova and openvswitch, br-int, my tap device and my qvb device all got set to MTU=9000. So I'm closer! But still one item is blocking me. I show this tracepath from my controller node direct to the VM (which is on a compute node on the local network): # tracepath 10.50.100.4 1?: [LOCALHOST] pmtu 9000 1: 10.50.100.4 0.682ms 1: 10.50.100.4 0.241ms 2: 10.50.100.4 0.297ms pmtu 1500 2: 10.50.100.4 1.664ms reached 10.50.100.4 is the VM. It looks like the path is jumbo clean up until that third hop. But the thing is, I don't know what the third hop is. ;) On my compute node I still see some stuff with MTU=1500, but I'm not sure if one of those is blocking me: # ifconfig br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) RX packets 2401498 bytes 359284253 (342.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 30 bytes 1572 (1.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid 0x20<link> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) RX packets 133 bytes 12934 (12.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) RX packets 165957142 bytes 20333410092 (18.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 23299881 bytes 5950708819 (5.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) RX packets 6014767 bytes 813880745 (776.1 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 79301 bytes 19052451 (18.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 22462729 bytes 1202484822 (1.1 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 22462729 bytes 1202484822 (1.1 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::4c1a:55ff:feba:14c3 prefixlen 64 scopeid 0x20<link> ether 56:a6:a6:db:83:c4 txqueuelen 0 (Ethernet) RX packets 16 bytes 1520 (1.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid 0x20<link> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) RX packets 15 bytes 1456 (1.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 inet6 fe80::54a6:a6ff:fedb:83c4 prefixlen 64 scopeid 0x20<link> ether 56:a6:a6:db:83:c4 txqueuelen 1000 (Ethernet) RX packets 86 bytes 9610 (9.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 133 bytes 12767 (12.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid 0x20<link> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) RX packets 377 bytes 57664 (56.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 333 bytes 38765 (37.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 inet6 fe80::b44a:bff:fe72:aaea prefixlen 64 scopeid 0x20<link> ether b6:4a:0b:72:aa:ea txqueuelen 1000 (Ethernet) RX packets 133 bytes 12767 (12.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 86 bytes 9610 (9.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid 0x20<link> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) RX packets 333 bytes 38765 (37.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 377 bytes 57664 (56.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::fc16:3eff:fefa:9945 prefixlen 64 scopeid 0x20<link> ether fe:16:3e:fa:99:45 txqueuelen 500 (Ethernet) RX packets 118 bytes 11561 (11.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 95 bytes 10316 (10.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 My network node has all interfaces set to MTU=9000. I thought maybe the bottleneck might be there but I don't think it is. Here's ifconfig from my network node: # ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 2042 bytes 238727 (233.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2042 bytes 238727 (233.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::207:43ff:fe10:deb8 prefixlen 64 scopeid 0x20<link> ether 00:07:43:10:de:b8 txqueuelen 1000 (Ethernet) RX packets 2156053308 bytes 325330839639 (302.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 223004 bytes 24769304 (23.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 72 p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.50.1.51 netmask 255.255.0.0 broadcast 10.50.255.255 inet6 fe80::260:ddff:fe44:2aea prefixlen 64 scopeid 0x20<link> ether 00:60:dd:44:2a:ea txqueuelen 1000 (Ethernet) RX packets 49352916 bytes 3501547231 (3.2 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 18876911 bytes 3768900461 (3.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::260:ddff:fe44:2aeb prefixlen 64 scopeid 0x20<link> ether 00:60:dd:44:2a:eb txqueuelen 1000 (Ethernet) RX packets 2491224974 bytes 348058319500 (324.1 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1597 bytes 204525 (199.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Any way I can figure out what the third hop is from my tracepath? Thanks as always for the sage advice! -erich On 10/07/2015 09:57 AM, Erich Weiler wrote: > Yeah, I made the changes and then recreated all the networks. For > some reason br-int and the individual virtual instance interfaces on > the compute node still show 1500 byte frames. > > Has anyone else configured jumbo frames in a Kilo environment? Or > maybe I'm just an outlier... ;) > > -erich > > On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote: >> Hi Erich, >> >> did you recreate the neutron networks after the configuration changes? >> >> Pedro Navarro Pérez >> OpenStack product specialist >> Red Hat Iberia >> Passeig de Gràcia 120, >> 08008 Barcelona >> Spain >> M +34 639 642 379 >> E pnavarro(a)redhat.com >> >> ----- Original Message ----- >> From: "Erich Weiler" <weiler(a)soe.ucsc.edu&gt; >> To: rdo-list(a)redhat.com >> Sent: Wednesday, 7 October, 2015 2:34:28 AM >> Subject: [Rdo-list] Jumbo MTU to instances in Kilo? >> >> Hi Y'all, >> >> I know someone must have figured this one out, but I can't seem to >> get >> 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes >> have >> MTU=9000 on their interfaces, so does the network node. dnsmasq >> also is configured to set MTU=9000 on instances, which works. But I >> still can't ping with large packets to my instance: >> >> [weiler@stacker ~]$ ping 10.50.100.2 PING 10.50.100.2 (10.50.100.2) >> 56(84) bytes of data. >> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms >> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms >> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms >> >> That works fine. This however doesn't work: >> >> [root@stacker ~]# ping -M do -s 8000 10.50.100.2 PING 10.50.100.2 >> (10.50.100.2) 8000(8028) bytes of data. >> From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500) >> ping: local error: Message too long, mtu=1500 >> ping: local error: Message too long, mtu=1500 >> ping: local error: Message too long, mtu=1500 >> ping: local error: Message too long, mtu=1500 >> >> It looks like somehow the br-int interface for OVS isn't set at >> 9000, but I can't figure out how to do that... >> >> Here's ifconfig on my compute node: >> >> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >> 0x20<link> >> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >> RX packets 2401432 bytes 359276713 (342.6 MiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 30 bytes 1572 (1.5 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid >> 0x20<link> >> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) >> RX packets 69 bytes 6866 (6.7 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 8 bytes 648 (648.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 >> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >> 0x20<link> >> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) >> RX packets 130174458 bytes 15334807929 (14.2 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 22919305 bytes 5859090420 (5.4 GiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 >> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >> 0x20<link> >> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >> RX packets 38429352 bytes 5152853436 (4.7 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 419842 bytes 101161981 (96.4 MiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> inet 127.0.0.1 netmask 255.0.0.0 >> inet6 ::1 prefixlen 128 scopeid 0x10<host> >> loop txqueuelen 0 (Local Loopback) >> RX packets 22141566 bytes 1185622090 (1.1 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 22141566 bytes 1185622090 (1.1 GiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid >> 0x20<link> >> ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet) >> RX packets 16 bytes 1472 (1.4 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 8 bytes 648 (648.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid >> 0x20<link> >> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) >> RX packets 15 bytes 1456 (1.4 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 8 bytes 648 (648.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid >> 0x20<link> >> ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet) >> RX packets 247 bytes 28323 (27.6 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 233 bytes 25355 (24.7 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid >> 0x20<link> >> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) >> RX packets 377 bytes 57664 (56.3 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 333 bytes 38765 (37.8 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid >> 0x20<link> >> ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet) >> RX packets 233 bytes 25355 (24.7 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 247 bytes 28323 (27.6 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid >> 0x20<link> >> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) >> RX packets 333 bytes 38765 (37.8 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 377 bytes 57664 (56.3 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid >> 0x20<link> >> ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet) >> RX packets 219 bytes 24239 (23.6 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 224 bytes 26661 (26.0 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >> inet 192.168.122.1 netmask 255.255.255.0 broadcast >> 192.168.122.255 >> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) >> RX packets 0 bytes 0 (0.0 B) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 0 bytes 0 (0.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> This is on RHEL 7.1. Any obvious way I can get all the intermediate >> bridges to MTU=9000? I've RTFM'd and googled to no avail... >> >> Here's the ovs-vsctl outout: >> >> [root@node-136 ~]# ovs-vsctl show >> 6f5a5f00-59e2-4420-aeaf-7ad464ead232 >> Bridge br-int >> fail_mode: secure >> Port br-int >> Interface br-int >> type: internal >> Port "qvo247da3ed-a4" >> tag: 1 >> Interface "qvo247da3ed-a4" >> Port "int-br-eth1" >> Interface "int-br-eth1" >> Port "int-br-enp3s0f0" >> Interface "int-br-enp3s0f0" >> type: patch >> options: {peer="phy-br-enp3s0f0"} >> Bridge "br-enp3s0f0" >> Port "enp3s0f0" >> Interface "enp3s0f0" >> Port "br-enp3s0f0" >> Interface "br-enp3s0f0" >> type: internal >> Port "phy-br-enp3s0f0" >> Interface "phy-br-enp3s0f0" >> type: patch >> options: {peer="int-br-enp3s0f0"} >> ovs_version: "2.3.1" >> >> Many thanks if anyone has any information on this topic! Or can >> point me to some documentation I missed... >> >> Thanks, >> erich >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>

Eric, also, to set jumbo frames on your env, you have to set mtu from VM to controller: # echo "dhcp-option-force=26,8900" > /etc/neutron/dnsmasq-neutron.conf # openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf # openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent veth_mtu 8900 # openstack-config --set /etc/neutron/l3_agent.ini DEFAULT network_device_mtu 9000 # openstack-config --set /etc/nova/nova.conf DEFAULT network_device_mtu 9000 <--- this on every nova-compute take a look at l3_agent.ini file, without network_device_mtu every new router will use default mtu at 1500 # ip netns exec qrouter-26f64a08-52ab-4643-b903-9aea6eae047a /bin/bash # ip a | grep mtu 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 69: ha-89546945-ab: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN 74: qr-f207f652-da: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN 81: qg-ab978cd0-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN HTH Amedeo -----Messaggio originale----- Da: rdo-list-bounces(a)redhat.com [mailto:rdo-list-bounces@redhat.com] Per conto di Erich Weiler Inviato: giovedì 8 ottobre 2015 01:53 A: Pedro Navarro Perez Cc: rdo-list(a)redhat.com Oggetto: Re: [Rdo-list] Jumbo MTU to instances in Kilo? Actually I was wrong, it WAS on the network node. The virtual router interfaces were not set to MTU=9000. On network node: [root@os-net-01 ~]# ip netns qdhcp-c395cff9-af7b-4456-91e3-3c55e6c2c5f5 qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 i[root@os-net-01 ~]# ip netns exec qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qg-fa1e2a28-25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.50.100.1 netmask 255.255.0.0 broadcast 10.50.255.255 inet6 fe80::f816:3eff:fe6a:608b prefixlen 64 scopeid 0x20<link> ether fa:16:3e:6a:60:8b txqueuelen 0 (Ethernet) RX packets 34071065 bytes 5046408745 (4.6 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 442 bytes 51915 (50.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qr-51904c89-b8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.100.0.1 netmask 255.255.0.0 broadcast 10.100.255.255 inet6 fe80::f816:3eff:fe37:eca6 prefixlen 64 scopeid 0x20<link> ether fa:16:3e:37:ec:a6 txqueuelen 0 (Ethernet) RX packets 702 bytes 75369 (73.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 814 bytes 92259 (90.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 I can fix it manually: [root@os-net-01 neutron]# ip netns exec qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig qg-fa1e2a28-25 mtu 9000 [root@os-net-01 neutron]# ip netns exec qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig qr-51904c89-b8 mtu 9000 [root@os-net-01 neutron]# ip netns exec qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qg-fa1e2a28-25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.50.100.1 netmask 255.255.0.0 broadcast 10.50.255.255 inet6 fe80::f816:3eff:fe6a:608b prefixlen 64 scopeid 0x20<link> ether fa:16:3e:6a:60:8b txqueuelen 0 (Ethernet) RX packets 34086053 bytes 5048637833 (4.7 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 442 bytes 51915 (50.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qr-51904c89-b8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.100.0.1 netmask 255.255.0.0 broadcast 10.100.255.255 inet6 fe80::f816:3eff:fe37:eca6 prefixlen 64 scopeid 0x20<link> ether fa:16:3e:37:ec:a6 txqueuelen 0 (Ethernet) RX packets 702 bytes 75369 (73.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 814 bytes 92259 (90.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 And then I have a jumbo clean path everywhere! All is good then. But... How to set this in a config file or something so I don't have to do it manually? I found this bug report: https://bugs.launchpad.net/neutron/+bug/1311097 Anyone know if that bug is still out there? Or how can I set virtual router interfaces MTU by default when I create the router? cheers, erich On 10/07/2015 04:35 PM, Erich Weiler wrote: > Actually I think I'm closer - on the compute nodes, I set this in > nova.conf: > > network_device_mtu=9000 > > even though there was a big note above it that said not to use it > because this option was deprecated. But after setting that option, > and restarting nova and openvswitch, br-int, my tap device and my qvb > device all got set to MTU=9000. So I'm closer! But still one item > is blocking me. I show this tracepath from my controller node direct > to the VM (which is on a compute node on the local network): > > # tracepath 10.50.100.4 > 1?: [LOCALHOST] pmtu 9000 > 1: 10.50.100.4 0.682ms > 1: 10.50.100.4 0.241ms > 2: 10.50.100.4 0.297ms pmtu > 1500 > 2: 10.50.100.4 1.664ms reached > > 10.50.100.4 is the VM. It looks like the path is jumbo clean up > until that third hop. But the thing is, I don't know what the third hop is. > ;) > > On my compute node I still see some stuff with MTU=1500, but I'm not > sure if one of those is blocking me: > > # ifconfig > br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> > ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) > RX packets 2401498 bytes 359284253 (342.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 30 bytes 1572 (1.5 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid 0x20<link> > ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) > RX packets 133 bytes 12934 (12.6 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 8 bytes 648 (648.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 > inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> > ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) > RX packets 165957142 bytes 20333410092 (18.9 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 23299881 bytes 5950708819 (5.5 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 > inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> > ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) > RX packets 6014767 bytes 813880745 (776.1 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 79301 bytes 19052451 (18.1 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > inet 127.0.0.1 netmask 255.0.0.0 > inet6 ::1 prefixlen 128 scopeid 0x10<host> > loop txqueuelen 0 (Local Loopback) > RX packets 22462729 bytes 1202484822 (1.1 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 22462729 bytes 1202484822 (1.1 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::4c1a:55ff:feba:14c3 prefixlen 64 scopeid 0x20<link> > ether 56:a6:a6:db:83:c4 txqueuelen 0 (Ethernet) > RX packets 16 bytes 1520 (1.4 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 8 bytes 648 (648.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid 0x20<link> > ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) > RX packets 15 bytes 1456 (1.4 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 8 bytes 648 (648.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu > 9000 > inet6 fe80::54a6:a6ff:fedb:83c4 prefixlen 64 scopeid 0x20<link> > ether 56:a6:a6:db:83:c4 txqueuelen 1000 (Ethernet) > RX packets 86 bytes 9610 (9.3 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 133 bytes 12767 (12.4 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu > 1500 > inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid 0x20<link> > ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) > RX packets 377 bytes 57664 (56.3 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 333 bytes 38765 (37.8 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu > 9000 > inet6 fe80::b44a:bff:fe72:aaea prefixlen 64 scopeid 0x20<link> > ether b6:4a:0b:72:aa:ea txqueuelen 1000 (Ethernet) > RX packets 133 bytes 12767 (12.4 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 86 bytes 9610 (9.3 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> > mtu > 1500 > inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid 0x20<link> > ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) > RX packets 333 bytes 38765 (37.8 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 377 bytes 57664 (56.3 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::fc16:3eff:fefa:9945 prefixlen 64 scopeid 0x20<link> > ether fe:16:3e:fa:99:45 txqueuelen 500 (Ethernet) > RX packets 118 bytes 11561 (11.2 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 95 bytes 10316 (10.0 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 > inet 192.168.122.1 netmask 255.255.255.0 broadcast > 192.168.122.255 > ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > My network node has all interfaces set to MTU=9000. I thought maybe the > bottleneck might be there but I don't think it is. Here's ifconfig > from my network node: > > # ifconfig > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > inet 127.0.0.1 netmask 255.0.0.0 > inet6 ::1 prefixlen 128 scopeid 0x10<host> > loop txqueuelen 0 (Local Loopback) > RX packets 2042 bytes 238727 (233.1 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 2042 bytes 238727 (233.1 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::207:43ff:fe10:deb8 prefixlen 64 scopeid 0x20<link> > ether 00:07:43:10:de:b8 txqueuelen 1000 (Ethernet) > RX packets 2156053308 bytes 325330839639 (302.9 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 223004 bytes 24769304 (23.6 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 72 > > p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet 10.50.1.51 netmask 255.255.0.0 broadcast 10.50.255.255 > inet6 fe80::260:ddff:fe44:2aea prefixlen 64 scopeid 0x20<link> > ether 00:60:dd:44:2a:ea txqueuelen 1000 (Ethernet) > RX packets 49352916 bytes 3501547231 (3.2 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 18876911 bytes 3768900461 (3.5 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 > inet6 fe80::260:ddff:fe44:2aeb prefixlen 64 scopeid 0x20<link> > ether 00:60:dd:44:2a:eb txqueuelen 1000 (Ethernet) > RX packets 2491224974 bytes 348058319500 (324.1 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 1597 bytes 204525 (199.7 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > Any way I can figure out what the third hop is from my tracepath? > > Thanks as always for the sage advice! > > -erich > > On 10/07/2015 09:57 AM, Erich Weiler wrote: >> Yeah, I made the changes and then recreated all the networks. For >> some reason br-int and the individual virtual instance interfaces on >> the compute node still show 1500 byte frames. >> >> Has anyone else configured jumbo frames in a Kilo environment? Or >> maybe I'm just an outlier... ;) >> >> -erich >> >> On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote: >>> Hi Erich, >>> >>> did you recreate the neutron networks after the configuration changes? >>> >>> Pedro Navarro Pérez >>> OpenStack product specialist >>> Red Hat Iberia >>> Passeig de Gràcia 120, >>> 08008 Barcelona >>> Spain >>> M +34 639 642 379 >>> E pnavarro(a)redhat.com >>> >>> ----- Original Message ----- >>> From: "Erich Weiler" <weiler(a)soe.ucsc.edu&gt; >>> To: rdo-list(a)redhat.com >>> Sent: Wednesday, 7 October, 2015 2:34:28 AM >>> Subject: [Rdo-list] Jumbo MTU to instances in Kilo? >>> >>> Hi Y'all, >>> >>> I know someone must have figured this one out, but I can't seem to >>> get >>> 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes >>> have >>> MTU=9000 on their interfaces, so does the network node. dnsmasq >>> also is configured to set MTU=9000 on instances, which works. But >>> I still can't ping with large packets to my instance: >>> >>> [weiler@stacker ~]$ ping 10.50.100.2 PING 10.50.100.2 (10.50.100.2) >>> 56(84) bytes of data. >>> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms >>> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms >>> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms >>> >>> That works fine. This however doesn't work: >>> >>> [root@stacker ~]# ping -M do -s 8000 10.50.100.2 PING 10.50.100.2 >>> (10.50.100.2) 8000(8028) bytes of data. >>> From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500) >>> ping: local error: Message too long, mtu=1500 >>> ping: local error: Message too long, mtu=1500 >>> ping: local error: Message too long, mtu=1500 >>> ping: local error: Message too long, mtu=1500 >>> >>> It looks like somehow the br-int interface for OVS isn't set at >>> 9000, but I can't figure out how to do that... >>> >>> Here's ifconfig on my compute node: >>> >>> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >>> 0x20<link> >>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >>> RX packets 2401432 bytes 359276713 (342.6 MiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 30 bytes 1572 (1.5 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid >>> 0x20<link> >>> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) >>> RX packets 69 bytes 6866 (6.7 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 8 bytes 648 (648.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 >>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >>> 0x20<link> >>> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) >>> RX packets 130174458 bytes 15334807929 (14.2 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 22919305 bytes 5859090420 (5.4 GiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 >>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >>> 0x20<link> >>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >>> RX packets 38429352 bytes 5152853436 (4.7 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 419842 bytes 101161981 (96.4 MiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >>> inet 127.0.0.1 netmask 255.0.0.0 >>> inet6 ::1 prefixlen 128 scopeid 0x10<host> >>> loop txqueuelen 0 (Local Loopback) >>> RX packets 22141566 bytes 1185622090 (1.1 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 22141566 bytes 1185622090 (1.1 GiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid >>> 0x20<link> >>> ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet) >>> RX packets 16 bytes 1472 (1.4 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 8 bytes 648 (648.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid >>> 0x20<link> >>> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) >>> RX packets 15 bytes 1456 (1.4 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 8 bytes 648 (648.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu 1500 >>> inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid >>> 0x20<link> >>> ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet) >>> RX packets 247 bytes 28323 (27.6 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 233 bytes 25355 (24.7 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu 1500 >>> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid >>> 0x20<link> >>> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) >>> RX packets 377 bytes 57664 (56.3 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 333 bytes 38765 (37.8 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu 1500 >>> inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid >>> 0x20<link> >>> ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet) >>> RX packets 233 bytes 25355 (24.7 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 247 bytes 28323 (27.6 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu 1500 >>> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid >>> 0x20<link> >>> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) >>> RX packets 333 bytes 38765 (37.8 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 377 bytes 57664 (56.3 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid >>> 0x20<link> >>> ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet) >>> RX packets 219 bytes 24239 (23.6 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 224 bytes 26661 (26.0 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >>> inet 192.168.122.1 netmask 255.255.255.0 broadcast >>> 192.168.122.255 >>> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) >>> RX packets 0 bytes 0 (0.0 B) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 0 bytes 0 (0.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>> 0 >>> >>> This is on RHEL 7.1. Any obvious way I can get all the >>> intermediate bridges to MTU=9000? I've RTFM'd and googled to no avail... >>> >>> Here's the ovs-vsctl outout: >>> >>> [root@node-136 ~]# ovs-vsctl show >>> 6f5a5f00-59e2-4420-aeaf-7ad464ead232 >>> Bridge br-int >>> fail_mode: secure >>> Port br-int >>> Interface br-int >>> type: internal >>> Port "qvo247da3ed-a4" >>> tag: 1 >>> Interface "qvo247da3ed-a4" >>> Port "int-br-eth1" >>> Interface "int-br-eth1" >>> Port "int-br-enp3s0f0" >>> Interface "int-br-enp3s0f0" >>> type: patch >>> options: {peer="phy-br-enp3s0f0"} >>> Bridge "br-enp3s0f0" >>> Port "enp3s0f0" >>> Interface "enp3s0f0" >>> Port "br-enp3s0f0" >>> Interface "br-enp3s0f0" >>> type: internal >>> Port "phy-br-enp3s0f0" >>> Interface "phy-br-enp3s0f0" >>> type: patch >>> options: {peer="int-br-enp3s0f0"} >>> ovs_version: "2.3.1" >>> >>> Many thanks if anyone has any information on this topic! Or can >>> point me to some documentation I missed... >>> >>> Thanks, >>> erich >>> >>> _______________________________________________ >>> Rdo-list mailing list >>> Rdo-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>> _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

Amedeo, Thanks for pointing this out. Although the KB article now includes this setting, some of our other documentation doesn't include this setting. I'll make sure it gets added. I'm curious whether anyone has tested out the new MTU-related options that were added to Kilo: advertise_mtu path_mtu segment_mtu physnet_mtus I haven't gotten a chance to test and document these new options myself. They serve to simplify configuration a bit, but also the new physnet_mtus option allows you to set a different MTU per interface: Example: physnet_mtus = physnet1:1550, physnet2:1500 Or, to set MTU for physnet2 and leave physnet1 as default: physnet_mtus = physnet2:1550 Lastly, has anyone ever run into problems when running (MTU - 50 bytes) as the veth_mtu with VXLAN? I see documentation all over recommending (MTU - 100 bytes), but I don't see why VXLAN should take that many extra bytes. I've done extensive testing at VM MTU 8950 over a 9000 MTU link, and never run into an issue. Is this just cargo-culting, or is there a reason to give VXLAN additional headroom in some scenarios? -- Dan Sneddon | Principal OpenStack Engineer dsneddon(a)redhat.com | redhat.com/openstack 650.254.4025 | dsneddon:irc @dxs:twitter On 10/09/2015 05:24 AM, Salvati Amedeo wrote: > Erich you are welcome in the club :D > > One side note: as we have rhosp and not rdo, we asked to rh to document this and they wrote a solution on their kb: > > https://access.redhat.com/solutions/1417133 > > Regards, > Amedeo > > -----Messaggio originale----- > Da: Erich Weiler [mailto:weiler@soe.ucsc.edu] > Inviato: giovedì 8 ottobre 2015 18:23 > A: Salvati Amedeo; Pedro Navarro Perez > Cc: rdo-list(a)redhat.com > Oggetto: Re: R: [Rdo-list] Jumbo MTU to instances in Kilo? > > Thanks Amedeo, > > The bit about the config item in the l3_agent.ini file is new to me - I couldn't find that in the documentation, or even as a comment in the file as a config option. If it is a config item as you point out, maybe it should have a commented section in l3_agent.ini? > > Thanks for the insight! > > cheers, > erich > > On 10/08/2015 03:02 AM, Salvati Amedeo wrote: >> Eric, >> >> also, to set jumbo frames on your env, you have to set mtu from VM to controller: >> >> # echo "dhcp-option-force=26,8900" > /etc/neutron/dnsmasq-neutron.conf >> # openstack-config --set /etc/neutron/dhcp_agent.ini DEFAULT dnsmasq_config_file /etc/neutron/dnsmasq-neutron.conf >> # openstack-config --set /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini agent veth_mtu 8900 >> # openstack-config --set /etc/neutron/l3_agent.ini DEFAULT network_device_mtu 9000 >> # openstack-config --set /etc/nova/nova.conf DEFAULT network_device_mtu 9000 <--- this on every nova-compute >> >> take a look at l3_agent.ini file, without network_device_mtu every new >> router will use default mtu at 1500 >> >> # ip netns exec qrouter-26f64a08-52ab-4643-b903-9aea6eae047a /bin/bash >> # ip a | grep mtu >> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN >> 69: ha-89546945-ab: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc >> noqueue state UNKNOWN >> 74: qr-f207f652-da: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc >> noqueue state UNKNOWN >> 81: qg-ab978cd0-ad: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc >> noqueue state UNKNOWN >> >> HTH >> Amedeo >> >> -----Messaggio originale----- >> Da: rdo-list-bounces(a)redhat.com [mailto:rdo-list-bounces@redhat.com] >> Per conto di Erich Weiler >> Inviato: giovedì 8 ottobre 2015 01:53 >> A: Pedro Navarro Perez >> Cc: rdo-list(a)redhat.com >> Oggetto: Re: [Rdo-list] Jumbo MTU to instances in Kilo? >> >> Actually I was wrong, it WAS on the network node. The virtual router interfaces were not set to MTU=9000. On network node: >> >> [root@os-net-01 ~]# ip netns >> qdhcp-c395cff9-af7b-4456-91e3-3c55e6c2c5f5 >> qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 >> >> i[root@os-net-01 ~]# ip netns exec >> qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> inet 127.0.0.1 netmask 255.0.0.0 >> inet6 ::1 prefixlen 128 scopeid 0x10<host> >> loop txqueuelen 0 (Local Loopback) >> RX packets 0 bytes 0 (0.0 B) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 0 bytes 0 (0.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qg-fa1e2a28-25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet 10.50.100.1 netmask 255.255.0.0 broadcast 10.50.255.255 >> inet6 fe80::f816:3eff:fe6a:608b prefixlen 64 scopeid 0x20<link> >> ether fa:16:3e:6a:60:8b txqueuelen 0 (Ethernet) >> RX packets 34071065 bytes 5046408745 (4.6 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 442 bytes 51915 (50.6 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qr-51904c89-b8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet 10.100.0.1 netmask 255.255.0.0 broadcast 10.100.255.255 >> inet6 fe80::f816:3eff:fe37:eca6 prefixlen 64 scopeid 0x20<link> >> ether fa:16:3e:37:ec:a6 txqueuelen 0 (Ethernet) >> RX packets 702 bytes 75369 (73.6 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 814 bytes 92259 (90.0 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> I can fix it manually: >> >> [root@os-net-01 neutron]# ip netns exec >> qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig qg-fa1e2a28-25 >> mtu >> 9000 >> [root@os-net-01 neutron]# ip netns exec >> qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig qr-51904c89-b8 >> mtu >> 9000 >> [root@os-net-01 neutron]# ip netns exec >> qrouter-0b52e3a6-135c-4481-b286-7c96229f6555 ifconfig >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> inet 127.0.0.1 netmask 255.0.0.0 >> inet6 ::1 prefixlen 128 scopeid 0x10<host> >> loop txqueuelen 0 (Local Loopback) >> RX packets 0 bytes 0 (0.0 B) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 0 bytes 0 (0.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qg-fa1e2a28-25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >> inet 10.50.100.1 netmask 255.255.0.0 broadcast 10.50.255.255 >> inet6 fe80::f816:3eff:fe6a:608b prefixlen 64 scopeid 0x20<link> >> ether fa:16:3e:6a:60:8b txqueuelen 0 (Ethernet) >> RX packets 34086053 bytes 5048637833 (4.7 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 442 bytes 51915 (50.6 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qr-51904c89-b8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >> inet 10.100.0.1 netmask 255.255.0.0 broadcast 10.100.255.255 >> inet6 fe80::f816:3eff:fe37:eca6 prefixlen 64 scopeid 0x20<link> >> ether fa:16:3e:37:ec:a6 txqueuelen 0 (Ethernet) >> RX packets 702 bytes 75369 (73.6 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 814 bytes 92259 (90.0 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> And then I have a jumbo clean path everywhere! All is good then. >> But... How to set this in a config file or something so I don't have to do it manually? >> >> I found this bug report: >> >> https://bugs.launchpad.net/neutron/+bug/1311097 >> >> Anyone know if that bug is still out there? Or how can I set virtual router interfaces MTU by default when I create the router? >> >> cheers, >> erich >> >> On 10/07/2015 04:35 PM, Erich Weiler wrote: >>> Actually I think I'm closer - on the compute nodes, I set this in >>> nova.conf: >>> >>> network_device_mtu=9000 >>> >>> even though there was a big note above it that said not to use it >>> because this option was deprecated. But after setting that option, >>> and restarting nova and openvswitch, br-int, my tap device and my qvb >>> device all got set to MTU=9000. So I'm closer! But still one item >>> is blocking me. I show this tracepath from my controller node direct >>> to the VM (which is on a compute node on the local network): >>> >>> # tracepath 10.50.100.4 >>> 1?: [LOCALHOST] pmtu 9000 >>> 1: 10.50.100.4 0.682ms >>> 1: 10.50.100.4 0.241ms >>> 2: 10.50.100.4 0.297ms pmtu >>> 1500 >>> 2: 10.50.100.4 1.664ms reached >>> >>> 10.50.100.4 is the VM. It looks like the path is jumbo clean up >>> until that third hop. But the thing is, I don't know what the third hop is. >>> ;) >>> >>> On my compute node I still see some stuff with MTU=1500, but I'm not >>> sure if one of those is blocking me: >>> >>> # ifconfig >>> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> >>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >>> RX packets 2401498 bytes 359284253 (342.6 MiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 30 bytes 1572 (1.5 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid 0x20<link> >>> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) >>> RX packets 133 bytes 12934 (12.6 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 8 bytes 648 (648.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 >>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> >>> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) >>> RX packets 165957142 bytes 20333410092 (18.9 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 23299881 bytes 5950708819 (5.5 GiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 >>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> >>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >>> RX packets 6014767 bytes 813880745 (776.1 MiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 79301 bytes 19052451 (18.1 MiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >>> inet 127.0.0.1 netmask 255.0.0.0 >>> inet6 ::1 prefixlen 128 scopeid 0x10<host> >>> loop txqueuelen 0 (Local Loopback) >>> RX packets 22462729 bytes 1202484822 (1.1 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 22462729 bytes 1202484822 (1.1 GiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::4c1a:55ff:feba:14c3 prefixlen 64 scopeid 0x20<link> >>> ether 56:a6:a6:db:83:c4 txqueuelen 0 (Ethernet) >>> RX packets 16 bytes 1520 (1.4 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 8 bytes 648 (648.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid 0x20<link> >>> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) >>> RX packets 15 bytes 1456 (1.4 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 8 bytes 648 (648.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu >>> 9000 >>> inet6 fe80::54a6:a6ff:fedb:83c4 prefixlen 64 scopeid 0x20<link> >>> ether 56:a6:a6:db:83:c4 txqueuelen 1000 (Ethernet) >>> RX packets 86 bytes 9610 (9.3 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 133 bytes 12767 (12.4 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu >>> 1500 >>> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid 0x20<link> >>> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) >>> RX packets 377 bytes 57664 (56.3 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 333 bytes 38765 (37.8 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu >>> 9000 >>> inet6 fe80::b44a:bff:fe72:aaea prefixlen 64 scopeid 0x20<link> >>> ether b6:4a:0b:72:aa:ea txqueuelen 1000 (Ethernet) >>> RX packets 133 bytes 12767 (12.4 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 86 bytes 9610 (9.3 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>> mtu >>> 1500 >>> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid 0x20<link> >>> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) >>> RX packets 333 bytes 38765 (37.8 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 377 bytes 57664 (56.3 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::fc16:3eff:fefa:9945 prefixlen 64 scopeid 0x20<link> >>> ether fe:16:3e:fa:99:45 txqueuelen 500 (Ethernet) >>> RX packets 118 bytes 11561 (11.2 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 95 bytes 10316 (10.0 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >>> inet 192.168.122.1 netmask 255.255.255.0 broadcast >>> 192.168.122.255 >>> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) >>> RX packets 0 bytes 0 (0.0 B) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 0 bytes 0 (0.0 B) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> My network node has all interfaces set to MTU=9000. I thought maybe the >>> bottleneck might be there but I don't think it is. Here's ifconfig >>> from my network node: >>> >>> # ifconfig >>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >>> inet 127.0.0.1 netmask 255.0.0.0 >>> inet6 ::1 prefixlen 128 scopeid 0x10<host> >>> loop txqueuelen 0 (Local Loopback) >>> RX packets 2042 bytes 238727 (233.1 KiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 2042 bytes 238727 (233.1 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::207:43ff:fe10:deb8 prefixlen 64 scopeid 0x20<link> >>> ether 00:07:43:10:de:b8 txqueuelen 1000 (Ethernet) >>> RX packets 2156053308 bytes 325330839639 (302.9 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 223004 bytes 24769304 (23.6 MiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> device interrupt 72 >>> >>> p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet 10.50.1.51 netmask 255.255.0.0 broadcast 10.50.255.255 >>> inet6 fe80::260:ddff:fe44:2aea prefixlen 64 scopeid 0x20<link> >>> ether 00:60:dd:44:2a:ea txqueuelen 1000 (Ethernet) >>> RX packets 49352916 bytes 3501547231 (3.2 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 18876911 bytes 3768900461 (3.5 GiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>> inet6 fe80::260:ddff:fe44:2aeb prefixlen 64 scopeid 0x20<link> >>> ether 00:60:dd:44:2a:eb txqueuelen 1000 (Ethernet) >>> RX packets 2491224974 bytes 348058319500 (324.1 GiB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 1597 bytes 204525 (199.7 KiB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> >>> Any way I can figure out what the third hop is from my tracepath? >>> >>> Thanks as always for the sage advice! >>> >>> -erich >>> >>> On 10/07/2015 09:57 AM, Erich Weiler wrote: >>>> Yeah, I made the changes and then recreated all the networks. For >>>> some reason br-int and the individual virtual instance interfaces on >>>> the compute node still show 1500 byte frames. >>>> >>>> Has anyone else configured jumbo frames in a Kilo environment? Or >>>> maybe I'm just an outlier... ;) >>>> >>>> -erich >>>> >>>> On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote: >>>>> Hi Erich, >>>>> >>>>> did you recreate the neutron networks after the configuration changes? >>>>> >>>>> Pedro Navarro Pérez >>>>> OpenStack product specialist >>>>> Red Hat Iberia >>>>> Passeig de Gràcia 120, >>>>> 08008 Barcelona >>>>> Spain >>>>> M +34 639 642 379 >>>>> E pnavarro(a)redhat.com >>>>> >>>>> ----- Original Message ----- >>>>> From: "Erich Weiler" <weiler(a)soe.ucsc.edu&gt; >>>>> To: rdo-list(a)redhat.com >>>>> Sent: Wednesday, 7 October, 2015 2:34:28 AM >>>>> Subject: [Rdo-list] Jumbo MTU to instances in Kilo? >>>>> >>>>> Hi Y'all, >>>>> >>>>> I know someone must have figured this one out, but I can't seem to >>>>> get >>>>> 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes >>>>> have >>>>> MTU=9000 on their interfaces, so does the network node. dnsmasq >>>>> also is configured to set MTU=9000 on instances, which works. But >>>>> I still can't ping with large packets to my instance: >>>>> >>>>> [weiler@stacker ~]$ ping 10.50.100.2 PING 10.50.100.2 (10.50.100.2) >>>>> 56(84) bytes of data. >>>>> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms >>>>> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms >>>>> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms >>>>> >>>>> That works fine. This however doesn't work: >>>>> >>>>> [root@stacker ~]# ping -M do -s 8000 10.50.100.2 PING 10.50.100.2 >>>>> (10.50.100.2) 8000(8028) bytes of data. >>>>> From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500) >>>>> ping: local error: Message too long, mtu=1500 >>>>> ping: local error: Message too long, mtu=1500 >>>>> ping: local error: Message too long, mtu=1500 >>>>> ping: local error: Message too long, mtu=1500 >>>>> >>>>> It looks like somehow the br-int interface for OVS isn't set at >>>>> 9000, but I can't figure out how to do that... >>>>> >>>>> Here's ifconfig on my compute node: >>>>> >>>>> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>>>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >>>>> RX packets 2401432 bytes 359276713 (342.6 MiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 30 bytes 1572 (1.5 KiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>>>> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) >>>>> RX packets 69 bytes 6866 (6.7 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 8 bytes 648 (648.0 B) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 >>>>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) >>>>> RX packets 130174458 bytes 15334807929 (14.2 GiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 22919305 bytes 5859090420 (5.4 GiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >>>>> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 >>>>> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >>>>> RX packets 38429352 bytes 5152853436 (4.7 GiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 419842 bytes 101161981 (96.4 MiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >>>>> inet 127.0.0.1 netmask 255.0.0.0 >>>>> inet6 ::1 prefixlen 128 scopeid 0x10<host> >>>>> loop txqueuelen 0 (Local Loopback) >>>>> RX packets 22141566 bytes 1185622090 (1.1 GiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 22141566 bytes 1185622090 (1.1 GiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>>>> inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet) >>>>> RX packets 16 bytes 1472 (1.4 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 8 bytes 648 (648.0 B) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>>>> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) >>>>> RX packets 15 bytes 1456 (1.4 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 8 bytes 648 (648.0 B) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>>>> mtu 1500 >>>>> inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet) >>>>> RX packets 247 bytes 28323 (27.6 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 233 bytes 25355 (24.7 KiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>>>> mtu 1500 >>>>> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) >>>>> RX packets 377 bytes 57664 (56.3 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 333 bytes 38765 (37.8 KiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>>>> mtu 1500 >>>>> inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet) >>>>> RX packets 233 bytes 25355 (24.7 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 247 bytes 28323 (27.6 KiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >>>>> mtu 1500 >>>>> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) >>>>> RX packets 333 bytes 38765 (37.8 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 377 bytes 57664 (56.3 KiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>>>> inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet) >>>>> RX packets 219 bytes 24239 (23.6 KiB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 224 bytes 26661 (26.0 KiB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >>>>> inet 192.168.122.1 netmask 255.255.255.0 broadcast >>>>> 192.168.122.255 >>>>> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) >>>>> RX packets 0 bytes 0 (0.0 B) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 0 bytes 0 (0.0 B) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions >>>>> 0 >>>>> >>>>> This is on RHEL 7.1. Any obvious way I can get all the >>>>> intermediate bridges to MTU=9000? I've RTFM'd and googled to no avail... >>>>> >>>>> Here's the ovs-vsctl outout: >>>>> >>>>> [root@node-136 ~]# ovs-vsctl show >>>>> 6f5a5f00-59e2-4420-aeaf-7ad464ead232 >>>>> Bridge br-int >>>>> fail_mode: secure >>>>> Port br-int >>>>> Interface br-int >>>>> type: internal >>>>> Port "qvo247da3ed-a4" >>>>> tag: 1 >>>>> Interface "qvo247da3ed-a4" >>>>> Port "int-br-eth1" >>>>> Interface "int-br-eth1" >>>>> Port "int-br-enp3s0f0" >>>>> Interface "int-br-enp3s0f0" >>>>> type: patch >>>>> options: {peer="phy-br-enp3s0f0"} >>>>> Bridge "br-enp3s0f0" >>>>> Port "enp3s0f0" >>>>> Interface "enp3s0f0" >>>>> Port "br-enp3s0f0" >>>>> Interface "br-enp3s0f0" >>>>> type: internal >>>>> Port "phy-br-enp3s0f0" >>>>> Interface "phy-br-enp3s0f0" >>>>> type: patch >>>>> options: {peer="int-br-enp3s0f0"} >>>>> ovs_version: "2.3.1" >>>>> >>>>> Many thanks if anyone has any information on this topic! Or can >>>>> point me to some documentation I missed... >>>>> >>>>> Thanks, >>>>> erich >>>>> >>>>> _______________________________________________ >>>>> Rdo-list mailing list >>>>> Rdo-list(a)redhat.com >>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>> >>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>>> >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> > > _______________________________________________ > Rdo-list mailing list > Rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >

> Lastly, has anyone ever run into problems when running (MTU - 50 bytes) > as the veth_mtu with VXLAN? I see documentation all over recommending > (MTU - 100 bytes), but I don't see why VXLAN should take that many > extra bytes. I've done extensive testing at VM MTU 8950 over a 9000 MTU > link, and never run into an issue. Is this just cargo-culting, or is > there a reason to give VXLAN additional headroom in some scenarios? Along those same lines... I'm running tenant segregation via regular VLAN segregation (not VXLAN or GRE), using all interfaces set to MTU=9000 (the instance all the way up to the main network gateway, all router interfaces, etc). My physical switches have MTU=9260 on all ports however. It seems to work and perform OK, but is there a recommendation on giving regular VLANs a little VLAN-tag headroom as well? Like, should I be setting my instance MTU to 8950 or something?