Re: [Rdo-list] Jumbo MTU to instances in Kilo?

Actually I think I'm closer - on the compute nodes, I set this in nova.conf: network_device_mtu=9000 even though there was a big note above it that said not to use it because this option was deprecated. But after setting that option, and restarting nova and openvswitch, br-int, my tap device and my qvb device all got set to MTU=9000. So I'm closer! But still one item is blocking me. I show this tracepath from my controller node direct to the VM (which is on a compute node on the local network): # tracepath 10.50.100.4 1?: [LOCALHOST] pmtu 9000 1: 10.50.100.4 0.682ms 1: 10.50.100.4 0.241ms 2: 10.50.100.4 0.297ms pmtu 1500 2: 10.50.100.4 1.664ms reached 10.50.100.4 is the VM. It looks like the path is jumbo clean up until that third hop. But the thing is, I don't know what the third hop is. ;) On my compute node I still see some stuff with MTU=1500, but I'm not sure if one of those is blocking me: # ifconfig br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) RX packets 2401498 bytes 359284253 (342.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 30 bytes 1572 (1.5 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid 0x20<link> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) RX packets 133 bytes 12934 (12.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) RX packets 165957142 bytes 20333410092 (18.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 23299881 bytes 5950708819 (5.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) RX packets 6014767 bytes 813880745 (776.1 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 79301 bytes 19052451 (18.1 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 22462729 bytes 1202484822 (1.1 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 22462729 bytes 1202484822 (1.1 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::4c1a:55ff:feba:14c3 prefixlen 64 scopeid 0x20<link> ether 56:a6:a6:db:83:c4 txqueuelen 0 (Ethernet) RX packets 16 bytes 1520 (1.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid 0x20<link> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) RX packets 15 bytes 1456 (1.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8 bytes 648 (648.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 inet6 fe80::54a6:a6ff:fedb:83c4 prefixlen 64 scopeid 0x20<link> ether 56:a6:a6:db:83:c4 txqueuelen 1000 (Ethernet) RX packets 86 bytes 9610 (9.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 133 bytes 12767 (12.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid 0x20<link> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) RX packets 377 bytes 57664 (56.3 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 333 bytes 38765 (37.8 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 inet6 fe80::b44a:bff:fe72:aaea prefixlen 64 scopeid 0x20<link> ether b6:4a:0b:72:aa:ea txqueuelen 1000 (Ethernet) RX packets 133 bytes 12767 (12.4 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 86 bytes 9610 (9.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid 0x20<link> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) RX packets 333 bytes 38765 (37.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 377 bytes 57664 (56.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::fc16:3eff:fefa:9945 prefixlen 64 scopeid 0x20<link> ether fe:16:3e:fa:99:45 txqueuelen 500 (Ethernet) RX packets 118 bytes 11561 (11.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 95 bytes 10316 (10.0 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255 ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 My network node has all interfaces set to MTU=9000. I thought maybe the bottleneck might be there but I don't think it is. Here's ifconfig from my network node: # ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 2042 bytes 238727 (233.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2042 bytes 238727 (233.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::207:43ff:fe10:deb8 prefixlen 64 scopeid 0x20<link> ether 00:07:43:10:de:b8 txqueuelen 1000 (Ethernet) RX packets 2156053308 bytes 325330839639 (302.9 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 223004 bytes 24769304 (23.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 72 p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.50.1.51 netmask 255.255.0.0 broadcast 10.50.255.255 inet6 fe80::260:ddff:fe44:2aea prefixlen 64 scopeid 0x20<link> ether 00:60:dd:44:2a:ea txqueuelen 1000 (Ethernet) RX packets 49352916 bytes 3501547231 (3.2 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 18876911 bytes 3768900461 (3.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet6 fe80::260:ddff:fe44:2aeb prefixlen 64 scopeid 0x20<link> ether 00:60:dd:44:2a:eb txqueuelen 1000 (Ethernet) RX packets 2491224974 bytes 348058319500 (324.1 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1597 bytes 204525 (199.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Any way I can figure out what the third hop is from my tracepath? Thanks as always for the sage advice! -erich On 10/07/2015 09:57 AM, Erich Weiler wrote: > Yeah, I made the changes and then recreated all the networks. For some > reason br-int and the individual virtual instance interfaces on the > compute node still show 1500 byte frames. > > Has anyone else configured jumbo frames in a Kilo environment? Or maybe > I'm just an outlier... ;) > > -erich > > On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote: >> Hi Erich, >> >> did you recreate the neutron networks after the configuration changes? >> >> Pedro Navarro Pérez >> OpenStack product specialist >> Red Hat Iberia >> Passeig de Gràcia 120, >> 08008 Barcelona >> Spain >> M +34 639 642 379 >> E pnavarro(a)redhat.com >> >> ----- Original Message ----- >> From: "Erich Weiler" <weiler(a)soe.ucsc.edu&gt; >> To: rdo-list(a)redhat.com >> Sent: Wednesday, 7 October, 2015 2:34:28 AM >> Subject: [Rdo-list] Jumbo MTU to instances in Kilo? >> >> Hi Y'all, >> >> I know someone must have figured this one out, but I can't seem to get >> 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes have >> MTU=9000 on their interfaces, so does the network node. dnsmasq also is >> configured to set MTU=9000 on instances, which works. But I still can't >> ping with large packets to my instance: >> >> [weiler@stacker ~]$ ping 10.50.100.2 >> PING 10.50.100.2 (10.50.100.2) 56(84) bytes of data. >> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms >> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms >> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms >> >> That works fine. This however doesn't work: >> >> [root@stacker ~]# ping -M do -s 8000 10.50.100.2 >> PING 10.50.100.2 (10.50.100.2) 8000(8028) bytes of data. >> From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500) >> ping: local error: Message too long, mtu=1500 >> ping: local error: Message too long, mtu=1500 >> ping: local error: Message too long, mtu=1500 >> ping: local error: Message too long, mtu=1500 >> >> It looks like somehow the br-int interface for OVS isn't set at 9000, >> but I can't figure out how to do that... >> >> Here's ifconfig on my compute node: >> >> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >> 0x20<link> >> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >> RX packets 2401432 bytes 359276713 (342.6 MiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 30 bytes 1572 (1.5 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid >> 0x20<link> >> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet) >> RX packets 69 bytes 6866 (6.7 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 8 bytes 648 (648.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000 >> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >> 0x20<link> >> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet) >> RX packets 130174458 bytes 15334807929 (14.2 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 22919305 bytes 5859090420 (5.4 GiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 >> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255 >> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid >> 0x20<link> >> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet) >> RX packets 38429352 bytes 5152853436 (4.7 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 419842 bytes 101161981 (96.4 MiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 >> inet 127.0.0.1 netmask 255.0.0.0 >> inet6 ::1 prefixlen 128 scopeid 0x10<host> >> loop txqueuelen 0 (Local Loopback) >> RX packets 22141566 bytes 1185622090 (1.1 GiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 22141566 bytes 1185622090 (1.1 GiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid >> 0x20<link> >> ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet) >> RX packets 16 bytes 1472 (1.4 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 8 bytes 648 (648.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid >> 0x20<link> >> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet) >> RX packets 15 bytes 1456 (1.4 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 8 bytes 648 (648.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid >> 0x20<link> >> ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet) >> RX packets 247 bytes 28323 (27.6 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 233 bytes 25355 (24.7 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid >> 0x20<link> >> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet) >> RX packets 377 bytes 57664 (56.3 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 333 bytes 38765 (37.8 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid >> 0x20<link> >> ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet) >> RX packets 233 bytes 25355 (24.7 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 247 bytes 28323 (27.6 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> >> mtu 1500 >> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid >> 0x20<link> >> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet) >> RX packets 333 bytes 38765 (37.8 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 377 bytes 57664 (56.3 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid >> 0x20<link> >> ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet) >> RX packets 219 bytes 24239 (23.6 KiB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 224 bytes 26661 (26.0 KiB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 >> inet 192.168.122.1 netmask 255.255.255.0 broadcast >> 192.168.122.255 >> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet) >> RX packets 0 bytes 0 (0.0 B) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 0 bytes 0 (0.0 B) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> >> This is on RHEL 7.1. Any obvious way I can get all the intermediate >> bridges to MTU=9000? I've RTFM'd and googled to no avail... >> >> Here's the ovs-vsctl outout: >> >> [root@node-136 ~]# ovs-vsctl show >> 6f5a5f00-59e2-4420-aeaf-7ad464ead232 >> Bridge br-int >> fail_mode: secure >> Port br-int >> Interface br-int >> type: internal >> Port "qvo247da3ed-a4" >> tag: 1 >> Interface "qvo247da3ed-a4" >> Port "int-br-eth1" >> Interface "int-br-eth1" >> Port "int-br-enp3s0f0" >> Interface "int-br-enp3s0f0" >> type: patch >> options: {peer="phy-br-enp3s0f0"} >> Bridge "br-enp3s0f0" >> Port "enp3s0f0" >> Interface "enp3s0f0" >> Port "br-enp3s0f0" >> Interface "br-enp3s0f0" >> type: internal >> Port "phy-br-enp3s0f0" >> Interface "phy-br-enp3s0f0" >> type: patch >> options: {peer="int-br-enp3s0f0"} >> ovs_version: "2.3.1" >> >> Many thanks if anyone has any information on this topic! Or can point >> me to some documentation I missed... >> >> Thanks, >> erich >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>