6:35 p.m.
Actually I think I'm closer - on the compute nodes, I set this in nova.conf:
network_device_mtu=9000
even though there was a big note above it that said not to use it
because this option was deprecated. But after setting that option, and
restarting nova and openvswitch, br-int, my tap device and my qvb device
all got set to MTU=9000. So I'm closer! But still one item is blocking
me. I show this tracepath from my controller node direct to the VM
(which is on a compute node on the local network):
# tracepath 10.50.100.4
1?: [LOCALHOST] pmtu 9000
1: 10.50.100.4 0.682ms
1: 10.50.100.4 0.241ms
2: 10.50.100.4 0.297ms pmtu
1500
2: 10.50.100.4 1.664ms reached
10.50.100.4 is the VM. It looks like the path is jumbo clean up until
that third hop. But the thing is, I don't know what the third hop is. ;)
On my compute node I still see some stuff with MTU=1500, but I'm not
sure if one of those is blocking me:
# ifconfig
br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
RX packets 2401498 bytes 359284253 (342.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 30 bytes 1572 (1.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid 0x20<link>
ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet)
RX packets 133 bytes 12934 (12.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet)
RX packets 165957142 bytes 20333410092 (18.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23299881 bytes 5950708819 (5.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255
inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid 0x20<link>
ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
RX packets 6014767 bytes 813880745 (776.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 79301 bytes 19052451 (18.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 22462729 bytes 1202484822 (1.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 22462729 bytes 1202484822 (1.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::4c1a:55ff:feba:14c3 prefixlen 64 scopeid 0x20<link>
ether 56:a6:a6:db:83:c4 txqueuelen 0 (Ethernet)
RX packets 16 bytes 1520 (1.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid 0x20<link>
ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet)
RX packets 15 bytes 1456 (1.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 648 (648.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
inet6 fe80::54a6:a6ff:fedb:83c4 prefixlen 64 scopeid 0x20<link>
ether 56:a6:a6:db:83:c4 txqueuelen 1000 (Ethernet)
RX packets 86 bytes 9610 (9.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 133 bytes 12767 (12.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid 0x20<link>
ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet)
RX packets 377 bytes 57664 (56.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 333 bytes 38765 (37.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
inet6 fe80::b44a:bff:fe72:aaea prefixlen 64 scopeid 0x20<link>
ether b6:4a:0b:72:aa:ea txqueuelen 1000 (Ethernet)
RX packets 133 bytes 12767 (12.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 86 bytes 9610 (9.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid 0x20<link>
ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet)
RX packets 333 bytes 38765 (37.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 377 bytes 57664 (56.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::fc16:3eff:fefa:9945 prefixlen 64 scopeid 0x20<link>
ether fe:16:3e:fa:99:45 txqueuelen 500 (Ethernet)
RX packets 118 bytes 11561 (11.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 95 bytes 10316 (10.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast
192.168.122.255
ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
My network node has all interfaces set to MTU=9000. I thought maybe the
bottleneck might be there but I don't think it is. Here's ifconfig
from my network node:
# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 2042 bytes 238727 (233.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2042 bytes 238727 (233.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::207:43ff:fe10:deb8 prefixlen 64 scopeid 0x20<link>
ether 00:07:43:10:de:b8 txqueuelen 1000 (Ethernet)
RX packets 2156053308 bytes 325330839639 (302.9 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 223004 bytes 24769304 (23.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 72
p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet 10.50.1.51 netmask 255.255.0.0 broadcast 10.50.255.255
inet6 fe80::260:ddff:fe44:2aea prefixlen 64 scopeid 0x20<link>
ether 00:60:dd:44:2a:ea txqueuelen 1000 (Ethernet)
RX packets 49352916 bytes 3501547231 (3.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18876911 bytes 3768900461 (3.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
inet6 fe80::260:ddff:fe44:2aeb prefixlen 64 scopeid 0x20<link>
ether 00:60:dd:44:2a:eb txqueuelen 1000 (Ethernet)
RX packets 2491224974 bytes 348058319500 (324.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1597 bytes 204525 (199.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Any way I can figure out what the third hop is from my tracepath?
Thanks as always for the sage advice!
-erich
On 10/07/2015 09:57 AM, Erich Weiler wrote:
Yeah, I made the changes and then recreated all the networks. For
some
reason br-int and the individual virtual instance interfaces on the
compute node still show 1500 byte frames.
Has anyone else configured jumbo frames in a Kilo environment? Or maybe
I'm just an outlier... ;)
-erich
On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote:
> Hi Erich,
>
> did you recreate the neutron networks after the configuration changes?
>
> Pedro Navarro Pérez
> OpenStack product specialist
> Red Hat Iberia
> Passeig de Gràcia 120,
> 08008 Barcelona
> Spain
> M +34 639 642 379
> E pnavarro(a)redhat.com
>
> ----- Original Message -----
> From: "Erich Weiler" <weiler(a)soe.ucsc.edu>
> To: rdo-list(a)redhat.com
> Sent: Wednesday, 7 October, 2015 2:34:28 AM
> Subject: [Rdo-list] Jumbo MTU to instances in Kilo?
>
> Hi Y'all,
>
> I know someone must have figured this one out, but I can't seem to get
> 9000 byte MTUs working. I have it set in plugin.ini, etc, my nodes have
> MTU=9000 on their interfaces, so does the network node. dnsmasq also is
> configured to set MTU=9000 on instances, which works. But I still can't
> ping with large packets to my instance:
>
> [weiler@stacker ~]$ ping 10.50.100.2
> PING 10.50.100.2 (10.50.100.2) 56(84) bytes of data.
> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms
> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms
> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms
>
> That works fine. This however doesn't work:
>
> [root@stacker ~]# ping -M do -s 8000 10.50.100.2
> PING 10.50.100.2 (10.50.100.2) 8000(8028) bytes of data.
> From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)
> ping: local error: Message too long, mtu=1500
> ping: local error: Message too long, mtu=1500
> ping: local error: Message too long, mtu=1500
> ping: local error: Message too long, mtu=1500
>
> It looks like somehow the br-int interface for OVS isn't set at 9000,
> but I can't figure out how to do that...
>
> Here's ifconfig on my compute node:
>
> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid
> 0x20<link>
> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
> RX packets 2401432 bytes 359276713 (342.6 MiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 30 bytes 1572 (1.5 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet6 fe80::64dc:94ff:fe35:db4c prefixlen 64 scopeid
> 0x20<link>
> ether 66:dc:94:35:db:4c txqueuelen 0 (Ethernet)
> RX packets 69 bytes 6866 (6.7 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 8 bytes 648 (648.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 9000
> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid
> 0x20<link>
> ether 0c:c4:7a:58:42:3e txqueuelen 1000 (Ethernet)
> RX packets 130174458 bytes 15334807929 (14.2 GiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 22919305 bytes 5859090420 (5.4 GiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000
> inet 10.50.1.236 netmask 255.255.0.0 broadcast 10.50.255.255
> inet6 fe80::ec4:7aff:fe58:423e prefixlen 64 scopeid
> 0x20<link>
> ether 0c:c4:7a:58:42:3e txqueuelen 0 (Ethernet)
> RX packets 38429352 bytes 5152853436 (4.7 GiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 419842 bytes 101161981 (96.4 MiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
> inet 127.0.0.1 netmask 255.0.0.0
> inet6 ::1 prefixlen 128 scopeid 0x10<host>
> loop txqueuelen 0 (Local Loopback)
> RX packets 22141566 bytes 1185622090 (1.1 GiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 22141566 bytes 1185622090 (1.1 GiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet6 fe80::5c8f:c0ff:fe79:bc11 prefixlen 64 scopeid
> 0x20<link>
> ether b6:1f:54:3f:3d:48 txqueuelen 0 (Ethernet)
> RX packets 16 bytes 1472 (1.4 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 8 bytes 648 (648.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet6 fe80::f484:f1ff:fe53:fb2e prefixlen 64 scopeid
> 0x20<link>
> ether c2:a6:d8:25:63:ea txqueuelen 0 (Ethernet)
> RX packets 15 bytes 1456 (1.4 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 8 bytes 648 (648.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
> mtu 1500
> inet6 fe80::b41f:54ff:fe3f:3d48 prefixlen 64 scopeid
> 0x20<link>
> ether b6:1f:54:3f:3d:48 txqueuelen 1000 (Ethernet)
> RX packets 247 bytes 28323 (27.6 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 233 bytes 25355 (24.7 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
> mtu 1500
> inet6 fe80::c0a6:d8ff:fe25:63ea prefixlen 64 scopeid
> 0x20<link>
> ether c2:a6:d8:25:63:ea txqueuelen 1000 (Ethernet)
> RX packets 377 bytes 57664 (56.3 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 333 bytes 38765 (37.8 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
> mtu 1500
> inet6 fe80::dcfa:f1ff:fe03:ee88 prefixlen 64 scopeid
> 0x20<link>
> ether de:fa:f1:03:ee:88 txqueuelen 1000 (Ethernet)
> RX packets 233 bytes 25355 (24.7 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 247 bytes 28323 (27.6 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
> mtu 1500
> inet6 fe80::f03e:35ff:fefe:e52 prefixlen 64 scopeid
> 0x20<link>
> ether f2:3e:35:fe:0e:52 txqueuelen 1000 (Ethernet)
> RX packets 333 bytes 38765 (37.8 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 377 bytes 57664 (56.3 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet6 fe80::fc16:3eff:fede:5eea prefixlen 64 scopeid
> 0x20<link>
> ether fe:16:3e:de:5e:ea txqueuelen 500 (Ethernet)
> RX packets 219 bytes 24239 (23.6 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 224 bytes 26661 (26.0 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
> inet 192.168.122.1 netmask 255.255.255.0 broadcast
> 192.168.122.255
> ether 52:54:00:c4:75:9f txqueuelen 0 (Ethernet)
> RX packets 0 bytes 0 (0.0 B)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> This is on RHEL 7.1. Any obvious way I can get all the intermediate
> bridges to MTU=9000? I've RTFM'd and googled to no avail...
>
> Here's the ovs-vsctl outout:
>
> [root@node-136 ~]# ovs-vsctl show
> 6f5a5f00-59e2-4420-aeaf-7ad464ead232
> Bridge br-int
> fail_mode: secure
> Port br-int
> Interface br-int
> type: internal
> Port "qvo247da3ed-a4"
> tag: 1
> Interface "qvo247da3ed-a4"
> Port "int-br-eth1"
> Interface "int-br-eth1"
> Port "int-br-enp3s0f0"
> Interface "int-br-enp3s0f0"
> type: patch
> options: {peer="phy-br-enp3s0f0"}
> Bridge "br-enp3s0f0"
> Port "enp3s0f0"
> Interface "enp3s0f0"
> Port "br-enp3s0f0"
> Interface "br-enp3s0f0"
> type: internal
> Port "phy-br-enp3s0f0"
> Interface "phy-br-enp3s0f0"
> type: patch
> options: {peer="int-br-enp3s0f0"}
> ovs_version: "2.3.1"
>
> Many thanks if anyone has any information on this topic! Or can point
> me to some documentation I missed...
>
> Thanks,
> erich
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list(a)redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe(a)redhat.com
>