[Rdo-list] Jumbo MTU to instances in Kilo?

Erich Weiler weiler at soe.ucsc.edu
Wed Oct 7 23:35:49 UTC 2015


Actually I think I'm closer - on the compute nodes, I set this in nova.conf:

network_device_mtu=9000

even though there was a big note above it that said not to use it 
because this option was deprecated.  But after setting that option, and 
restarting nova and openvswitch, br-int, my tap device and my qvb device 
all got set to MTU=9000.  So I'm closer!  But still one item is blocking 
me.  I show this tracepath from my controller node direct to the VM 
(which is on a compute node on the local network):

# tracepath 10.50.100.4
  1?: [LOCALHOST]                                         pmtu 9000
  1:  10.50.100.4                                           0.682ms
  1:  10.50.100.4                                           0.241ms
  2:  10.50.100.4                                           0.297ms pmtu 
1500
  2:  10.50.100.4                                           1.664ms reached

10.50.100.4 is the VM.  It looks like the path is jumbo clean up until 
that third hop.  But the thing is, I don't know what the third hop is.  ;)

On my compute node I still see some stuff with MTU=1500, but I'm not 
sure if one of those is blocking me:

# ifconfig
br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet6 fe80::ec4:7aff:fe58:423e  prefixlen 64  scopeid 0x20<link>
         ether 0c:c4:7a:58:42:3e  txqueuelen 0  (Ethernet)
         RX packets 2401498  bytes 359284253 (342.6 MiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 30  bytes 1572 (1.5 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet6 fe80::64dc:94ff:fe35:db4c  prefixlen 64  scopeid 0x20<link>
         ether 66:dc:94:35:db:4c  txqueuelen 0  (Ethernet)
         RX packets 133  bytes 12934 (12.6 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 8  bytes 648 (648.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 9000
         inet6 fe80::ec4:7aff:fe58:423e  prefixlen 64  scopeid 0x20<link>
         ether 0c:c4:7a:58:42:3e  txqueuelen 1000  (Ethernet)
         RX packets 165957142  bytes 20333410092 (18.9 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 23299881  bytes 5950708819 (5.5 GiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet 10.50.1.236  netmask 255.255.0.0  broadcast 10.50.255.255
         inet6 fe80::ec4:7aff:fe58:423e  prefixlen 64  scopeid 0x20<link>
         ether 0c:c4:7a:58:42:3e  txqueuelen 0  (Ethernet)
         RX packets 6014767  bytes 813880745 (776.1 MiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 79301  bytes 19052451 (18.1 MiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
         inet 127.0.0.1  netmask 255.0.0.0
         inet6 ::1  prefixlen 128  scopeid 0x10<host>
         loop  txqueuelen 0  (Local Loopback)
         RX packets 22462729  bytes 1202484822 (1.1 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 22462729  bytes 1202484822 (1.1 GiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qbr922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet6 fe80::4c1a:55ff:feba:14c3  prefixlen 64  scopeid 0x20<link>
         ether 56:a6:a6:db:83:c4  txqueuelen 0  (Ethernet)
         RX packets 16  bytes 1520 (1.4 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 8  bytes 648 (648.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet6 fe80::f484:f1ff:fe53:fb2e  prefixlen 64  scopeid 0x20<link>
         ether c2:a6:d8:25:63:ea  txqueuelen 0  (Ethernet)
         RX packets 15  bytes 1456 (1.4 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 8  bytes 648 (648.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvb922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 9000
         inet6 fe80::54a6:a6ff:fedb:83c4  prefixlen 64  scopeid 0x20<link>
         ether 56:a6:a6:db:83:c4  txqueuelen 1000  (Ethernet)
         RX packets 86  bytes 9610 (9.3 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 133  bytes 12767 (12.4 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
         inet6 fe80::c0a6:d8ff:fe25:63ea  prefixlen 64  scopeid 0x20<link>
         ether c2:a6:d8:25:63:ea  txqueuelen 1000  (Ethernet)
         RX packets 377  bytes 57664 (56.3 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 333  bytes 38765 (37.8 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvo922bd9f5-bb: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 9000
         inet6 fe80::b44a:bff:fe72:aaea  prefixlen 64  scopeid 0x20<link>
         ether b6:4a:0b:72:aa:ea  txqueuelen 1000  (Ethernet)
         RX packets 133  bytes 12767 (12.4 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 86  bytes 9610 (9.3 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
         inet6 fe80::f03e:35ff:fefe:e52  prefixlen 64  scopeid 0x20<link>
         ether f2:3e:35:fe:0e:52  txqueuelen 1000  (Ethernet)
         RX packets 333  bytes 38765 (37.8 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 377  bytes 57664 (56.3 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tap922bd9f5-bb: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet6 fe80::fc16:3eff:fefa:9945  prefixlen 64  scopeid 0x20<link>
         ether fe:16:3e:fa:99:45  txqueuelen 500  (Ethernet)
         RX packets 118  bytes 11561 (11.2 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 95  bytes 10316 (10.0 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
         inet 192.168.122.1  netmask 255.255.255.0  broadcast 
192.168.122.255
         ether 52:54:00:c4:75:9f  txqueuelen 0  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

My network node has all interfaces set to MTU=9000.  I thought maybe the 
bottleneck might be there but I don't think it is.   Here's ifconfig 
from my network node:

# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
         inet 127.0.0.1  netmask 255.0.0.0
         inet6 ::1  prefixlen 128  scopeid 0x10<host>
         loop  txqueuelen 0  (Local Loopback)
         RX packets 2042  bytes 238727 (233.1 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 2042  bytes 238727 (233.1 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p1p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet6 fe80::207:43ff:fe10:deb8  prefixlen 64  scopeid 0x20<link>
         ether 00:07:43:10:de:b8  txqueuelen 1000  (Ethernet)
         RX packets 2156053308  bytes 325330839639 (302.9 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 223004  bytes 24769304 (23.6 MiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
         device interrupt 72

p2p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet 10.50.1.51  netmask 255.255.0.0  broadcast 10.50.255.255
         inet6 fe80::260:ddff:fe44:2aea  prefixlen 64  scopeid 0x20<link>
         ether 00:60:dd:44:2a:ea  txqueuelen 1000  (Ethernet)
         RX packets 49352916  bytes 3501547231 (3.2 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 18876911  bytes 3768900461 (3.5 GiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p2p2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
         inet6 fe80::260:ddff:fe44:2aeb  prefixlen 64  scopeid 0x20<link>
         ether 00:60:dd:44:2a:eb  txqueuelen 1000  (Ethernet)
         RX packets 2491224974  bytes 348058319500 (324.1 GiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 1597  bytes 204525 (199.7 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Any way I can figure out what the third hop is from my tracepath?

Thanks as always for the sage advice!

-erich

On 10/07/2015 09:57 AM, Erich Weiler wrote:
> Yeah, I made the changes and then recreated all the networks.  For some
> reason br-int and the individual virtual instance interfaces on the
> compute node still show 1500 byte frames.
>
> Has anyone else configured jumbo frames in a Kilo environment?  Or maybe
> I'm just an outlier...  ;)
>
> -erich
>
> On 10/07/2015 01:46 AM, Pedro Navarro Perez wrote:
>> Hi Erich,
>>
>> did you recreate the neutron networks after the configuration changes?
>>
>> Pedro Navarro Pérez
>> OpenStack product specialist
>> Red Hat Iberia
>> Passeig de Gràcia 120,
>> 08008 Barcelona
>> Spain
>> M +34 639 642 379
>> E pnavarro at redhat.com
>>
>> ----- Original Message -----
>> From: "Erich Weiler" <weiler at soe.ucsc.edu>
>> To: rdo-list at redhat.com
>> Sent: Wednesday, 7 October, 2015 2:34:28 AM
>> Subject: [Rdo-list] Jumbo MTU to instances in Kilo?
>>
>> Hi Y'all,
>>
>> I know someone must have figured this one out, but I can't seem to get
>> 9000 byte MTUs working.  I have it set in plugin.ini, etc, my nodes have
>> MTU=9000 on their interfaces, so does the network node.  dnsmasq also is
>> configured to set MTU=9000 on instances, which works.  But I still can't
>> ping with large packets to my instance:
>>
>> [weiler at stacker ~]$ ping 10.50.100.2
>> PING 10.50.100.2 (10.50.100.2) 56(84) bytes of data.
>> 64 bytes from 10.50.100.2: icmp_seq=1 ttl=63 time=2.95 ms
>> 64 bytes from 10.50.100.2: icmp_seq=2 ttl=63 time=1.14 ms
>> 64 bytes from 10.50.100.2: icmp_seq=3 ttl=63 time=0.661 ms
>>
>> That works fine.  This however doesn't work:
>>
>> [root at stacker ~]# ping -M do -s 8000 10.50.100.2
>> PING 10.50.100.2 (10.50.100.2) 8000(8028) bytes of data.
>>   From 10.50.100.2 icmp_seq=1 Frag needed and DF set (mtu = 1500)
>> ping: local error: Message too long, mtu=1500
>> ping: local error: Message too long, mtu=1500
>> ping: local error: Message too long, mtu=1500
>> ping: local error: Message too long, mtu=1500
>>
>> It looks like somehow the br-int interface for OVS isn't set at 9000,
>> but I can't figure out how to do that...
>>
>> Here's ifconfig on my compute node:
>>
>> br-enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
>>           inet6 fe80::ec4:7aff:fe58:423e  prefixlen 64  scopeid
>> 0x20<link>
>>           ether 0c:c4:7a:58:42:3e  txqueuelen 0  (Ethernet)
>>           RX packets 2401432  bytes 359276713 (342.6 MiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 30  bytes 1572 (1.5 KiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> br-int: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>           inet6 fe80::64dc:94ff:fe35:db4c  prefixlen 64  scopeid
>> 0x20<link>
>>           ether 66:dc:94:35:db:4c  txqueuelen 0  (Ethernet)
>>           RX packets 69  bytes 6866 (6.7 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 8  bytes 648 (648.0 B)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> enp3s0f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 9000
>>           inet6 fe80::ec4:7aff:fe58:423e  prefixlen 64  scopeid
>> 0x20<link>
>>           ether 0c:c4:7a:58:42:3e  txqueuelen 1000  (Ethernet)
>>           RX packets 130174458  bytes 15334807929 (14.2 GiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 22919305  bytes 5859090420 (5.4 GiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> enp3s0f0.50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
>>           inet 10.50.1.236  netmask 255.255.0.0  broadcast 10.50.255.255
>>           inet6 fe80::ec4:7aff:fe58:423e  prefixlen 64  scopeid
>> 0x20<link>
>>           ether 0c:c4:7a:58:42:3e  txqueuelen 0  (Ethernet)
>>           RX packets 38429352  bytes 5152853436 (4.7 GiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 419842  bytes 101161981 (96.4 MiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
>>           inet 127.0.0.1  netmask 255.0.0.0
>>           inet6 ::1  prefixlen 128  scopeid 0x10<host>
>>           loop  txqueuelen 0  (Local Loopback)
>>           RX packets 22141566  bytes 1185622090 (1.1 GiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 22141566  bytes 1185622090 (1.1 GiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> qbr247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>           inet6 fe80::5c8f:c0ff:fe79:bc11  prefixlen 64  scopeid
>> 0x20<link>
>>           ether b6:1f:54:3f:3d:48  txqueuelen 0  (Ethernet)
>>           RX packets 16  bytes 1472 (1.4 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 8  bytes 648 (648.0 B)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> qbrf42ea01f-fe: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>           inet6 fe80::f484:f1ff:fe53:fb2e  prefixlen 64  scopeid
>> 0x20<link>
>>           ether c2:a6:d8:25:63:ea  txqueuelen 0  (Ethernet)
>>           RX packets 15  bytes 1456 (1.4 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 8  bytes 648 (648.0 B)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> qvb247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>>           inet6 fe80::b41f:54ff:fe3f:3d48  prefixlen 64  scopeid
>> 0x20<link>
>>           ether b6:1f:54:3f:3d:48  txqueuelen 1000  (Ethernet)
>>           RX packets 247  bytes 28323 (27.6 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 233  bytes 25355 (24.7 KiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> qvbf42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>>           inet6 fe80::c0a6:d8ff:fe25:63ea  prefixlen 64  scopeid
>> 0x20<link>
>>           ether c2:a6:d8:25:63:ea  txqueuelen 1000  (Ethernet)
>>           RX packets 377  bytes 57664 (56.3 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 333  bytes 38765 (37.8 KiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> qvo247da3ed-a4: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>>           inet6 fe80::dcfa:f1ff:fe03:ee88  prefixlen 64  scopeid
>> 0x20<link>
>>           ether de:fa:f1:03:ee:88  txqueuelen 1000  (Ethernet)
>>           RX packets 233  bytes 25355 (24.7 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 247  bytes 28323 (27.6 KiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> qvof42ea01f-fe: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>
>> mtu 1500
>>           inet6 fe80::f03e:35ff:fefe:e52  prefixlen 64  scopeid
>> 0x20<link>
>>           ether f2:3e:35:fe:0e:52  txqueuelen 1000  (Ethernet)
>>           RX packets 333  bytes 38765 (37.8 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 377  bytes 57664 (56.3 KiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> tap247da3ed-a4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>>           inet6 fe80::fc16:3eff:fede:5eea  prefixlen 64  scopeid
>> 0x20<link>
>>           ether fe:16:3e:de:5e:ea  txqueuelen 500  (Ethernet)
>>           RX packets 219  bytes 24239 (23.6 KiB)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 224  bytes 26661 (26.0 KiB)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
>>           inet 192.168.122.1  netmask 255.255.255.0  broadcast
>> 192.168.122.255
>>           ether 52:54:00:c4:75:9f  txqueuelen 0  (Ethernet)
>>           RX packets 0  bytes 0 (0.0 B)
>>           RX errors 0  dropped 0  overruns 0  frame 0
>>           TX packets 0  bytes 0 (0.0 B)
>>           TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> This is on RHEL 7.1.  Any obvious way I can get all the intermediate
>> bridges to MTU=9000?  I've RTFM'd and googled to no avail...
>>
>> Here's the ovs-vsctl outout:
>>
>> [root at node-136 ~]# ovs-vsctl show
>> 6f5a5f00-59e2-4420-aeaf-7ad464ead232
>>       Bridge br-int
>>           fail_mode: secure
>>           Port br-int
>>               Interface br-int
>>                   type: internal
>>           Port "qvo247da3ed-a4"
>>               tag: 1
>>               Interface "qvo247da3ed-a4"
>>           Port "int-br-eth1"
>>               Interface "int-br-eth1"
>>           Port "int-br-enp3s0f0"
>>               Interface "int-br-enp3s0f0"
>>                   type: patch
>>                   options: {peer="phy-br-enp3s0f0"}
>>       Bridge "br-enp3s0f0"
>>           Port "enp3s0f0"
>>               Interface "enp3s0f0"
>>           Port "br-enp3s0f0"
>>               Interface "br-enp3s0f0"
>>                   type: internal
>>           Port "phy-br-enp3s0f0"
>>               Interface "phy-br-enp3s0f0"
>>                   type: patch
>>                   options: {peer="int-br-enp3s0f0"}
>>       ovs_version: "2.3.1"
>>
>> Many thanks if anyone has any information on this topic!  Or can point
>> me to some documentation I missed...
>>
>> Thanks,
>> erich
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>>
>> To unsubscribe: rdo-list-unsubscribe at redhat.com
>>




More information about the dev mailing list