[Rdo-list] Outbound packet traffic with nova-network

[Michael Solberg]

On 06/27/2013 01:13 PM, Russell Bryant wrote:
> On 06/26/2013 11:58 PM, Michael Solberg wrote:
>> Hi list.
>>
>> I'm having some trouble with my network setup and I thought maybe the
>> group could help.  I have three compute nodes with two network
>> interfaces.  There's a private 192.168.32.0/24 network (fixed range) on
>> eth1 and a 10.17.12.128/25 network (floating range) on eth0 on these nodes.
>>
>> Networking works as expected.  Connectivity is good on the 192 network
>> and traffic is forwarded from the floating range inbound to the fixed
>> range correctly.  That is, I can ssh into an instance from the outside
>> world using the floating address.  However, when traffic is leaving the
>> instance, it doesn't seem to get translated.  Here's a concrete example:
>>
>> Instance has a fixed address of 192.168.32.4.
>> Instance is assigned a floating address of 10.17.12.139.
>> Instance is running on a hypervisor with the address 10.17.12.12.
>>
>> Pings from the instance to 10.17.12.12 return fine, but pings from the
>> instance out to the internet don't work.  When I run a tcpdump, I see
>> that the pings reach the destination with the source address set to
>> 192.168.32.4:
>>
>> [root at 10.17.12.11 ~]# tcpdump -n -i eth0 icmp
>> tcpdump: WARNING: eth0: no IPv4 address assigned
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 23:55:38.238739 IP 192.168.32.4 > 10.17.12.11: ICMP echo request, id
>> 45321, seq 1, length 64
>> 23:55:38.238774 IP 10.17.12.11 > 192.168.32.4: ICMP echo reply, id
>> 45321, seq 1, length 64
>>
>> Is this expected?  If so, is there any way to get nova-network to
>> translate the outbound traffic so that it looks like it's coming from
>> 10.17.12.139 instead of 192.168.32.4?
>
> Can you provide more information on your network topology?  What
> nova-network mode are you using?

I'm using single-host FlatDHCPManager.  Here's a crude diagram of the 
traffic flow:

(eth0 in VM) 192.168.32.4 -> (on hypervisor) vnet0 -> br100 -> (eth0 
10.17.12.12) -> (across switch to other host) 10.17.12.11 (eth0).

The host at 10.17.12.11 didn't have a route back to 192.168.32.4, so the 
reply packets were just dropped.

> Your example is for "pings from the instance out to the internet", but
> you're pinging an address (10.17.12.11) that's on the same private
> network as the compute node (10.17.12.12).  My guess is that is
> confusing things.

I was able to get the behavior I wanted by adding a masquerade rule to 
the POSTROUTING table:

iptables -t nat -I POSTROUTING 1 -s 192.168.32.0/22 ! -d 192.168.32.0/22 
-j MASQUERADE

Now all traffic out from the virtual machine gets NAT'd when it leaves 
the nova network host:

[root at virt-two ~]# tcpdump -n -i eth0 icmp
tcpdump: WARNING: eth0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
08:39:30.685952 IP 10.17.12.12 > 10.17.12.11: ICMP echo request, id 
45596, seq 1, length 64
08:39:30.686007 IP 10.17.12.11 > 10.17.12.12: ICMP echo reply, id 45596, 
seq 1, length 64

10.17.12.12 routes the packets back to 192.168.32.4.

Michael.