Hi, Gary

Finally I found the root cause:  the vlan handling of RHEL 6.4:

In my setup, on both gateway and compute node, I use NIC p3p1 as internal traffic port, it’s added into br-int.

Analysis:

I do tcpdump on compute node p3p1 port, find the frame sending out the port does have vlan tag, but when I do tcpdump on gateway node (receiver) p3p1 port, I see the frame has no vlan tag on it. Somehow the OS eats the vlan  tag of incoming frame from p3p1 port. That causes openvswitch on gateway node can't handle these abnormal frames.  Test shows the same results for traffic from gateway node to compute node.

What I have done to fix this:

On both gateway and compute node, I add a vlan interface p3p1.195 (vlan 195 is actually not used).  After this, the OS can handle vlan tagging on frames received from p3p1 correctly. It seems the NIC does not know how to handle vlan tags unless there’s at least one vlan configured on the port.

It can be a OS bug of just a configuration problem.  I don't know if there’s a better way to activate vlan handling on the port without actually creating a vlan interface on it. ? Any ideas ?

About your findings.

1. quantum-ovs-cleanup script, yes, it has a bug, I did the same fix too and enable startup run for it.

2. Yes, I use dnsmasq 2.48 since it comes from OS image. I do find running it has problem, vm can’t get right gateway IP, always uses dhcp server(dhcp agent) IP as gateway IP. After I upgrade it to 2.65, it works normally.

3. Yes, I notice the same that quantum client does not support security groups, even a client from folsom release suppot that, this should be updated.

4. The 192.168.122.x network in dnsmasq was default network created by libvirtd, I already removed after libvirtd installation by running:

virsh net-destroy default
virsh net-undefine default

I don’t think it’s the cause of the problem

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: rdo-list-bounces@redhat.com [mailto:rdo-list-bounces@redhat.com] On Behalf Of ext Gary Kotton
Sent: Sunday, April 28, 2013 9:48 PM
To: gkotton@redhat.com
Cc: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

Hi,
In addition to that I have discovered that there are the following:

[root@dhcp-4-126 ~]# ps aux |grep dns
nobody    2320  0.0  0.0  12888   576 ?        S    09:31   0:00 /usr/sbin/dnsmasq --strict-order --local=// --domain-needed --pid-file=/var/run/libvirt/network/default.pid --conf-file= --except-interface lo --bind-interfaces --listen-address 192.168.122.1 --dhcp-range 192.168.122.2,192.168.122.254 --dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases --dhcp-lease-max=253 --dhcp-no-override --dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile --addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts
nobody    2718  0.0  0.0  12884   600 ?        S    09:32   0:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapd0ed5836-38 --except-interface=lo --pid-file=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/host --dhcp-optsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --conf-file= --domain=openstacklocal
root      2719  0.0  0.0  12884   208 ?        S    09:32   0:00 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapd0ed5836-38 --except-interface=lo --pid-file=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/host --dhcp-optsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --conf-file= --domain=openstacklocal
root      6054  0.0  0.0 103248   840 pts/0    S+   09:39   0:00 grep dns

When the process 2320 is killed the VM receives its address.

So in short we have some hardening to do :)
With patches for the issues below and shutting down the aforementioned process I have a VM getting a address.

Thanks
Gary


On 04/28/2013 01:55 PM, Gary Kotton wrote:

Hi,
I have found a few problems and hopefully one or more may be related to the case that you have experienced:
1. When using OVS it is important you run the service ovs-quantum-cleanup when the host boots. This is due to the fact that OVS will store all tap device. This causes havoc when restarting hosts (in particular ones that have dhcp and l3 agents). So please make sure you have run "chkconfig quantum-ovs-cleanup on" on all hosts that are running the OVS. You can verify if this is the case by checking of the DHCP agent has created an IP address on the host. [Please note that we have a problem here - in the file /etc/init.d/quantum-ovs-cleanup "--config-file /usr/share/$proj/$proj-dist.conf" needs to be removed].
2. Which dnsmasq version are you using? If this is 2.48 then there is a problem with the DHCP agent running. We are in the process of resolving this. If you make use of a version with tag support then this will work.
3. The quantum client needs to be updated to support the security groups.
Hopefully we will have solutions for all of the above ASAP.
Thanks
Gary



On 04/28/2013 12:26 PM, Gary Kotton wrote:

Hi,
I have been able to reproduce the problem. I'll get back to you as soon as I have any information.
Thanks
Gary

On 04/28/2013 11:56 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

Yes, I did run quantum-dhcp-setup on network node.

Thanks, good luck there.

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: ext Gary Kotton [mailto:gkotton@redhat.com]
Sent: Sunday, April 28, 2013 4:47 PM
To: Zhang, Kimi (NSN - CN/Cheng Du)
Cc: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

Thanks.
One more question - on the network node, did you run quantum-dhcp-setup?
I am nearly ready with my setup. Hopefully I'll have a reproduction or some additional questions.
Thanks
Gary

On 04/28/2013 11:41 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

Sure, my answers below. :)

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: ext Gary Kotton [mailto:gkotton@redhat.com]
Sent: Sunday, April 28, 2013 4:31 PM
To: Zhang, Kimi (NSN - CN/Cheng Du)
Cc: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

Hi,
I have a few questions (please be patient with me):
1. On the compute node, which services are running?

<A> nova-compute, nova-novncproxy, quantum-openvswitch-agent, openvswitch

2. Can you please print the iptables on the compute node?

<A> I disabled it already, here's output before I do it.

[root@computer-2 ~]# iptables-save

# Generated by iptables-save v1.4.7 on Sun Apr 28 16:37:18 2013

*filter

:INPUT ACCEPT [22634:3487580]

:FORWARD ACCEPT [22:704]

:OUTPUT ACCEPT [22619:5860198]

:nova-compute-FORWARD - [0:0]

:nova-compute-INPUT - [0:0]

:nova-compute-OUTPUT - [0:0]

:nova-compute-inst-26 - [0:0]

:nova-compute-local - [0:0]

:nova-compute-provider - [0:0]

:nova-compute-sg-fallback - [0:0]

:nova-filter-top - [0:0]

-A INPUT -j nova-compute-INPUT

-A FORWARD -j nova-filter-top

-A FORWARD -j nova-compute-FORWARD

-A OUTPUT -j nova-filter-top

-A OUTPUT -j nova-compute-OUTPUT

-A nova-compute-FORWARD -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT

-A nova-compute-INPUT -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m udp --sport 68 --dport 67 -j ACCEPT

-A nova-compute-inst-26 -m state --state INVALID -j DROP

-A nova-compute-inst-26 -m state --state RELATED,ESTABLISHED -j ACCEPT

-A nova-compute-inst-26 -j nova-compute-provider

-A nova-compute-inst-26 -s 172.1.1.3/32 -p udp -m udp --sport 67 --dport 68 -j ACCEPT

-A nova-compute-inst-26 -s 172.1.1.0/24 -j ACCEPT

-A nova-compute-inst-26 -p icmp -j ACCEPT

-A nova-compute-inst-26 -p tcp -m tcp --dport 22 -j ACCEPT

-A nova-compute-inst-26 -j nova-compute-sg-fallback

-A nova-compute-local -d 172.1.1.5/32 -j nova-compute-inst-26

-A nova-compute-sg-fallback -j DROP

-A nova-filter-top -j nova-compute-local

COMMIT

# Completed on Sun Apr 28 16:37:18 2013

# Generated by iptables-save v1.4.7 on Sun Apr 28 16:37:18 2013

*mangle

:PREROUTING ACCEPT [22733:3519752]

:INPUT ACCEPT [22733:3519752]

:FORWARD ACCEPT [175:50468]

:OUTPUT ACCEPT [22705:5868566]

:POSTROUTING ACCEPT [22880:5919034]

:nova-compute-POSTROUTING - [0:0]

-A POSTROUTING -j nova-compute-POSTROUTING

COMMIT

# Completed on Sun Apr 28 16:37:18 2013

# Generated by iptables-save v1.4.7 on Sun Apr 28 16:37:18 2013

*nat

:PREROUTING ACCEPT [16:14570]

:POSTROUTING ACCEPT [338:22855]

:OUTPUT ACCEPT [331:20579]

:nova-compute-OUTPUT - [0:0]

:nova-compute-POSTROUTING - [0:0]

:nova-compute-PREROUTING - [0:0]

:nova-compute-float-snat - [0:0]

:nova-compute-snat - [0:0]

:nova-postrouting-bottom - [0:0]

-A PREROUTING -j nova-compute-PREROUTING

-A POSTROUTING -j nova-compute-POSTROUTING

-A POSTROUTING -j nova-postrouting-bottom

-A OUTPUT -j nova-compute-OUTPUT

-A nova-compute-snat -j nova-compute-float-snat

-A nova-postrouting-bottom -j nova-compute-snat

COMMIT

# Completed on Sun Apr 28 16:37:18 2013

3. Can you please print the flow table rules (ovs-dpctl dump-flows br-int)?

<A> I suppose you mean ovs-ofctl dump-flows br-int ?

[root@computer-2 ~]# ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=4125.444s, table=0, n_packets=1707, n_bytes=90606, idle_age=12, priority=1 actions=NORMAL

cookie=0x0, duration=4123.006s, table=0, n_packets=143, n_bytes=8688, idle_age=20, priority=2,in_port=1 actions=drop

cookie=0x0, duration=3349.566s, table=0, n_packets=0, n_bytes=0, idle_age=3349, priority=3,in_port=1,dl_vlan=1001 actions=mod_vlan_vid:1,NORMAL

Here’s also ovs-dpctl show:

[root@computer-2 ~]# ovs-dpctl show

system@br-p3p1:

        lookups: hit:3967 missed:314 lost:0

        flows: 1

        port 0: br-p3p1 (internal)

        port 1: p3p1

        port 2: phy-br-p3p1

system@br-int:

        lookups: hit:1575 missed:302 lost:0

        flows: 0

        port 0: br-int (internal)

        port 1: int-br-p3p1

        port 4: qvo39242f22-ec

Thanks
Gary

On 04/28/2013 11:17 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

Hi， Gary

I tried capture packet while keeping VM to restart it’s network.

I can see dhcp request broadcast packet on tap, qbr, qvb and qvo interfaces.

Failed to see packet on int-br-p3p1 on bridge br-int.

Not sure if it has something to do with openflow setting?  I attach some ovs-ofctl outputs

I have not  seen “veth” port anywhere…

---Record---

[root@computer-2 ~]# brctl show

bridge name     bridge id               STP enabled     interfaces

qbr39242f22-ec          8000.c6f95e6a859a       no              qvb39242f22-ec

                                                        tap39242f22-ec

virbr0          8000.525400c47f62       yes             virbr0-nic

[root@computer-2 ~]# ovs-vsctl show

5660d1b5-1f26-46fc-bcb7-0ccfd06fe57b

    Bridge br-int

        Port br-int

            Interface br-int

                type: internal

        Port "int-br-p3p1"

            Interface "int-br-p3p1"

        Port "qvo39242f22-ec"

            tag: 1

            Interface "qvo39242f22-ec"

    Bridge "br-p3p1"

        Port "phy-br-p3p1"

            Interface "phy-br-p3p1"

        Port "p3p1"

            Interface "p3p1"

        Port "br-p3p1"

            Interface "br-p3p1"

                type: internal

    ovs_version: "1.9.0"

[root@computer-2 ~]# tcpdump -i  tap39242f22-ec    port 67             

tcpdump: WARNING: tap39242f22-ec: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on tap39242f22-ec, link-type EN10MB (Ethernet), capture size 65535 bytes

16:12:21.455212 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300

16:12:21.455289 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300

^C

2 packets captured

2 packets received by filter

0 packets dropped by kernel

[root@computer-2 ~]# tcpdump -i  qbr39242f22-ec     port 67             

tcpdump: WARNING: qbr39242f22-ec: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on qbr39242f22-ec, link-type EN10MB (Ethernet), capture size 65535 bytes

16:12:34.456228 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300

^C

1 packets captured

1 packets received by filter

0 packets dropped by kernel

[root@computer-2 ~]# tcpdump -i   qvb39242f22-ec     port 67             

tcpdump: WARNING: qvb39242f22-ec: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on qvb39242f22-ec, link-type EN10MB (Ethernet), capture size 65535 bytes

16:12:43.460251 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300

^C

1 packets captured

1 packets received by filter

0 packets dropped by kernel

[root@computer-2 ~]# tcpdump -i   qvo39242f22-ec   port 67               

tcpdump: WARNING: qvo39242f22-ec: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on qvo39242f22-ec, link-type EN10MB (Ethernet), capture size 65535 bytes

16:13:03.712272 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300

16:13:08.455932 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300

^C

2 packets captured

2 packets received by filter

0 packets dropped by kernel

[root@computer-2 ~]# tcpdump -i int-br-p3p1   port 67                

tcpdump: WARNING: int-br-p3p1: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on int-br-p3p1, link-type EN10MB (Ethernet), capture size 65535 bytes

^C

0 packets captured

0 packets received by filter

0 packets dropped by kernel

---output of ovs-ofctl---

[root@computer-2 ~]# ovs-ofctl show br-int

OFPT_FEATURES_REPLY (xid=0x1): dpid:000086401820f142

n_tables:255, n_buffers:256

capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP

actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE

1(int-br-p3p1): addr:de:42:e4:9d:b7:1d

     config:     0

     state:      0

     current:    10GB-FD COPPER

     speed: 10000 Mbps now, 100 Mbps max

4(qvo39242f22-ec): addr:ea:5d:b8:7e:4a:78

     config:     0

     state:      0

     current:    10GB-FD COPPER

     speed: 10000 Mbps now, 100 Mbps max

LOCAL(br-int): addr:86:40:18:20:f1:42

     config:     PORT_DOWN

     state:      LINK_DOWN

     speed: 100 Mbps now, 100 Mbps max

OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0

[root@computer-2 ~]#

[root@computer-2 ~]# ovs-ofctl show br-p3p1

OFPT_FEATURES_REPLY (xid=0x1): dpid:0000a0369f15d424

n_tables:255, n_buffers:256

capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP

actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST ENQUEUE

1(p3p1): addr:a0:36:9f:15:d4:24

     config:     0

     state:      0

     current:    10GB-FD

     advertised: 10GB-FD FIBER

     supported:  10GB-FD FIBER

     speed: 10000 Mbps now, 10000 Mbps max

2(phy-br-p3p1): addr:be:3c:f9:8d:d9:d0

     config:     0

     state:      0

     current:    10GB-FD COPPER

     speed: 10000 Mbps now, 100 Mbps max

LOCAL(br-p3p1): addr:a0:36:9f:15:d4:24

     config:     PORT_DOWN

     state:      LINK_DOWN

     speed: 100 Mbps now, 100 Mbps max

OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0

[root@computer-2 ~]# ovs-ofctl dump-flows br-int

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=4125.444s, table=0, n_packets=1707, n_bytes=90606, idle_age=12, priority=1 actions=NORMAL

cookie=0x0, duration=4123.006s, table=0, n_packets=143, n_bytes=8688, idle_age=20, priority=2,in_port=1 actions=drop

cookie=0x0, duration=3349.566s, table=0, n_packets=0, n_bytes=0, idle_age=3349, priority=3,in_port=1,dl_vlan=1001 actions=mod_vlan_vid:1,NORMAL

[root@computer-2 ~]# ovs-ofctl dump-flows br-p3p1

NXST_FLOW reply (xid=0x4):

cookie=0x0, duration=4129.629s, table=0, n_packets=2175, n_bytes=138652, idle_age=0, priority=1 actions=NORMAL

cookie=0x0, duration=4127.415s, table=0, n_packets=16, n_bytes=1224, idle_age=1045, priority=2,in_port=2 actions=drop

cookie=0x0, duration=3354.578s, table=0, n_packets=1697, n_bytes=96638, idle_age=17, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:1001,NORMAL

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: rdo-list-bounces@redhat.com [mailto:rdo-list-bounces@redhat.com] On Behalf Of ext Zhang, Kimi (NSN - CN/Cheng Du)
Sent: Sunday, April 28, 2013 3:40 PM
To: gkotton@redhat.com; rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

Very nice pic, I am going to try to capture packet on each port.

I did not configure to use quantum to manage firewall , just leave it to nova-compute, will try your configs later.

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: rdo-list-bounces@redhat.com [mailto:rdo-list-bounces@redhat.com] On Behalf Of ext Gary Kotton
Sent: Sunday, April 28, 2013 3:33 PM
To: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

Hi,
Can you also please check that firewall_driver = quantum.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver is configured in plugin.ini file.And security_group_api = quantum is set in nova.conf
Thanks
Gary

On 04/28/2013 10:21 AM, Gary Kotton wrote:

On 04/28/2013 10:16 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

Hi, Gary

I tried to disable iptables on both network and compute nodes, still does not work out L

Can you please look at https://docs.google.com/drawings/d/1wax2Nlk-LRJeOXwF_6X9L05cAf9HKl2FI_0B51rG4XE/edit?usp=sharing

When using the OVS there are a number of devices. Would it be possible that you try and capture on each device so that we can try and see where the packet is discarded.

I will have a setup ready in about an hour.

From quantum openvswitch agent logs, following messages keeps coming out repeatly every 2-3 seconds, not sure if they matter or not?

The messages below are OK - this is how the OVS agent works. It polls the OVS every interval to check if new ports are created.

2013-04-28 15:15:39    DEBUG [quantum.openstack.common.rpc.amqp] Making synchronous call on q-plugin ...

2013-04-28 15:15:39    DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is 92f4e83cf92c46f1b9304c879f9b7a41

2013-04-28 15:15:39    DEBUG [quantum.openstack.common.rpc.amqp] UNIQUE_ID is b27f9545ca9d4745961ac574abdc103b.

2013-04-28 15:15:40    DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']

2013-04-28 15:15:40    DEBUG [quantum.agent.linux.utils]

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']

Exit code: 0

Stdout: 'int-br-p3p1\n'

Stderr: ''

2013-04-28 15:15:40    DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1', 'external_ids']

2013-04-28 15:15:41    DEBUG [quantum.agent.linux.utils]

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1', 'external_ids']

Exit code: 0

Stdout: '{}\n'

Stderr: ''

2013-04-28 15:15:42    DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']

2013-04-28 15:15:42    DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']

2013-04-28 15:15:42    DEBUG [quantum.agent.linux.utils]

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']

Exit code: 0

Stdout: 'int-br-p3p1\n'

Stderr: ''

2013-04-28 15:15:42    DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1', 'external_ids']

2013-04-28 15:15:42    DEBUG [quantum.agent.linux.utils]

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']

Exit code: 0

Stdout: 'int-br-p3p1\n'

Stderr: ''

2013-04-28 15:15:42    DEBUG [quantum.agent.linux.utils] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1', 'external_ids']

2013-04-28 15:15:43    DEBUG [quantum.agent.linux.utils]

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1', 'external_ids']

Exit code: 0

Stdout: '{}\n'

Stderr: ''

2013-04-28 15:15:43    DEBUG [quantum.agent.linux.utils]

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1', 'external_ids']

Exit code: 0

Stdout: '{}\n'

Stderr: ''

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: ext Gary Kotton [mailto:gkotton@redhat.com]
Sent: Sunday, April 28, 2013 3:08 PM
To: Zhang, Kimi (NSN - CN/Cheng Du)
Cc: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

On 04/28/2013 10:04 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

I tried that too, no lucky.

From tcpdump ,it seems br-int does not forward any packet to interfaces connect to br-p3p1, which connects to physical network…

There could be a number of issues here:
1. The iptables are dropping the traffic (I am in the process of getting a setup up and running)
2. The network connectivity

In order to ensure that it is not the first one can you try and see which iptables rules are matched or disable the iptables?

Regards,

Kimi Zhang

MP: +86 186 0800 8182

Call me(NCS): sip:+86018608008182

From: ext Gary Kotton [mailto:gkotton@redhat.com]
Sent: Sunday, April 28, 2013 3:01 PM
To: Zhang, Kimi (NSN - CN/Cheng Du)
Cc: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

On 04/28/2013 09:54 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

Hi, Gary

Yes, I’m aware of that packstack does not support quantum yet.  The whole setup was installed manually.

I did run quantum-server-setup and quantum-host-setup, I tried linuxbridge plugin too, it has no issue for VM to get IP address, but openvswitch has issues on this…

ok.

if you configure and IP address manually on the VM are you able to ping the port of the DHCP agent?

you can get the IP from quantum port-list

Regards,

Kimi

From: rdo-list-bounces@redhat.com [mailto:rdo-list-bounces@redhat.com] On Behalf Of ext Gary Kotton
Sent: Sunday, April 28, 2013 2:50 PM
To: rdo-list@redhat.com
Subject: Re: [Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan

Hi Kimi,
Thanks for the mail. Please see the inline comments below. Please note that at the moment we do not have packstack support for Quantum so there is a little manual plumbing that needs to be done (not sure if you have done this already).
On the host where the quantum service is running you need to run quantum-server-setup and on the compute nodes you need to run quantum-host-setup (please note that the relevant keystone credentials need to be set too).
Thanks
Gary

On 04/28/2013 09:38 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:

converted from rtf

When I start VM instance, the VM can’t get IP address. Could someone help me on this ?

I will try

3 nodes Setup with RHEL 6.4 OS + rdo grizzly repository.

· Controller node:

Services: Keystone+Glance+Cinder+Quantum server + Nova services

Network: bond0(10.68.125.11 for O&M)

· Network node:

Services: quantum-openvswitch-agent,  quantum-l3-agent, quantum-dhcp-agent, quantum-metadata-agent

Network: bond0(10.68.125.15 for O&M) , p3p1 for VM internal network, p3p2 for external network

Please note that RHEL currently does not support namespaces so there are a number of limitations. We are addressing this at the moment. If namespaces are not used then it is suggested that one does not run the DHCP agent and the L3 agent on the same host. The reason for this is that there is no network isolation.

· Compute node:

Services: nove-compute and quantum-openvswitch-agent

Network: bond0(10.68.125.16 for O&M), p3p1 for VM internal network

· Switch setup tagging for vlan 1000-2999 for p3p1 ports(VM network) of network and compute nodes.

1. Quantum.conf:

[DEFAULT]

debug = True

verbose = True

lock_path = $state_path/lock

bind_host = 0.0.0.0

bind_port = 9696

core_plugin = quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPluginV2

api_paste_config = api-paste.ini

rpc_backend = quantum.openstack.common.rpc.impl_kombu

Are you using rabbit or qpid?

control_exchange = quantum

rabbit_host = 10.68.125.11

notification_driver = quantum.openstack.common.notifier.rpc_notifier

default_notification_level = INFO

notification_topics = notifications

[QUOTAS]

[DEFAULT_SERVICETYPE]

[AGENT]

polling_interval = 2

root_helper = sudo quantum-rootwrap /etc/quantum/rootwrap.conf

[keystone_authtoken]

auth_host = 10.68.125.11

auth_port = 35357

auth_protocol = http

signing_dir = /var/lib/quantum/keystone-signing

admin_tenant_name = service

admin_user = quantum

admin_password = password

2. ovs_quantum_plugin.ini

[DATABASE]

sql_connection = mysql://quantum:quantum@10.68.125.11:3306/ovs_quantum

reconnect_interval = 2

[OVS]

tenant_network_type = vlan

network_vlan_ranges = physnet1:1000:2999

bridge_mappings = physnet1:br-p3p1

[AGENT]

polling_interval = 2

[SECURITYGROUP]

3. nova.conf

[DEFAULT]

verbose=true

logdir = /var/log/nova

state_path = /var/lib/nova

lock_path = /var/lib/nova/tmp

volumes_dir = /etc/nova/volumes

dhcpbridge = /usr/bin/nova-dhcpbridge

dhcpbridge_flagfile = /etc/nova/nova.conf

force_dhcp_release = True

injected_network_template = /usr/share/nova/interfaces.template

libvirt_nonblocking = True

libvirt_inject_partition = -1

network_manager = nova.network.manager.FlatDHCPManager

iscsi_helper = tgtadm

compute_driver = libvirt.LibvirtDriver

libvirt_type=kvm

libvirt_ovs_bridge=br-int

firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver

manager=nova.conductor.manager.ConductorManager

rpc_backend = nova.openstack.common.rpc.impl_kombu

rabbit_host = 10.68.125.11

rootwrap_config = /etc/nova/rootwrap.conf

use_deprecated_auth=false

auth_strategy=keystone

glance_api_servers=10.68.125.11:9292

image_service=nova.image.glance.GlanceImageService

novnc_enabled=true

novncproxy_base_url=http://10.68.125.11:6080/vnc_auto.html

novncproxy_port=6080

vncserver_proxyclient_address=10.68.125.16

vncserver_listen=0.0.0.0

libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver

libvirt_use_virtio_for_bridges=True

network_api_class=nova.network.quantumv2.api.API

quantum_url=http://10.68.125.11:9696

quantum_auth_strategy=keystone

quantum_admin_tenant_name=service

quantum_admin_username=quantum

quantum_admin_password=password

quantum_admin_auth_url=http://10.68.125.11:35357/v2.0

linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver

libvirt_vif_type=ethernet

service_quantum_metadata_proxy = True

quantum_metadata_proxy_shared_secret = helloOpenStack

metadata_host = 10.68.125.11

metadata_listen = 0.0.0.0

metadata_listen_port = 8775

[keystone_authtoken]

admin_tenant_name = service

admin_user = nova

admin_password = password

auth_host = 10.68.125.11

auth_port = 35357

auth_protocol = http

signing_dir = /tmp/keystone-signing-nova

4. ovs-vsctl show on network node:

aeeb6cf7-271b-405a-aa17-1b95bcd9e301

    Bridge "br-p3p1"

        Port "p3p1"

            Interface "p3p1"

        Port "phy-br-p3p1"

            Interface "phy-br-p3p1"

        Port "br-p3p1"

            Interface "br-p3p1"

                type: internal

    Bridge br-ex

        Port br-ex

            Interface br-ex

                type: internal

        Port "qg-a83c0abd-f4"

            Interface "qg-a83c0abd-f4"

                type: internal

        Port "p3p2"

            Interface "p3p2"

    Bridge br-int

        Port br-int

            Interface br-int

                type: internal

        Port "int-br-p3p1"

            Interface "int-br-p3p1"

        Port "tap1f386a2a-12"

            tag: 1

            Interface "tap1f386a2a-12"

                type: internal

ovs_version: "1.9.0"

5. ovs-vsctl show on compute node:

8d6c2637-ff69-4a2d-a7db-e4f181273bc0

    Bridge "br-p3p1"

        Port "br-p3p1"

            Interface "br-p3p1"

                type: internal

        Port "phy-br-p3p1"

            Interface "phy-br-p3p1"

        Port "p3p1"

            Interface "p3p1"

    Bridge br-int

        Port "qvo56a4572c-dc"

            tag: 2

            Interface "qvo56a4572c-dc"

        Port "int-br-p3p1"

            Interface "int-br-p3p1"

        Port br-int

            Interface br-int

                type: internal

ovs_version: "1.9.0"

On compute node, I can see dhcp request packet from tcpdump on qvo56a4572c-dc, but it seems the packet is not forwarded out since I can’t see packet from int-br-p3p1 on br-int or any port from br-p3p1.

Any chance to get the DHCP and the L3 agent configuration files? Please check that use_namespaces = False in both of these files.

Are there any log errors?

Thank you!

Regards,

Kimi

_______________________________________________
Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

_______________________________________________
Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

_______________________________________________
Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list

_______________________________________________
Rdo-list mailing list
Rdo-list@redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list