[Rdo-list] [Grizzly] Network problem with Quantum + Openvswitch + Vlan
Gary Kotton
gkotton at redhat.com
Sun Apr 28 13:47:30 UTC 2013
Hi,
In addition to that I have discovered that there are the following:
[root at dhcp-4-126 ~]# ps aux |grep dns
nobody 2320 0.0 0.0 12888 576 ? S 09:31 0:00
/usr/sbin/dnsmasq --strict-order --local=// --domain-needed
--pid-file=/var/run/libvirt/network/default.pid --conf-file=
--except-interface lo --bind-interfaces --listen-address 192.168.122.1
--dhcp-range 192.168.122.2,192.168.122.254
--dhcp-leasefile=/var/lib/libvirt/dnsmasq/default.leases
--dhcp-lease-max=253 --dhcp-no-override
--dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile
--addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts
nobody 2718 0.0 0.0 12884 600 ? S 09:32 0:00
/usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order
--bind-interfaces --interface=tapd0ed5836-38 --except-interface=lo
--pid-file=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/host
--dhcp-optsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/opts
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update
--leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --conf-file=
--domain=openstacklocal
root 2719 0.0 0.0 12884 208 ? S 09:32 0:00
/usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order
--bind-interfaces --interface=tapd0ed5836-38 --except-interface=lo
--pid-file=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/host
--dhcp-optsfile=/var/lib/quantum/dhcp/45f9b635-c996-4230-89df-b8c6ac1adb71/opts
--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update
--leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s --conf-file=
--domain=openstacklocal
root 6054 0.0 0.0 103248 840 pts/0 S+ 09:39 0:00 grep dns
When the process 2320 is killed the VM receives its address.
So in short we have some hardening to do :)
With patches for the issues below and shutting down the aforementioned
process I have a VM getting a address.
Thanks
Gary
On 04/28/2013 01:55 PM, Gary Kotton wrote:
> Hi,
> I have found a few problems and hopefully one or more may be related
> to the case that you have experienced:
> 1. When using OVS it is important you run the service
> ovs-quantum-cleanup when the host boots. This is due to the fact that
> OVS will store all tap device. This causes havoc when restarting hosts
> (in particular ones that have dhcp and l3 agents). So please make sure
> you have run "chkconfig quantum-ovs-cleanup on" on all hosts that are
> running the OVS. You can verify if this is the case by checking of the
> DHCP agent has created an IP address on the host. [Please note that we
> have a problem here - in the file /etc/init.d/quantum-ovs-cleanup
> "--config-file /usr/share/$proj/$proj-dist.conf" needs to be removed].
> 2. Which dnsmasq version are you using? If this is 2.48 then there is
> a problem with the DHCP agent running. We are in the process of
> resolving this. If you make use of a version with tag support then
> this will work.
> 3. The quantum client needs to be updated to support the security groups.
> Hopefully we will have solutions for all of the above ASAP.
> Thanks
> Gary
>
>
>
> On 04/28/2013 12:26 PM, Gary Kotton wrote:
>> Hi,
>> I have been able to reproduce the problem. I'll get back to you as
>> soon as I have any information.
>> Thanks
>> Gary
>>
>> On 04/28/2013 11:56 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> Yes, I did run quantum-dhcp-setup on network node.
>>>
>>> Thanks, good luck there.
>>>
>>> Regards,
>>>
>>> Kimi Zhang
>>>
>>> MP: +86 186 0800 8182
>>>
>>> Call me(NCS): sip:+86018608008182
>>>
>>> *From:*ext Gary Kotton [mailto:gkotton at redhat.com]
>>> *Sent:* Sunday, April 28, 2013 4:47 PM
>>> *To:* Zhang, Kimi (NSN - CN/Cheng Du)
>>> *Cc:* rdo-list at redhat.com
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> Thanks.
>>> One more question - on the network node, did you run quantum-dhcp-setup?
>>> I am nearly ready with my setup. Hopefully I'll have a reproduction
>>> or some additional questions.
>>> Thanks
>>> Gary
>>>
>>> On 04/28/2013 11:41 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> Sure, my answers below. :)
>>>
>>> Regards,
>>>
>>> Kimi Zhang
>>>
>>> MP: +86 186 0800 8182
>>>
>>> Call me(NCS): sip:+86018608008182
>>>
>>> *From:*ext Gary Kotton [mailto:gkotton at redhat.com]
>>> *Sent:* Sunday, April 28, 2013 4:31 PM
>>> *To:* Zhang, Kimi (NSN - CN/Cheng Du)
>>> *Cc:* rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> Hi,
>>> I have a few questions (please be patient with me):
>>> 1. On the compute node, which services are running?
>>>
>>> <A> nova-compute, nova-novncproxy, quantum-openvswitch-agent,
>>> openvswitch
>>>
>>>
>>> 2. Can you please print the iptables on the compute node?
>>>
>>> <A> I disabled it already, here's output before I do it.
>>>
>>> [root at computer-2 ~]# iptables-save
>>>
>>> # Generated by iptables-save v1.4.7 on Sun Apr 28 16:37:18 2013
>>>
>>> *filter
>>>
>>> :INPUT ACCEPT [22634:3487580]
>>>
>>> :FORWARD ACCEPT [22:704]
>>>
>>> :OUTPUT ACCEPT [22619:5860198]
>>>
>>> :nova-compute-FORWARD - [0:0]
>>>
>>> :nova-compute-INPUT - [0:0]
>>>
>>> :nova-compute-OUTPUT - [0:0]
>>>
>>> :nova-compute-inst-26 - [0:0]
>>>
>>> :nova-compute-local - [0:0]
>>>
>>> :nova-compute-provider - [0:0]
>>>
>>> :nova-compute-sg-fallback - [0:0]
>>>
>>> :nova-filter-top - [0:0]
>>>
>>> -A INPUT -j nova-compute-INPUT
>>>
>>> -A FORWARD -j nova-filter-top
>>>
>>> -A FORWARD -j nova-compute-FORWARD
>>>
>>> -A OUTPUT -j nova-filter-top
>>>
>>> -A OUTPUT -j nova-compute-OUTPUT
>>>
>>> -A nova-compute-FORWARD -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp
>>> -m udp --sport 68 --dport 67 -j ACCEPT
>>>
>>> -A nova-compute-INPUT -s 0.0.0.0/32 -d 255.255.255.255/32 -p udp -m
>>> udp --sport 68 --dport 67 -j ACCEPT
>>>
>>> -A nova-compute-inst-26 -m state --state INVALID -j DROP
>>>
>>> -A nova-compute-inst-26 -m state --state RELATED,ESTABLISHED -j ACCEPT
>>>
>>> -A nova-compute-inst-26 -j nova-compute-provider
>>>
>>> -A nova-compute-inst-26 -s 172.1.1.3/32 -p udp -m udp --sport 67
>>> --dport 68 -j ACCEPT
>>>
>>> -A nova-compute-inst-26 -s 172.1.1.0/24 -j ACCEPT
>>>
>>> -A nova-compute-inst-26 -p icmp -j ACCEPT
>>>
>>> -A nova-compute-inst-26 -p tcp -m tcp --dport 22 -j ACCEPT
>>>
>>> -A nova-compute-inst-26 -j nova-compute-sg-fallback
>>>
>>> -A nova-compute-local -d 172.1.1.5/32 -j nova-compute-inst-26
>>>
>>> -A nova-compute-sg-fallback -j DROP
>>>
>>> -A nova-filter-top -j nova-compute-local
>>>
>>> COMMIT
>>>
>>> # Completed on Sun Apr 28 16:37:18 2013
>>>
>>> # Generated by iptables-save v1.4.7 on Sun Apr 28 16:37:18 2013
>>>
>>> *mangle
>>>
>>> :PREROUTING ACCEPT [22733:3519752]
>>>
>>> :INPUT ACCEPT [22733:3519752]
>>>
>>> :FORWARD ACCEPT [175:50468]
>>>
>>> :OUTPUT ACCEPT [22705:5868566]
>>>
>>> :POSTROUTING ACCEPT [22880:5919034]
>>>
>>> :nova-compute-POSTROUTING - [0:0]
>>>
>>> -A POSTROUTING -j nova-compute-POSTROUTING
>>>
>>> COMMIT
>>>
>>> # Completed on Sun Apr 28 16:37:18 2013
>>>
>>> # Generated by iptables-save v1.4.7 on Sun Apr 28 16:37:18 2013
>>>
>>> *nat
>>>
>>> :PREROUTING ACCEPT [16:14570]
>>>
>>> :POSTROUTING ACCEPT [338:22855]
>>>
>>> :OUTPUT ACCEPT [331:20579]
>>>
>>> :nova-compute-OUTPUT - [0:0]
>>>
>>> :nova-compute-POSTROUTING - [0:0]
>>>
>>> :nova-compute-PREROUTING - [0:0]
>>>
>>> :nova-compute-float-snat - [0:0]
>>>
>>> :nova-compute-snat - [0:0]
>>>
>>> :nova-postrouting-bottom - [0:0]
>>>
>>> -A PREROUTING -j nova-compute-PREROUTING
>>>
>>> -A POSTROUTING -j nova-compute-POSTROUTING
>>>
>>> -A POSTROUTING -j nova-postrouting-bottom
>>>
>>> -A OUTPUT -j nova-compute-OUTPUT
>>>
>>> -A nova-compute-snat -j nova-compute-float-snat
>>>
>>> -A nova-postrouting-bottom -j nova-compute-snat
>>>
>>> COMMIT
>>>
>>> # Completed on Sun Apr 28 16:37:18 2013
>>>
>>>
>>> 3. Can you please print the flow table rules (ovs-dpctl dump-flows
>>> br-int)?
>>>
>>> <A> I suppose you mean ovs-ofctl dump-flows br-int ?
>>>
>>> [root at computer-2 ~]# ovs-ofctl dump-flows br-int
>>>
>>> NXST_FLOW reply (xid=0x4):
>>>
>>> cookie=0x0, duration=4125.444s, table=0, n_packets=1707,
>>> n_bytes=90606, idle_age=12, priority=1 actions=NORMAL
>>>
>>> cookie=0x0, duration=4123.006s, table=0, n_packets=143,
>>> n_bytes=8688, idle_age=20, priority=2,in_port=1 actions=drop
>>>
>>> cookie=0x0, duration=3349.566s, table=0, n_packets=0, n_bytes=0,
>>> idle_age=3349, priority=3,in_port=1,dl_vlan=1001
>>> actions=mod_vlan_vid:1,NORMAL
>>>
>>> Here's also ovs-dpctl show:
>>>
>>> [root at computer-2 ~]# ovs-dpctl show
>>>
>>> system at br-p3p1:
>>>
>>> lookups: hit:3967 missed:314 lost:0
>>>
>>> flows: 1
>>>
>>> port 0: br-p3p1 (internal)
>>>
>>> port 1: p3p1
>>>
>>> port 2: phy-br-p3p1
>>>
>>> system at br-int:
>>>
>>> lookups: hit:1575 missed:302 lost:0
>>>
>>> flows: 0
>>>
>>> port 0: br-int (internal)
>>>
>>> port 1: int-br-p3p1
>>>
>>> port 4: qvo39242f22-ec
>>>
>>>
>>> Thanks
>>> Gary
>>>
>>> On 04/28/2013 11:17 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> Hi,Gary
>>>
>>> I tried capture packet while keeping VM to restart it's network.
>>>
>>> I can see dhcp request broadcast packet on tap, qbr, qvb and qvo
>>> interfaces.
>>>
>>> Failed to see packet on int-br-p3p1 on bridge br-int.
>>>
>>> Not sure if it has something to do with openflow setting? I attach
>>> some ovs-ofctl outputs
>>>
>>> I have not seen "veth" port anywhere...
>>>
>>> ---Record---
>>>
>>> [root at computer-2 ~]# brctl show
>>>
>>> bridge name bridge id STP enabled interfaces
>>>
>>> qbr39242f22-ec 8000.c6f95e6a859a no
>>> qvb39242f22-ec
>>>
>>> tap39242f22-ec
>>>
>>> virbr0 8000.525400c47f62 yes virbr0-nic
>>>
>>> [root at computer-2 ~]# ovs-vsctl show
>>>
>>> 5660d1b5-1f26-46fc-bcb7-0ccfd06fe57b
>>>
>>> Bridge br-int
>>>
>>> Port br-int
>>>
>>> Interface br-int
>>>
>>> type: internal
>>>
>>> Port "int-br-p3p1"
>>>
>>> Interface "int-br-p3p1"
>>>
>>> Port "qvo39242f22-ec"
>>>
>>> tag: 1
>>>
>>> Interface "qvo39242f22-ec"
>>>
>>> Bridge "br-p3p1"
>>>
>>> Port "phy-br-p3p1"
>>>
>>> Interface "phy-br-p3p1"
>>>
>>> Port "p3p1"
>>>
>>> Interface "p3p1"
>>>
>>> Port "br-p3p1"
>>>
>>> Interface "br-p3p1"
>>>
>>> type: internal
>>>
>>> ovs_version: "1.9.0"
>>>
>>> [root at computer-2 ~]# tcpdump -i tap39242f22-ec port 67
>>>
>>> tcpdump: WARNING: tap39242f22-ec: no IPv4 address assigned
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>>
>>> listening on tap39242f22-ec, link-type EN10MB (Ethernet), capture
>>> size 65535 bytes
>>>
>>> 16:12:21.455212 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>> BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300
>>>
>>> 16:12:21.455289 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>> BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300
>>>
>>> ^C
>>>
>>> 2 packets captured
>>>
>>> 2 packets received by filter
>>>
>>> 0 packets dropped by kernel
>>>
>>> [root at computer-2 ~]# tcpdump -i qbr39242f22-ec port 67
>>>
>>> tcpdump: WARNING: qbr39242f22-ec: no IPv4 address assigned
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>>
>>> listening on qbr39242f22-ec, link-type EN10MB (Ethernet), capture
>>> size 65535 bytes
>>>
>>> 16:12:34.456228 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>> BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300
>>>
>>> ^C
>>>
>>> 1 packets captured
>>>
>>> 1 packets received by filter
>>>
>>> 0 packets dropped by kernel
>>>
>>> [root at computer-2 ~]# tcpdump -i qvb39242f22-ec port 67
>>>
>>> tcpdump: WARNING: qvb39242f22-ec: no IPv4 address assigned
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>>
>>> listening on qvb39242f22-ec, link-type EN10MB (Ethernet), capture
>>> size 65535 bytes
>>>
>>> 16:12:43.460251 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>> BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300
>>>
>>> ^C
>>>
>>> 1 packets captured
>>>
>>> 1 packets received by filter
>>>
>>> 0 packets dropped by kernel
>>>
>>> [root at computer-2 ~]# tcpdump -i qvo39242f22-ec port 67
>>>
>>> tcpdump: WARNING: qvo39242f22-ec: no IPv4 address assigned
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>>
>>> listening on qvo39242f22-ec, link-type EN10MB (Ethernet), capture
>>> size 65535 bytes
>>>
>>> 16:13:03.712272 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>> BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300
>>>
>>> 16:13:08.455932 IP 0.0.0.0.bootpc > 255.255.255.255.bootps:
>>> BOOTP/DHCP, Request from fa:16:3e:15:82:82 (oui Unknown), length 300
>>>
>>> ^C
>>>
>>> 2 packets captured
>>>
>>> 2 packets received by filter
>>>
>>> 0 packets dropped by kernel
>>>
>>> [root at computer-2 ~]# tcpdump -i int-br-p3p1 port 67
>>>
>>> tcpdump: WARNING: int-br-p3p1: no IPv4 address assigned
>>>
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>>
>>> listening on int-br-p3p1, link-type EN10MB (Ethernet), capture size
>>> 65535 bytes
>>>
>>> ^C
>>>
>>> 0 packets captured
>>>
>>> 0 packets received by filter
>>>
>>> 0 packets dropped by kernel
>>>
>>> ---output of ovs-ofctl---
>>>
>>> [root at computer-2 ~]# ovs-ofctl show br-int
>>>
>>> OFPT_FEATURES_REPLY (xid=0x1): dpid:000086401820f142
>>>
>>> n_tables:255, n_buffers:256
>>>
>>> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
>>>
>>> actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
>>> SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST
>>> ENQUEUE
>>>
>>> 1(int-br-p3p1): addr:de:42:e4:9d:b7:1d
>>>
>>> config: 0
>>>
>>> state: 0
>>>
>>> current: 10GB-FD COPPER
>>>
>>> speed: 10000 Mbps now, 100 Mbps max
>>>
>>> 4(qvo39242f22-ec): addr:ea:5d:b8:7e:4a:78
>>>
>>> config: 0
>>>
>>> state: 0
>>>
>>> current: 10GB-FD COPPER
>>>
>>> speed: 10000 Mbps now, 100 Mbps max
>>>
>>> LOCAL(br-int): addr:86:40:18:20:f1:42
>>>
>>> config: PORT_DOWN
>>>
>>> state: LINK_DOWN
>>>
>>> speed: 100 Mbps now, 100 Mbps max
>>>
>>> OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0
>>>
>>> [root at computer-2 ~]#
>>>
>>> [root at computer-2 ~]# ovs-ofctl show br-p3p1
>>>
>>> OFPT_FEATURES_REPLY (xid=0x1): dpid:0000a0369f15d424
>>>
>>> n_tables:255, n_buffers:256
>>>
>>> capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
>>>
>>> actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
>>> SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST
>>> ENQUEUE
>>>
>>> 1(p3p1): addr:a0:36:9f:15:d4:24
>>>
>>> config: 0
>>>
>>> state: 0
>>>
>>> current: 10GB-FD
>>>
>>> advertised: 10GB-FD FIBER
>>>
>>> supported: 10GB-FD FIBER
>>>
>>> speed: 10000 Mbps now, 10000 Mbps max
>>>
>>> 2(phy-br-p3p1): addr:be:3c:f9:8d:d9:d0
>>>
>>> config: 0
>>>
>>> state: 0
>>>
>>> current: 10GB-FD COPPER
>>>
>>> speed: 10000 Mbps now, 100 Mbps max
>>>
>>> LOCAL(br-p3p1): addr:a0:36:9f:15:d4:24
>>>
>>> config: PORT_DOWN
>>>
>>> state: LINK_DOWN
>>>
>>> speed: 100 Mbps now, 100 Mbps max
>>>
>>> OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0
>>>
>>> [root at computer-2 ~]# ovs-ofctl dump-flows br-int
>>>
>>> NXST_FLOW reply (xid=0x4):
>>>
>>> cookie=0x0, duration=4125.444s, table=0, n_packets=1707,
>>> n_bytes=90606, idle_age=12, priority=1 actions=NORMAL
>>>
>>> cookie=0x0, duration=4123.006s, table=0, n_packets=143,
>>> n_bytes=8688, idle_age=20, priority=2,in_port=1 actions=drop
>>>
>>> cookie=0x0, duration=3349.566s, table=0, n_packets=0, n_bytes=0,
>>> idle_age=3349, priority=3,in_port=1,dl_vlan=1001
>>> actions=mod_vlan_vid:1,NORMAL
>>>
>>> [root at computer-2 ~]# ovs-ofctl dump-flows br-p3p1
>>>
>>> NXST_FLOW reply (xid=0x4):
>>>
>>> cookie=0x0, duration=4129.629s, table=0, n_packets=2175,
>>> n_bytes=138652, idle_age=0, priority=1 actions=NORMAL
>>>
>>> cookie=0x0, duration=4127.415s, table=0, n_packets=16, n_bytes=1224,
>>> idle_age=1045, priority=2,in_port=2 actions=drop
>>>
>>> cookie=0x0, duration=3354.578s, table=0, n_packets=1697,
>>> n_bytes=96638, idle_age=17, priority=4,in_port=2,dl_vlan=1
>>> actions=mod_vlan_vid:1001,NORMAL
>>>
>>> Regards,
>>>
>>> Kimi Zhang
>>>
>>> MP: +86 186 0800 8182
>>>
>>> Call me(NCS): sip:+86018608008182
>>>
>>> *From:*rdo-list-bounces at redhat.com
>>> <mailto:rdo-list-bounces at redhat.com>
>>> [mailto:rdo-list-bounces at redhat.com] *On Behalf Of *ext Zhang, Kimi
>>> (NSN - CN/Cheng Du)
>>> *Sent:* Sunday, April 28, 2013 3:40 PM
>>> *To:* gkotton at redhat.com <mailto:gkotton at redhat.com>;
>>> rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> Very nice pic, I am going to try to capture packet on each port.
>>>
>>> I did not configure to use quantum to manage firewall , just leave
>>> it to nova-compute, will try your configs later.
>>>
>>> Regards,
>>>
>>> Kimi Zhang
>>>
>>> MP: +86 186 0800 8182
>>>
>>> Call me(NCS): sip:+86018608008182
>>>
>>> *From:*rdo-list-bounces at redhat.com
>>> <mailto:rdo-list-bounces at redhat.com>
>>> [mailto:rdo-list-bounces at redhat.com] *On Behalf Of *ext Gary Kotton
>>> *Sent:* Sunday, April 28, 2013 3:33 PM
>>> *To:* rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> Hi,
>>> Can you also please check that firewall_driver =
>>> quantum.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver is
>>> configured in plugin.ini file.And security_group_api = quantum is
>>> set in nova.conf
>>> Thanks
>>> Gary
>>>
>>> On 04/28/2013 10:21 AM, Gary Kotton wrote:
>>>
>>> On 04/28/2013 10:16 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> Hi, Gary
>>>
>>> I tried to disable iptables on both network and compute nodes, still
>>> does not work out L
>>>
>>>
>>> Can you please look at
>>> https://docs.google.com/drawings/d/1wax2Nlk-LRJeOXwF_6X9L05cAf9HKl2FI_0B51rG4XE/edit?usp=sharing
>>>
>>> When using the OVS there are a number of devices. Would it be
>>> possible that you try and capture on each device so that we can try
>>> and see where the packet is discarded.
>>>
>>> I will have a setup ready in about an hour.
>>>
>>>
>>>
>>> From quantum openvswitch agent logs, following messages keeps coming
>>> out repeatly every 2-3 seconds, not sure if they matter or not?
>>>
>>>
>>> The messages below are OK - this is how the OVS agent works. It
>>> polls the OVS every interval to check if new ports are created.
>>>
>>>
>>>
>>> 2013-04-28 15:15:39 DEBUG [quantum.openstack.common.rpc.amqp]
>>> Making synchronous call on q-plugin ...
>>>
>>> 2013-04-28 15:15:39 DEBUG [quantum.openstack.common.rpc.amqp]
>>> MSG_ID is 92f4e83cf92c46f1b9304c879f9b7a41
>>>
>>> 2013-04-28 15:15:39 DEBUG [quantum.openstack.common.rpc.amqp]
>>> UNIQUE_ID is b27f9545ca9d4745961ac574abdc103b.
>>>
>>> 2013-04-28 15:15:40 DEBUG [quantum.agent.linux.utils] Running
>>> command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
>>>
>>> 2013-04-28 15:15:40 DEBUG [quantum.agent.linux.utils]
>>>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
>>>
>>> Exit code: 0
>>>
>>> Stdout: 'int-br-p3p1\n'
>>>
>>> Stderr: ''
>>>
>>> 2013-04-28 15:15:40 DEBUG [quantum.agent.linux.utils] Running
>>> command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1',
>>> 'external_ids']
>>>
>>> 2013-04-28 15:15:41 DEBUG [quantum.agent.linux.utils]
>>>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1',
>>> 'external_ids']
>>>
>>> Exit code: 0
>>>
>>> Stdout: '{}\n'
>>>
>>> Stderr: ''
>>>
>>> 2013-04-28 15:15:42 DEBUG [quantum.agent.linux.utils] Running
>>> command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
>>>
>>> 2013-04-28 15:15:42 DEBUG [quantum.agent.linux.utils] Running
>>> command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
>>>
>>> 2013-04-28 15:15:42 DEBUG [quantum.agent.linux.utils]
>>>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
>>>
>>> Exit code: 0
>>>
>>> Stdout: 'int-br-p3p1\n'
>>>
>>> Stderr: ''
>>>
>>> 2013-04-28 15:15:42 DEBUG [quantum.agent.linux.utils] Running
>>> command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1',
>>> 'external_ids']
>>>
>>> 2013-04-28 15:15:42 DEBUG [quantum.agent.linux.utils]
>>>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'list-ports', 'br-int']
>>>
>>> Exit code: 0
>>>
>>> Stdout: 'int-br-p3p1\n'
>>>
>>> Stderr: ''
>>>
>>> 2013-04-28 15:15:42 DEBUG [quantum.agent.linux.utils] Running
>>> command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1',
>>> 'external_ids']
>>>
>>> 2013-04-28 15:15:43 DEBUG [quantum.agent.linux.utils]
>>>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1',
>>> 'external_ids']
>>>
>>> Exit code: 0
>>>
>>> Stdout: '{}\n'
>>>
>>> Stderr: ''
>>>
>>> 2013-04-28 15:15:43 DEBUG [quantum.agent.linux.utils]
>>>
>>> Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf',
>>> 'ovs-vsctl', '--timeout=2', 'get', 'Interface', 'int-br-p3p1',
>>> 'external_ids']
>>>
>>> Exit code: 0
>>>
>>> Stdout: '{}\n'
>>>
>>> Stderr: ''
>>>
>>> Regards,
>>>
>>> Kimi Zhang
>>>
>>> MP: +86 186 0800 8182
>>>
>>> Call me(NCS): sip:+86018608008182
>>>
>>> *From:*ext Gary Kotton [mailto:gkotton at redhat.com]
>>> *Sent:* Sunday, April 28, 2013 3:08 PM
>>> *To:* Zhang, Kimi (NSN - CN/Cheng Du)
>>> *Cc:* rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> On 04/28/2013 10:04 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> I tried that too, no lucky.
>>>
>>> From tcpdump ,it seems br-int does not forward any packet to
>>> interfaces connect to br-p3p1, which connects to physical network...
>>>
>>>
>>> There could be a number of issues here:
>>> 1. The iptables are dropping the traffic (I am in the process of
>>> getting a setup up and running)
>>> 2. The network connectivity
>>>
>>> In order to ensure that it is not the first one can you try and see
>>> which iptables rules are matched or disable the iptables?
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Kimi Zhang
>>>
>>> MP: +86 186 0800 8182
>>>
>>> Call me(NCS): sip:+86018608008182
>>>
>>> *From:*ext Gary Kotton [mailto:gkotton at redhat.com]
>>> *Sent:* Sunday, April 28, 2013 3:01 PM
>>> *To:* Zhang, Kimi (NSN - CN/Cheng Du)
>>> *Cc:* rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> On 04/28/2013 09:54 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> Hi, Gary
>>>
>>> Yes, I'm aware of that packstack does not support quantum yet. The
>>> whole setup was installed manually.
>>>
>>> I did run quantum-server-setup and quantum-host-setup, I tried
>>> linuxbridge plugin too, it has no issue for VM to get IP address,
>>> but openvswitch has issues on this...
>>>
>>>
>>> ok.
>>>
>>> if you configure and IP address manually on the VM are you able to
>>> ping the port of the DHCP agent?
>>>
>>> you can get the IP from quantum port-list
>>>
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> Kimi
>>>
>>> *From:*rdo-list-bounces at redhat.com
>>> <mailto:rdo-list-bounces at redhat.com>
>>> [mailto:rdo-list-bounces at redhat.com] *On Behalf Of *ext Gary Kotton
>>> *Sent:* Sunday, April 28, 2013 2:50 PM
>>> *To:* rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>> *Subject:* Re: [Rdo-list] [Grizzly] Network problem with Quantum +
>>> Openvswitch + Vlan
>>>
>>> Hi Kimi,
>>> Thanks for the mail. Please see the inline comments below. Please
>>> note that at the moment we do not have packstack support for Quantum
>>> so there is a little manual plumbing that needs to be done (not sure
>>> if you have done this already).
>>> On the host where the quantum service is running you need to run
>>> quantum-server-setup and on the compute nodes you need to run
>>> quantum-host-setup (please note that the relevant keystone
>>> credentials need to be set too).
>>> Thanks
>>> Gary
>>>
>>> On 04/28/2013 09:38 AM, Zhang, Kimi (NSN - CN/Cheng Du) wrote:
>>>
>>> converted from rtf
>>>
>>> When I start VM instance, the VM can't get IP address. Could someone
>>> help me on this ?
>>>
>>>
>>> I will try
>>>
>>>
>>>
>>>
>>>
>>> 3 nodes Setup with RHEL 6.4 OS + rdo grizzly repository.
>>>
>>> ·Controller node:
>>>
>>> Services: Keystone+Glance+Cinder+Quantum server + Nova services
>>>
>>> Network: bond0(10.68.125.11 for O&M)
>>>
>>> ·Network node:
>>>
>>> Services: quantum-openvswitch-agent, quantum-l3-agent,
>>> quantum-dhcp-agent, quantum-metadata-agent
>>>
>>> Network: bond0(10.68.125.15 for O&M) , p3p1 for VM internal network,
>>> p3p2 for external network
>>>
>>>
>>> Please note that RHEL currently does not support namespaces so there
>>> are a number of limitations. We are addressing this at the moment.
>>> If namespaces are not used then it is suggested that one does not
>>> run the DHCP agent and the L3 agent on the same host. The reason for
>>> this is that there is no network isolation.
>>>
>>>
>>>
>>>
>>>
>>>
>>> ·Compute node:
>>>
>>> Services: nove-compute and quantum-openvswitch-agent
>>>
>>> Network: bond0(10.68.125.16 for O&M), p3p1 for VM internal network
>>>
>>> ·Switch setup tagging for vlan 1000-2999 for p3p1 ports(VM network)
>>> of network and compute nodes.
>>>
>>> 1.Quantum.conf:
>>>
>>> [DEFAULT]
>>>
>>> debug = True
>>>
>>> verbose = True
>>>
>>> lock_path = $state_path/lock
>>>
>>> bind_host = 0.0.0.0
>>>
>>> bind_port = 9696
>>>
>>> core_plugin =
>>> quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPluginV2
>>>
>>> api_paste_config = api-paste.ini
>>>
>>> rpc_backend = quantum.openstack.common.rpc.impl_kombu
>>>
>>>
>>> Are you using rabbit or qpid?
>>>
>>>
>>>
>>>
>>>
>>>
>>> control_exchange = quantum
>>>
>>> rabbit_host = 10.68.125.11
>>>
>>> notification_driver = quantum.openstack.common.notifier.rpc_notifier
>>>
>>> default_notification_level = INFO
>>>
>>> notification_topics = notifications
>>>
>>> [QUOTAS]
>>>
>>> [DEFAULT_SERVICETYPE]
>>>
>>> [AGENT]
>>>
>>> polling_interval = 2
>>>
>>> root_helper = sudo quantum-rootwrap /etc/quantum/rootwrap.conf
>>>
>>> [keystone_authtoken]
>>>
>>> auth_host = 10.68.125.11
>>>
>>> auth_port = 35357
>>>
>>> auth_protocol = http
>>>
>>> signing_dir = /var/lib/quantum/keystone-signing
>>>
>>> admin_tenant_name = service
>>>
>>> admin_user = quantum
>>>
>>> admin_password = password
>>>
>>> 2.ovs_quantum_plugin.ini
>>>
>>> [DATABASE]
>>>
>>> sql_connection =
>>> mysql://quantum:quantum@10.68.125.11:3306/ovs_quantum
>>> <mailto:mysql://quantum:quantum@10.68.125.11:3306/ovs_quantum>
>>>
>>> reconnect_interval = 2
>>>
>>> [OVS]
>>>
>>> tenant_network_type = vlan
>>>
>>> network_vlan_ranges = physnet1:1000:2999
>>>
>>> bridge_mappings = physnet1:br-p3p1
>>>
>>> [AGENT]
>>>
>>> polling_interval = 2
>>>
>>> [SECURITYGROUP]
>>>
>>> 3.nova.conf
>>>
>>> [DEFAULT]
>>>
>>> verbose=true
>>>
>>> logdir = /var/log/nova
>>>
>>> state_path = /var/lib/nova
>>>
>>> lock_path = /var/lib/nova/tmp
>>>
>>> volumes_dir = /etc/nova/volumes
>>>
>>> dhcpbridge = /usr/bin/nova-dhcpbridge
>>>
>>> dhcpbridge_flagfile = /etc/nova/nova.conf
>>>
>>> force_dhcp_release = True
>>>
>>> injected_network_template = /usr/share/nova/interfaces.template
>>>
>>> libvirt_nonblocking = True
>>>
>>> libvirt_inject_partition = -1
>>>
>>> network_manager = nova.network.manager.FlatDHCPManager
>>>
>>> iscsi_helper = tgtadm
>>>
>>> compute_driver = libvirt.LibvirtDriver
>>>
>>> libvirt_type=kvm
>>>
>>> libvirt_ovs_bridge=br-int
>>>
>>> firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
>>>
>>> manager=nova.conductor.manager.ConductorManager
>>>
>>> rpc_backend = nova.openstack.common.rpc.impl_kombu
>>>
>>> rabbit_host = 10.68.125.11
>>>
>>> rootwrap_config = /etc/nova/rootwrap.conf
>>>
>>> use_deprecated_auth=false
>>>
>>> auth_strategy=keystone
>>>
>>> glance_api_servers=10.68.125.11:9292
>>>
>>> image_service=nova.image.glance.GlanceImageService
>>>
>>> novnc_enabled=true
>>>
>>> novncproxy_base_url=http://10.68.125.11:6080/vnc_auto.html
>>>
>>> novncproxy_port=6080
>>>
>>> vncserver_proxyclient_address=10.68.125.16
>>>
>>> vncserver_listen=0.0.0.0
>>>
>>> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver
>>>
>>> libvirt_use_virtio_for_bridges=True
>>>
>>> network_api_class=nova.network.quantumv2.api.API
>>>
>>> quantum_url=http://10.68.125.11:9696
>>>
>>> quantum_auth_strategy=keystone
>>>
>>> quantum_admin_tenant_name=service
>>>
>>> quantum_admin_username=quantum
>>>
>>> quantum_admin_password=password
>>>
>>> quantum_admin_auth_url=http://10.68.125.11:35357/v2.0
>>>
>>> linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
>>>
>>> libvirt_vif_type=ethernet
>>>
>>> service_quantum_metadata_proxy = True
>>>
>>> quantum_metadata_proxy_shared_secret = helloOpenStack
>>>
>>> metadata_host = 10.68.125.11
>>>
>>> metadata_listen = 0.0.0.0
>>>
>>> metadata_listen_port = 8775
>>>
>>> [keystone_authtoken]
>>>
>>> admin_tenant_name = service
>>>
>>> admin_user = nova
>>>
>>> admin_password = password
>>>
>>> auth_host = 10.68.125.11
>>>
>>> auth_port = 35357
>>>
>>> auth_protocol = http
>>>
>>> signing_dir = /tmp/keystone-signing-nova
>>>
>>> 4.ovs-vsctl show on network node:
>>>
>>> aeeb6cf7-271b-405a-aa17-1b95bcd9e301
>>>
>>> Bridge "br-p3p1"
>>>
>>> Port "p3p1"
>>>
>>> Interface "p3p1"
>>>
>>> Port "phy-br-p3p1"
>>>
>>> Interface "phy-br-p3p1"
>>>
>>> Port "br-p3p1"
>>>
>>> Interface "br-p3p1"
>>>
>>> type: internal
>>>
>>> Bridge br-ex
>>>
>>> Port br-ex
>>>
>>> Interface br-ex
>>>
>>> type: internal
>>>
>>> Port "qg-a83c0abd-f4"
>>>
>>> Interface "qg-a83c0abd-f4"
>>>
>>> type: internal
>>>
>>> Port "p3p2"
>>>
>>> Interface "p3p2"
>>>
>>> Bridge br-int
>>>
>>> Port br-int
>>>
>>> Interface br-int
>>>
>>> type: internal
>>>
>>> Port "int-br-p3p1"
>>>
>>> Interface "int-br-p3p1"
>>>
>>> Port "tap1f386a2a-12"
>>>
>>> tag: 1
>>>
>>> Interface "tap1f386a2a-12"
>>>
>>> type: internal
>>>
>>> ovs_version: "1.9.0"
>>>
>>> 5.ovs-vsctl show on compute node:
>>>
>>> 8d6c2637-ff69-4a2d-a7db-e4f181273bc0
>>>
>>> Bridge "br-p3p1"
>>>
>>> Port "br-p3p1"
>>>
>>> Interface "br-p3p1"
>>>
>>> type: internal
>>>
>>> Port "phy-br-p3p1"
>>>
>>> Interface "phy-br-p3p1"
>>>
>>> Port "p3p1"
>>>
>>> Interface "p3p1"
>>>
>>> Bridge br-int
>>>
>>> Port "qvo56a4572c-dc"
>>>
>>> tag: 2
>>>
>>> Interface "qvo56a4572c-dc"
>>>
>>> Port "int-br-p3p1"
>>>
>>> Interface "int-br-p3p1"
>>>
>>> Port br-int
>>>
>>> Interface br-int
>>>
>>> type: internal
>>>
>>> ovs_version: "1.9.0"
>>>
>>> On compute node, I can see dhcp request packet from tcpdump on
>>> qvo56a4572c-dc, but it seems the packet is not forwarded out since I
>>> can't see packet from int-br-p3p1 on br-int or any port from br-p3p1.
>>>
>>>
>>> Any chance to get the DHCP and the L3 agent configuration files?
>>> Please check that use_namespaces = False in both of these files.
>>>
>>> Are there any log errors?
>>>
>>>
>>>
>>>
>>>
>>>
>>> Thank you!
>>>
>>> Regards,
>>>
>>> Kimi
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Rdo-list mailing list
>>> Rdo-list at redhat.com <mailto:Rdo-list at redhat.com>
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Rdo-list mailing list
>>> Rdo-list at redhat.com <mailto:Rdo-list at redhat.com>
>>> https://www.redhat.com/mailman/listinfo/rdo-list
>>>
>>
>>
>>
>> _______________________________________________
>> Rdo-list mailing list
>> Rdo-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/rdo-list
>
>
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20130428/eec41b18/attachment.html>
More information about the dev
mailing list