[Rdo-list] Fwd: [Neutron] router can't ping external gateway
Marius Cornea
mcornea at redhat.com
Tue May 19 12:32:08 UTC 2015
Delete and check if other computers in the network are receiving broadcasts:
ip netns exec qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip -s -s neigh flush 192.168.5.1
tcpdump -i <if_name> arp #on one of the computers in the 192.168.5.0 network
ip netns exec qrouter-85fa9459-503d-4996-86f3-6042604fed74 ping 192.168.5.1
See if any ARP requests reach the computer where you run tcpdump.
I'm still thinking about some blocking stuff happening in the vswitch since the ICMP requests are sent to the eth0 interface so they should reach the vswitch port.
----- Original Message -----
> From: "ICHIBA Sara" <ichi.sara at gmail.com>
> To: "Marius Cornea" <mcornea at redhat.com>
> Cc: rdo-list at redhat.com
> Sent: Tuesday, May 19, 2015 2:15:06 PM
> Subject: Re: [Rdo-list] Fwd: [Neutron] router can't ping external gateway
>
> [root at localhost ~(keystone_admin)]# ip netns exec
> qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip n | grep '192.168.5.1 '
> 192.168.5.1 dev qg-e1b584b4-db lladdr 00:23:48:9e:85:7c STALE
>
>
>
>
>
> 2015-05-19 14:12 GMT+02:00 Marius Cornea <mcornea at redhat.com>:
>
> > Is there an ARP entry for 192.168.5.1 ?
> >
> > ip n | grep '192.168.5.1 ' in the router namespace
> >
> >
> >
> > ----- Original Message -----
> > > From: "ICHIBA Sara" <ichi.sara at gmail.com>
> > > To: rdo-list at redhat.com
> > > Sent: Tuesday, May 19, 2015 1:42:11 PM
> > > Subject: [Rdo-list] Fwd: [Neutron] router can't ping external gateway
> > >
> > >
> > > ---------- Forwarded message ----------
> > > From: ICHIBA Sara < ichi.sara at gmail.com >
> > > Date: 2015-05-19 13:41 GMT+02:00
> > > Subject: Re: [Rdo-list] [Neutron] router can't ping external gateway
> > > To: Marius Cornea < mcornea at redhat.com >
> > >
> > >
> > > The forged transmissions on the vswitch are accepted. What's next?
> > >
> > > 2015-05-19 13:29 GMT+02:00 Marius Cornea < mcornea at redhat.com > :
> > >
> > >
> > > Oh, ESXi...I remember that the vswitch had some security features in
> > place.
> > > You can check those and I think the one that you're looking for is called
> > > forged retransmits.
> > >
> > > Thanks,
> > > Marius
> > >
> > > ----- Original Message -----
> > > > From: "ICHIBA Sara" < ichi.sara at gmail.com >
> > > > To: "Marius Cornea" < mcornea at redhat.com >
> > > > Cc: rdo-list at redhat.com
> > > > Sent: Tuesday, May 19, 2015 1:17:20 PM
> > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external gateway
> > > >
> > > > the ICMP requests arrives to the eth0 interface
> > > > [root at localhost ~]# tcpdump -i eth0 icmp
> > > > tcpdump: WARNING: eth0: no IPv4 address assigned
> > > > tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > decode
> > > > listening on eth0, link-type EN10MB (Ethernet), capture size 65535
> > bytes
> > > > 13:14:13.205573 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 1, length 64
> > > > 13:14:14.205303 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 2, length 64
> > > > 13:14:15.205391 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 3, length 64
> > > > 13:14:16.205397 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 4, length 64
> > > > 13:14:17.205408 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 5, length 64
> > > > 13:14:18.205412 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 6, length 64
> > > > 13:14:19.205392 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 7, length 64
> > > > 13:14:20.205357 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id
> > > > 31055,
> > > > seq 8, length 64
> > > > 13:14:33.060267
> > > >
> > > >
> > > > what should I do next?
> > > >
> > > > P.S: My compute and controller hosts are ESXi VMs and I can ssh to
> > both of
> > > > them without a problem.
> > > >
> > > > 2015-05-19 13:00 GMT+02:00 Marius Cornea < mcornea at redhat.com >:
> > > >
> > > > > Also, I'm seeing that you have 2 default routes on your host. I'm not
> > > > > sure
> > > > > it affects the setup but try keeping only one: e.g. 'ip route del
> > default
> > > > > via 192.168.4.1' to delete the eth1 one.
> > > > >
> > > > > ======[root at localhost ~(keystone_admin)]# ip r
> > > > > default via 192.168.5.1 dev br-ex
> > > > > default via 192.168.4.1 dev eth1
> > > > >
> > > > > ----- Original Message -----
> > > > > > From: "Marius Cornea" < mcornea at redhat.com >
> > > > > > To: "ICHIBA Sara" < ichi.sara at gmail.com >
> > > > > > Cc: rdo-list at redhat.com
> > > > > > Sent: Tuesday, May 19, 2015 12:50:45 PM
> > > > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external
> > gateway
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Try to see if any of the ICMP requests leave the eth0 interface
> > like
> > > > > 'tcpdump
> > > > > > -i eth0 icmp' while pinging 192.168.5.1 from the router namespace.
> > > > > >
> > > > > > Thanks,
> > > > > > Marius
> > > > > >
> > > > > > ----- Original Message -----
> > > > > > > From: "ICHIBA Sara" < ichi.sara at gmail.com >
> > > > > > > To: "Boris Derzhavets" < bderzhavets at hotmail.com >,
> > > > > > > rdo-list at redhat.com
> > > > > > > Sent: Tuesday, May 19, 2015 12:12:30 PM
> > > > > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external
> > gateway
> > > > > > >
> > > > > > > ====updates
> > > > > > >
> > > > > > > I have deleted my networks, rebooted my machines and configured
> > an
> > > > > other
> > > > > > > network. Now I can see the qr bridge mapped to the router but
> > still
> > > > > can't
> > > > > > > ping the external gateway:
> > > > > > >
> > > > > > > ====[root at localhost ~(keystone_admin)]# ip netns exec
> > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip r
> > > > > > > default via 192.168.5.1 dev qg-e1b584b4-db
> > > > > > > 10.0.0.0/24 dev qr-7b330e0e-5c proto kernel scope link src
> > 10.0.0.1
> > > > > > > 192.168.5.0/24 dev qg-e1b584b4-db proto kernel scope link src
> > > > > 192.168.5.70
> > > > > > >
> > > > > > > ====[root at localhost ~(keystone_admin)]# ip netns exec
> > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip a
> > > > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
> > UNKNOWN
> > > > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > > > > > > inet 127.0.0.1/8 scope host lo
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > inet6 ::1/128 scope host
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > 12: qg-e1b584b4-db: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> > qdisc
> > > > > > > noqueue
> > > > > > > state UNKNOWN
> > > > > > > link/ether fa:16:3e:68:83:f8 brd ff:ff:ff:ff:ff:ff
> > > > > > > inet 192.168.5.70/24 brd 192.168.5.255 scope global
> > qg-e1b584b4-db
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > inet 192.168.5.73/32 brd 192.168.5.73 scope global
> > qg-e1b584b4-db
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > inet6 fe80::f816:3eff:fe68:83f8/64 scope link
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > 13: qr-7b330e0e-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> > qdisc
> > > > > > > noqueue
> > > > > > > state UNKNOWN
> > > > > > > link/ether fa:16:3e:92:9c:90 brd ff:ff:ff:ff:ff:ff
> > > > > > > inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-7b330e0e-5c
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > inet6 fe80::f816:3eff:fe92:9c90/64 scope link
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > >
> > > > > > >
> > > > > > > =====[root at localhost ~(keystone_admin)]# ip netns exec
> > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ping 192.168.5.1
> > > > > > > PING 192.168.5.1 (192.168.5.1) 56(84) bytes of data.
> > > > > > > From 192.168.5.70 icmp_seq=10 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=11 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=12 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=13 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=14 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=15 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=16 Destination Host Unreachable
> > > > > > > From 192.168.5.70 icmp_seq=17 Destination Host Unreachable
> > > > > > >
> > > > > > >
> > > > > > > =====[root at localhost ~(keystone_admin)]# ovs-vsctl show
> > > > > > > 19de58db-509d-4de8-bd88-9222019b13f1
> > > > > > > Bridge br-int
> > > > > > > fail_mode: secure
> > > > > > > Port "tap2decc1bc-bf"
> > > > > > > tag: 2
> > > > > > > Interface "tap2decc1bc-bf"
> > > > > > > type: internal
> > > > > > > Port br-int
> > > > > > > Interface br-int
> > > > > > > type: internal
> > > > > > > Port patch-tun
> > > > > > > Interface patch-tun
> > > > > > > type: patch
> > > > > > > options: {peer=patch-int}
> > > > > > > Port "qr-7b330e0e-5c"
> > > > > > > tag: 2
> > > > > > > Interface "qr-7b330e0e-5c"
> > > > > > > type: internal
> > > > > > > Port "qvo164afbd4-0c"
> > > > > > > tag: 2
> > > > > > > Interface "qvo164afbd4-0c"
> > > > > > > Bridge br-ex
> > > > > > > Port "eth0"
> > > > > > > Interface "eth0"
> > > > > > > Port br-ex
> > > > > > > Interface br-ex
> > > > > > > type: internal
> > > > > > > Port "qg-e1b584b4-db"
> > > > > > > Interface "qg-e1b584b4-db"
> > > > > > > type: internal
> > > > > > > Bridge br-tun
> > > > > > > Port br-tun
> > > > > > > Interface br-tun
> > > > > > > type: internal
> > > > > > > Port "vxlan-c0a80520"
> > > > > > > Interface "vxlan-c0a80520"
> > > > > > > type: vxlan
> > > > > > > options: {df_default="true", in_key=flow,
> > local_ip="192.168.5.33",
> > > > > > > out_key=flow, remote_ip="192.168.5.32"}
> > > > > > > Port patch-int
> > > > > > > Interface patch-int
> > > > > > > type: patch
> > > > > > > options: {peer=patch-tun}
> > > > > > > ovs_version: "2.3.1"
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2015-05-19 11:58 GMT+02:00 ICHIBA Sara < ichi.sara at gmail.com > :
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > can you show me your plugin.ini file? /etc/neutron/plugin.ini
> > and the
> > > > > other
> > > > > > > file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
> > > > > > >
> > > > > > >
> > > > > > > 2015-05-19 10:47 GMT+02:00 Boris Derzhavets <
> > bderzhavets at hotmail.com
> > > > > > :
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > There is one thing , which I clearly see . It is
> > qrouter-namespace
> > > > > > > misconfiguration. There is no qr-xxxxx bridge attached to br-int
> > > > > > > Picture , in general, should look like this
> > > > > > >
> > > > > > > ubuntu at ubuntu-System:~$ sudo ip netns exec
> > > > > > > qrouter-6cb93ddd-2637-449d-8b10-7c07da49ee8c route -n
> > > > > > >
> > > > > > > Kernel IP routing table
> > > > > > > Destination Gateway Genmask Flags Metric Ref Use Iface
> > > > > > > 0.0.0.0 192.168.12.15 0.0.0.0 UG 0 0 0 qg-a753a8f5-c8
> > > > > > > 10.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-393d9f71-53
> > > > > > > 192.168.12.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-a753a8f5-c8
> > > > > > >
> > > > > > > ubuntu at ubuntu-System:~$ sudo ip netns exec
> > > > > > > qrouter-6cb93ddd-2637-449d-8b10-7c07da49ee8c ifconfig
> > > > > > > lo Link encap:Local Loopback
> > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > UP LOOPBACK RUNNING MTU:65536 Metric:1
> > > > > > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0
> > > > > > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
> > > > > > >
> > > > > > > qg-a753a8f5-c8 Link encap:Ethernet HWaddr fa:16:3e:a2:11:b4
> > > > > > > inet addr:192.168.12.150 Bcast:192.168.12.255 Mask:255.255.255.0
> > > > > > > inet6 addr: fe80::f816:3eff:fea2:11b4/64 Scope:Link
> > > > > > > UP BROADCAST RUNNING MTU:1500 Metric:1
> > > > > > > RX packets:24504 errors:0 dropped:0 overruns:0 frame:0
> > > > > > > TX packets:17367 errors:0 dropped:0 overruns:0 carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:24328699 (24.3 MB) TX bytes:1443691 (1.4 MB)
> > > > > > >
> > > > > > > qr-393d9f71-53 Link encap:Ethernet HWaddr fa:16:3e:9e:ec:01
> > > > > > > inet addr:10.254.1.1 Bcast:10.254.1.255 Mask:255.255.255.0
> > > > > > > inet6 addr: fe80::f816:3eff:fe9e:ec01/64 Scope:Link
> > > > > > > UP BROADCAST RUNNING MTU:1500 Metric:1
> > > > > > > RX packets:22487 errors:0 dropped:5 overruns:0 frame:0
> > > > > > > TX packets:24736 errors:0 dropped:0 overruns:0 carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:2379287 (2.3 MB) TX bytes:24338711 (24.3 MB)
> > > > > > >
> > > > > > > I would also advise you to post a question also on
> > ask.openstack.org
> > > > > > >
> > > > > > > Boris.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Date: Tue, 19 May 2015 09:48:58 +0200
> > > > > > > From: ichi.sara at gmail.com
> > > > > > > To: rdo-list at redhat.com
> > > > > > > Subject: [Rdo-list] [Neutron] router can't ping external gateway
> > > > > > >
> > > > > > >
> > > > > > > Hey people,
> > > > > > > I have an issue with my networking. I connected my openstack to
> > an
> > > > > external
> > > > > > > network I did all the changes required. But still my router can't
> > > > > reach the
> > > > > > > external gateway.
> > > > > > >
> > > > > > > =====ifcfg-br-ex
> > > > > > > DEVICE=br-ex
> > > > > > > DEVICETYPE=ovs
> > > > > > > TYPE=OVSBridge
> > > > > > > BOOTPROTO=static
> > > > > > > IPADDR=192.168.5.33
> > > > > > > NETMASK=255.255.255.0
> > > > > > > ONBOOT=yes
> > > > > > > GATEWAY=192.168.5.1
> > > > > > > DNS1=8.8.8.8
> > > > > > > DNS2=192.168.5.1
> > > > > > >
> > > > > > >
> > > > > > > ====ifcfg-eth0
> > > > > > > DEVICE=eth0
> > > > > > > HWADDR=00:0c:29:a2:b1:b9
> > > > > > > ONBOOT=yes
> > > > > > > TYPE=OVSPort
> > > > > > > NM_CONTROLLED=yes
> > > > > > > DEVICETYPE=ovs
> > > > > > > OVS_BRIDGE=br-ex
> > > > > > >
> > > > > > > ======[root at localhost ~(keystone_admin)]# ovs-vsctl show
> > > > > > > 19de58db-509d-4de8-bd88-9222019b13f1
> > > > > > > Bridge br-int
> > > > > > > fail_mode: secure
> > > > > > > Port "tap8652132e-b8"
> > > > > > > tag: 1
> > > > > > > Interface "tap8652132e-b8"
> > > > > > > type: internal
> > > > > > > Port br-int
> > > > > > > Interface br-int
> > > > > > > type: internal
> > > > > > > Port patch-tun
> > > > > > > Interface patch-tun
> > > > > > > type: patch
> > > > > > > options: {peer=patch-int}
> > > > > > > Bridge br-ex
> > > > > > > Port "qg-5f8ebe30-40"
> > > > > > > Interface "qg-5f8ebe30-40"
> > > > > > > type: internal
> > > > > > > Port "eth0"
> > > > > > > Interface "eth0"
> > > > > > > Port br-ex
> > > > > > > Interface br-ex
> > > > > > > type: internal
> > > > > > > Bridge br-tun
> > > > > > > Port "vxlan-c0a80520"
> > > > > > > Interface "vxlan-c0a80520"
> > > > > > > type: vxlan
> > > > > > > options: {df_default="true", in_key=flow,
> > local_ip="192.168.5.33",
> > > > > > > out_key=flow, remote_ip="192.168.5.32"}
> > > > > > > Port br-tun
> > > > > > > Interface br-tun
> > > > > > > type: internal
> > > > > > > Port patch-int
> > > > > > > Interface patch-int
> > > > > > > type: patch
> > > > > > > options: {peer=patch-tun}
> > > > > > > ovs_version: "2.3.1"
> > > > > > >
> > > > > > > =====[root at localhost ~(keystone_admin)]# ping 192.168.5.1
> > > > > > > PING 192.168.5.1 (192.168.5.1) 56(84) bytes of data.
> > > > > > > 64 bytes from 192.168.5.1 : icmp_seq=1 ttl=64 time=1.76 ms
> > > > > > > 64 bytes from 192.168.5.1 : icmp_seq=2 ttl=64 time=1.88 ms
> > > > > > > 64 bytes from 192.168.5.1 : icmp_seq=3 ttl=64 time=1.45 ms
> > > > > > > ^C
> > > > > > > --- 192.168.5.1 ping statistics ---
> > > > > > > 3 packets transmitted, 3 received, 0% packet loss, time 2002ms
> > > > > > > rtt min/avg/max/mdev = 1.452/1.699/1.880/0.187 ms
> > > > > > > [root at localhost ~(keystone_admin)]#
> > > > > > >
> > > > > > > ======[root at localhost ~(keystone_admin)]# ip netns exec
> > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip a
> > > > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
> > UNKNOWN
> > > > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > > > > > > inet 127.0.0.1/8 scope host lo
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > inet6 ::1/128 scope host
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > 14: qg-5f8ebe30-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> > qdisc
> > > > > > > noqueue
> > > > > > > state UNKNOWN
> > > > > > > link/ether fa:16:3e:c2:1b:5e brd ff:ff:ff:ff:ff:ff
> > > > > > > inet 192.168.5.70/24 brd 192.168.5.255 scope global
> > qg-5f8ebe30-40
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > inet6 fe80::f816:3eff:fec2:1b5e/64 scope link
> > > > > > > valid_lft forever preferred_lft forever
> > > > > > > [root at localhost ~(keystone_admin)]#
> > > > > > >
> > > > > > >
> > > > > > > ======[root at localhost ~(keystone_admin)]# ip r
> > > > > > > default via 192.168.5.1 dev br-ex
> > > > > > > default via 192.168.4.1 dev eth1
> > > > > > > 169.254.0.0/16 dev eth0 scope link metric 1002
> > > > > > > 169.254.0.0/16 dev eth1 scope link metric 1003
> > > > > > > 169.254.0.0/16 dev br-ex scope link metric 1005
> > > > > > > 192.168.4.0/24 dev eth1 proto kernel scope link src 192.168.4.14
> > > > > > > 192.168.5.0/24 dev br-ex proto kernel scope link src
> > 192.168.5.33
> > > > > > > [root at localhost ~(keystone_admin)]#
> > > > > > >
> > > > > > >
> > > > > > > ======[root at localhost ~(keystone_admin)]# ip netns exec
> > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip r
> > > > > > > default via 192.168.5.1 dev qg-5f8ebe30-40
> > > > > > > 192.168.5.0/24 dev qg-5f8ebe30-40 proto kernel scope link src
> > > > > 192.168.5.70
> > > > > > > [root at localhost ~(keystone_admin)]#
> > > > > > >
> > > > > > >
> > > > > > > ======[root at localhost ~(keystone_admin)]# ip netns exec
> > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ping 192.168.5.1
> > > > > > > PING 192.168.5.1 (192.168.5.1) 56(84) bytes of data.
> > > > > > > ^C
> > > > > > > --- 192.168.5.1 ping statistics ---
> > > > > > > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms
> > > > > > >
> > > > > > > any hints??
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________ Rdo-list mailing
> > list
> > > > > > > Rdo-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/rdo-list
> > > > > To
> > > > > > > unsubscribe: rdo-list-unsubscribe at redhat.com
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Rdo-list mailing list
> > > > > > > Rdo-list at redhat.com
> > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list
> > > > > > >
> > > > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com
> > > > > >
> > > > > > _______________________________________________
> > > > > > Rdo-list mailing list
> > > > > > Rdo-list at redhat.com
> > > > > > https://www.redhat.com/mailman/listinfo/rdo-list
> > > > > >
> > > > > > To unsubscribe: rdo-list-unsubscribe at redhat.com
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Rdo-list mailing list
> > > Rdo-list at redhat.com
> > > https://www.redhat.com/mailman/listinfo/rdo-list
> > >
> > > To unsubscribe: rdo-list-unsubscribe at redhat.com
> >
>
More information about the dev
mailing list