Re: [Rdo-list] Fwd: [Neutron] router can't ping external gateway

Delete and check if other computers in the network are receiving broadcasts: ip netns exec qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip -s -s neigh flush 192.168.5.1 tcpdump -i <if_name> arp #on one of the computers in the 192.168.5.0 network ip netns exec qrouter-85fa9459-503d-4996-86f3-6042604fed74 ping 192.168.5.1 See if any ARP requests reach the computer where you run tcpdump. I'm still thinking about some blocking stuff happening in the vswitch since the ICMP requests are sent to the eth0 interface so they should reach the vswitch port. ----- Original Message ----- > From: "ICHIBA Sara" <ichi.sara(a)gmail.com&gt; > To: "Marius Cornea" <mcornea(a)redhat.com&gt; > Cc: rdo-list(a)redhat.com > Sent: Tuesday, May 19, 2015 2:15:06 PM > Subject: Re: [Rdo-list] Fwd: [Neutron] router can't ping external gateway > > [root@localhost ~(keystone_admin)]# ip netns exec > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip n | grep '192.168.5.1 ' > 192.168.5.1 dev qg-e1b584b4-db lladdr 00:23:48:9e:85:7c STALE > > > > > > 2015-05-19 14:12 GMT+02:00 Marius Cornea <mcornea(a)redhat.com&gt;: > > > Is there an ARP entry for 192.168.5.1 ? > > > > ip n | grep '192.168.5.1 ' in the router namespace > > > > > > > > ----- Original Message ----- > > > From: "ICHIBA Sara" <ichi.sara(a)gmail.com&gt; > > > To: rdo-list(a)redhat.com > > > Sent: Tuesday, May 19, 2015 1:42:11 PM > > > Subject: [Rdo-list] Fwd: [Neutron] router can't ping external gateway > > > > > > > > > ---------- Forwarded message ---------- > > > From: ICHIBA Sara < ichi.sara(a)gmail.com > > > > Date: 2015-05-19 13:41 GMT+02:00 > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external gateway > > > To: Marius Cornea < mcornea(a)redhat.com > > > > > > > > > > The forged transmissions on the vswitch are accepted. What's next? > > > > > > 2015-05-19 13:29 GMT+02:00 Marius Cornea < mcornea(a)redhat.com > : > > > > > > > > > Oh, ESXi...I remember that the vswitch had some security features in > > place. > > > You can check those and I think the one that you're looking for is called > > > forged retransmits. > > > > > > Thanks, > > > Marius > > > > > > ----- Original Message ----- > > > > From: "ICHIBA Sara" < ichi.sara(a)gmail.com > > > > > To: "Marius Cornea" < mcornea(a)redhat.com > > > > > Cc: rdo-list(a)redhat.com > > > > Sent: Tuesday, May 19, 2015 1:17:20 PM > > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external gateway > > > > > > > > the ICMP requests arrives to the eth0 interface > > > > [root@localhost ~]# tcpdump -i eth0 icmp > > > > tcpdump: WARNING: eth0: no IPv4 address assigned > > > > tcpdump: verbose output suppressed, use -v or -vv for full protocol > > decode > > > > listening on eth0, link-type EN10MB (Ethernet), capture size 65535 > > bytes > > > > 13:14:13.205573 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 1, length 64 > > > > 13:14:14.205303 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 2, length 64 > > > > 13:14:15.205391 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 3, length 64 > > > > 13:14:16.205397 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 4, length 64 > > > > 13:14:17.205408 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 5, length 64 > > > > 13:14:18.205412 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 6, length 64 > > > > 13:14:19.205392 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 7, length 64 > > > > 13:14:20.205357 IP 192.168.5.70 > 192.168.5.1 : ICMP echo request, id > > > > 31055, > > > > seq 8, length 64 > > > > 13:14:33.060267 > > > > > > > > > > > > what should I do next? > > > > > > > > P.S: My compute and controller hosts are ESXi VMs and I can ssh to > > both of > > > > them without a problem. > > > > > > > > 2015-05-19 13:00 GMT+02:00 Marius Cornea < mcornea(a)redhat.com >: > > > > > > > > > Also, I'm seeing that you have 2 default routes on your host. I'm not > > > > > sure > > > > > it affects the setup but try keeping only one: e.g. 'ip route del > > default > > > > > via 192.168.4.1' to delete the eth1 one. > > > > > > > > > > ======[root@localhost ~(keystone_admin)]# ip r > > > > > default via 192.168.5.1 dev br-ex > > > > > default via 192.168.4.1 dev eth1 > > > > > > > > > > ----- Original Message ----- > > > > > > From: "Marius Cornea" < mcornea(a)redhat.com > > > > > > > To: "ICHIBA Sara" < ichi.sara(a)gmail.com > > > > > > > Cc: rdo-list(a)redhat.com > > > > > > Sent: Tuesday, May 19, 2015 12:50:45 PM > > > > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external > > gateway > > > > > > > > > > > > Hi, > > > > > > > > > > > > Try to see if any of the ICMP requests leave the eth0 interface > > like > > > > > 'tcpdump > > > > > > -i eth0 icmp' while pinging 192.168.5.1 from the router namespace. > > > > > > > > > > > > Thanks, > > > > > > Marius > > > > > > > > > > > > ----- Original Message ----- > > > > > > > From: "ICHIBA Sara" < ichi.sara(a)gmail.com > > > > > > > > To: "Boris Derzhavets" < bderzhavets(a)hotmail.com >, > > > > > > > rdo-list(a)redhat.com > > > > > > > Sent: Tuesday, May 19, 2015 12:12:30 PM > > > > > > > Subject: Re: [Rdo-list] [Neutron] router can't ping external > > gateway > > > > > > > > > > > > > > ====updates > > > > > > > > > > > > > > I have deleted my networks, rebooted my machines and configured > > an > > > > > other > > > > > > > network. Now I can see the qr bridge mapped to the router but > > still > > > > > can't > > > > > > > ping the external gateway: > > > > > > > > > > > > > > ====[root@localhost ~(keystone_admin)]# ip netns exec > > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip r > > > > > > > default via 192.168.5.1 dev qg-e1b584b4-db > > > > > > > 10.0.0.0/24 dev qr-7b330e0e-5c proto kernel scope link src > > 10.0.0.1 > > > > > > > 192.168.5.0/24 dev qg-e1b584b4-db proto kernel scope link src > > > > > 192.168.5.70 > > > > > > > > > > > > > > ====[root@localhost ~(keystone_admin)]# ip netns exec > > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip a > > > > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > UNKNOWN > > > > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > > > > > inet 127.0.0.1/8 scope host lo > > > > > > > valid_lft forever preferred_lft forever > > > > > > > inet6 ::1/128 scope host > > > > > > > valid_lft forever preferred_lft forever > > > > > > > 12: qg-e1b584b4-db: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > > qdisc > > > > > > > noqueue > > > > > > > state UNKNOWN > > > > > > > link/ether fa:16:3e:68:83:f8 brd ff:ff:ff:ff:ff:ff > > > > > > > inet 192.168.5.70/24 brd 192.168.5.255 scope global > > qg-e1b584b4-db > > > > > > > valid_lft forever preferred_lft forever > > > > > > > inet 192.168.5.73/32 brd 192.168.5.73 scope global > > qg-e1b584b4-db > > > > > > > valid_lft forever preferred_lft forever > > > > > > > inet6 fe80::f816:3eff:fe68:83f8/64 scope link > > > > > > > valid_lft forever preferred_lft forever > > > > > > > 13: qr-7b330e0e-5c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > > qdisc > > > > > > > noqueue > > > > > > > state UNKNOWN > > > > > > > link/ether fa:16:3e:92:9c:90 brd ff:ff:ff:ff:ff:ff > > > > > > > inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-7b330e0e-5c > > > > > > > valid_lft forever preferred_lft forever > > > > > > > inet6 fe80::f816:3eff:fe92:9c90/64 scope link > > > > > > > valid_lft forever preferred_lft forever > > > > > > > > > > > > > > > > > > > > > =====[root@localhost ~(keystone_admin)]# ip netns exec > > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ping 192.168.5.1 > > > > > > > PING 192.168.5.1 (192.168.5.1) 56(84) bytes of data. > > > > > > > From 192.168.5.70 icmp_seq=10 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=11 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=12 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=13 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=14 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=15 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=16 Destination Host Unreachable > > > > > > > From 192.168.5.70 icmp_seq=17 Destination Host Unreachable > > > > > > > > > > > > > > > > > > > > > =====[root@localhost ~(keystone_admin)]# ovs-vsctl show > > > > > > > 19de58db-509d-4de8-bd88-9222019b13f1 > > > > > > > Bridge br-int > > > > > > > fail_mode: secure > > > > > > > Port "tap2decc1bc-bf" > > > > > > > tag: 2 > > > > > > > Interface "tap2decc1bc-bf" > > > > > > > type: internal > > > > > > > Port br-int > > > > > > > Interface br-int > > > > > > > type: internal > > > > > > > Port patch-tun > > > > > > > Interface patch-tun > > > > > > > type: patch > > > > > > > options: {peer=patch-int} > > > > > > > Port "qr-7b330e0e-5c" > > > > > > > tag: 2 > > > > > > > Interface "qr-7b330e0e-5c" > > > > > > > type: internal > > > > > > > Port "qvo164afbd4-0c" > > > > > > > tag: 2 > > > > > > > Interface "qvo164afbd4-0c" > > > > > > > Bridge br-ex > > > > > > > Port "eth0" > > > > > > > Interface "eth0" > > > > > > > Port br-ex > > > > > > > Interface br-ex > > > > > > > type: internal > > > > > > > Port "qg-e1b584b4-db" > > > > > > > Interface "qg-e1b584b4-db" > > > > > > > type: internal > > > > > > > Bridge br-tun > > > > > > > Port br-tun > > > > > > > Interface br-tun > > > > > > > type: internal > > > > > > > Port "vxlan-c0a80520" > > > > > > > Interface "vxlan-c0a80520" > > > > > > > type: vxlan > > > > > > > options: {df_default="true", in_key=flow, > > local_ip="192.168.5.33", > > > > > > > out_key=flow, remote_ip="192.168.5.32"} > > > > > > > Port patch-int > > > > > > > Interface patch-int > > > > > > > type: patch > > > > > > > options: {peer=patch-tun} > > > > > > > ovs_version: "2.3.1" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2015-05-19 11:58 GMT+02:00 ICHIBA Sara < ichi.sara(a)gmail.com > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > can you show me your plugin.ini file? /etc/neutron/plugin.ini > > and the > > > > > other > > > > > > > file /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini > > > > > > > > > > > > > > > > > > > > > 2015-05-19 10:47 GMT+02:00 Boris Derzhavets < > > bderzhavets(a)hotmail.com > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is one thing , which I clearly see . It is > > qrouter-namespace > > > > > > > misconfiguration. There is no qr-xxxxx bridge attached to br-int > > > > > > > Picture , in general, should look like this > > > > > > > > > > > > > > ubuntu@ubuntu-System:~$ sudo ip netns exec > > > > > > > qrouter-6cb93ddd-2637-449d-8b10-7c07da49ee8c route -n > > > > > > > > > > > > > > Kernel IP routing table > > > > > > > Destination Gateway Genmask Flags Metric Ref Use Iface > > > > > > > 0.0.0.0 192.168.12.15 0.0.0.0 UG 0 0 0 qg-a753a8f5-c8 > > > > > > > 10.254.1.0 0.0.0.0 255.255.255.0 U 0 0 0 qr-393d9f71-53 > > > > > > > 192.168.12.0 0.0.0.0 255.255.255.0 U 0 0 0 qg-a753a8f5-c8 > > > > > > > > > > > > > > ubuntu@ubuntu-System:~$ sudo ip netns exec > > > > > > > qrouter-6cb93ddd-2637-449d-8b10-7c07da49ee8c ifconfig > > > > > > > lo Link encap:Local Loopback > > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0 > > > > > > > inet6 addr: ::1/128 Scope:Host > > > > > > > UP LOOPBACK RUNNING MTU:65536 Metric:1 > > > > > > > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > > > > > > > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > > > > > > > collisions:0 txqueuelen:0 > > > > > > > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > > > > > > > > > > > > > > qg-a753a8f5-c8 Link encap:Ethernet HWaddr fa:16:3e:a2:11:b4 > > > > > > > inet addr:192.168.12.150 Bcast:192.168.12.255 Mask:255.255.255.0 > > > > > > > inet6 addr: fe80::f816:3eff:fea2:11b4/64 Scope:Link > > > > > > > UP BROADCAST RUNNING MTU:1500 Metric:1 > > > > > > > RX packets:24504 errors:0 dropped:0 overruns:0 frame:0 > > > > > > > TX packets:17367 errors:0 dropped:0 overruns:0 carrier:0 > > > > > > > collisions:0 txqueuelen:0 > > > > > > > RX bytes:24328699 (24.3 MB) TX bytes:1443691 (1.4 MB) > > > > > > > > > > > > > > qr-393d9f71-53 Link encap:Ethernet HWaddr fa:16:3e:9e:ec:01 > > > > > > > inet addr:10.254.1.1 Bcast:10.254.1.255 Mask:255.255.255.0 > > > > > > > inet6 addr: fe80::f816:3eff:fe9e:ec01/64 Scope:Link > > > > > > > UP BROADCAST RUNNING MTU:1500 Metric:1 > > > > > > > RX packets:22487 errors:0 dropped:5 overruns:0 frame:0 > > > > > > > TX packets:24736 errors:0 dropped:0 overruns:0 carrier:0 > > > > > > > collisions:0 txqueuelen:0 > > > > > > > RX bytes:2379287 (2.3 MB) TX bytes:24338711 (24.3 MB) > > > > > > > > > > > > > > I would also advise you to post a question also on > > ask.openstack.org > > > > > > > > > > > > > > Boris. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Date: Tue, 19 May 2015 09:48:58 +0200 > > > > > > > From: ichi.sara(a)gmail.com > > > > > > > To: rdo-list(a)redhat.com > > > > > > > Subject: [Rdo-list] [Neutron] router can't ping external gateway > > > > > > > > > > > > > > > > > > > > > Hey people, > > > > > > > I have an issue with my networking. I connected my openstack to > > an > > > > > external > > > > > > > network I did all the changes required. But still my router can't > > > > > reach the > > > > > > > external gateway. > > > > > > > > > > > > > > =====ifcfg-br-ex > > > > > > > DEVICE=br-ex > > > > > > > DEVICETYPE=ovs > > > > > > > TYPE=OVSBridge > > > > > > > BOOTPROTO=static > > > > > > > IPADDR=192.168.5.33 > > > > > > > NETMASK=255.255.255.0 > > > > > > > ONBOOT=yes > > > > > > > GATEWAY=192.168.5.1 > > > > > > > DNS1=8.8.8.8 > > > > > > > DNS2=192.168.5.1 > > > > > > > > > > > > > > > > > > > > > ====ifcfg-eth0 > > > > > > > DEVICE=eth0 > > > > > > > HWADDR=00:0c:29:a2:b1:b9 > > > > > > > ONBOOT=yes > > > > > > > TYPE=OVSPort > > > > > > > NM_CONTROLLED=yes > > > > > > > DEVICETYPE=ovs > > > > > > > OVS_BRIDGE=br-ex > > > > > > > > > > > > > > ======[root@localhost ~(keystone_admin)]# ovs-vsctl show > > > > > > > 19de58db-509d-4de8-bd88-9222019b13f1 > > > > > > > Bridge br-int > > > > > > > fail_mode: secure > > > > > > > Port "tap8652132e-b8" > > > > > > > tag: 1 > > > > > > > Interface "tap8652132e-b8" > > > > > > > type: internal > > > > > > > Port br-int > > > > > > > Interface br-int > > > > > > > type: internal > > > > > > > Port patch-tun > > > > > > > Interface patch-tun > > > > > > > type: patch > > > > > > > options: {peer=patch-int} > > > > > > > Bridge br-ex > > > > > > > Port "qg-5f8ebe30-40" > > > > > > > Interface "qg-5f8ebe30-40" > > > > > > > type: internal > > > > > > > Port "eth0" > > > > > > > Interface "eth0" > > > > > > > Port br-ex > > > > > > > Interface br-ex > > > > > > > type: internal > > > > > > > Bridge br-tun > > > > > > > Port "vxlan-c0a80520" > > > > > > > Interface "vxlan-c0a80520" > > > > > > > type: vxlan > > > > > > > options: {df_default="true", in_key=flow, > > local_ip="192.168.5.33", > > > > > > > out_key=flow, remote_ip="192.168.5.32"} > > > > > > > Port br-tun > > > > > > > Interface br-tun > > > > > > > type: internal > > > > > > > Port patch-int > > > > > > > Interface patch-int > > > > > > > type: patch > > > > > > > options: {peer=patch-tun} > > > > > > > ovs_version: "2.3.1" > > > > > > > > > > > > > > =====[root@localhost ~(keystone_admin)]# ping 192.168.5.1 > > > > > > > PING 192.168.5.1 (192.168.5.1) 56(84) bytes of data. > > > > > > > 64 bytes from 192.168.5.1 : icmp_seq=1 ttl=64 time=1.76 ms > > > > > > > 64 bytes from 192.168.5.1 : icmp_seq=2 ttl=64 time=1.88 ms > > > > > > > 64 bytes from 192.168.5.1 : icmp_seq=3 ttl=64 time=1.45 ms > > > > > > > ^C > > > > > > > --- 192.168.5.1 ping statistics --- > > > > > > > 3 packets transmitted, 3 received, 0% packet loss, time 2002ms > > > > > > > rtt min/avg/max/mdev = 1.452/1.699/1.880/0.187 ms > > > > > > > [root@localhost ~(keystone_admin)]# > > > > > > > > > > > > > > ======[root@localhost ~(keystone_admin)]# ip netns exec > > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip a > > > > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state > > UNKNOWN > > > > > > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > > > > > > inet 127.0.0.1/8 scope host lo > > > > > > > valid_lft forever preferred_lft forever > > > > > > > inet6 ::1/128 scope host > > > > > > > valid_lft forever preferred_lft forever > > > > > > > 14: qg-5f8ebe30-40: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 > > qdisc > > > > > > > noqueue > > > > > > > state UNKNOWN > > > > > > > link/ether fa:16:3e:c2:1b:5e brd ff:ff:ff:ff:ff:ff > > > > > > > inet 192.168.5.70/24 brd 192.168.5.255 scope global > > qg-5f8ebe30-40 > > > > > > > valid_lft forever preferred_lft forever > > > > > > > inet6 fe80::f816:3eff:fec2:1b5e/64 scope link > > > > > > > valid_lft forever preferred_lft forever > > > > > > > [root@localhost ~(keystone_admin)]# > > > > > > > > > > > > > > > > > > > > > ======[root@localhost ~(keystone_admin)]# ip r > > > > > > > default via 192.168.5.1 dev br-ex > > > > > > > default via 192.168.4.1 dev eth1 > > > > > > > 169.254.0.0/16 dev eth0 scope link metric 1002 > > > > > > > 169.254.0.0/16 dev eth1 scope link metric 1003 > > > > > > > 169.254.0.0/16 dev br-ex scope link metric 1005 > > > > > > > 192.168.4.0/24 dev eth1 proto kernel scope link src 192.168.4.14 > > > > > > > 192.168.5.0/24 dev br-ex proto kernel scope link src > > 192.168.5.33 > > > > > > > [root@localhost ~(keystone_admin)]# > > > > > > > > > > > > > > > > > > > > > ======[root@localhost ~(keystone_admin)]# ip netns exec > > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ip r > > > > > > > default via 192.168.5.1 dev qg-5f8ebe30-40 > > > > > > > 192.168.5.0/24 dev qg-5f8ebe30-40 proto kernel scope link src > > > > > 192.168.5.70 > > > > > > > [root@localhost ~(keystone_admin)]# > > > > > > > > > > > > > > > > > > > > > ======[root@localhost ~(keystone_admin)]# ip netns exec > > > > > > > qrouter-85fa9459-503d-4996-86f3-6042604fed74 ping 192.168.5.1 > > > > > > > PING 192.168.5.1 (192.168.5.1) 56(84) bytes of data. > > > > > > > ^C > > > > > > > --- 192.168.5.1 ping statistics --- > > > > > > > 5 packets transmitted, 0 received, 100% packet loss, time 3999ms > > > > > > > > > > > > > > any hints?? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ Rdo-list mailing > > list > > > > > > > Rdo-list(a)redhat.com > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > > To > > > > > > > unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Rdo-list mailing list > > > > > > > Rdo-list(a)redhat.com > > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > > > > > > > > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > > > > > > > > > _______________________________________________ > > > > > > Rdo-list mailing list > > > > > > Rdo-list(a)redhat.com > > > > > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > > > > > > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Rdo-list mailing list > > > Rdo-list(a)redhat.com > > > https://www.redhat.com/mailman/listinfo/rdo-list > > > > > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com > > >