On 11/13/2015 12:56 PM, Dan Sneddon wrote: One more quick thing to note: In order to use floating IPs, you will also have to attach the external router to the tenant networks where floating IPs will be used. When you go through the steps to create a tenant network, also attach it to the router: 1) Create the network: neutron net-create tenant-net-1 2) Create the subnet: neutron subnet-create --name tenant-subnet-1 tenant-net-1 172.21.0.0/22 3) Attach the external router to the network: neutron router-interface-add tenant-router-1 subnet=tenant-subnet-1 (since no specific port was given in the router-interface-add command, Neutron will automatically choose the first address in the given subnet, so 172.21.0.1 in this example) -- Dan Sneddon | Principal OpenStack Engineer dsneddon(a)redhat.com | redhat.com/openstack 650.254.4025 | dsneddon:irc @dxs:twitter

I happened to have deployed following the exact same guide a cloud on bare metal composed of 3 controllers, 2 haproxy nodes and N computes, with external provider networks. What I did was: -) (at https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/neutron-c...): On controller and compute, define the external vlan interface and its bridge: cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-bond0.102 DEVICE=bond0.102 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-bond0.102 ONBOOT=yes BOOTPROTO=none VLAN=yes MTU="1500" NM_CONTROLLED=no EOF cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-br-bond0.102 DEVICE=br-bond0.102 DEVICETYPE=ovs OVSBOOTPROTO=none TYPE=OVSBridge ONBOOT=yes BOOTPROTO=static MTU="1500" NM_CONTROLLED=no EOF then, make sure that everywhere exists /etc/neutron/plugins/ml2/openvswitch_agent.ini as: [ovs] enable_tunneling = True tunnel_id_ranges = 1:1000 tenant_network_type = vxlan integration_bridge = br-int tunnel_bridge = br-tun local_ip = <IP on bond0> bridge_mappings = physnet1:br-bond0.102 network_vlan_ranges = physnet1 [agent] tunnel_types = vxlan vxlan_udp_port = 4789 l2_population = False [securitygroup] firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver restart neutron-openvswitch-agent everywhere to make it work: Hope it helps Alessandro

________________________________ From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.comSent: Saturday, November 14, 2015 7:58 AM To: Alessandro Vozza Cc: rdo-list(a)redhat.com Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md ________________________________ From: Alessandro Vozza <alessandro(a)namecheap.comSent: Saturday, November 14, 2015 4:26 AM To: Boris Derzhavets Cc: Dan Sneddon; rdo-list(a)redhat.com Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md I happened to have deployed following the exact same guide a cloud on bare metal composed of 3 controllers, 2 haproxy nodes and N computes, with external provider networks. What I did was: -) (at https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/neutron-c...): [https://avatars0.githubusercontent.com/u/114726?v=3&s=400]<https:/... beekhof/osp-ha-deploy osp-ha-deploy - scripts for deploying a HA install of OSP Read more...<https://github.com/beekhof/osp-ha-deploy/blob/master/keepalive... Actually, using bondings and external network provider of VLAN type, controller nodes may have 3 VLAN's : bond0.100(management network), bond0.101 (tunnel network) and bond0.102(external-network). Then tune haproxy with keepalived for a virtual ip setup on bond0.100 This excellent approach, but question which is my concern is a bit different. Thanks Boris On controller and compute, define the external vlan interface and its bridge: cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-bond0.102 DEVICE=bond0.102 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-bond0.102 ONBOOT=yes BOOTPROTO=none VLAN=yes MTU="1500" NM_CONTROLLED=no EOF cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-br-bond0.102 DEVICE=br-bond0.102 DEVICETYPE=ovs OVSBOOTPROTO=none TYPE=OVSBridge ONBOOT=yes BOOTPROTO=static MTU="1500" NM_CONTROLLED=no EOF then, make sure that everywhere exists /etc/neutron/plugins/ml2/openvswitch_agent.ini as: [ovs] enable_tunneling = True tunnel_id_ranges = 1:1000 tenant_network_type = vxlan integration_bridge = br-int tunnel_bridge = br-tun local_ip = <IP on bond0bridge_mappings = physnet1:br-bond0.102 network_vlan_ranges = physnet1 [agent] tunnel_types = vxlan vxlan_udp_port = 4789 l2_population = False [securitygroup] firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver restart neutron-openvswitch-agent everywhere to make it work: [cid:762776E3-E3BC-4FA0-B1CF-3DAC1E1EBD91@fritz.box] Hope it helps Alessandro I would guess , that external net was created similar to this way controller# neutron net-create --router:external=True \ --provider:network_type=vlan --provider:segmentation_id=102 ext-network controller# neutron subnet-create --name ext-subnet --disable-dhcp \ --allocation-pool start 10.10.10.100,end=10.10.10.150 \ ext_network 10.10.10.0/24 of VLAN type not FLAT Could you be so kind to share yours ml2_conf.ini on controllers in cluster. I also believe that tuning your switch contains a kind of :- switchport trunk allowed vlan 100,102,104 Thank you Boris. 14 Nov 2015, at 09:35, Boris Derzhavets <bderzhavets@hotmail.com<mailto:bderzhavets@hotmail.com>> wrote: ________________________________________ From: Dan Sneddon <dsneddon@redhat.com<mailto:dsneddon@redhat.com>Sent: Friday, November 13, 2015 4:10 PM To: Boris Derzhavets; rdo-list@redhat.com<mailto:rdo-list@redhat.comSubject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md On 11/13/2015 12:56 PM, Dan Sneddon wrote: Hi Boris, Let's keep this on-list, there may be others who are having similar issues who could find this discussion useful. Answers inline... On 11/13/2015 12:17 PM, Boris Derzhavets wrote: ________________________________________ From: Dan Sneddon <dsneddon@redhat.com<mailto:dsneddon@redhat.com>Sent: Friday, November 13, 2015 2:46 PM To: Boris Derzhavets; Javier Pena Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.comSubject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md On 11/13/2015 11:38 AM, Boris Derzhavets wrote: I understand that in usual situation , creating ifcfg-br-ex and ifcfg-eth2 ( as OVS bridge and OVS port) , `service network restart` should be run to make eth2 (no IP) OVS port of br-ex (any IP which belongs ext net and is available) What bad does NetworkManager when external network provider is used ? Disabling it, I break routing via eth0's interfaces of cluster nodes to 10.10.10.0/24 ( ext net), so nothing is supposed to work :- http://blog.oddbit.com/2014/05/28/multiple-external-networks-wit/ http://dbaxps.blogspot.com/2015/10/multiple-external-networks-with-single... Either I am missing something here. ________________________________________ From: rdo-list-bounces@redhat.com<mailto:rdo-list-bounces@redhat.com> <rdo-list-bounces@redhat.com<mailto:rdo-list-bounces@redhat.com>> on behalf of Boris Derzhavets <bderzhavets@hotmail.com<mailto:bderzhavets@hotmail.com>Sent: Friday, November 13, 2015 1:09 PM To: Javier Pena Cc: rdo-list@redhat.com<mailto:rdo-list@redhat.comSubject: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md Working on this task I was able to build 3 node HAProxy/Keepalived Controller's cluster , create compute node , launch CirrOS VM, However, I cannot ping floating IP of VM running on compute ( total 4 CentOS 7.1 VMs, nested kvm enabled ) Looks like provider external networks doesn't work for me. But , to have eth0 without IP (due to `ovs-vsctl add-port br-eth0 eth0 ) still allowing to ping 10.10.10.1, I need NetworkManager active, rather then network.service [root@hacontroller1 network-scripts]# systemctl status NetworkManager NetworkManager.service - Network Manager Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled) Active: active (running) since Fri 2015-11-13 20:39:21 MSK; 12min ago Main PID: 808 (NetworkManager) CGroup: /system.slice/NetworkManager.service ├─ 808 /usr/sbin/NetworkManager --no-daemon └─2325 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0... Nov 13 20:39:22 hacontroller1.example.com<http://hacontroller1.example.com> NetworkManager[808]: <info> NetworkManager state is n...L Nov 13 20:39:22 hacontroller1.example.com<http://hacontroller1.example.com> dhclient[2325]: bound to 10.10.10.216 -- renewal in 1...s. Nov 13 20:39:22 hacontroller1.example.com<http://hacontroller1.example.com> NetworkManager[808]: <info> (eth0): Activation: succe.... Nov 13 20:39:25 hacontroller1.example.com<http://hacontroller1.example.com> NetworkManager[808]: <info> startup complete [root@hacontroller1 network-scripts]# systemctl status network.service network.service - LSB: Bring up/down networking Loaded: loaded (/etc/rc.d/init.d/network) Active: inactive (dead) [root@hacontroller1 network-scripts]# cat ifcfg-eth0 TYPE="Ethernet" BOOTPROTO="static" NAME="eth0" DEVICE=eth0 ONBOOT="yes" [root@hacontroller1 network-scripts]# ping -c 3 10.10.10.1 PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data. 64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.087 ms 64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.128 ms 64 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=0.117 ms --- 10.10.10.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.087/0.110/0.128/0.021 ms If I disable NetworkManager and enable network this feature will be lost. Eth0 would have to have static IP or dhcp lease, to provide route to 10.10.10.0/24. Thank you. Boris. _______________________________________________ Rdo-list mailing list Rdo-list@redhat.com<mailto:Rdo-list@redhat.comhttps://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com OK, a few things here. First of all, you don't actually need to have an IP address on the host system to use a VLAN or interface as an external provider network. The Neutron router will have an IP on the right network, and within its namespace will be able to reach the 10.10.10.x network. It looks to me like NetworkManager is running dhclient for eth0, even though you have BOOTPROTO="static". This is causing an IP address to be added to eth0, so you are able to ping 10.10.10.x from the host. When you turn off NetworkManager, this unexpected behavior goes away, *but you should still be able to use provider networks*. Here I am quoting Lars Kellogg Stedman http://blog.oddbit.com/2014/05/28/multiple-external-networks-wit/ The bottom statement in blog post above states :- "This assumes that eth1 is connected to a network using 10.1.0.0/24 and eth2 is connected to a network using 10.2.0.0/24, and that each network has a gateway sitting at the corresponding .1 address." Right, what Lars means is that eth1 is physically connected to a network with the 10.1.0.0/24 subnet, and eth2 is physically connected to a network with the 10.2.0.0/24 subnet. You might notice that in Lars's instructions, he never puts a host IP on either interface. Try creating a Neutron router with an IP on 10.10.10.x, and then you should be able to ping that network from the router namespace. " When I issue `neutron router-creater --ha True --tenant-id xxxxxx RouterHA` , i cannot specify router's IP " Let me refer you to this page, which explains the basics of creating and managing Neutron networks: http://docs.openstack.org/user-guide/cli_create_and_manage_networks.html You will have to create an external network, which you will associate with a physical network via a bridge mapping. The default bridge mapping for br-ex is datacentre:br-ex. Using the name of the physical network "datacentre", we can create an 1. Javier is using external network provider ( and so did I , following him) #. /root/keystonerc_admin # neutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external # neutron subnet-create --gateway 10.10.10.1 --allocation-pool start=10.10.10.100,end=10.10.10.150 --disable-dhcp --name public_subnet public 10.10.10.0/24 HA Neutron router and tenant's subnet have been created. Then interface to tenant's network was activated as well as gateway to public. Security rules were implemented as usual. Cloud VM was launched, it obtained private IP and committed cloud-init OK. Then I assigned FIP from public to cloud VM , it should be ping able from from F23 Visualization Host 2. All traffic to/from external network flows through br-int when provider external networks has been involved. No br-ex is needed. When in Javier does `ovs-vsctl add-port br-eth0 eth0` , eth0 (which is inside VM ,running Controller node) should be on 10.10.10.X/24. It doesn't happen when service network is active (and NM disabled) . In this case eth0 doesn't have any kind of IP assigned to provide route to Libvirt's subnet 10.10.10.X/24 ( pre created by myself) In meantime I am under impression that ovs bridge br-eth0 and OVS port eth0 would work when IP is assigned to port eth0, not to bridge. OVS release =>2.3.1 seems to allow that. Tested here (VM's case ) :- http://dbaxps.blogspot.com/2015/10/multiple-external-networks-with-single... If neither one of br-eth0 and eth0 would have IP then packets won't be forwarded to external net external network: [If the external network is on VLAN 104] neutron net-create ext-net --router:external \ --provider:physical_network datacentre \ --provider:network_type vlan \ --provider:segmentation_id 104 [If the external net is on the native VLAN (flat)] neutron net-create ext-net --router:external \ --provider:physical_network datacentre \ --provider:network_type flat Next, you must create a subnet for the network, including the range of floating IPs (allocation pool): neutron subnet-create --name ext-subnet \ --enable_dhcp=False \ --allocation-pool start=10.10.10.50,end=10.10.10.100 \ --gateway 10.10.10.1 \ ext-net 10.10.10.0/24 Next, you have to create a router: neutron router-create ext-router You then add an interface to the router. Since Neutron will assign the first address in the subnet to the router by default (10.10.10.1), you will want to first create a port with a specific IP, then assign that port to the router. neutron port-create ext-net --fixed-ip ip_address=10.10.10.254 You will need to note the UUID of the newly created port. You can also see this with "neutron port-list". Now, create the router interface with the port you just created: neutron router-interface-add ext-router port=<UUID If you want to be able to ping 10.10.10.x from the host, then you should put either a static IP or DHCP on the bridge, not on eth0. This should work whether you are running NetworkManager or network.service. "I do can ping 10.0.0.x from F23 KVM Server (running cluster's VMs as Controller's nodes), it's just usual non-default libvirt subnet,matching exactly external network creating in Javier's "Howto". It was created via `virsh net-define openstackvms.xml`, but I cannot ping FIPs belong to cloud VM on this subnet." I think you will have better luck once you create the external network and router. You can then use namespaces to ping the network from the router: First, obtain the qrouter-<UUID> from the list of namespaces: sudo ip netns list Then, find the qrouter-<UUID> and ping from there: ip netns exec qrouter-XXXX-XXXX-XXX-XXX ping 10.10.10.1 One more quick thing to note: In order to use floating IPs, you will also have to attach the external router to the tenant networks where floating IPs will be used. When you go through the steps to create a tenant network, also attach it to the router: 1) Create the network: neutron net-create tenant-net-1 2) Create the subnet: neutron subnet-create --name tenant-subnet-1 tenant-net-1 172.21.0.0/22 3) Attach the external router to the network: neutron router-interface-add tenant-router-1 subnet=tenant-subnet-1 (since no specific port was given in the router-interface-add command, Neutron will automatically choose the first address in the given subnet, so 172.21.0.1 in this example) -- Dan Sneddon | Principal OpenStack Engineer dsneddon@redhat.com<mailto:dsneddon@redhat.com> | redhat.com/openstack<http://redhat.com/openstack650.254.4025 | dsneddon:irc @dxs:twitter _______________________________________________ Rdo-list mailing list Rdo-list@redhat.com<mailto:Rdo-list@redhat.comhttps://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe@redhat.com<mailto:rdo-list-unsubscribe@redhat.com

Alessandro, I did neutron work flow check on controllers 1,2 hosting HA neutron router. FIRST [root@hacontroller1 ~(keystone_admin)]# ovs-ofctl show br-eth0 OFPT_FEATURES_REPLY (xid=0x2): dpid:0000baf0db1a854f n_tables:254, n_buffers:256 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 1(eth0): addr:52:54:00:aa:0e:fc config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 2(phy-br-eth0): addr:46:c0:e0:30:72:92 <====== config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(br-eth0): addr:ba:f0:db:1a:85:4f config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 [root@hacontroller1 ~(keystone_admin)]# ovs-ofctl dump-flows br-eth0 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=15577.057s, table=0, n_packets=50441, n_bytes=3262529, idle_age=2, priority=4,in_port=2,dl_vlan=3 actions=strip_vlan,NORMAL <===== cookie=0x0, duration=15765.938s, table=0, n_packets=31225, n_bytes=1751795, idle_age=0, priority=2,in_port=2 actions=drop cookie=0x0, duration=15765.974s, table=0, n_packets=39982, n_bytes=42838752, idle_age=1, priority=0 actions=NORMAL Check `ovs-vsctl show` Bridge br-int fail_mode: secure Port "tapc8488877-45" tag: 4 Interface "tapc8488877-45" type: internal Port br-int Interface br-int type: internal Port patch-tun Interface patch-tun type: patch options: {peer=patch-int} Port "tap14aa6eeb-70" tag: 2 Interface "tap14aa6eeb-70" type: internal Port "qr-8f5b3f4a-45" tag: 2 Interface "qr-8f5b3f4a-45" type: internal Port "int-br-eth0" Interface "int-br-eth0" type: patch options: {peer="phy-br-eth0"} Port "qg-34893aa0-17" <===== tag: 3 SECOND [root@hacontroller2 ~(keystone_demo)]# ovs-ofctl show br-eth0 OFPT_FEATURES_REPLY (xid=0x2): dpid:0000b6bfa2bafd45 n_tables:254, n_buffers:256 capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst 1(eth0): addr:52:54:00:73:df:29 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max 2(phy-br-eth0): addr:be:89:61:87:56:20 <======= config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max LOCAL(br-eth0): addr:b6:bf:a2:ba:fd:45 config: 0 state: 0 speed: 0 Mbps now, 0 Mbps max OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 [root@hacontroller2 ~(keystone_demo)]# ovs-ofctl dump-flows br-eth0 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=15810.746s, table=0, n_packets=0, n_bytes=0, idle_age=15810, priority=4,in_port=2,dl_vlan=2 actions=strip_vlan,NORMAL <======== cookie=0x0, duration=16105.662s, table=0, n_packets=31849, n_bytes=1786827, idle_age=0, priority=2,in_port=2 actions=drop cookie=0x0, duration=16105.696s, table=0, n_packets=39762, n_bytes=2100763, idle_age=0, priority=0 actions=NORMAL Check `ovs-vsctl show` Bridge br-int fail_mode: secure Port "qg-34893aa0-17" tag: 2 <===== Interface "qg-34893aa0-17" type: internal <https://www.blogger.com/profile/08011293873468656575> It looks like qrouter's namespace output interface qg-xxxxxx sends vlan tagged packets to eth0 (which has VLAN=yes) , but OVS bridge br-eth0 is not aware of vlan tagging (as you wrote) , it strips tags before sending packets outside into external flat network. In case of external network provider qg-xxxxxx are on Br-int, that is normal. That's why your patch works so stable. If my logic is wrong,please, let me know. Thank you once again. Boris.

Answers inline... On 11/14/2015 12:35 AM, Boris Derzhavets wrote: That looks like it would be OK if physnet1 is a flat connection (native VLAN on the interface). If you want to create a provider network on, for example VLAN 104, you can use this command: neutron net-create --provider:physical_network physnet1 --provider:network_type vlan --provider:segmentation_id 104 --router:external public Your subnet-create statement looks correct. That's OK that eth0 doesn't have any kind of IP assigned or routes. The IP gets assigned to the Neutron router, and the routing table exists only inside of the router namespace. Once you have created the router, you will see a "qrouter-XXXX" entry for the router when you run the command: sudo ip netns list Copy the name of the namespace that starts with "qrouter" (you might have more than one if you have more than one Neutron router), then try pinging the external network from inside the namespace: sudo ip netns exec qrouter-c333bd80-ccc3-43ba-99e4-8df471ed8b9e ping 10.10.10.1 For Provider networks, you shouldn't have to assign an IP address to eth0 or to the bridge. The IP address lives on the router inside of the router namespace. -- Dan Sneddon | Principal OpenStack Engineer dsneddon(a)redhat.com | redhat.com/openstack 650.254.4025 | dsneddon:irc @dxs:twitter

*** WARNING: THE ATTACHED DOCUMENT(S) CONTAIN MACROS *** *** MACROS MAY CONTAIN MALICIOUS CODE *** *** Open only if you can verify and trust the sender *** *** Please contact infosec(a)redhat.com if you have questions or concerns ** I would guess that page for Kilo :- https://github.com/beekhof/osp-ha-deploy/blob/Kilo-RDO7/keepalived/neutro... systemctl enable openvswitch systemctl start openvswitch ovs-vsctl add-br br-int ovs-vsctl add-br br-eth0 Note: we have seeing issues when trying to configure an IP on br-eth0 (specially ARP problems), so it is not recommended. ovs-vsctl add-port br-eth0 eth0 would better update in same way as for Liberty - https://github.com/beekhof/osp-ha-deploy/blob/master/keepalived/neutron-c.... 4 days ago updated by Alessandro Vozza as follows Assuming eth0 is your interface attached to the external network, create two files in /etc/sysconfig/network-scripts/ as follows (change MTU if you need): cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-eth0 ONBOOT=yes BOOTPROTO=none VLAN=yes MTU="9000" NM_CONTROLLED=no EOF cat <<EOF > /etc/sysconfig/network-scripts/ifcfg-br-eth0 DEVICE=br-eth0 DEVICETYPE=ovs OVSBOOTPROTO=none TYPE=OVSBridge ONBOOT=yes BOOTPROTO=static MTU="9000" NM_CONTROLLED=no EOF In other words patch https://github.com/beekhof/osp-ha-deploy/commit/b2e01e86ca93cfad9ad01d533... should be applied to Kilo OVS bridging configuration. I believe so. Thank you very much. Boris ________________________________________ From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.comSent: Monday, November 16, 2015 3:10 PM To: Dan Sneddon; rdo-list(a)redhat.com Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md ________________________________________ From: Dan Sneddon <dsneddon(a)redhat.comSent: Monday, November 16, 2015 11:39 AM To: Boris Derzhavets; rdo-list(a)redhat.com Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md Answers inline... Thank you so much for your support. In meantime I can manage HAProxy/Keepalived 3 VMs Controller and one Compute VM ( nested kvm enabled ) via Nova && Neutron CLI with no problems (RDO Liberty). Dashboard is extremely slow ( i7 4790 CPU, 32 GB RAM). I still believe that problem is 4 Core desktop CPUs limitations. As soon as 3 Controllers get in sync and start working ( 4 VCPUs each one ,4 GB RAM ) graphics slows down immediately. Testing RDO Manager on desktops is hardly possible. On 11/14/2015 12:35 AM, Boris Derzhavets wrote: > ________________________________________ > From: Dan Sneddon <dsneddon(a)redhat.com> Sent: Friday, November 13, 2015 4:10 PM > To: Boris Derzhavets; rdo-list(a)redhat.com > Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md > On 11/13/2015 12:56 PM, Dan Sneddon wrote: >> Hi Boris, >>> Let's keep this on-list, there may be others who are having similar >> issues who could find this discussion useful. >>> Answers inline... >>> On 11/13/2015 12:17 PM, Boris Derzhavets wrote: >>>>>>> ________________________________________ >>> From: Dan Sneddon <dsneddon(a)redhat.com>>> Sent: Friday, November 13, 2015 2:46 PM >>> To: Boris Derzhavets; Javier Pena >>> Cc: rdo-list(a)redhat.com >>> Subject: Re: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md >>>>> On 11/13/2015 11:38 AM, Boris Derzhavets wrote: >>>> I understand that in usual situation , creating ifcfg-br-ex and ifcfg-eth2 ( as OVS bridge and OVS port) , >>>> `service network restart` should be run to make eth2 (no IP) OVS port of br-ex (any IP which belongs ext net and is available) >>>> What bad does NetworkManager when external network provider is used ? >>>> Disabling it, I break routing via eth0's interfaces of cluster nodes to 10.10.10.0/24 ( ext net), >>>> so nothing is supposed to work :- >>>> http://blog.oddbit.com/2014/05/28/multiple-external-networks-wit/ >>>> http://dbaxps.blogspot.com/2015/10/multiple-external-networks-with-single... >>>> Either I am missing something here. >>>> ________________________________________ >>>> From: rdo-list-bounces(a)redhat.com <rdo-list-bounces(a)redhat.com&gt; on behalf of Boris Derzhavets <bderzhavets(a)hotmail.com>>>> Sent: Friday, November 13, 2015 1:09 PM >>>> To: Javier Pena >>>> Cc: rdo-list(a)redhat.com >>>> Subject: [Rdo-list] Attempt to reproduce https://github.com/beekhof/osp-ha-deploy/blob/master/HA-keepalived.md >>>>>>> Working on this task I was able to build 3 node HAProxy/Keepalived Controller's cluster , create compute node , launch CirrOS VM, >>>> However, I cannot ping floating IP of VM running on compute ( total 4 CentOS 7.1 VMs, nested kvm enabled ) >>>> Looks like provider external networks doesn't work for me. >>>>>>> But , to have eth0 without IP (due to `ovs-vsctl add-port br-eth0 eth0 ) still allowing to ping 10.10.10.1, >>>> I need NetworkManager active, rather then network.service >>>>>>> [root@hacontroller1 network-scripts]# systemctl status NetworkManager >>>> NetworkManager.service - Network Manager >>>> Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled) >>>> Active: active (running) since Fri 2015-11-13 20:39:21 MSK; 12min ago >>>> Main PID: 808 (NetworkManager) >>>> CGroup: /system.slice/NetworkManager.service >>>> ├─ 808 /usr/sbin/NetworkManager --no-daemon >>>> └─2325 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0... >>>>>>> Nov 13 20:39:22 hacontroller1.example.com NetworkManager[808]: <info> NetworkManager state is n...L >>>> Nov 13 20:39:22 hacontroller1.example.com dhclient[2325]: bound to 10.10.10.216 -- renewal in 1...s. >>>> Nov 13 20:39:22 hacontroller1.example.com NetworkManager[808]: <info> (eth0): Activation: succe.... >>>> Nov 13 20:39:25 hacontroller1.example.com NetworkManager[808]: <info> startup complete >>>>>>> [root@hacontroller1 network-scripts]# systemctl status network.service >>>> network.service - LSB: Bring up/down networking >>>> Loaded: loaded (/etc/rc.d/init.d/network) >>>> Active: inactive (dead) >>>>>>> [root@hacontroller1 network-scripts]# cat ifcfg-eth0 >>>> TYPE="Ethernet" >>>> BOOTPROTO="static" >>>> NAME="eth0" >>>> DEVICE=eth0 >>>> ONBOOT="yes" >>>>>>> [root@hacontroller1 network-scripts]# ping -c 3 10.10.10.1 >>>> PING 10.10.10.1 (10.10.10.1) 56(84) bytes of data. >>>> 64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=0.087 ms >>>> 64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=0.128 ms >>>> 64 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=0.117 ms >>>>>>> --- 10.10.10.1 ping statistics --- >>>> 3 packets transmitted, 3 received, 0% packet loss, time 1999ms >>>> rtt min/avg/max/mdev = 0.087/0.110/0.128/0.021 ms >>>>>>> If I disable NetworkManager and enable network this feature will be lost. Eth0 would have to have static IP or dhcp lease, >>>> to provide route to 10.10.10.0/24. >>>>>>> Thank you. >>>> Boris. >>>>>>> _______________________________________________ >>>> Rdo-list mailing list >>>> Rdo-list(a)redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>>>>>> OK, a few things here. First of all, you don't actually need to have an >>> IP address on the host system to use a VLAN or interface as an external >>> provider network. The Neutron router will have an IP on the right >>> network, and within its namespace will be able to reach the 10.10.10.x >>> network. >>>>>> It looks to me like NetworkManager is running dhclient for eth0, even >>>> though you have BOOTPROTO="static". This is causing an IP address to be >>>> added to eth0, so you are able to ping 10.10.10.x from the host. When >>>> you turn off NetworkManager, this unexpected behavior goes away, *but >>>> you should still be able to use provider networks*. >>>>> Here I am quoting Lars Kellogg Stedman >>> http://blog.oddbit.com/2014/05/28/multiple-external-networks-wit/ >>> The bottom statement in blog post above states :- >>> "This assumes that eth1 is connected to a network using 10.1.0.0/24 and eth2 is connected to a network using 10.2.0.0/24, and that each network has a gateway sitting at the corresponding .1 address." >>> Right, what Lars means is that eth1 is physically connected to a >> network with the 10.1.0.0/24 subnet, and eth2 is physically connected >> to a network with the 10.2.0.0/24 subnet. >>> You might notice that in Lars's instructions, he never puts a host IP >> on either interface. >>>>> Try creating a Neutron router with an IP on 10.10.10.x, and then you >>>> should be able to ping that network from the router namespace. >>>>> " When I issue `neutron router-creater --ha True --tenant-id xxxxxx RouterHA` , i cannot specify router's >>> IP " >>> Let me refer you to this page, which explains the basics of creating >> and managing Neutron networks: >>> http://docs.openstack.org/user-guide/cli_create_and_manage_networks.html >>> You will have to create an external network, which you will associate >> with a physical network via a bridge mapping. The default bridge >> mapping for br-ex is datacentre:br-ex. >>> Using the name of the physical network "datacentre", we can create an > 1. Javier is using external network provider ( and so did I , following him) > #. /root/keystonerc_admin > # neutron net-create public --provider:network_type flat --provider:physical_network physnet1 --router:external > # neutron subnet-create --gateway 10.10.10.1 --allocation-pool start=10.10.10.100,end=10.10.10.150 --disable-dhcp --name public_subnet public 10.10.10.0/24 That looks like it would be OK if physnet1 is a flat connection (native VLAN on the interface). If you want to create a provider network on, for example VLAN 104, you can use this command: neutron net-create --provider:physical_network physnet1 --provider:network_type vlan --provider:segmentation_id 104 --router:external public Your subnet-create statement looks correct. > HA Neutron router and tenant's subnet have been created. > Then interface to tenant's network was activated as well as gateway to public. > Security rules were implemented as usual. > Cloud VM was launched, it obtained private IP and committed cloud-init OK. > Then I assigned FIP from public to cloud VM , it should be ping able from from F23 Visualization > Host > 2. All traffic to/from external network flows through br-int when provider external networks has been involved. No br-ex is needed. > When in Javier does `ovs-vsctl add-port br-eth0 eth0` , eth0 (which is inside VM ,running Controller node) > should be on 10.10.10.X/24. It doesn't happen when service network is active (and NM disabled) . > In this case eth0 doesn't have any kind of IP assigned to provide route to Libvirt's subnet 10.10.10.X/24 ( pre created by myself) That's OK that eth0 doesn't have any kind of IP assigned or routes. The IP gets assigned to the Neutron router, and the routing table exists only inside of the router namespace. Once you have created the router, you will see a "qrouter-XXXX" entry for the router when you run the command: sudo ip netns list Copy the name of the namespace that starts with "qrouter" (you might have more than one if you have more than one Neutron router), then try pinging the external network from inside the namespace: sudo ip netns exec qrouter-c333bd80-ccc3-43ba-99e4-8df471ed8b9e ping 10.10.10.1 For Provider networks, you shouldn't have to assign an IP address to eth0 or to the bridge. The IP address lives on the router inside of the router namespace. -- Dan Sneddon | Principal OpenStack Engineer dsneddon(a)redhat.com | redhat.com/openstack 650.254.4025 | dsneddon:irc @dxs:twitter >> external network: >>> [If the external network is on VLAN 104] >> neutron net-create ext-net --router:external \ >> --provider:physical_network datacentre \ >> --provider:network_type vlan \ >> --provider:segmentation_id 104 >>> [If the external net is on the native VLAN (flat)] >> neutron net-create ext-net --router:external \ >> --provider:physical_network datacentre \ >> --provider:network_type flat >>> Next, you must create a subnet for the network, including the range of >> floating IPs (allocation pool): >>> neutron subnet-create --name ext-subnet \ >> --enable_dhcp=False \ >> --allocation-pool start=10.10.10.50,end=10.10.10.100 \ >> --gateway 10.10.10.1 \ >> ext-net 10.10.10.0/24 >>> Next, you have to create a router: >>> neutron router-create ext-router >>> You then add an interface to the router. Since Neutron will assign the >> first address in the subnet to the router by default (10.10.10.1), you >> will want to first create a port with a specific IP, then assign that >> port to the router. >>> neutron port-create ext-net --fixed-ip ip_address=10.10.10.254 >>> You will need to note the UUID of the newly created port. You can also >> see this with "neutron port-list". Now, create the router interface >> with the port you just created: >>> neutron router-interface-add ext-router port=<UUID>>>>> If you want to be able to ping 10.10.10.x from the host, then you >>>> should put either a static IP or DHCP on the bridge, not on eth0. This >>>> should work whether you are running NetworkManager or network.service. >>>>> "I do can ping 10.0.0.x from F23 KVM Server (running cluster's VMs as Controller's nodes), >>> it's just usual non-default libvirt subnet,matching exactly external network creating in Javier's "Howto". >>> It was created via `virsh net-define openstackvms.xml`, but I cannot ping FIPs belong to >>> cloud VM on this subnet." >>> I think you will have better luck once you create the external network >> and router. You can then use namespaces to ping the network from the >> router: >>> First, obtain the qrouter-<UUID> from the list of namespaces: >>> sudo ip netns list >>> Then, find the qrouter-<UUID> and ping from there: >>> ip netns exec qrouter-XXXX-XXXX-XXX-XXX ping 10.10.10.1 >> One more quick thing to note: > In order to use floating IPs, you will also have to attach the external > router to the tenant networks where floating IPs will be used. > When you go through the steps to create a tenant network, also attach > it to the router: > 1) Create the network: > neutron net-create tenant-net-1 > 2) Create the subnet: > neutron subnet-create --name tenant-subnet-1 tenant-net-1 172.21.0.0/22 > 3) Attach the external router to the network: > neutron router-interface-add tenant-router-1 subnet=tenant-subnet-1 > (since no specific port was given in the router-interface-add command, > Neutron will automatically choose the first address in the given > subnet, so 172.21.0.1 in this example) > -- > Dan Sneddon | Principal OpenStack Engineer > dsneddon(a)redhat.com | redhat.com/openstack > 650.254.4025 | dsneddon:irc @dxs:twitter _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com