Hi Everyone, I am having problems with my neutron setup and hopefully with your help I can get it figured out. I have a 4 node blade setup with two nics each, all of them running CentOS 6.6. One host is foreman, the other three are for openstack. Since foreman is managing the blades, they have their IP addresses assigned via DHCP to eth0. After the install I noticed that the eth0 device was not attaching to the br-ex device. After lots of work, I was able to get that connected using these configs: ifcfg-br-ex: DEVICE="eth0" #BOOTPROTO="dhcp" BOOTPROTO="none" DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-ex #DHCP_HOSTNAME="openstack-1.quicksand.bitc.morphotrust.com" #HOSTNAME="openstack-1.quicksand.bitc.morphotrust.com" HWADDR="E4:1F:13:78:D8:90" #IPV6INIT="yes" MTU="1500" #NM_CONTROLLED="yes" NM_CONTROLLED="no" ONBOOT="yes" #TYPE="Ethernet" UUID="ebd620ad-7e48-4a08-9875-c596b4c4648c" VLAN=yes ifcfg-eth0: DEVICE="eth0" #BOOTPROTO="dhcp" BOOTPROTO="none" DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-ex #DHCP_HOSTNAME="openstack-1.quicksand.bitc.morphotrust.com" #HOSTNAME="openstack-1.quicksand.bitc.morphotrust.com" HWADDR="E4:1F:13:78:D8:90" #IPV6INIT="yes" MTU="1500" #NM_CONTROLLED="yes" NM_CONTROLLED="no" ONBOOT="yes" #TYPE="Ethernet" UUID="ebd620ad-7e48-4a08-9875-c596b4c4648c" VLAN=yes I can see eth0 attached to the br-ex, along with the external router port in ovs-vsctl show: Bridge br-ex Port br-ex Interface br-ex type: internal Port "qg-161de698-16" Interface "qg-161de698-16" type: internal Port "eth0" Interface "eth0" Now my problem, I can not get the guest VM to talk out. It can ping to the router port IP (10.30.1.10) but nothing past it. And from my network I can ping to the gateway of that network (10.30.1.1). What else should I check? I feel this is a problem with openvswitch, but I just dont know what to look at. Thanks for any help you can offer. --Brian

Hi Brian, On 11-12-14 16:15, brian lee wrote: > It looks like my cute and paste did not work right. My br-ex device > looks like this: > > DEVICE=br-ex > OVSBOOTPROTO="dhcp" > OVSDHCPINTERFACES="eth0" > ONBOOT=yes > NM_CONTROLLED=no > TYPE=OVSBridge > DEVICETYPE=ovs > DEVICE=br-ex > OVSBOOTPROTO="dhcp" > OVSDHCPINTERFACES="eth0" > ONBOOT=yes > NM_CONTROLLED=no > TYPE=OVSBridge > DEVICETYPE=ovs > > Sorry about the confusion. > I use RDO Juno and here are my interfaces: [root@neutron1-1 network-scripts]# cat ifcfg-br-ex DEVICE=br-ex TYPE=OVSBridge DEVICETYPE=ovs OVSBOOTPROTO=dhcp OVSDHCPINTERFACES=eth1 MACADDR="00:01:02:03:04:05" OVS_EXTRA="set bridge $DEVICE other-config:hwaddr=$MACADDR" ONBOOT=yes NM_CONTROLLED=no [root@neutron1-1 network-scripts]# cat ifcfg-eth1 DEVICE=eth1 TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=br-ex ONBOOT=yes BOOTPROTO=none NM_CONTROLLED=no HTH, Patrick _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list

I have been working on this for days now and I just can not figure it out. Attached is a bit from horizon where it is showing both interfaces on the router as down. How can I find out what is preventing them from starting? ​ --Brian On Thu, Dec 11, 2014 at 10:28 AM, brian lee <brian(a)brianlee.org <mailto:brian@brianlee.org>> wrote: Man my copy and paste just is not liking me. Anyways, I saw posting about forcing the mac address every time, but I have not had a problem. My problem is the port does not become active. I included the device settings as a reference. This is the status of the port: +-----------------------+-------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | openstack-1.quicksand.bitc.morphotrust.com <http://openstack-1.quicksand.bitc.morphotrust.com> | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | device_id | 7319781c-6186-4684-ba60-260b5ecee97c | | device_owner | network:router_gateway | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "7761c2ee-e392-48ff-b69a-f0f10bbcb6db", "ip_address": "10.30.1.10"} | | id | 161de698-1666-4c0d-9248-8de900797301 | | mac_address | fa:16:3e:c9:ff:64 | | name | | | network_id | b10fc224-2332-49f5-b555-9090c3dc7f44 | | security_groups | | | status | DOWN | | tenant_id | | +-----------------------+-------------------------------------------------------------------------------------+ I am just not able to get that port up. And since its not up I cant ping/ssh to the VMs. What do I need to do for vlans on my physical switch? --Brian On Thu, Dec 11, 2014 at 10:01 AM, Patrick Laimbock <patrick(a)laimbock.com <mailto:patrick@laimbock.com>> wrote: Hi Brian, On 11-12-14 16:15, brian lee wrote: It looks like my cute and paste did not work right. My br-ex device looks like this: DEVICE=br-ex OVSBOOTPROTO="dhcp" OVSDHCPINTERFACES="eth0" ONBOOT=yes NM_CONTROLLED=no TYPE=OVSBridge DEVICETYPE=ovs DEVICE=br-ex OVSBOOTPROTO="dhcp" OVSDHCPINTERFACES="eth0" ONBOOT=yes NM_CONTROLLED=no TYPE=OVSBridge DEVICETYPE=ovs Sorry about the confusion. I use RDO Juno and here are my interfaces: [root@neutron1-1 network-scripts]# cat ifcfg-br-ex DEVICE=br-ex TYPE=OVSBridge DEVICETYPE=ovs OVSBOOTPROTO=dhcp OVSDHCPINTERFACES=eth1 MACADDR="00:01:02:03:04:05" OVS_EXTRA="set bridge $DEVICE other-config:hwaddr=$MACADDR" ONBOOT=yes NM_CONTROLLED=no [root@neutron1-1 network-scripts]# cat ifcfg-eth1 DEVICE=eth1 TYPE=OVSPort DEVICETYPE=ovs OVS_BRIDGE=br-ex ONBOOT=yes BOOTPROTO=none NM_CONTROLLED=no HTH, Patrick _________________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com <mailto:Rdo-list@redhat.com> https://www.redhat.com/__mailman/listinfo/rdo-list <https://www.redhat.com/mailman/listinfo/rdo-list>

Hi Patrick, Thanks for the info, it is slowly coming together for me, I hope. I do have a few more question and I hope it will clear up more. First let me describe my environment more. I am using foreman to manage the physical hosts, and once openstack is running it will manage the VMs as well. So that is why I have a DHCP address for the host, its a static lease from foreman. My physical environment is in a blade center that has two switches in it. One switch is for eth0 and the other is for eth1. For the controller host (Everything but nova compute) the switch is configured for trunked vlan 111 (Management) and 110 (tenets) for both eth0 and eth1. For the compute nodes, the switches are configured for vlan 111 only. I am thinking on my controller host I need to configure the eth0.110 device, give it a static IP and connect it to the br-ex, does that sound right? I do also have some confusion about vxlan and how it is used. Is that only in the "overlay" network? From what I understand it can have tens of thousands of vlans, which the physical switches can not support. How does the OS/physical network handle that? Do you have to use a non-admin project to create the private network? Thanks again for the feedback, I feel I am getting close to resolving this. --Brian On Thu, Dec 11, 2014 at 8:20 PM, Patrick Laimbock <patrick(a)laimbock.com&gt; wrote: > > Hi Brian, > > Maybe there's a really simple solution but I don't have enough info to > tell. So here's a "slightly" longer suggestion. > > For VLAN support on the *physical* network your switch will need to > support 802.1Q. When you say VLANs what do you mean? If you want to use > VLANs for tenant separation (so in the overlay network, not the physical > network) then Open vSwitch will take of that and AFAIK (I don't use VLANs) > you don't need to enable VLANs on your ifcfg devices. Unless your physical > network requires VLANs off course. > > The interfaces you pasted had VLAN=yes but not a VLAN designation (like > DEVICE=eth0.10 where .10 indicates VLAN 10) and although configured for a > static setting (DHCP commented out) there was no IP address defined. > > So maybe take a step back. Delete all the networks and routers (might > need to do that from the CLI if things are stuck), on your Neutron node > backup & delete ifcfg-br-ex and restore a working ifcfg-eth0, then restart > the network and restart the Open vSwitch service on your neutron node so it > detects previous stuff is gone (check with ovs-vsctl show), then start with > defining the ifcfg-br-ex device and make sure your network is OK first > (check with ip address show and restart the network and check again). Then > add ethX to br-ex: > # ovs-vsctl add-port br-ex ethX ; service network restart > Make sure you have access to a local console so you don't get locked out > if your network fails to restart. Then restart the Open vSwitch service. > > Then move on to create the tenant stuff you'll need. I don't know how you > installed RDO. If you used Packstack and want VLAN tenant separation then > you have already provided VLAN info and you should use that when setting > things up with something like: > > As regular user: > the router > the private network > the private subnet > add private subnet to router > > As admin: > the public network (to be used for example to access the Internet) > the public subnet > add public gateway on the router > > As regular user: > Create some floating IPs > Start an instance of for example the Cirros image > Assign a floating IP address > Once booted log into it via the console, ping local & remote addresses. > Hopefully shout "YES!" :) > > FWIW: If you want VLANs for tenant separation then VXLAN and GRE are much > easier: Read Rhyz's explanation (5th comment) why: > https://openstack.redhat.com/forum/discussion/626/help- > with-neutron-networking/p1 > > HTH, > Patrick > > On 12-12-14 02:00, brian lee wrote: > >> I have been working on this for days now and I just can not figure it >> out. Attached is a bit from horizon where it is showing both interfaces >> on the router as down. How can I find out what is preventing them from >> starting? >> >> ​ >> >> --Brian >> >> On Thu, Dec 11, 2014 at 10:28 AM, brian lee <brian(a)brianlee.org >> <mailto:brian@brianlee.org>> wrote: >> >> Man my copy and paste just is not liking me. Anyways, I saw posting >> about forcing the mac address every time, but I have not had a >> problem. >> My problem is the port does not become active. I included the device >> settings as a reference. This is the status of the port: >> >> +-----------------------+----------------------------------- >> --------------------------------------------------+ >> | Field | Value >> | >> +-----------------------+----------------------------------- >> --------------------------------------------------+ >> | admin_state_up | True >> | >> | allowed_address_pairs | >> | >> | binding:host_id | openstack-1.quicksand.bitc.morphotrust.com >> <http://openstack-1.quicksand.bitc.morphotrust.com> >> | >> | binding:profile | {} >> | >> | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": >> true} | >> | binding:vif_type | ovs >> | >> | binding:vnic_type | normal >> | >> | device_id | 7319781c-6186-4684-ba60-260b5ecee97c >> | >> | device_owner | network:router_gateway >> | >> | extra_dhcp_opts | >> | >> | fixed_ips | {"subnet_id": >> "7761c2ee-e392-48ff-b69a-f0f10bbcb6db", "ip_address": "10.30.1.10"} >> | >> | id | 161de698-1666-4c0d-9248-8de900797301 >> | >> | mac_address | fa:16:3e:c9:ff:64 >> | >> | name | >> | >> | network_id | b10fc224-2332-49f5-b555-9090c3dc7f44 >> | >> | security_groups | >> | >> | status | DOWN >> | >> | tenant_id | >> | >> +-----------------------+----------------------------------- >> --------------------------------------------------+ >> >> I am just not able to get that port up. And since its not up I cant >> ping/ssh to the VMs. What do I need to do for vlans on my physical >> switch? >> >> --Brian >> >> On Thu, Dec 11, 2014 at 10:01 AM, Patrick Laimbock >> <patrick(a)laimbock.com <mailto:patrick@laimbock.com>> wrote: >> >> Hi Brian, >> >> On 11-12-14 16:15, brian lee wrote: >> >> It looks like my cute and paste did not work right. My br-ex >> device >> looks like this: >> >> DEVICE=br-ex >> OVSBOOTPROTO="dhcp" >> OVSDHCPINTERFACES="eth0" >> ONBOOT=yes >> NM_CONTROLLED=no >> TYPE=OVSBridge >> DEVICETYPE=ovs >> DEVICE=br-ex >> OVSBOOTPROTO="dhcp" >> OVSDHCPINTERFACES="eth0" >> ONBOOT=yes >> NM_CONTROLLED=no >> TYPE=OVSBridge >> DEVICETYPE=ovs >> >> Sorry about the confusion. >> >> >> I use RDO Juno and here are my interfaces: >> >> [root@neutron1-1 network-scripts]# cat ifcfg-br-ex >> DEVICE=br-ex >> TYPE=OVSBridge >> DEVICETYPE=ovs >> OVSBOOTPROTO=dhcp >> OVSDHCPINTERFACES=eth1 >> MACADDR="00:01:02:03:04:05" >> OVS_EXTRA="set bridge $DEVICE other-config:hwaddr=$MACADDR" >> ONBOOT=yes >> NM_CONTROLLED=no >> >> >> [root@neutron1-1 network-scripts]# cat ifcfg-eth1 >> DEVICE=eth1 >> TYPE=OVSPort >> DEVICETYPE=ovs >> OVS_BRIDGE=br-ex >> ONBOOT=yes >> BOOTPROTO=none >> NM_CONTROLLED=no >> >> HTH, >> Patrick >> >> >> _________________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com <mailto:Rdo-list@redhat.com> >> https://www.redhat.com/__mailman/listinfo/rdo-list >> <https://www.redhat.com/mailman/listinfo/rdo-list> >> >> >

PROGRESS! I was able to get my external network interface talking to the network. This is a first for me. So I can ping the router_gateway, even though it says its down. But when I spin up a VM, it is not getting a address for the private network, and associating a float doesn't connect to the VM either. So its getting closer. What does the network config need to look like on the compute nodes? --Brian On Thu, Dec 11, 2014 at 9:24 PM, brian lee <brian(a)brianlee.org&gt; wrote: > > Another follow up: What needs to be configured on the compute nodes? > > --Brian > > On Thu, Dec 11, 2014 at 9:07 PM, brian lee <brian(a)brianlee.org&gt; wrote: >> >> Hi Patrick, >> >> Thanks for the info, it is slowly coming together for me, I hope. I do >> have a few more question and I hope it will clear up more. First let me >> describe my environment more. I am using foreman to manage the physical >> hosts, and once openstack is running it will manage the VMs as well. So >> that is why I have a DHCP address for the host, its a static lease from >> foreman. >> >> My physical environment is in a blade center that has two switches in >> it. One switch is for eth0 and the other is for eth1. For the controller >> host (Everything but nova compute) the switch is configured for trunked >> vlan 111 (Management) and 110 (tenets) for both eth0 and eth1. For the >> compute nodes, the switches are configured for vlan 111 only. >> >> I am thinking on my controller host I need to configure the eth0.110 >> device, give it a static IP and connect it to the br-ex, does that sound >> right? >> >> I do also have some confusion about vxlan and how it is used. Is that >> only in the "overlay" network? From what I understand it can have tens of >> thousands of vlans, which the physical switches can not support. How does >> the OS/physical network handle that? >> >> Do you have to use a non-admin project to create the private network? >> >> Thanks again for the feedback, I feel I am getting close to resolving >> this. >> >> --Brian >> >> On Thu, Dec 11, 2014 at 8:20 PM, Patrick Laimbock <patrick(a)laimbock.com&gt; >> wrote: >>> >>> Hi Brian, >>> >>> Maybe there's a really simple solution but I don't have enough info to >>> tell. So here's a "slightly" longer suggestion. >>> >>> For VLAN support on the *physical* network your switch will need to >>> support 802.1Q. When you say VLANs what do you mean? If you want to use >>> VLANs for tenant separation (so in the overlay network, not the physical >>> network) then Open vSwitch will take of that and AFAIK (I don't use VLANs) >>> you don't need to enable VLANs on your ifcfg devices. Unless your physical >>> network requires VLANs off course. >>> >>> The interfaces you pasted had VLAN=yes but not a VLAN designation (like >>> DEVICE=eth0.10 where .10 indicates VLAN 10) and although configured for a >>> static setting (DHCP commented out) there was no IP address defined. >>> >>> So maybe take a step back. Delete all the networks and routers (might >>> need to do that from the CLI if things are stuck), on your Neutron node >>> backup & delete ifcfg-br-ex and restore a working ifcfg-eth0, then restart >>> the network and restart the Open vSwitch service on your neutron node so it >>> detects previous stuff is gone (check with ovs-vsctl show), then start with >>> defining the ifcfg-br-ex device and make sure your network is OK first >>> (check with ip address show and restart the network and check again). Then >>> add ethX to br-ex: >>> # ovs-vsctl add-port br-ex ethX ; service network restart >>> Make sure you have access to a local console so you don't get locked >>> out if your network fails to restart. Then restart the Open vSwitch service. >>> >>> Then move on to create the tenant stuff you'll need. I don't know how >>> you installed RDO. If you used Packstack and want VLAN tenant separation >>> then you have already provided VLAN info and you should use that when >>> setting things up with something like: >>> >>> As regular user: >>> the router >>> the private network >>> the private subnet >>> add private subnet to router >>> >>> As admin: >>> the public network (to be used for example to access the Internet) >>> the public subnet >>> add public gateway on the router >>> >>> As regular user: >>> Create some floating IPs >>> Start an instance of for example the Cirros image >>> Assign a floating IP address >>> Once booted log into it via the console, ping local & remote addresses. >>> Hopefully shout "YES!" :) >>> >>> FWIW: If you want VLANs for tenant separation then VXLAN and GRE are >>> much easier: Read Rhyz's explanation (5th comment) why: >>> https://openstack.redhat.com/forum/discussion/626/help- >>> with-neutron-networking/p1 >>> >>> HTH, >>> Patrick >>> >>> On 12-12-14 02:00, brian lee wrote: >>> >>>> I have been working on this for days now and I just can not figure it >>>> out. Attached is a bit from horizon where it is showing both interfaces >>>> on the router as down. How can I find out what is preventing them from >>>> starting? >>>> >>>> ​ >>>> >>>> --Brian >>>> >>>> On Thu, Dec 11, 2014 at 10:28 AM, brian lee <brian(a)brianlee.org >>>> <mailto:brian@brianlee.org>> wrote: >>>> >>>> Man my copy and paste just is not liking me. Anyways, I saw posting >>>> about forcing the mac address every time, but I have not had a >>>> problem. >>>> My problem is the port does not become active. I included the >>>> device >>>> settings as a reference. This is the status of the port: >>>> >>>> +-----------------------+----------------------------------- >>>> --------------------------------------------------+ >>>> | Field | Value >>>> | >>>> +-----------------------+----------------------------------- >>>> --------------------------------------------------+ >>>> | admin_state_up | True >>>> | >>>> | allowed_address_pairs | >>>> | >>>> | binding:host_id | openstack-1.quicksand.bitc. >>>> morphotrust.com >>>> <http://openstack-1.quicksand.bitc.morphotrust.com> >>>> | >>>> | binding:profile | {} >>>> | >>>> | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": >>>> true} | >>>> | binding:vif_type | ovs >>>> | >>>> | binding:vnic_type | normal >>>> | >>>> | device_id | 7319781c-6186-4684-ba60-260b5ecee97c >>>> | >>>> | device_owner | network:router_gateway >>>> | >>>> | extra_dhcp_opts | >>>> | >>>> | fixed_ips | {"subnet_id": >>>> "7761c2ee-e392-48ff-b69a-f0f10bbcb6db", "ip_address": >>>> "10.30.1.10"} | >>>> | id | 161de698-1666-4c0d-9248-8de900797301 >>>> | >>>> | mac_address | fa:16:3e:c9:ff:64 >>>> | >>>> | name | >>>> | >>>> | network_id | b10fc224-2332-49f5-b555-9090c3dc7f44 >>>> | >>>> | security_groups | >>>> | >>>> | status | DOWN >>>> | >>>> | tenant_id | >>>> | >>>> +-----------------------+----------------------------------- >>>> --------------------------------------------------+ >>>> >>>> I am just not able to get that port up. And since its not up I cant >>>> ping/ssh to the VMs. What do I need to do for vlans on my physical >>>> switch? >>>> >>>> --Brian >>>> >>>> On Thu, Dec 11, 2014 at 10:01 AM, Patrick Laimbock >>>> <patrick(a)laimbock.com <mailto:patrick@laimbock.com>> wrote: >>>> >>>> Hi Brian, >>>> >>>> On 11-12-14 16:15, brian lee wrote: >>>> >>>> It looks like my cute and paste did not work right. My >>>> br-ex >>>> device >>>> looks like this: >>>> >>>> DEVICE=br-ex >>>> OVSBOOTPROTO="dhcp" >>>> OVSDHCPINTERFACES="eth0" >>>> ONBOOT=yes >>>> NM_CONTROLLED=no >>>> TYPE=OVSBridge >>>> DEVICETYPE=ovs >>>> DEVICE=br-ex >>>> OVSBOOTPROTO="dhcp" >>>> OVSDHCPINTERFACES="eth0" >>>> ONBOOT=yes >>>> NM_CONTROLLED=no >>>> TYPE=OVSBridge >>>> DEVICETYPE=ovs >>>> >>>> Sorry about the confusion. >>>> >>>> >>>> I use RDO Juno and here are my interfaces: >>>> >>>> [root@neutron1-1 network-scripts]# cat ifcfg-br-ex >>>> DEVICE=br-ex >>>> TYPE=OVSBridge >>>> DEVICETYPE=ovs >>>> OVSBOOTPROTO=dhcp >>>> OVSDHCPINTERFACES=eth1 >>>> MACADDR="00:01:02:03:04:05" >>>> OVS_EXTRA="set bridge $DEVICE other-config:hwaddr=$MACADDR" >>>> ONBOOT=yes >>>> NM_CONTROLLED=no >>>> >>>> >>>> [root@neutron1-1 network-scripts]# cat ifcfg-eth1 >>>> DEVICE=eth1 >>>> TYPE=OVSPort >>>> DEVICETYPE=ovs >>>> OVS_BRIDGE=br-ex >>>> ONBOOT=yes >>>> BOOTPROTO=none >>>> NM_CONTROLLED=no >>>> >>>> HTH, >>>> Patrick >>>> >>>> >>>> _________________________________________________ >>>> Rdo-list mailing list >>>> Rdo-list(a)redhat.com <mailto:Rdo-list@redhat.com> >>>> https://www.redhat.com/__mailman/listinfo/rdo-list >>>> <https://www.redhat.com/mailman/listinfo/rdo-list> >>>> >>>> >>>

Another follow up: What needs to be configured on the compute nodes?