[Rdo-list] [Neutron] Fwd: Icehouse with Neutron and ML2 on RHEL

FYI... ---------- Forwarded message ---------- From: Phil Hopkins <phil.hopkins(a)rackspace.com&gt; Date: Sat, Mar 1, 2014 at 8:13 PM Subject: Re: Icehouse with Neutron and ML2 on RHEL To: Matt Kassawara <mkassawara(a)gmail.com&gt; Cc: Carl Perry <edolnx(a)gmail.com&gt;, Edgar Magana <emagana(a)plumgrid.com&gt;, Nick Chase <nchase(a)mirantis.com&gt; Very interesting, something is broke bad there. There should be a new build of the packages this week. Lets hope that fixes this.problem. Phil Sent from my iPhone On Mar 1, 2014, at 8:29 PM, "Matt Kassawara" <mkassawara(a)gmail.com&gt; wrote: The following tables appear after starting neutron-server with a fresh database: mysql> show tables; +---------------------+ | Tables_in_neutron | +---------------------+ | agents | | ml2_gre_allocations | | ml2_gre_endpoints | | networks | | quotas | | securitygroups | +---------------------+ 6 rows in set (0.00 sec) The log shows plenty of errors regarding missing database information. I'm particularly curious about this one: 2014-03-01 18:58:55.133 2877 INFO neutron.db.api [-] Database registration exception: (OperationalError) (1005, "Can't create table 'neutron.networkdhcpagentbindings' (errno: 150)") '\nCREATE TABLE networkdhcpagentbindings (\n\tnetwork_id VARCHAR(36) NOT NULL, \n\tdhcp_agent_id VARCHAR(36) NOT NULL, \n\tPRIMARY KEY (network_id, dhcp_agent_id), \n\tFOREIGN KEY(network_id) REFERENCES networks (id) ON DELETE CASCADE, \n\tFOREIGN KEY(dhcp_agent_id) REFERENCES agents (id) ON DELETE CASCADE\n)ENGINE=InnoDB\n\n' () Interestingly, trying this SQL command manually also fails: mysql> create table networkdhcpagentbindings (network_id varchar(36) not null, dhcp_agent_id varchar(36) not null, primary key (network_id, dhcp_agent_id), foreign key(network_id) references networks (id) on delete cascade, foreign key(dhcp_agent_id) references agents (id) on delete cascade ) engine=InnoDB; ERROR 1005 (HY000): Can't create table 'neutron.networkdhcpagentbindings' (errno: 150) However, it works without "engine=InnoDB" appended at the end which leads me to some sort of referential integrity problem... or I'm mistyping something. :) mysql> show engine innodb status; LATEST FOREIGN KEY ERROR ------------------------ 140301 19:24:40 Error in foreign key constraint of table neutron/networkdhcpagentbindings: foreign key(dhcp_agent_id) references agents (id) on delete cascade) engine=InnoDB: Cannot resolve table name close to: (id) on delete cascade) engine=InnoDB In comparison, here's the neutron database tables from my functional Ubuntu installation: mysql> show tables; +---------------------------+ | Tables_in_neutron | +---------------------------+ | agents | | allowedaddresspairs | | dnsnameservers | | externalnetworks | | extradhcpopts | | floatingips | | ipallocationpools | | ipallocations | | ipavailabilityranges | | ml2_gre_allocations | | ml2_gre_endpoints | | ml2_network_segments | | ml2_port_bindings | | networkdhcpagentbindings | | networks | | ports | | quotas | | routerl3agentbindings | | routerroutes | | routers | | securitygroupportbindings | | securitygrouprules | | securitygroups | | subnetroutes | | subnets | +---------------------------+ 25 rows in set (0.00 sec) On Sat, Mar 1, 2014 at 6:46 PM, Phil Hopkins <phil.hopkins(a)rackspace.com&gt;wrote: > I was thinking nova not neutron. Try dropping the neutron database, > recreate it and restart the neutron-server service. > > That should rebuild the tables. Check and see if they are created. If so > then see if the error reoccurs. > > Also see if any errors show up when the neutron-server service is > restarted. Maybe it is having trouble creating the database tables. > > Phil > > Sent from my iPhone > > On Mar 1, 2014, at 7:00 PM, "Matt Kassawara" <mkassawara(a)gmail.com&gt; wrote: > > I did that a few times. Unlike the other similar utilities, > neutron-db-manage doesn't offer a "sync" command. > > > On Sat, Mar 1, 2014 at 5:52 PM, Phil Hopkins > <phil.hopkins(a)rackspace.com&gt;wrote: > >> You might try rerunning the db sync to ensure the database tables are >> built correctly. >> >> Phil >> >> Sent from my iPhone >> >> On Mar 1, 2014, at 6:21 PM, "Matt Kassawara" <mkassawara(a)gmail.com&gt; >> wrote: >> >> Fresh. >> >> >> On Sat, Mar 1, 2014 at 5:19 PM, Phil Hopkins >> <phil.hopkins(a)rackspace.com&gt;wrote: >> >>> Was this a fresh install or an upgrade? >>> >>> Phil >>> >>> Sent from my iPhone >>> >>> On Mar 1, 2014, at 5:04 PM, "Matt Kassawara" <mkassawara(a)gmail.com&gt; >>> wrote: >>> >>> If you thought getting Icehouse with Neutron/ML2 working on Ubuntu >>> took some effort, the RHEL situation appears much worse. All of the >>> Neutron >>> daemons spew database errors about missing tables and/or inability to >>> create them. >>> >>> For example: >>> >>> [root@hst-osctl8 neutron]# neutron net-list >>> Request Failed: internal server error while processing your request. >>> >>> From the server log... >>> >>> 2014-03-01 15:37:07.970 2584 INFO urllib3.connectionpool [-] Starting >>> new HTTP connection (1): hst-osctl8 >>> 2014-03-01 15:37:08.133 2584 ERROR neutron.api.v2.resource >>> [req-9ec4eec0-bf1b-4720-af03-fbdba6a03c14 None] index failed >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource Traceback >>> (most recent call last): >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib/python2.6/site-packages/neutron/api/v2/resource.py", line 84, >>> in >>> resource >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource result = >>> method(request=request, **args) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 279, in >>> index >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource return >>> self._items(request, True, parent_id) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib/python2.6/site-packages/neutron/api/v2/base.py", line 233, in >>> _items >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource obj_list >>> = obj_getter(request.context, **kwargs) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line >>> 367, >>> in get_networks >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource limit, >>> marker, page_reverse) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line >>> 1029, in get_networks >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource >>> page_reverse=page_reverse) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib/python2.6/site-packages/neutron/db/db_base_plugin_v2.py", line >>> 196, in _get_collection >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource items = >>> [dict_func(c, fields) for c in query] >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", >>> line 2227, in __iter__ >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource return >>> self._execute_and_instances(context) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", >>> line 2242, in _execute_and_instances >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource result = >>> conn.execute(querycontext.statement, self._params) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", >>> line 1449, in execute >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource params) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", >>> line 1584, in _execute_clauseelement >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource >>> compiled_sql, distilled_params >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", >>> line 1698, in _execute_context >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource context) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", >>> line 1691, in _execute_context >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource context) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", >>> line 331, in do_execute >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource >>> cursor.execute(statement, parameters) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in >>> execute >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource >>> self.errorhandler(self, exc, value) >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource File >>> "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in >>> defaulterrorhandler >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource raise >>> errorclass, errorvalue >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource >>> ProgrammingError: (ProgrammingError) (1146, "Table >>> 'neutron.externalnetworks' doesn't exist") 'SELECT networks.tenant_id AS >>> networks_tenant_id, networks.id AS networks_id, networks.name AS >>> networks_name, networks.status AS networks_status, >>> networks.admin_state_up >>> AS networks_admin_state_up, networks.shared AS networks_shared, >>> externalnetworks_1.network_id AS externalnetworks_1_network_id \nFROM >>> networks LEFT OUTER JOIN externalnetworks ON networks.id = >>> externalnetworks.network_id LEFT OUTER JOIN externalnetworks AS >>> externalnetworks_1 ON networks.id = externalnetworks_1.network_id' () >>> 2014-03-01 15:37:08.133 2584 TRACE neutron.api.v2.resource >>> >>> Although I haven't needed to use it before, I poked around at various >>> permutations of neutron-db-manage, all of which failed with various SQL >>> errors. I also dissected the functional neutron-server package on Ubuntu >>> and found no post-installation script calls to neutron-db-manage. On a >>> related note, Steve Gordon told me that RH should release icehouse-3 >>> packages this week which might address this issue. >>> >>> >> >