Re: [Rdo-list] First Mitaka test day summary

The deploy command was: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ntp-server clock.redhat.com -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml

To try and investigate this subject further, I enabled nested virtualization on the host, and changed the libvirt type to qemu. Unfortunately that also required turning off all the VMs on teh host for the kvm module to be reloaded - and now I have problems with keystone that doesn't want to come back up. I mailed the list about this problem in a separate thread. Thanks. Udi. On Thu, Jan 28, 2016 at 2:01 PM, John Trowbridge <trown(a)redhat.com&gt; wrote: > > > On 01/28/2016 04:35 AM, Udi Kalifon wrote: >> I can't launch instances in Mitaka on a virtual environment. I made >> sure that nested virtualization is enabled on the host. If I do nova >> show on the failed instance I see this: >> >> {"message": "No valid host was found. There are not enough hosts >> available.", "code": 500, "details": " File >> \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line >> 372, in build_instances >> context, request_spec, filter_properties) >> File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", >> line 416, in _schedule_instances >> hosts = self.scheduler_client.select_destinations(context, >> spec_obj) >> File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", >> line 372, in wrapped >> return func(*args, **kwargs) >> File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", >> line 51, in select_destinations >> return self.queryclient.select_destinations(context, spec_obj) >> File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", >> line 37, in __run_method >> return getattr(self.instance, __name)(*args, **kwargs) >> File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", >> line 32, in select_destinations >> return self.scheduler_rpcapi.select_destinations(context, spec_obj) >> File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", >> line 121, in select_destinations >> return cctxt.call(ctxt, 'select_destinations', **msg_args) >> File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", >> line 158, in call >> retry=self.retry) >> File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", >> line 90, in _send >> timeout=timeout, retry=retry) >> File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", >> line 466, in send >> retry=retry) >> File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", >> line 457, in _send >> raise result >> ", "created": "2016-01-28T09:20:47Z"} >> >> >> On the controller, I see a lot of db-related errors in nova-scheduler.log: >> >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db [-] >> Unexpected error while reporting service status >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> Traceback (most recent call last): >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", >> line 88, in _report_state >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> service.service_ref.save() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line >> 221, in wrapper >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return fn(self, *args, **kwargs) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 282, >> in save >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self._check_minimum_version() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 258, >> in _check_minimum_version >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> minver = self.get_minimum_version(self._context, self.binary) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line >> 179, in wrapper >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> result = fn(cls, context, *args, **kwargs) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 330, >> in get_minimum_version >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> use_slave=use_slave) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/db/api.py", line 118, in >> service_get_minimum_version >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> use_slave=use_slave) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line >> 468, in service_get_minimum_version >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> filter(models.Service.forced_down == false()).\ >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2503, in scalar >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> ret = self.one() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2472, in one >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> ret = list(self) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2515, in __iter__ >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return self._execute_and_instances(context) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2528, in _execute_and_instances >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> close_with_result=True) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2519, in _connection_from_session >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db **kw) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 882, in connection >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> execution_options=execution_options) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 887, in _connection_for_bind >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> engine, execution_options) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 334, in _connection_for_bind >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> conn = bind.contextual_connect() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2036, in contextual_connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db **kwargs) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 92, in __init__ >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self.dispatch.engine_connect(self, self.__branch) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line >> 258, in __call__ >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> fn(*args, **kw) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line >> 80, in _connect_ping_listener >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> connection.scalar(select([1])) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 844, in scalar >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return self.execute(object, *multiparams, **params).scalar() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 914, in execute >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return meth(self, multiparams, params) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line >> 323, in _execute_on_connection >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return connection._execute_clauseelement(self, multiparams, params) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 1010, in _execute_clauseelement >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> compiled_sql, distilled_params >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 1078, in _execute_context >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db None, None) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 1335, in _handle_dbapi_exception >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> util.raise_from_cause(newraise, exc_info) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line >> 199, in raise_from_cause >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> reraise(type(exception), exception, tb=exc_tb) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 1071, in _execute_context >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> conn = self._revalidate_connection() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 393, in _revalidate_connection >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self.__connection = self.engine.raw_connection(_connection=self) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2099, in raw_connection >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self.pool.unique_connection, _connection) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2075, in _wrap_pool_connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> util.reraise(*sys.exc_info()) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2069, in _wrap_pool_connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db return fn() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in >> unique_connection >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return _ConnectionFairy._checkout(self) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 708, in >> _checkout >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> fairy = _ConnectionRecord.checkout(pool) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in >> checkout >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> rec.checkin() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", >> line 60, in __exit__ >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> compat.reraise(exc_type, exc_value, exc_tb) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in >> checkout >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> dbapi_connection = rec.get_connection() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 590, in >> get_connection >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self.connection = self.__connect() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in >> __connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> connection = self.__pool._invoke_creator(self) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", >> line 97, in connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return dialect.connect(*cargs, **cparams) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", >> line 377, in connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return self.dbapi.connect(*cargs, **cparams) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in >> Connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> return Connection(*args, **kwargs) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 657, >> in __init__ >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self.connect() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 850, >> in connect >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> self._get_server_information() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1061, >> in _get_server_information >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> packet = self._read_packet() >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 895, >> in _read_packet >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> packet_header = self._read_bytes(4) >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 922, >> in _read_bytes >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> 2013, "Lost connection to MySQL server during query") >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost >> connection to MySQL server during query') [SQL: u'SELECT 1'] >> 2016-01-28 08:25:02.645 1121 ERROR nova.servicegroup.drivers.db >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db [-] >> Unexpected error while reporting service status >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> Traceback (most recent call last): >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", >> line 88, in _report_state >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> service.service_ref.save() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line >> 221, in wrapper >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> return fn(self, *args, **kwargs) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 282, >> in save >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> self._check_minimum_version() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 258, >> in _check_minimum_version >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> minver = self.get_minimum_version(self._context, self.binary) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line >> 179, in wrapper >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> result = fn(cls, context, *args, **kwargs) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 330, >> in get_minimum_version >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> use_slave=use_slave) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/db/api.py", line 118, in >> service_get_minimum_version >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> use_slave=use_slave) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line >> 468, in service_get_minimum_version >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> filter(models.Service.forced_down == false()).\ >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2503, in scalar >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> ret = self.one() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2472, in one >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> ret = list(self) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2515, in __iter__ >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> return self._execute_and_instances(context) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2528, in _execute_and_instances >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> close_with_result=True) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2519, in _connection_from_session >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db **kw) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 882, in connection >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> execution_options=execution_options) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 887, in _connection_for_bind >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> engine, execution_options) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 334, in _connection_for_bind >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> conn = bind.contextual_connect() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2034, in contextual_connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> self._wrap_pool_connect(self.pool.connect, None), >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2073, in _wrap_pool_connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db e, >> dialect, self) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 1399, in _handle_dbapi_exception_noconnection >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> util.raise_from_cause(newraise, exc_info) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line >> 199, in raise_from_cause >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> reraise(type(exception), exception, tb=exc_tb) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2069, in _wrap_pool_connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db return fn() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in >> connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> return _ConnectionFairy._checkout(self) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 708, in >> _checkout >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> fairy = _ConnectionRecord.checkout(pool) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in >> checkout >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> rec.checkin() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", >> line 60, in __exit__ >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> compat.reraise(exc_type, exc_value, exc_tb) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in >> checkout >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> dbapi_connection = rec.get_connection() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in >> get_connection >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> self.connection = self.__connect() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in >> __connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> connection = self.__pool._invoke_creator(self) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", >> line 97, in connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> return dialect.connect(*cargs, **cparams) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", >> line 377, in connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> return self.dbapi.connect(*cargs, **cparams) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 88, in >> Connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> return Connection(*args, **kwargs) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 657, >> in __init__ >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> self.connect() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 850, >> in connect >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> self._get_server_information() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1061, >> in _get_server_information >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> packet = self._read_packet() >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 895, >> in _read_packet >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> packet_header = self._read_bytes(4) >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 922, >> in _read_bytes >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> 2013, "Lost connection to MySQL server during query") >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost >> connection to MySQL server during query') >> 2016-01-28 08:25:08.191 1121 ERROR nova.servicegroup.drivers.db >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db [-] >> Unexpected error while reporting service status >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> Traceback (most recent call last): >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", >> line 88, in _report_state >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> service.service_ref.save() >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line >> 221, in wrapper >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> return fn(self, *args, **kwargs) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 282, >> in save >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> self._check_minimum_version() >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 258, >> in _check_minimum_version >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> minver = self.get_minimum_version(self._context, self.binary) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line >> 179, in wrapper >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> result = fn(cls, context, *args, **kwargs) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 330, >> in get_minimum_version >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> use_slave=use_slave) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/db/api.py", line 118, in >> service_get_minimum_version >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> use_slave=use_slave) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line >> 468, in service_get_minimum_version >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> filter(models.Service.forced_down == false()).\ >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2503, in scalar >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> ret = self.one() >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2472, in one >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> ret = list(self) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2515, in __iter__ >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> return self._execute_and_instances(context) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2528, in _execute_and_instances >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> close_with_result=True) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line >> 2519, in _connection_from_session >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db **kw) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 882, in connection >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> execution_options=execution_options) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 887, in _connection_for_bind >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> engine, execution_options) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line >> 334, in _connection_for_bind >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> conn = bind.contextual_connect() >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2034, in contextual_connect >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> self._wrap_pool_connect(self.pool.connect, None), >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line >> 2069, in _wrap_pool_connect >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db return fn() >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 376, in >> connect >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> return _ConnectionFairy._checkout(self) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 729, in >> _checkout >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db fairy) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line >> 258, in __call__ >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> fn(*args, **kw) >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db File >> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line >> 352, in checkout >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db if >> connection_record.info['pid'] != pid: >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db KeyError: 'pid' >> 2016-01-28 08:25:16.966 1121 ERROR nova.servicegroup.drivers.db >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool [-] >> Exception during reset or similar >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool Traceback >> (most recent call last): >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 631, in >> _finalize_fairy >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool >> fairy._reset(pool) >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 765, in >> _reset >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool >> pool._dialect.do_rollback(self) >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", >> line 2519, in do_rollback >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool >> dbapi_connection.rollback() >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 723, >> in rollback >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool >> self._execute_command(COMMAND.COM_QUERY, "ROLLBACK") >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 956, >> in _execute_command >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool raise >> err.InterfaceError("(0, '')") >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool >> InterfaceError: (0, '') >> 2016-01-28 08:25:16.969 1121 ERROR sqlalchemy.pool.QueuePool >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool [-] >> Exception closing connection <pymysql.connections.Connection object at >> 0x510f550> >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool Traceback >> (most recent call last): >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 290, in >> _close_connection >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool >> self._dialect.do_close(connection) >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", >> line 418, in do_close >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool >> dbapi_connection.close() >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool File >> "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 662, >> in close >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool raise >> err.Error("Already closed") >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool Error: >> Already closed >> 2016-01-28 08:25:16.971 1121 ERROR sqlalchemy.pool.QueuePool >> 2016-01-28 08:25:27.009 1121 INFO nova.servicegroup.drivers.db [-] >> Recovered from being unable to report status. >> 2016-01-28 08:27:21.165 31979 WARNING >> oslo_reports.guru_meditation_report [-] Guru mediation now registers >> SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 >> will no longer be registered in a future release, so please use >> SIGUSR2 to generate reports. >> 2016-01-28 08:27:21.544 31979 WARNING oslo_config.cfg >> [req-d3d0b357-c23b-4e8b-b322-32f5b225dc19 - - - - -] Option >> "use_local" from group "conductor" is deprecated for removal. Its >> value may be silently ignored in the future. >> 2016-01-28 08:27:21.606 31979 INFO oslo_service.periodic_task >> [req-d3d0b357-c23b-4e8b-b322-32f5b225dc19 - - - - -] Skipping periodic >> task _periodic_update_dns because its interval is negative >> 2016-01-28 08:27:21.635 31979 INFO nova.service [-] Starting scheduler >> node (version 13.0.0-dev80.el7.centos) >> >> Was anyone successful in launching instances in Mitaka? I installed >> with the director, it's an HA deployment without network isolation in >> a virtualized environment. >> > > CI is passing on that configuration, and I have had success with it > manually. Could you post your deploy command? > >> Thanks a lot, >> Udi. >> >> On Wed, Jan 27, 2016 at 11:17 PM, John Trowbridge <trown(a)redhat.com&gt; wrote: >>> >>> >>> On 01/27/2016 01:25 PM, Udi Kalifon wrote: >>>> Hello. >>>> >>>> The good news is that I succeeded to deploy :). I haven't yet tried to >>>> test the overcloud for any sanity, but in past test days I was never >>>> able to report any success - so maybe it's a sign that things are >>>> stabilizing. >>> >>> That's awesome! >>> >>>> >>>> I deployed with rdo-manager on a virtual setup according to the >>>> instructions in https://www.rdoproject.org/rdo-manager/. I wasn't able >>>> to deploy with network isolation, because I assume that my templates >>>> from 7.x require changes, but I haven't seen any documentation on >>>> what's changed. If you can point me in the right direction to get >>>> network isolation working for this environment I will test it >>>> tomorrow. >>>> >>>> Some of the problems I hit today: >>>> >>>> 1) The link to the quickstart guide from the testday page >>>> https://www.rdoproject.org/testday/mitaka/milestone2/ points to a very >>>> old github page. The correct link should be the one I already >>>> mentioned: https://www.rdoproject.org/rdo-manager/ >>> >>> I fixed that link, thanks! >>> >>>> >>>> 2) The prerequisites to installing ansible are not documented. On a >>>> fresh CentOS 7 I had to install python-virtualenv, git, and gcc. I >>>> then ran "easy_install pip" and "pip install >>>> git+https://github.com/ansible/ansible.git@v2.0.0-0.6.rc1#egg=ansible" >>>> to be able to run the playbook which installs the virtual environment. >>> >>> The quickstart.sh will do all of that for you, but if you wanted to >>> submit a patch for more detailed instructions for manually setting up >>> the virtualenv, that would great. For tripleo-quickstart, gerrit is >>> setup and follows the same gerrit workflow as everything else. >>> >>>> >>>> Thanks, >>>> Udi. >>>> >>>> _______________________________________________ >>>> Rdo-list mailing list >>>> Rdo-list(a)redhat.com >>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>> >>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>> >>> >>> _______________________________________________ >>> Rdo-list mailing list >>> Rdo-list(a)redhat.com >>> https://www.redhat.com/mailman/listinfo/rdo-list >>> >>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com