Hi,

I am trying to implement AIO POC with Trove enabled using packstack on centos 7. I have created image for mongodb and data store is also configured.

I am having issue when I try to create trove instance, instance status is error. Though trove was deployed by packstack, I cross checked all trove config files as per this guide https://docs.openstack.org/trove/latest/install/install-rdo.html they seems ok.

trove-api.log has following error,

2017-12-08 13:14:48.021 15950 INFO trove.instance.service [-] Creating a database instance for tenant 'b20cf7ccbd1a4636bb8a8bdf8cd3973d'

2017-12-08 13:14:48.048 15950 WARNING trove.common.cfg [-] Manager name ('datastore_manager') not defined, using 'mysql' options instead.

2017-12-08 13:14:48.049 15950 WARNING trove.common.cfg [-] Manager name ('datastore_manager') not defined, using 'mysql' options instead.

2017-12-08 13:14:48.062 15950 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.

2017-12-08 13:14:49.426 15950 INFO eventlet.wsgi [-] 172.16.5.184 - - [08/Dec/2017 13:14:49] "POST /v1.0/b20cf7ccbd1a4636bb8a8bdf8cd3973d/instances HTTP/1.1" 200 1085 1.411768

2017-12-08 13:14:49.851 15963 INFO trove.instance.service [-] Listing database instances for tenant 'b20cf7ccbd1a4636bb8a8bdf8cd3973d'

2017-12-08 13:14:49.855 15963 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.

2017-12-08 13:14:51.215 15963 INFO eventlet.wsgi [-] 172.16.5.184 - - [08/Dec/2017 13:14:51] "GET /v1.0/b20cf7ccbd1a4636bb8a8bdf8cd3973d/instances?limit=20 HTTP/1.1" 200 949 1.738005

2017-12-08 13:14:51.230 15963 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.

2017-12-08 13:14:52.299 15963 INFO eventlet.wsgi [-] 172.16.5.184 - - [08/Dec/2017 13:14:52] "GET /v1.0/b20cf7ccbd1a4636bb8a8bdf8cd3973d/flavors HTTP/1.1" 200 1936 1.080438

2017-12-08 13:14:52.306 15955 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option.

2017-12-08 13:14:53.313 15955 INFO trove.instance.service [-] Showing database instance '62b40b11-9440-41ad-929f-7fd8b0bb6208' for tenant 'b20cf7ccbd1a4636bb8a8bdf8cd3973d'

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api [-] Error calling get_filesystem_stats: MessagingTimeout: Timed out waiting for a reply to message ID 5827c0618531497f9e7da6cf01de28b1

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api Traceback (most recent call last):

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/trove/guestagent/api.py", line 90, in _call

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api result = cctxt.call(self.context, method_name, **kwargs)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api retry=self.retry)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api timeout=timeout, retry=retry)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api retry=retry)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in _send

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api result = self._waiter.wait(msg_id, timeout)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api message = self.waiters.get(msg_id, timeout=timeout)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api 'to message ID %s' % msg_id)

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api MessagingTimeout: Timed out waiting for a reply to message ID 5827c0618531497f9e7da6cf01de28b1

2017-12-08 13:14:58.422 15955 ERROR trove.guestagent.api

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models [-] An error occurred communicating with the guest: Timed out waiting for a reply to message ID 5827c0618531497f9e7da6cf01de28b1.: GuestError: An error occurred communicating with the guest: Timed out waiting for a reply to message ID 5827c0618531497f9e7da6cf01de28b1.

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models Traceback (most recent call last):

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models File "/usr/lib/python2.7/site-packages/trove/instance/models.py", line 561, in load_guest_info

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models volume_info = guest.get_volume_info()

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models File "/usr/lib/python2.7/site-packages/trove/guestagent/api.py", line 423, in get_volume_info

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models version=version, fs_path=None)

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models File "/usr/lib/python2.7/site-packages/trove/guestagent/api.py", line 99, in _call

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models raise exception.GuestError(original_message=str(e))

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models GuestError: An error occurred communicating with the guest: Timed out waiting for a reply to message ID 5827c0618531497f9e7da6cf01de28b1.

2017-12-08 13:14:58.424 15955 ERROR trove.instance.models

2017-12-08 13:14:58.483 15955 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.

2017-12-08 13:14:59.795 15955 INFO eventlet.wsgi [-] 172.16.5.184 - - [08/Dec/2017 13:14:59] "GET /v1.0/b20cf7ccbd1a4636bb8a8bdf8cd3973d/instances/62b40b11-9440-41ad-929f-7fd8b0bb6208 HTTP/1.1" 200 2327 7.491198

Whereas trove-taskmanager.log has following error,

2017-12-08 13:07:25.719 15981 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.

2017-12-08 13:07:26.715 15981 ERROR trove.instance.models [-] Could not find nova server_id(None).: NotFound: Instance None could not be found. (HTTP 404) (Request-ID: req-dba69b39-19ed-49bc-8f32-d37d7d356a24)

2017-12-08 13:07:26.716 15981 ERROR trove.common.exception [-] Cannot find compute instance None for instance 3f9f4c53-7ba6-4e78-9278-ffd9918de42b.: NotFound: Instance None could not be found. (HTTP 404) (Request-ID: req-dba69b39-19ed-49bc-8f32-d37d7d356a24)

2017-12-08 13:07:26.716 15981 ERROR trove.instance.models [-] Could not load compute instance None.: ComputeInstanceNotFound: Resource 3f9f4c53-7ba6-4e78-9278-ffd9918de42b can not be retrieved.

2017-12-08 13:07:26.724 15981 WARNING novaclient.v2.client [-] Property `management_url` is deprecated for SessionClient. It should be set via `endpoint_override` variable while class initialization.: UnprocessableEntity: Instance 3f9f4c53-7ba6-4e78-9278-ffd9918de42b is not ready.

2017-12-08 13:14:49.336 15981 INFO trove.taskmanager.models [-] Creating instance 62b40b11-9440-41ad-929f-7fd8b0bb6208.

2017-12-08 13:14:53.094 15981 ERROR trove.taskmanager.models [-] Failed to create volume for instance 62b40b11-9440-41ad-929f-7fd8b0bb6208

The resource could not be found.

(HTTP 404)

Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 811, in _build_volume_info

volume_size, volume_type, datastore_manager)

File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 848, in _create_volume

volume_type=volume_type)

File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 290, in create

return self._create('/volumes', body, 'volume')

File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 324, in _create

resp, body = self.api.client.post(url, body=body)

File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 434, in post

return self._cs_request(url, 'POST', **kwargs)

File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 387, in _cs_request

resp, body = self.request(url, method, **kwargs)

File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 369, in request

raise exceptions.from_response(resp, body)

NotFound: The resource could not be found.

(HTTP 404)

: NotFound: The resource could not be found.

(HTTP 404)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server [-] Exception during message handling: TroveError: Failed to create volume for instance 62b40b11-9440-41ad-929f-7fd8b0bb6208

The resource could not be found.

(HTTP 404)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server Traceback (most recent call last):

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 384, in create_instance

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server locality)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/trove/taskmanager/manager.py", line 364, in _create_instance

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server scheduler_hints)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 505, in create_instance

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server scheduler_hints)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 782, in _create_server_volume_individually

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server volume_type=volume_type)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 815, in _build_volume_info

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server self._log_and_raise(e, msg, err)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/trove/taskmanager/models.py", line 839, in _log_and_raise

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server raise TroveError(message=full_message)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server TroveError: Failed to create volume for instance 62b40b11-9440-41ad-929f-7fd8b0bb6208

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server The resource could not be found.

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server (HTTP 404)

2017-12-08 13:14:53.247 15981 ERROR oslo_messaging.rpc.server

Regards,

/Sonali