Hi ,


I am getting the ConnectFailure error message upon triggering “nova image-list” command. nova-api process should be listening on 8774. It doesn't look like it is not running. Also I do not find any error logs in nova-api.log nova-compute.log and nova-conductor.log. I am using openstack-mitaka release on host (Cent OS 7.2). How can I debug and know what prevents it from running ? please suggest.


Note: This was working while back and got this issue all of a sudden.


Here are some logs.


[root@localhost ~(keystone_admin)]# nova image-list

ERROR (ConnectFailure): Unable to establish connection to http://172.18.121.48:8774/v2/4bc608763cee41d9a8df26d3ef919825

[root@localhost ~(keystone_admin)]#

 

[root@localhost ~(keystone_admin)]# service openstack-nova-api restart

Redirecting to /bin/systemctl restart  openstack-nova-api.service

Job for openstack-nova-api.service failed because the control process exited with error code. See "systemctl status openstack-nova-api.service" and "journalctl -xe" for details.

[root@localhost ~(keystone_admin)]# systemctl status openstack-nova-api.service

â openstack-nova-api.service - OpenStack Nova API Server

   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled)

   Active: activating (start) since Wed 2016-06-08 07:59:20 EDT; 2s ago

 Main PID: 179955 (nova-api)

   CGroup: /system.slice/openstack-nova-api.service

           ââ179955 /usr/bin/python2 /usr/bin/nova-api

 

Jun 08 07:59:20 localhost systemd[1]: Starting OpenStack Nova API Server...

Jun 08 07:59:22 localhost python2[179955]: detected unhandled Python exception in '/usr/bin/nova-api'

[root@localhost ~(keystone_admin)]#

 

[root@localhost ~(keystone_admin)]#  keystone endpoint-list

/usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.

  'python-keystoneclient.', DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.

  'the 2.0.0 release.', DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead

  super(Client, self).__init__(**kwargs)

/usr/lib/python2.7/site-packages/debtcollector/renames.py:45: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead

  return f(*args, **kwargs)

/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:371: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.

  'the 2.0.0 release.', DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/session.py:140: DeprecationWarning: keystoneclient.session.Session is deprecated as of the 2.1.0 release in favor of keystoneauth1.session.Session. It will be removed in future releases.

  DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py:56: DeprecationWarning: keystoneclient auth plugins are deprecated as of the 2.1.0 release in favor of keystoneauth1 plugins. They will be removed in future releases.

  'in future releases.', DeprecationWarning)

+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+-------------------------------------------------+----------------------------------+

|                id                |   region  |                    publicurl                    |                   internalurl                   |                     adminurl                    |            service_id            |

+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+-------------------------------------------------+----------------------------------+

| 02fcec9a7b834128b3e30403c4ed0de7 | RegionOne | http://172.18.121.48:8080/v1/AUTH_%(tenant_id)s | http://172.18.121.48:8080/v1/AUTH_%(tenant_id)s | http://172.18.121.48:8080/v1/AUTH_%(tenant_id)s | 5533324a63d8402888040832640a19d0 |

| 295802909413422cb7c22dc1e268bce9 | RegionOne |    http://172.18.121.48:8774/v2/%(tenant_id)s   |    http://172.18.121.48:8774/v2/%(tenant_id)s   |    http://172.18.121.48:8774/v2/%(tenant_id)s   | f7fe68bf4cec47a4a3c942f3916dc377 |

| 2a125f10b0d04f8a9306dede85b65514 | RegionOne |            http://172.18.121.48:9696            |            http://172.18.121.48:9696            |            http://172.18.121.48:9696            | b2a60cdc144e40a49757f13c2264f030 |

| 2d1a91d39f3d421cb1b2fe73fba5fd3a | RegionOne |            http://172.18.121.48:8777            |            http://172.18.121.48:8777            |            http://172.18.121.48:8777            | e6d750ac5ef3433799d4fe39518a3fe6 |

| 47b634f3e18e4caf914521a1a4157008 | RegionOne |            http://172.18.121.48:8042            |            http://172.18.121.48:8042            |            http://172.18.121.48:8042            | 07cd8adf66254b4ab9b07be03a24084b |

| 595913f7227b44dc8753db3b0cf6acdc | RegionOne |            http://172.18.121.48:8041            |            http://172.18.121.48:8041            |            http://172.18.121.48:8041            | f43240abe5f3476ea64a8bd381fe4da7 |

| 64381b509bc84639b6a4710e6d99a23b | RegionOne |    http://172.18.121.48:8776/v1/%(tenant_id)s   |    http://172.18.121.48:8776/v1/%(tenant_id)s   |    http://172.18.121.48:8776/v1/%(tenant_id)s   | 7edc7bedf93d4f388185699b9793ec7f |

| 727d25775be54c9f8453f697ae5cb625 | RegionOne |          http://172.18.121.48:5000/v2.0         |          http://172.18.121.48:5000/v2.0         |         http://172.18.121.48:35357/v2.0         | 25e99a2a98f244d9a73bf965acdd39da |

| 9049338c57574b2d8ff8308b1a4265a5 | RegionOne |    http://172.18.121.48:8776/v2/%(tenant_id)s   |    http://172.18.121.48:8776/v2/%(tenant_id)s   |    http://172.18.121.48:8776/v2/%(tenant_id)s   | 6e070f0629094b72b66025250fdbda64 |

| c051c0f9649143f6b29eaf0895940abe | RegionOne |            http://172.18.121.48:9292            |            http://172.18.121.48:9292            |            http://172.18.121.48:9292            | 40874e10139a47eb88dfec2114047a34 |

| ee4a00c1e8334cb8921fa3f2a7c82f1b | RegionOne |           http://172.18.121.48:8774/v3          |           http://172.18.121.48:8774/v3          |           http://172.18.121.48:8774/v3          | 24c92ce4cd354e3db6c5ad59b8beeae8 |

| fa60b5ba0ab7436ab1ffebb2982d3ccc | RegionOne |      http://127.0.0.1:8776/v3/%(tenant_id)s     |      http://127.0.0.1:8776/v3/%(tenant_id)s     |      http://127.0.0.1:8776/v3/%(tenant_id)s     | b0b6b97cf9d649c9800300cc64b0e866 |

+----------------------------------+-----------+-------------------------------------------------+-------------------------------------------------+-------------------------------------------------+----------------------------------+

[root@localhost ~(keystone_admin)]#

 

 

[root@localhost ~(keystone_admin)]# netstat -ntlp | grep 8774

[root@localhost ~(keystone_admin)]#

 

[root@localhost ~(keystone_admin)]# ps -ef | grep nova-api

nova     156427      1 86 07:51 ?        00:00:01 /usr/bin/python2 /usr/bin/nova-api

 [root@localhost ~(keystone_admin)]#

 

[root@localhost ~(keystone_admin)]#  lsof -i :8774

[root@localhost ~(keystone_admin)]#

 

 

[root@localhost ~(keystone_admin)]# keystone  user-list

/usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.

  'python-keystoneclient.', DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.

  'the 2.0.0 release.', DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead

  super(Client, self).__init__(**kwargs)

/usr/lib/python2.7/site-packages/debtcollector/renames.py:45: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead

  return f(*args, **kwargs)

/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:371: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.

  'the 2.0.0 release.', DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/session.py:140: DeprecationWarning: keystoneclient.session.Session is deprecated as of the 2.1.0 release in favor of keystoneauth1.session.Session. It will be removed in future releases.

  DeprecationWarning)

/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py:56: DeprecationWarning: keystoneclient auth plugins are deprecated as of the 2.1.0 release in favor of keystoneauth1 plugins. They will be removed in future releases.

  'in future releases.', DeprecationWarning)

+----------------------------------+------------+---------+----------------------+

|                id                |    name    | enabled |        email         |

+----------------------------------+------------+---------+----------------------+

| 266f5859848e4f39b9725203dda5c3f2 |   admin    |   True  |    root@localhost    |

| 79a6ff3cc7cc4d018247c750adbc18e7 |    aodh    |   True  |    aodh@localhost    |

| 90f28a2a80054132a901d39da307213f | ceilometer |   True  | ceilometer@localhost |

| 16fa5ffa60e147d89ad84646b6519278 |   cinder   |   True  |   cinder@localhost   |

| c6312ec6c2c444288a412f32173fcd99 |   glance   |   True  |   glance@localhost   |

| ac8fb9c33d404a1697d576d428db90b3 |  gnocchi   |   True  |  gnocchi@localhost   |

| 1a5b4da4ed974ac8a6c78b752ac8fab6 |  neutron   |   True  |  neutron@localhost   |

| f21e8a15da5c40b7957416de4fa91b62 |    nova    |   True  |    nova@localhost    |

| b843358d7ae44944b11af38ce4b61f4d |   swift    |   True  |   swift@localhost    |

+----------------------------------+------------+---------+----------------------+

[root@localhost ~(keystone_admin)]#

 

[root@localhost ~(keystone_admin)]# nova-manage service list

Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.

Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".

Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".

DEPRECATED: Use the nova service-* commands from python-novaclient instead or the os-services REST resource. The service subcommand will be removed in the 14.0 release.

Binary           Host                                 Zone             Status     State Updated_At

nova-osapi_compute 0.0.0.0                              internal         enabled    XXX   None

nova-metadata    0.0.0.0                              internal         enabled    XXX   None

nova-cert        localhost                               internal         enabled    XXX   2016-06-08 06:12:38

nova-consoleauth localhost                               internal         enabled    XXX   2016-06-08 06:12:37

nova-scheduler   localhost                               internal         enabled    XXX   2016-06-08 06:12:38

nova-conductor   localhost                               internal         enabled    XXX   2016-06-08 06:12:37

nova-compute     localhost                               nova             enabled    XXX   2016-06-08 06:12:43

[root@localhost ~(keystone_admin)]#

 

[root@localhost ~(keystone_admin)]# ls -l /var/log/nova/

total 4

-rw-r--r--. 1 nova nova   0 Jun  8 05:22 nova-api.log

-rw-r--r--. 1 nova nova   0 Jun  8 05:22 nova-cert.log

-rw-r--r--. 1 nova nova   0 Jun  8 05:22 nova-compute.log

-rw-r--r--. 1 nova nova   0 Jun  8 05:22 nova-conductor.log

-rw-r--r--. 1 nova nova   0 Jun  8 05:22 nova-consoleauth.log

-rw-r--r--. 1 root root   0 Jun  8 05:22 nova-manage.log

-rw-r--r--. 1 nova nova 995 Jun  8 05:32 nova-novncproxy.log

-rw-r--r--. 1 nova nova   0 Jun  8 05:23 nova-scheduler.log

[root@localhost ~(keystone_admin)]#

 

 

Regards,

Chinmaya