7:29 a.m.
Hi All
I have been trying to set instack with RDO. I have successfully
installed undercloud and moving on to overcloud. Now, when I run
"instack-deploy-overcloud", I get the following error:
|+ OVERCLOUD_YAML_PATH=overcloud.yaml
+ heat stack-create-f
overcloud.yaml-PAdminToken=b003d63242f5db3e1ad4864ae66911e02ba19bcb-PAdminPassword=7bfe4d4a18280752ad07f259a69a3ed00db2ab44
-PCinderPassword=df0893b4355f3511a6d67538dd592d02d1bc11d3-PGlancePassword=066f65f878157b438a916ccbd44e0b7037ee118f
-PHeatPassword=58fda0e4d6708e0164167b11fe6fca6ab6b35ec6
-PNeutronPassword=80853ad029feb77bb7c60d035542f21aa5c24177
-PNovaPassword=331474580be53b78e40c91dfdfc2323578a035e7
-PNeutronPublicInterface=eth0-PSwiftPassword=b0eca57b45ebf3dd5cae071dc3880888fb1d4840-PSwiftHashSuffix=a8d87f3952d6f91da589fbef801bb92141fd1461-PNovaComputeLibvirtType=qemu-P'GlanceLogFile='\'''\'''
-PNeutronDnsmasqOptions=dhcp-option-force=26,1400 overcloud
+--------------------------------------+------------+--------------------+----------------------+
| id| stack_name| stack_status| creation_time|
+--------------------------------------+------------+--------------------+----------------------+
| 0ca028e7-682b-41ef-8af0-b2eb67bee272| overcloud| CREATE_IN_PROGRESS|
2014-07-18T10:50:48Z |
+--------------------------------------+------------+--------------------+----------------------+
+ tripleo wait_for_stack_ready220 10 overcloud
Command output matched'CREATE_FAILED'. Exiting...
|Now, i understand that the stack isn't being created. So, I tried to check out the
state of the stack:
|[stack@localhost~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id| stack_name| stack_status| creation_time|
+--------------------------------------+------------+---------------+----------------------+
| 0ca028e7-682b-41ef-8af0-b2eb67bee272| overcloud| CREATE_FAILED| 2014-07-18T10:50:48Z
|
+--------------------------------------+------------+---------------+----------------------+
|
i even tried to create stack manually, but ended up getting the same error.
Update: Here is the heat log:
|2014-07-18 06:51:11.884 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:12.921 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:16.058 30750 ERROR heat.engine.resource[-] CREATE: Server
"SwiftStorage0" [07e42c3d-0f1b-4bb9-b980-ffbb74ac770d] Stack
"overcloud" [0ca028e7-682b-41ef-8af0-b2eb67bee272]
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourceTraceback (most recent
calllast):
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourceFile
"/usr/lib/python2.7/site-packages/heat/engine/resource.py", line420, in
_do_action
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourcewhile not
check(handle_data):
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourceFile
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line545,
in check_create_complete
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourcereturn
self._check_active(server)
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourceFile
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line561,
in _check_active
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourceraise exc
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resourceError: Creation of server
overcloud-SwiftStorage0-qdjqbif6peva failed.
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource
2014-07-18 06:51:16.255 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:16.939 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:17.368 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:17.638 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:18.158 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back tousing default
2014-07-18 06:51:18.613 30750 WARNING heat.common.keystoneclient[-] stack_user_domain
IDnot set in heat.conf falling back...
Earlier posted in wrong/different forum.
Ref:
https://ask.openstack.org/en/question/43017/heat-is-not-able-to-create-st...
|
--Pradeep
11:23 a.m.
On Mon, Jul 21, 2014 at 05:59:25PM +0530, Pradeep Kumar Surisetty wrote:
Hi All
I have been trying to set instack with RDO. I have successfully installed
undercloud and moving on to overcloud. Now, when I run
"instack-deploy-overcloud", I get the following error:
+ OVERCLOUD_YAML_PATH=overcloud.yaml
+ heat stack-create -f overcloud.yaml -P
AdminToken=b003d63242f5db3e1ad4864ae66911e02ba19bcb -P
AdminPassword=7bfe4d4a18280752ad07f259a69a3ed00db2ab44 -P
CinderPassword=df0893b4355f3511a6d67538dd592d02d1bc11d3 -P
GlancePassword=066f65f878157b438a916ccbd44e0b7037ee!
118f -P HeatPassword=58fda0e4d6708e0164167b11fe6fca6ab6b35ec6 -P
NeutronPassword=80853ad029feb77bb7c60d035542f21aa5c24177 -P
NovaPassword=331474580be53b78e40c91dfdfc2323578a035e7 -P NeutronPublicInterface=eth0 -P
SwiftPassword=b0eca57b45ebf3dd5cae071dc3880888fb1d4840 -P
SwiftHashSuffix=a8d87f3952d6f91da589fbef801bb92141fd1461 -P NovaComputeLibvirtType=qemu -P
'GlanceLogFile='\'''\''' -P
NeutronDnsmasqOptions=dhcp-option-force=26,1400 overcloud
+--------------------------------------+------------+--------------------+----------------------+
| id | stack_name | stack_status | creation_time
|
+--------------------------------------+------------+--------------------+----------------------+
| 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud | CREATE_IN_PROGRESS |
2014-07-18T10:50:48Z |
+--------------------------------------+------------+--------------------+----------------------+
+ tripleo wait_for_stack_ready 220 10 overcloud
Command output matched 'CREATE_FAILED'. Exiting...
Now, i understand that the stack isn't being created. So, I tried to check out the
state of the stack:
[stack@localhost ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id | stack_name | stack_status | creation_time
|
+--------------------------------------+------------+---------------+----------------------+
| 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud | CREATE_FAILED |
2014-07-18T10:50:48Z |
+--------------------------------------+------------+---------------+----------------------+
i even tried to create stack manually, but ended up getting the same
error.
Update: Here is the heat log:
2014-07-18 06:51:11.884 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:12.921 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:16.058 30750 ERROR heat.engine.resource [-] CREATE : Server
"SwiftStorage0" [07e42c3d-0f1b-4bb9-b980-ffbb74ac770d] Stack
"overcloud" [0ca028e7-682b-41ef-8af0-b2eb67bee272]
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Traceback (most recent call
last):
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 420, in
_do_action
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource while not
check(handle_data):
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line 545, in
check_create_complete
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource return
self._check_active(server)
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line 561, in
_check_active
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource raise exc
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Error: Creation of server
overcloud-SwiftStorage0-qdjqbif6peva failed.
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource
2014-07-18 06:51:16.255 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:16.939 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:17.368 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:17.638 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:18.158 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back to using default
2014-07-18 06:51:18.613 30750 WARNING heat.common.keystoneclient [-] stack_user_domain
ID not set in heat.conf falling back ...
Hi Pradeep,
Can you run a "nova show <instance-id>" on the failed instance? And also
provide any tracebacks or errors from the nova compute log under /var/log/nova?
--
-- James Slagle
--
1:26 a.m.
Hi James,
Here are the details of the failed instance. Interestingly enough, when I
re-ran the
overcloud deployment, this time, swift didn't fail, it was actually a
notcompute instance.
Here are the details:
[stack@localhost ~]$ nova list
+--------------------------------------+--------------------------------------+--------+------------+-------------+---------------------+
| ID | Name
| Status | Task State | Power State | Networks |
+--------------------------------------+--------------------------------------+--------+------------+-------------+---------------------+
| b815d299-4817-4491-b352-d09ab618bd77 |
overcloud-BlockStorage0-zwfiycn67hpc | ACTIVE | - | Running |
ctlplane=192.0.2.15 |
| 9b646a2a-d4b3-438b-94de-4e14bdbe1432 |
overcloud-NovaCompute0-ubp6vlfjjepu | ACTIVE | - | Running |
ctlplane=192.0.2.14 |
| c0ad2f8f-ef91-4a6d-b229-c52b4a89bedd |
overcloud-SwiftStorage0-xns634un3z7k | ACTIVE | - | Running |
ctlplane=192.0.2.16 |
| 11d91a42-4244-48d7-8c4b-92db3a9c43b6 | overcloud-notCompute0-bw4mq7v2sh5y
| ERROR | - | NOSTATE | |
+--------------------------------------+--------------------------------------+--------+------------+-------------+---------------------+
[stack@localhost ~]$ nova show 11d91a42-4244-48d7-8c4b-92db3a9c43b6
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value
|
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone | nova
|
| OS-EXT-SRV-ATTR:host | -
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | -
|
| OS-EXT-SRV-ATTR:instance_name | instance-00000016
|
| OS-EXT-STS:power_state | 0
|
| OS-EXT-STS:task_state | -
|
| OS-EXT-STS:vm_state | error
|
| OS-SRV-USG:launched_at | -
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| config_drive |
|
| created | 2014-07-21T11:18:32Z
|
| fault | {"message": "No valid host was
found. ", "code": 500, "details": " File
\"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\",
line 108, in schedule_run_instance |
| | raise exception.NoValidHost
(reason=\"\")
|
| | ", "created":
"2014-07-21T11:18:32Z"}
|
| flavor | baremetal
(b564fd03-bc8d-42f4-8c5b-264cfa62a655)
|
| hostId |
|
| id |
11d91a42-4244-48d7-8c4b-92db3a9c43b6
|
| image | overcloud-control
(cb938db4-2f1a-44d6-96f9-016c8cc7b406)
|
| key_name | default
|
| metadata | {}
|
| name | overcloud-notCompute0-bw4mq7v2sh5y
|
| os-extended-volumes:volumes_attached | []
|
| status | ERROR
|
| tenant_id | ae8b85d781ad443792f2a3516f38ed88
|
| updated | 2014-07-21T11:18:32Z
|
| user_id | 921abf3732ce40d0b1502e9aa13c6c2a
|
+--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The only relevant log I could found is with nova scheduler:
2014-07-21 04:25:41.473 31078 WARNING nova.scheduler.driver
[req-a00295d2-7308-4bb9-a40d-0740fe852bf5 921abf3732ce40d0b1502e9aa13c6c2a
ae8b85d781ad443792f2a3516f38ed88] [instance:
b3b3f6f9-25fc-45db-b14e-f7daae1f3216] Setting instance to ERROR state.
Though I would like to point out here that I am seeing 5 VMs in my setup
instead of 4.
[stack@devstack ~]$ virsh list --all
Id Name State
----------------------------------------------------
2 instack running
9 baremetal_1 running
10 baremetal_2 running
11 baremetal_3 running
- baremetal_0 shut off
Regards,
Peeyush Gupta
From: James Slagle <jslagle(a)redhat.com
To: Pradeep Kumar Surisetty <psuriset(a)linux.vnet.ibm.com Cc: rdo-list(a)redhat.com, deepthi(a)linux.vnet.ibm.com, Peeyush
Gupta/India/IBM@IBMIN, Pradeep K Surisetty/India/IBM@IBMIN,
anantyog(a)linux.vnet.ibm.com
Date: 07/21/2014 09:50 PM
Subject: Re: [Rdo-list] [RDO][Instack] heat is not able to create stack
with instack
On Mon, Jul 21, 2014 at 05:59:25PM +0530, Pradeep Kumar Surisetty wrote:
+--------------------------------------+------------+--------------------+----------------------+
+--------------------------------------+------------+--------------------+----------------------+
+--------------------------------------+------------+--------------------+----------------------+
> + tripleo wait_for_stack_ready 220 10 overcloud
> Command output matched 'CREATE_FAILED'. Exiting...
> Now, i understand that the stack isn't being created.
So, I tried to
check out the state of the stack:
> [stack@localhost ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
+--------------------------------------+------------+---------------+----------------------+
+--------------------------------------+------------+---------------+----------------------+
> i even tried to create stack
manually, but ended up getting the same
> error.
> Update: Here is the heat log:
> 2014-07-18 06:51:11.884 30750 WARNING
heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
Hi All
I have been trying to set instack with RDO. I have successfully
installed
undercloud and moving on to overcloud. Now, when I run
"instack-deploy-overcloud", I get the following error:
+ OVERCLOUD_YAML_PATH=overcloud.yaml
+ heat stack-create -f overcloud.yaml -P
AdminToken=b003d63242f5db3e1ad4864ae66911e02ba19bcb -P
AdminPassword=7bfe4d4a18280752ad07f259a69a3ed00db2ab44 -P
CinderPassword=df0893b4355f3511a6d67538dd592d02d1bc11d3 -P
GlancePassword=066f65f878157b438a916ccbd44e0b7037ee!
118f -P HeatPassword=58fda0e4d6708e0164167b11fe6fca6ab6b35ec6 -P
NeutronPassword=80853ad029feb77bb7c60d035542f21aa5c24177 -P
NovaPassword=331474580be53b78e40c91dfdfc2323578a035e7 -P
NeutronPublicInterface=eth0 -P
SwiftPassword=b0eca57b45ebf3dd5cae071dc3880888fb1d4840 -P
SwiftHashSuffix=a8d87f3952d6f91da589fbef801bb92141fd1461 -P
NovaComputeLibvirtType=qemu -P 'GlanceLogFile='\'''\'''
-P
NeutronDnsmasqOptions=dhcp-option-force=26,1400 overcloud
| id | stack_name | stack_status
| creation_time |
| 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud |
CREATE_IN_PROGRESS
| 2014-07-18T10:50:48Z |
| id | stack_name | stack_status
|
creation_time |
| 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud | CREATE_FAILED
|
2014-07-18T10:50:48Z |
2014-07-18 06:51:12.921 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
2014-07-18 06:51:16.058 30750 ERROR heat.engine.resource [-] CREATE
:
Server "SwiftStorage0" [07e42c3d-0f1b-4bb9-b980-ffbb74ac770d] Stack
"overcloud" [0ca028e7-682b-41ef-8af0-b2eb67bee272]
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Traceback
(most
recent call last):
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 420, in
_do_action
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource while
not
check(handle_data):
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line
545, in check_create_complete
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource return
self._check_active(server)
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line
561, in _check_active
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource raise
exc
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Error: Creation
of server
overcloud-SwiftStorage0-qdjqbif6peva failed.
2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource
2014-07-18 06:51:16.255 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
2014-07-18 06:51:16.939 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
2014-07-18 06:51:17.368 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
2014-07-18 06:51:17.638 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
2014-07-18 06:51:18.158 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
2014-07-18 06:51:18.613 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back ...
Hi Pradeep,
Can you run a "nova show <instance-id>" on the failed instance? And also
provide any tracebacks or errors from the nova compute log
under /var/log/nova?
--
-- James Slagle
--
3:40 p.m.
On Tue, Jul 22, 2014 at 11:56:28AM +0530, Peeyush Gupta wrote:
| fault | {"message":
"No valid host was
found. ", "code": 500, "details": " File
\"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\",
line 108, in schedule_run_instance |
| | raise exception.NoValidHost
(reason=\"\")
|
2014-07-21 04:25:41.473 31078 WARNING nova.scheduler.driver
[req-a00295d2-7308-4bb9-a40d-0740fe852bf5 921abf3732ce40d0b1502e9aa13c6c2a
ae8b85d781ad443792f2a3516f38ed88] [instance:
b3b3f6f9-25fc-45db-b14e-f7daae1f3216] Setting instance to ERROR state.
Unfortunately these error messages really aren't enough to debug the issue. Can
you set debug=True in /etc/nova/nova.conf aynd restart the following services:
openstack-nova-api
openstack-nova-compute
openstack-nova-scheduler
openstack-nova-conductor
Delete the overcloud with instack-delete-overcloud, and then try a deployment
again. If you get another failure, hopefully there will be more useful data in
the nova logs.
Though I would like to point out here that I am seeing 5 VMs in my setup
instead of 4.
You should have 5 vm's.
[stack@devstack ~]$ virsh list --all
Id Name State
----------------------------------------------------
2 instack running
9 baremetal_1 running
10 baremetal_2 running
11 baremetal_3 running
- baremetal_0 shut off
Regards,
Peeyush Gupta
From: James Slagle <jslagle(a)redhat.com>
To: Pradeep Kumar Surisetty <psuriset(a)linux.vnet.ibm.com>
Cc: rdo-list(a)redhat.com, deepthi(a)linux.vnet.ibm.com, Peeyush
Gupta/India/IBM@IBMIN, Pradeep K Surisetty/India/IBM@IBMIN,
anantyog(a)linux.vnet.ibm.com
Date: 07/21/2014 09:50 PM
Subject: Re: [Rdo-list] [RDO][Instack] heat is not able to create stack
with instack
On Mon, Jul 21, 2014 at 05:59:25PM +0530, Pradeep Kumar Surisetty wrote:
> Hi All
>
> I have been trying to set instack with RDO. I have successfully
installed
> undercloud and moving on to overcloud. Now, when I run
> "instack-deploy-overcloud", I get the following error:
>
> + OVERCLOUD_YAML_PATH=overcloud.yaml
> + heat stack-create -f overcloud.yaml -P
AdminToken=b003d63242f5db3e1ad4864ae66911e02ba19bcb -P
AdminPassword=7bfe4d4a18280752ad07f259a69a3ed00db2ab44 -P
CinderPassword=df0893b4355f3511a6d67538dd592d02d1bc11d3 -P
GlancePassword=066f65f878157b438a916ccbd44e0b7037ee!
> 118f -P HeatPassword=58fda0e4d6708e0164167b11fe6fca6ab6b35ec6 -P
NeutronPassword=80853ad029feb77bb7c60d035542f21aa5c24177 -P
NovaPassword=331474580be53b78e40c91dfdfc2323578a035e7 -P
NeutronPublicInterface=eth0 -P
SwiftPassword=b0eca57b45ebf3dd5cae071dc3880888fb1d4840 -P
SwiftHashSuffix=a8d87f3952d6f91da589fbef801bb92141fd1461 -P
NovaComputeLibvirtType=qemu -P 'GlanceLogFile='\'''\'''
-P
NeutronDnsmasqOptions=dhcp-option-force=26,1400 overcloud
>
+--------------------------------------+------------+--------------------+----------------------+
> | id | stack_name | stack_status
| creation_time |
>
+--------------------------------------+------------+--------------------+----------------------+
> | 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud | CREATE_IN_PROGRESS
| 2014-07-18T10:50:48Z |
>
+--------------------------------------+------------+--------------------+----------------------+
> + tripleo wait_for_stack_ready 220 10 overcloud
> Command output matched 'CREATE_FAILED'. Exiting...
>
> Now, i understand that the stack isn't being created. So, I tried to
check out the state of the stack:
>
> [stack@localhost ~]$ heat stack-list
>
+--------------------------------------+------------+---------------+----------------------+
> | id | stack_name | stack_status |
creation_time |
>
+--------------------------------------+------------+---------------+----------------------+
> | 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud | CREATE_FAILED |
2014-07-18T10:50:48Z |
>
+--------------------------------------+------------+---------------+----------------------+
>
>
> i even tried to create stack manually, but ended up getting the same
> error.
>
> Update: Here is the heat log:
>
> 2014-07-18 06:51:11.884 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:12.921 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:16.058 30750 ERROR heat.engine.resource [-] CREATE :
Server "SwiftStorage0" [07e42c3d-0f1b-4bb9-b980-ffbb74ac770d] Stack
"overcloud" [0ca028e7-682b-41ef-8af0-b2eb67bee272]
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Traceback (most
recent call last):
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 420, in
_do_action
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource while not
check(handle_data):
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line
545, in check_create_complete
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource return
self._check_active(server)
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line
561, in _check_active
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource raise exc
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Error: Creation
of server overcloud-SwiftStorage0-qdjqbif6peva failed.
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource
> 2014-07-18 06:51:16.255 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:16.939 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:17.368 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:17.638 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:18.158 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:18.613 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back ...
Hi Pradeep,
Can you run a "nova show <instance-id>" on the failed instance? And also
provide any tracebacks or errors from the nova compute log
under /var/log/nova?
--
-- James Slagle
--
--
-- James Slagle
--
5:09 a.m.
Hi James,
I reinstalled the undercloud and retraced all the steps with debug=True
in /etc/nova/nova/conf.
Here are the logs:
Nova-compute logs:
2014-07-23 05:58:03.357 2915 WARNING nova.virt.baremetal.driver
[req-648fe5c2-1152-44e6-83f2-87686e7ce87a bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] Destroy called on non-existing instance
dd613a79-a790-434e-a40e-ea787c6d5035
2014-07-23 05:58:03.806 2915 ERROR nova.compute.manager
[req-648fe5c2-1152-44e6-83f2-87686e7ce87a bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] Error: PXE deploy failed for instance
dd613a79-a790-434e-a40e-ea787c6d5035
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] Traceback (most recent call last):
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1305, in
_build_instance
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] set_access_ip=set_access_ip)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in
decorated_function
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] return function(self, context,
*args, **kwargs)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1717, in
_spawn
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] LOG.exception(_('Instance failed
to spawn'), instance=instance)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] six.reraise(self.type_,
self.value, self.tb)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714, in
_spawn
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] block_device_info)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035]
block_device_info=block_device_info)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] _update_state(context, node,
None, baremetal_states.DELETED)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] six.reraise(self.type_,
self.value, self.tb)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 281,
in _spawn
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] self.driver.activate_node
(context, node, instance)
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py", line 500, in
activate_node
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] locals['error'] % instance
['uuid'])
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] InstanceDeployFailure: PXE deploy
failed for instance dd613a79-a790-434e-a40e-ea787c6d5035
2014-07-23 05:58:03.806 2915 TRACE nova.compute.manager [instance:
dd613a79-a790-434e-a40e-ea787c6d5035]
2014-07-23 05:59:27.981 2915 ERROR nova.virt.baremetal.driver
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] Error deploying instance
ba65d12e-b664-4ad9-93fa-992a2886ab01 on baremetal node
68bf3440-7eb6-4b95-b439-74dd8edbe1bf.
2014-07-23 05:59:31.794 2915 ERROR nova.compute.manager
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Instance failed to spawn
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Traceback (most recent call last):
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714, in
_spawn
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] block_device_info)
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01]
block_device_info=block_device_info)
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] _update_state(context, node,
None, baremetal_states.DELETED)
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] six.reraise(self.type_,
self.value, self.tb)
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 281,
in _spawn
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] self.driver.activate_node
(context, node, instance)
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py", line 500, in
activate_node
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] locals['error'] % instance
['uuid'])
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] InstanceDeployFailure: PXE deploy
failed for instance ba65d12e-b664-4ad9-93fa-992a2886ab01
2014-07-23 05:59:31.794 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01]
2014-07-23 05:59:31.870 2915 WARNING nova.virt.baremetal.driver
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] Destroy called on non-existing instance
ba65d12e-b664-4ad9-93fa-992a2886ab01
2014-07-23 05:59:32.239 2915 ERROR nova.compute.manager
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Error: PXE deploy failed for instance
ba65d12e-b664-4ad9-93fa-992a2886ab01
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Traceback (most recent call last):
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1305, in
_build_instance
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] set_access_ip=set_access_ip)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in
decorated_function
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] return function(self, context,
*args, **kwargs)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1717, in
_spawn
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] LOG.exception(_('Instance failed
to spawn'), instance=instance)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] six.reraise(self.type_,
self.value, self.tb)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714, in
_spawn
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] block_device_info)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01]
block_device_info=block_device_info)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] _update_state(context, node,
None, baremetal_states.DELETED)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] six.reraise(self.type_,
self.value, self.tb)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 281,
in _spawn
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] self.driver.activate_node
(context, node, instance)
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py", line 500, in
activate_node
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] locals['error'] % instance
['uuid'])
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] InstanceDeployFailure: PXE deploy
failed for instance ba65d12e-b664-4ad9-93fa-992a2886ab01
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01]
2014-07-23 06:05:43.572 2915 WARNING nova.compute.manager [-] Found 4 in
the database and 0 on the hypervisor.
Nova-api logs:
2014-07-23 05:32:42.670 32685 WARNING keystoneclient.middleware.auth_token
[-] Configuring admin URI using auth fragments. This is deprecated, use
'identity_uri' instead.
2014-07-23 05:32:43.404 32685 WARNING keystoneclient.middleware.auth_token
[-] Configuring admin URI using auth fragments. This is deprecated, use
'identity_uri' instead.
2014-07-23 05:32:44.431 32685 WARNING keystoneclient.middleware.auth_token
[-] Configuring admin URI using auth fragments. This is deprecated, use
'identity_uri' instead.
2014-07-23 05:32:46.417 32685 WARNING keystoneclient.middleware.auth_token
[-] Configuring admin URI using auth fragments. This is deprecated, use
'identity_uri' instead.
2014-07-23 05:45:12.903 2947 ERROR nova.wsgi [-] Could not bind to
0.0.0.0:8773
2014-07-23 05:45:12.903 2947 CRITICAL nova [-] error: [Errno 98] Address
already in use
Nova-scheduler logs:
2014-07-23 05:32:45.277 32697 INFO oslo.messaging._drivers.impl_qpid [-]
Connected to AMQP server on 127.0.0.1:5672
2014-07-23 05:45:17.997 2962 INFO oslo.messaging._drivers.impl_qpid [-]
Connected to AMQP server on 127.0.0.1:5672
2014-07-23 05:51:44.675 2962 INFO oslo.messaging._drivers.impl_qpid [-]
Connected to AMQP server on 127.0.0.1:5672
2014-07-23 05:52:06.321 2962 ERROR nova.scheduler.filter_scheduler
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Error from last host: undercloud
(node dde604c0-d7f9-4212-bb26-c9e166b3df7e): [u'Traceback (most recent call
last):\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1272, in
_build_instance\n with rt.instance_claim(context, instance, limits):\n',
u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line
249, in inner\n return f(*args, **kwargs)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line
122, in instance_claim\n overhead=overhead, limits=limits)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/claims.py", line 95, in
__init__\n self._claim_test(resources, limits)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/claims.py", line 148, in
_claim_test\n "; ".join(reasons))\n', u'ComputeResourcesUnavailable:
Insufficient compute resources: Free memory 0.00 MB < requested 2048
MB.\n']
2014-07-23 05:55:53.648 2962 ERROR nova.scheduler.filter_scheduler
[req-af4756b6-9630-4790-b782-629f3033a055 bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
923d2903-d202-487f-82a9-cd7288e0cfe0] Error from last host: undercloud
(node 72b92fba-fd38-40f6-8b80-7216ed8dc72c): [u'Traceback (most recent call
last):\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1305, in
_build_instance\n set_access_ip=set_access_ip)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in
decorated_function\n return function(self, context, *args, **kwargs)\n',
u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
1717, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
instance=instance)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714,
in _spawn\n block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn\n block_device_info=block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn\n _update_state(context, node, None,
baremetal_states.DELETED)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line
281, in _spawn\n self.driver.activate_node(context, node, instance)\n',
u' File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py",
line 500, in activate_node\n locals[\'error\'] %
instance[\'uuid\'])\n',
u'InstanceDeployFailure: PXE deploy failed for instance
923d2903-d202-487f-82a9-cd7288e0cfe0\n']
2014-07-23 05:55:53.673 2962 WARNING nova.scheduler.driver
[req-af4756b6-9630-4790-b782-629f3033a055 bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
923d2903-d202-487f-82a9-cd7288e0cfe0] Setting instance to ERROR state.
2014-07-23 05:57:01.605 2962 ERROR nova.scheduler.filter_scheduler
[req-3122d59f-d1d2-4b8d-a1d6-59db30d59acd bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
00626822-43b1-46c6-b896-f1b69a738ac7] Error from last host: undercloud
(node 4cb58619-b383-4f97-be81-7dd0bfa51f8e): [u'Traceback (most recent call
last):\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1305, in
_build_instance\n set_access_ip=set_access_ip)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in
decorated_function\n return function(self, context, *args, **kwargs)\n',
u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
1717, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
instance=instance)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714,
in _spawn\n block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn\n block_device_info=block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn\n _update_state(context, node, None,
baremetal_states.DELETED)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line
281, in _spawn\n self.driver.activate_node(context, node, instance)\n',
u' File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py",
line 500, in activate_node\n locals[\'error\'] %
instance[\'uuid\'])\n',
u'InstanceDeployFailure: PXE deploy failed for instance
00626822-43b1-46c6-b896-f1b69a738ac7\n']
2014-07-23 05:57:01.638 2962 WARNING nova.scheduler.driver
[req-3122d59f-d1d2-4b8d-a1d6-59db30d59acd bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
00626822-43b1-46c6-b896-f1b69a738ac7] Setting instance to ERROR state.
2014-07-23 05:58:03.819 2962 ERROR nova.scheduler.filter_scheduler
[req-648fe5c2-1152-44e6-83f2-87686e7ce87a bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] Error from last host: undercloud
(node dde604c0-d7f9-4212-bb26-c9e166b3df7e): [u'Traceback (most recent call
last):\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1305, in
_build_instance\n set_access_ip=set_access_ip)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in
decorated_function\n return function(self, context, *args, **kwargs)\n',
u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
1717, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
instance=instance)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714,
in _spawn\n block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn\n block_device_info=block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn\n _update_state(context, node, None,
baremetal_states.DELETED)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line
281, in _spawn\n self.driver.activate_node(context, node, instance)\n',
u' File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py",
line 500, in activate_node\n locals[\'error\'] %
instance[\'uuid\'])\n',
u'InstanceDeployFailure: PXE deploy failed for instance
dd613a79-a790-434e-a40e-ea787c6d5035\n']
2014-07-23 05:58:03.829 2962 WARNING nova.scheduler.driver
[req-648fe5c2-1152-44e6-83f2-87686e7ce87a bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
dd613a79-a790-434e-a40e-ea787c6d5035] Setting instance to ERROR state.
2014-07-23 05:59:32.248 2962 ERROR nova.scheduler.filter_scheduler
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Error from last host: undercloud
(node 68bf3440-7eb6-4b95-b439-74dd8edbe1bf): [u'Traceback (most recent call
last):\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1305, in
_build_instance\n set_access_ip=set_access_ip)\n', u' File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in
decorated_function\n return function(self, context, *args, **kwargs)\n',
u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
1717, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
instance=instance)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1714,
in _spawn\n block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 245,
in spawn\n block_device_info=block_device_info)\n', u' File
"/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line 303,
in _spawn\n _update_state(context, node, None,
baremetal_states.DELETED)\n', u' File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line
68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', u'
File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/driver.py", line
281, in _spawn\n self.driver.activate_node(context, node, instance)\n',
u' File "/usr/lib/python2.7/site-packages/nova/virt/baremetal/pxe.py",
line 500, in activate_node\n locals[\'error\'] %
instance[\'uuid\'])\n',
u'InstanceDeployFailure: PXE deploy failed for instance
ba65d12e-b664-4ad9-93fa-992a2886ab01\n']
2014-07-23 05:59:32.267 2962 WARNING nova.scheduler.driver
[req-bc984c41-18c0-4d4c-8dd8-2dc6e3b7044b bd2463eed41a47dbaad173888e8e4e81
5db26e088245493b9d07dcd52be97734] [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] Setting instance to ERROR state.
I am not getting the "No valid host" error anymore. It's now PXE deploy
failure.
Regards,
Peeyush Gupta
From: James Slagle <jslagle(a)redhat.com>
To: Peeyush Gupta/India/IBM@IBMIN
Cc: anantyog(a)linux.vnet.ibm.com, deepthi(a)linux.vnet.ibm.com,
Pradeep K Surisetty/India/IBM@IBMIN, Pradeep Kumar Surisetty
<psuriset(a)linux.vnet.ibm.com>, rdo-list(a)redhat.com
Date: 07/23/2014 02:08 AM
Subject: Re: [Rdo-list] [RDO][Instack] heat is not able to create stack
with instack
On Tue, Jul 22, 2014 at 11:56:28AM +0530, Peeyush Gupta wrote:
| fault | {"message":
"No valid host was
found. ", "code": 500, "details": " File
\"/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py\",
line 108, in schedule_run_instance |
| | raise exception.NoValidHost
(reason=\"\")
|
2014-07-21 04:25:41.473 31078 WARNING nova.scheduler.driver
[req-a00295d2-7308-4bb9-a40d-0740fe852bf5
921abf3732ce40d0b1502e9aa13c6c2a
ae8b85d781ad443792f2a3516f38ed88] [instance:
b3b3f6f9-25fc-45db-b14e-f7daae1f3216] Setting instance to ERROR state.
Unfortunately these error messages really aren't enough to debug the issue.
Can
you set debug=True in /etc/nova/nova.conf aynd restart the following
services:
openstack-nova-api
openstack-nova-compute
openstack-nova-scheduler
openstack-nova-conductor
Delete the overcloud with instack-delete-overcloud, and then try a
deployment
again. If you get another failure, hopefully there will be more useful data
in
the nova logs.
Though I would like to point out here that I am seeing 5 VMs in my setup
instead of 4.
You should have 5 vm's.
[stack@devstack ~]$ virsh list --all
Id Name State
----------------------------------------------------
2 instack running
9 baremetal_1 running
10 baremetal_2 running
11 baremetal_3 running
- baremetal_0 shut off
Regards,
Peeyush Gupta
From: James Slagle <jslagle(a)redhat.com>
To: Pradeep Kumar Surisetty <psuriset(a)linux.vnet.ibm.com>
Cc: rdo-list(a)redhat.com, deepthi(a)linux.vnet.ibm.com, Peeyush
Gupta/India/IBM@IBMIN, Pradeep K Surisetty/India/IBM@IBMIN,
anantyog(a)linux.vnet.ibm.com
Date: 07/21/2014 09:50 PM
Subject: Re: [Rdo-list] [RDO][Instack] heat is not able to create
stack
with instack
On Mon, Jul 21, 2014 at 05:59:25PM +0530, Pradeep Kumar Surisetty wrote:
> Hi All
>
> I have been trying to set instack with RDO. I have successfully
installed
> undercloud and moving on to overcloud. Now, when I run
> "instack-deploy-overcloud", I get the following error:
>
> + OVERCLOUD_YAML_PATH=overcloud.yaml
> + heat stack-create -f overcloud.yaml -P
AdminToken=b003d63242f5db3e1ad4864ae66911e02ba19bcb -P
AdminPassword=7bfe4d4a18280752ad07f259a69a3ed00db2ab44 -P
CinderPassword=df0893b4355f3511a6d67538dd592d02d1bc11d3 -P
GlancePassword=066f65f878157b438a916ccbd44e0b7037ee!
> 118f -P HeatPassword=58fda0e4d6708e0164167b11fe6fca6ab6b35ec6 -P
NeutronPassword=80853ad029feb77bb7c60d035542f21aa5c24177 -P
NovaPassword=331474580be53b78e40c91dfdfc2323578a035e7 -P
NeutronPublicInterface=eth0 -P
SwiftPassword=b0eca57b45ebf3dd5cae071dc3880888fb1d4840 -P
SwiftHashSuffix=a8d87f3952d6f91da589fbef801bb92141fd1461 -P
NovaComputeLibvirtType=qemu -P 'GlanceLogFile='\'''\'''
-P
NeutronDnsmasqOptions=dhcp-option-force=26,1400 overcloud
>
+--------------------------------------+------------+--------------------+----------------------+
> | id | stack_name | stack_status
| creation_time |
>
+--------------------------------------+------------+--------------------+----------------------+
> | 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud |
CREATE_IN_PROGRESS
| 2014-07-18T10:50:48Z |
>
+--------------------------------------+------------+--------------------+----------------------+
> + tripleo wait_for_stack_ready 220 10 overcloud
> Command output matched 'CREATE_FAILED'. Exiting...
>
> Now, i understand that the stack isn't being created. So, I tried to
check out the state of the stack:
>
> [stack@localhost ~]$ heat stack-list
>
+--------------------------------------+------------+---------------+----------------------+
> | id | stack_name | stack_status |
creation_time |
>
+--------------------------------------+------------+---------------+----------------------+
> | 0ca028e7-682b-41ef-8af0-b2eb67bee272 | overcloud | CREATE_FAILED |
2014-07-18T10:50:48Z |
>
+--------------------------------------+------------+---------------+----------------------+
>
>
> i even tried to create stack manually, but ended up getting the same
> error.
>
> Update: Here is the heat log:
>
> 2014-07-18 06:51:11.884 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:12.921 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:16.058 30750 ERROR heat.engine.resource [-] CREATE :
Server "SwiftStorage0" [07e42c3d-0f1b-4bb9-b980-ffbb74ac770d] Stack
"overcloud" [0ca028e7-682b-41ef-8af0-b2eb67bee272]
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Traceback
(most
recent call last):
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 420, in
_do_action
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource while not
check(handle_data):
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line
545, in check_create_complete
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource return
self._check_active(server)
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource File
"/usr/lib/python2.7/site-packages/heat/engine/resources/server.py", line
561, in _check_active
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource raise exc
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource Error:
Creation
of server overcloud-SwiftStorage0-qdjqbif6peva failed.
> 2014-07-18 06:51:16.058 30750 TRACE heat.engine.resource
> 2014-07-18 06:51:16.255 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:16.939 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:17.368 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:17.638 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:18.158 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back to using default
> 2014-07-18 06:51:18.613 30750 WARNING heat.common.keystoneclient [-]
stack_user_domain ID not set in heat.conf falling back ...
Hi Pradeep,
Can you run a "nova show <instance-id>" on the failed instance? And also
provide any tracebacks or errors from the nova compute log
under /var/log/nova?
--
-- James Slagle
--
--
-- James Slagle
--
12:15 p.m.
On Wed, Jul 23, 2014 at 03:39:38PM +0530, Peeyush Gupta wrote:
Hi James,
I reinstalled the undercloud and retraced all the steps with debug=True
in /etc/nova/nova/conf.
Here are the logs:
Nova-compute logs:
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01] InstanceDeployFailure: PXE deploy
failed for instance ba65d12e-b664-4ad9-93fa-992a2886ab01
2014-07-23 05:59:32.239 2915 TRACE nova.compute.manager [instance:
ba65d12e-b664-4ad9-93fa-992a2886ab01]
2014-07-23 06:05:43.572 2915 WARNING nova.compute.manager [-] Found 4 in
the database and 0 on the hypervisor.
The WARNING here looks suspicous. Did you use instack-delete-overcloud to
delete a previous overcloud? Before trying another deployment, can you run
instack-delete-overcloud? That also deletes baremetal nodes, which will be
re-created the next time you run instack-deploy-overcloud*.
Nova-api logs:
'identity_uri' instead.
2014-07-23 05:45:12.903 2947 ERROR nova.wsgi [-] Could not bind to
0.0.0.0:8773
2014-07-23 05:45:12.903 2947 CRITICAL nova [-] error: [Errno 98] Address
already in use
For this one, I'd suggest stopping the openstack-nova-api service and verifying
that there are indeed no nova-api processes running. If there are, kill those
off. Then start openstack-nova-api.
Nova-scheduler logs:
_claim_test\n "; ".join(reasons))\n',
u'ComputeResourcesUnavailable:
Insufficient compute resources: Free memory 0.00 MB < requested 2048
MB.\n']
Deleting the previous baremetal nodes by using instack-delete-overcloud should
hopefully clear this error up as well.
--
-- James Slagle
--
3772
days inactive
3776
days old
5 comments
3 participants
participants (3)
-
James Slagle -
Peeyush Gupta -
Pradeep Kumar Surisetty