4:30 a.m.
Hi,
as mentioned in a previous email, I am deploying baremetal nodes using
the quickstart. At the moment I can introspect nodes correctly, but am
unable to deploy to them.
I performed the checks as mentioned in
/tripleo-docs/doc/source/troubleshooting/troubleshooting-overcloud.rst:
The flavor list I have is unchanged:
[stack@undercloud ~]$ openstack flavor list
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk |
Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| 2e72ffb5-c6d7-46fd-ad75-448c0ad6855f | baremetal | 4096 | 40 |
0 | 1 | True |
| 6b8b37e4-618d-4841-b5e3-f556ef27fd4d | oooq_compute | 8192 | 49 |
0 | 1 | True |
| 973b58c3-8730-4b1f-96b2-fda253c15dbc | oooq_control | 8192 | 49 |
0 | 1 | True |
| e22dc516-f53f-4a71-9793-29c614999801 | oooq_ceph | 8192 | 49 |
0 | 1 | True |
| e3dce62a-ac8d-41ba-9f97-84554b247faa | block-storage | 4096 | 40 |
0 | 1 | True |
| f5fe9ba6-cf5c-4ef3-adc2-34f3b4381915 | control | 4096 | 40 |
0 | 1 | True |
| fabf81d8-44cb-4c25-8ed0-2afd124425db | compute | 4096 | 40 |
0 | 1 | True |
| fe512696-2294-40cb-9d20-12415f45c1a9 | ceph-storage | 4096 | 40 |
0 | 1 | True |
| ffc859af-dbfd-4e27-99fb-9ab02f4afa79 | swift-storage | 4096 | 40 |
0 | 1 | True |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
In instackenv.json the nodes have been assigned as:
[stack@undercloud ~]$ cat instackenv.json
{
"nodes":[
{
"_comment": "ooo1",
"pm_type":"pxe_ipmitool",
"mac": [
"00:26:9e:9b:c3:36"
],
"cpu": "16",
"memory": "65536",
"disk": "370",
"arch": "x86_64",
"pm_user":"root",
"pm_password":"admin",
"pm_addr":"10.0.108.126",
"capabilities": "profile:control,boot_option:local"
},
{
"_comment": "ooo2",
"pm_type":"pxe_ipmitool",
"mac": [
"00:26:9e:9c:38:a6"
],
"cpu": "16",
"memory": "65536",
"disk": "370",
"arch": "x86_64",
"pm_user":"root",
"pm_password":"admin",
"pm_addr":"10.0.108.127",
"capabilities": "profile:compute,boot_option:local"
}
]
}
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power
State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| 0956df36-b642-44b8-a67f-0df88270372b | None | None | power
off | manageable | False |
| cc311355-f373-4e5c-99be-31ba3185639d | None | None | power
off | manageable | False |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
And manually I perform the introspection:
[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: 0956df36-b642-44b8-a67f-0df88270372b
Starting introspection of node: cc311355-f373-4e5c-99be-31ba3185639d
Waiting for introspection to finish...
Introspection for UUID 0956df36-b642-44b8-a67f-0df88270372b finished
successfully.
Introspection for UUID cc311355-f373-4e5c-99be-31ba3185639d finished
successfully.
Setting manageable nodes to available...
Node 0956df36-b642-44b8-a67f-0df88270372b has been set to available.
Node cc311355-f373-4e5c-99be-31ba3185639d has been set to available.
Introspection completed.
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power
State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| 0956df36-b642-44b8-a67f-0df88270372b | None | None | power
off | available | False |
| cc311355-f373-4e5c-99be-31ba3185639d | None | None | power
off | available | False |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
After this, I start the deployment. I have defined the compute and
control flavor to be of the respective type.
[stack@undercloud ~]$ ./overcloud-deploy.sh
<snip>
+ openstack overcloud deploy --templates --timeout 60 --control-scale
1 --control-flavor control --compute-scale 1 --compute-flavor compute
--ntp-server pool.ntp.org -e /tmp/deploy_env.yaml
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-06-20 08:18:33 [overcloud]: CREATE_IN_PROGRESS Stack CREATE started
2016-06-20 08:18:33 [HorizonSecret]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [RabbitCookie]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [PcsdPassword]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [MysqlClusterUniquePart]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [MysqlRootPassword]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [Networks]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:34 [VipConfig]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:34 [HeatAuthEncryptionKey]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:34 [overcloud-VipConfig-i4dgmk37z6hg]:
CREATE_IN_PROGRESS Stack CREATE started
2016-06-20 08:18:34 [overcloud-Networks-4pb3htxq7rkd]:
CREATE_IN_PROGRESS Stack CREATE started
<snip>
2016-06-20 08:19:06 [Controller]: CREATE_FAILED ResourceInError:
resources.Controller: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:06 [Controller]: DELETE_IN_PROGRESS state changed
2016-06-20 08:19:06 [NovaCompute]: CREATE_FAILED ResourceInError:
resources.NovaCompute: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:06 [NovaCompute]: DELETE_IN_PROGRESS state changed
2016-06-20 08:19:09 [Controller]: DELETE_COMPLETE state changed
2016-06-20 08:19:09 [NovaCompute]: DELETE_COMPLETE state changed
2016-06-20 08:19:12 [Controller]: CREATE_IN_PROGRESS state changed
2016-06-20 08:19:12 [NovaCompute]: CREATE_IN_PROGRESS state changed
2016-06-20 08:19:14 [Controller]: CREATE_FAILED ResourceInError:
resources.Controller: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:14 [Controller]: DELETE_IN_PROGRESS state changed
2016-06-20 08:19:14 [NovaCompute]: CREATE_FAILED ResourceInError:
resources.NovaCompute: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:14 [NovaCompute]: DELETE_IN_PROGRESS state changed
But as you can see, the deployment fails.
I check the introspection information and verify that the disk, local
memory and cpus are matching or exceeding the flavor:
[stack@undercloud ~]$ ironic node-show 0956df36-b642-44b8-a67f-0df88270372b
+------------------------+-------------------------------------------------------------------------+
| Property | Value
|
+------------------------+-------------------------------------------------------------------------+
| chassis_uuid |
|
| clean_step | {}
|
| console_enabled | False
|
| created_at | 2016-06-20T05:51:17+00:00
|
| driver | pxe_ipmitool
|
| driver_info | {u'ipmi_password': u'******',
u'ipmi_address': u'10.0.108.126', |
| | u'ipmi_username': u'root',
u'deploy_kernel': |
| | u'07c794a6-b427-4e75-ba58-7c555abbf2f8',
u'deploy_ramdisk': u'67a66b7b- |
| | 637f-4b25-bcef-ed39ae32a1f4'}
|
| driver_internal_info | {}
|
| extra | {u'hardware_swift_object':
u'extra_hardware-0956df36-b642-44b8-a67f- |
| | 0df88270372b'}
|
| inspection_finished_at | None
|
| inspection_started_at | None
|
| instance_info | {}
|
| instance_uuid | None
|
| last_error | None
|
| maintenance | False
|
| maintenance_reason | None
|
| name | None
|
| power_state | power off
|
| properties | {u'memory_mb': u'65536', u'cpu_arch':
u'x86_64', u'local_gb': u'371', |
| | u'cpus': u'16', u'capabilities':
u'profile:control,boot_option:local'} |
| provision_state | available
|
| provision_updated_at | 2016-06-20T07:32:46+00:00
|
| raid_config |
|
| reservation | None
|
| target_power_state | None
|
| target_provision_state | None
|
| target_raid_config |
|
| updated_at | 2016-06-20T07:32:46+00:00
|
| uuid | 0956df36-b642-44b8-a67f-0df88270372b
|
+------------------------+-------------------------------------------------------------------------+
And also the hypervisor stats are set, but only matching the node count.
[stack@undercloud ~]$ nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 0 |
| free_disk_gb | 0 |
| free_ram_mb | 0 |
| local_gb | 0 |
| local_gb_used | 0 |
| memory_mb | 0 |
| memory_mb_used | 0 |
| running_vms | 0 |
| vcpus | 0 |
| vcpus_used | 0 |
+----------------------+-------+
Registering the nodes as profile:baremetal has the same effect.
What other parameters are used in making the decision if a node can be
deployed to? I probably miss a small detail... what can I check to
make sure the deployment starts?
regards,
Gerard
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
9:34 a.m.
Unfortunately, "No valid host" is the most generic error in OpenStack.
Maybe someday Nova will provide a better error message for that error,
but in the meantime we need to check the scheduler logs
(/var/log/nova/nova-scheduler.log) for more clues.
I usually grep for 'returned 0 host' to find which filter is failing to
match any hosts. That narrows the search space to investigate further
why the filter fails.
On 06/20/2016 05:30 AM, Gerard Braad wrote:
Hi,
as mentioned in a previous email, I am deploying baremetal nodes using
the quickstart. At the moment I can introspect nodes correctly, but am
unable to deploy to them.
I performed the checks as mentioned in
/tripleo-docs/doc/source/troubleshooting/troubleshooting-overcloud.rst:
The flavor list I have is unchanged:
[stack@undercloud ~]$ openstack flavor list
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk |
Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
| 2e72ffb5-c6d7-46fd-ad75-448c0ad6855f | baremetal | 4096 | 40 |
0 | 1 | True |
| 6b8b37e4-618d-4841-b5e3-f556ef27fd4d | oooq_compute | 8192 | 49 |
0 | 1 | True |
| 973b58c3-8730-4b1f-96b2-fda253c15dbc | oooq_control | 8192 | 49 |
0 | 1 | True |
| e22dc516-f53f-4a71-9793-29c614999801 | oooq_ceph | 8192 | 49 |
0 | 1 | True |
| e3dce62a-ac8d-41ba-9f97-84554b247faa | block-storage | 4096 | 40 |
0 | 1 | True |
| f5fe9ba6-cf5c-4ef3-adc2-34f3b4381915 | control | 4096 | 40 |
0 | 1 | True |
| fabf81d8-44cb-4c25-8ed0-2afd124425db | compute | 4096 | 40 |
0 | 1 | True |
| fe512696-2294-40cb-9d20-12415f45c1a9 | ceph-storage | 4096 | 40 |
0 | 1 | True |
| ffc859af-dbfd-4e27-99fb-9ab02f4afa79 | swift-storage | 4096 | 40 |
0 | 1 | True |
+--------------------------------------+---------------+------+------+-----------+-------+-----------+
In instackenv.json the nodes have been assigned as:
[stack@undercloud ~]$ cat instackenv.json
{
"nodes":[
{
"_comment": "ooo1",
"pm_type":"pxe_ipmitool",
"mac": [
"00:26:9e:9b:c3:36"
],
"cpu": "16",
"memory": "65536",
"disk": "370",
"arch": "x86_64",
"pm_user":"root",
"pm_password":"admin",
"pm_addr":"10.0.108.126",
"capabilities": "profile:control,boot_option:local"
},
{
"_comment": "ooo2",
"pm_type":"pxe_ipmitool",
"mac": [
"00:26:9e:9c:38:a6"
],
"cpu": "16",
"memory": "65536",
"disk": "370",
"arch": "x86_64",
"pm_user":"root",
"pm_password":"admin",
"pm_addr":"10.0.108.127",
"capabilities": "profile:compute,boot_option:local"
}
]
}
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power
State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| 0956df36-b642-44b8-a67f-0df88270372b | None | None | power
off | manageable | False |
| cc311355-f373-4e5c-99be-31ba3185639d | None | None | power
off | manageable | False |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
And manually I perform the introspection:
[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: 0956df36-b642-44b8-a67f-0df88270372b
Starting introspection of node: cc311355-f373-4e5c-99be-31ba3185639d
Waiting for introspection to finish...
Introspection for UUID 0956df36-b642-44b8-a67f-0df88270372b finished
successfully.
Introspection for UUID cc311355-f373-4e5c-99be-31ba3185639d finished
successfully.
Setting manageable nodes to available...
Node 0956df36-b642-44b8-a67f-0df88270372b has been set to available.
Node cc311355-f373-4e5c-99be-31ba3185639d has been set to available.
Introspection completed.
[stack@undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power
State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| 0956df36-b642-44b8-a67f-0df88270372b | None | None | power
off | available | False |
| cc311355-f373-4e5c-99be-31ba3185639d | None | None | power
off | available | False |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
After this, I start the deployment. I have defined the compute and
control flavor to be of the respective type.
[stack@undercloud ~]$ ./overcloud-deploy.sh
<snip>
+ openstack overcloud deploy --templates --timeout 60 --control-scale
1 --control-flavor control --compute-scale 1 --compute-flavor compute
--ntp-server pool.ntp.org -e /tmp/deploy_env.yaml
Deploying templates in the directory /usr/share/openstack-tripleo-heat-templates
2016-06-20 08:18:33 [overcloud]: CREATE_IN_PROGRESS Stack CREATE started
2016-06-20 08:18:33 [HorizonSecret]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [RabbitCookie]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [PcsdPassword]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [MysqlClusterUniquePart]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [MysqlRootPassword]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:33 [Networks]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:34 [VipConfig]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:34 [HeatAuthEncryptionKey]: CREATE_IN_PROGRESS state changed
2016-06-20 08:18:34 [overcloud-VipConfig-i4dgmk37z6hg]:
CREATE_IN_PROGRESS Stack CREATE started
2016-06-20 08:18:34 [overcloud-Networks-4pb3htxq7rkd]:
CREATE_IN_PROGRESS Stack CREATE started
<snip>
2016-06-20 08:19:06 [Controller]: CREATE_FAILED ResourceInError:
resources.Controller: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:06 [Controller]: DELETE_IN_PROGRESS state changed
2016-06-20 08:19:06 [NovaCompute]: CREATE_FAILED ResourceInError:
resources.NovaCompute: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:06 [NovaCompute]: DELETE_IN_PROGRESS state changed
2016-06-20 08:19:09 [Controller]: DELETE_COMPLETE state changed
2016-06-20 08:19:09 [NovaCompute]: DELETE_COMPLETE state changed
2016-06-20 08:19:12 [Controller]: CREATE_IN_PROGRESS state changed
2016-06-20 08:19:12 [NovaCompute]: CREATE_IN_PROGRESS state changed
2016-06-20 08:19:14 [Controller]: CREATE_FAILED ResourceInError:
resources.Controller: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:14 [Controller]: DELETE_IN_PROGRESS state changed
2016-06-20 08:19:14 [NovaCompute]: CREATE_FAILED ResourceInError:
resources.NovaCompute: Went to status ERROR due to "Message: No valid
host was found. There are not enough hosts available., Code: 500"
2016-06-20 08:19:14 [NovaCompute]: DELETE_IN_PROGRESS state changed
But as you can see, the deployment fails.
I check the introspection information and verify that the disk, local
memory and cpus are matching or exceeding the flavor:
[stack@undercloud ~]$ ironic node-show 0956df36-b642-44b8-a67f-0df88270372b
+------------------------+-------------------------------------------------------------------------+
| Property | Value
|
+------------------------+-------------------------------------------------------------------------+
| chassis_uuid |
|
| clean_step | {}
|
| console_enabled | False
|
| created_at | 2016-06-20T05:51:17+00:00
|
| driver | pxe_ipmitool
|
| driver_info | {u'ipmi_password': u'******',
u'ipmi_address': u'10.0.108.126', |
| | u'ipmi_username': u'root',
u'deploy_kernel': |
| | u'07c794a6-b427-4e75-ba58-7c555abbf2f8',
u'deploy_ramdisk': u'67a66b7b- |
| | 637f-4b25-bcef-ed39ae32a1f4'}
|
| driver_internal_info | {}
|
| extra | {u'hardware_swift_object':
u'extra_hardware-0956df36-b642-44b8-a67f- |
| | 0df88270372b'}
|
| inspection_finished_at | None
|
| inspection_started_at | None
|
| instance_info | {}
|
| instance_uuid | None
|
| last_error | None
|
| maintenance | False
|
| maintenance_reason | None
|
| name | None
|
| power_state | power off
|
| properties | {u'memory_mb': u'65536', u'cpu_arch':
u'x86_64', u'local_gb': u'371', |
| | u'cpus': u'16', u'capabilities':
u'profile:control,boot_option:local'} |
| provision_state | available
|
| provision_updated_at | 2016-06-20T07:32:46+00:00
|
| raid_config |
|
| reservation | None
|
| target_power_state | None
|
| target_provision_state | None
|
| target_raid_config |
|
| updated_at | 2016-06-20T07:32:46+00:00
|
| uuid | 0956df36-b642-44b8-a67f-0df88270372b
|
+------------------------+-------------------------------------------------------------------------+
And also the hypervisor stats are set, but only matching the node count.
[stack@undercloud ~]$ nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 0 |
| free_disk_gb | 0 |
| free_ram_mb | 0 |
| local_gb | 0 |
| local_gb_used | 0 |
| memory_mb | 0 |
| memory_mb_used | 0 |
| running_vms | 0 |
| vcpus | 0 |
| vcpus_used | 0 |
+----------------------+-------+
Registering the nodes as profile:baremetal has the same effect.
What other parameters are used in making the decision if a node can be
deployed to? I probably miss a small detail... what can I check to
make sure the deployment starts?
regards,
Gerard
9:52 a.m.
On Mon, Jun 20, 2016 at 10:34 PM, John Trowbridge <trown(a)redhat.com> wrote:
I usually grep for 'returned 0 host' to find which filter is
failing
Filter RamFilter returned 0 hosts
regards,
Gerard
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
10:46 a.m.
Finally was able to get some time to look more into the log files
On Mon, Jun 20, 2016 at 10:52 PM, Gerard Braad <me(a)gbraad.nl> wrote:
On Mon, Jun 20, 2016 at 10:34 PM, John Trowbridge
<trown(a)redhat.com> wrote:
> I usually grep for 'returned 0 host' to find which filter is failing
after the Compute capability filter:
2016-06-20 09:15:18.275 1186 DEBUG nova.filters
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] Filter ComputeCapabilitiesFilter returned 1 host(s)
get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
It seems a node that claims to have less than 4G of memory. The node
actually contains 64G of memory and this is also reported by the
introspection (node-show)
2016-06-20 09:15:18.276 1186 DEBUG nova.scheduler.filters.ram_filter
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] (undercloud, 0956df36-b642-44b8-a67f-0df88270372b) ram: 0MB disk:
0MB io_ops: 0 instances: 0 does not have 4096 MB usable ram before
overcommit, it only has 0 MB. host_passes
/usr/lib/python2.7/site-packages/nova/scheduler/filters/ram_filter.py:45
2016-06-20 09:15:18.276 1186 INFO nova.filters
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] Filter RamFilter returned 0 hosts
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
11:03 a.m.
On 06/20/2016 11:46 AM, Gerard Braad wrote:
Finally was able to get some time to look more into the log files
On Mon, Jun 20, 2016 at 10:52 PM, Gerard Braad <me(a)gbraad.nl> wrote:
> On Mon, Jun 20, 2016 at 10:34 PM, John Trowbridge <trown(a)redhat.com> wrote:
>> I usually grep for 'returned 0 host' to find which filter is failing
after the Compute capability filter:
2016-06-20 09:15:18.275 1186 DEBUG nova.filters
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] Filter ComputeCapabilitiesFilter returned 1 host(s)
get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
It seems a node that claims to have less than 4G of memory. The node
actually contains 64G of memory and this is also reported by the
introspection (node-show)
2016-06-20 09:15:18.276 1186 DEBUG nova.scheduler.filters.ram_filter
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] (undercloud, 0956df36-b642-44b8-a67f-0df88270372b) ram: 0MB disk:
0MB io_ops: 0 instances: 0 does not have 4096 MB usable ram before
overcommit, it only has 0 MB. host_passes
/usr/lib/python2.7/site-packages/nova/scheduler/filters/ram_filter.py:45
2016-06-20 09:15:18.276 1186 INFO nova.filters
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] Filter RamFilter returned 0 hosts
What does `nova flavor-list` return?
I've seen this error and gotten past it by making sure my flavors have
only 4G of RAM instead of trying to make them represent the hardware.
John
11:13 a.m.
Hi John,
I do not use the oooq_compute (or oooq_control) flavour, but
specifically use the compute (and control).
On Tue, Jun 21, 2016 at 12:03 AM, John Fulton <johfulto(a)redhat.com> wrote:
What does `nova flavor-list` return?
[stack@undercloud ~]$ nova flavor-list
+--------------------------------------+---------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID | Name | Memory_MB |
Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+--------------------------------------+---------------+-----------+------+-----------+------+-------+-------------+-----------+
| 2e72ffb5-c6d7-46fd-ad75-448c0ad6855f | baremetal | 4096 |
40 | 0 | | 1 | 1.0 | True |
| 6b8b37e4-618d-4841-b5e3-f556ef27fd4d | oooq_compute | 8192 |
49 | 0 | | 1 | 1.0 | True |
| 973b58c3-8730-4b1f-96b2-fda253c15dbc | oooq_control | 8192 |
49 | 0 | | 1 | 1.0 | True |
| e22dc516-f53f-4a71-9793-29c614999801 | oooq_ceph | 8192 |
49 | 0 | | 1 | 1.0 | True |
| e3dce62a-ac8d-41ba-9f97-84554b247faa | block-storage | 4096 |
40 | 0 | | 1 | 1.0 | True |
| f5fe9ba6-cf5c-4ef3-adc2-34f3b4381915 | control | 4096 |
40 | 0 | | 1 | 1.0 | True |
| fabf81d8-44cb-4c25-8ed0-2afd124425db | compute | 4096 |
40 | 0 | | 1 | 1.0 | True |
| fe512696-2294-40cb-9d20-12415f45c1a9 | ceph-storage | 4096 |
40 | 0 | | 1 | 1.0 | True |
| ffc859af-dbfd-4e27-99fb-9ab02f4afa79 | swift-storage | 4096 |
40 | 0 | | 1 | 1.0 | True |
+--------------------------------------+---------------+-----------+------+-----------+------+-------+-------------+-----------+
There are both at: 4G / 4096
regards,
Gerard
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
1:55 p.m.
From: "Gerard Braad" <me(a)gbraad.nl>
To: "John Trowbridge" <trown(a)redhat.com>
Cc: "rdo-list" <rdo-list(a)redhat.com>
Sent: Monday, June 20, 2016 11:46:38 AM
Subject: Re: [rdo-list] [tripleo] [oooq] Deployment to baremetal fails; "No valid
host was found"
Finally was able to get some time to look more into the log files
On Mon, Jun 20, 2016 at 10:52 PM, Gerard Braad <me(a)gbraad.nl> wrote:
> On Mon, Jun 20, 2016 at 10:34 PM, John Trowbridge <trown(a)redhat.com> wrote:
>> I usually grep for 'returned 0 host' to find which filter is failing
after the Compute capability filter:
2016-06-20 09:15:18.275 1186 DEBUG nova.filters
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] Filter ComputeCapabilitiesFilter returned 1 host(s)
get_filtered_objects
/usr/lib/python2.7/site-packages/nova/filters.py:104
It seems a node that claims to have less than 4G of memory. The node
actually contains 64G of memory and this is also reported by the
introspection (node-show)
2016-06-20 09:15:18.276 1186 DEBUG nova.scheduler.filters.ram_filter
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] (undercloud, 0956df36-b642-44b8-a67f-0df88270372b) ram: 0MB disk:
0MB io_ops: 0 instances: 0 does not have 4096 MB usable ram before
overcommit, it only has 0 MB. host_passes
/usr/lib/python2.7/site-packages/nova/scheduler/filters/ram_filter.py:45
2016-06-20 09:15:18.276 1186 INFO nova.filters
[req-665aaa1f-7788-4aea-85cd-c08097d934c5
0b48c6614e73469eb2350fe09c33ee10 2fea396ebcab48ed8c150ef31506f66f - -
-] Filter RamFilter returned 0 hosts
If you are deploying with Mitaka, please check you have the fix linked here:
https://bugs.launchpad.net/tripleo/+bug/1567395
(https://github.com/rdo-packages/ironic-python-agent-distgit/commit/198e5b...).
Other possible 'No Host found' culprits:
If you are creating your own flavor, ensure that the correct properties are set:
example:
openstack flavor set --property "cpu_arch"="x86_64" --property
"capabilities:boot_option"="local" <flavor>;
The disk size reported after introspection looks fine - so you probably don't require
disk hints but it's something to keep in mind:
https://bugzilla.redhat.com/show_bug.cgi?id=1288192
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
_______________________________________________
rdo-list mailing list
rdo-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/rdo-list
To unsubscribe: rdo-list-unsubscribe(a)redhat.com
8:35 p.m.
On Tue, Jun 21, 2016 at 2:55 AM, Ronelle Landy <rlandy(a)redhat.com> wrote:
If you are deploying with Mitaka
Yes, I am deploying Mitaka...
However, I have also tried to use Liberty, but this results in a
failure of the introspection:
[stack@undercloud ~]$ openstack baremetal introspection bulk start
Setting nodes for introspection to manageable...
Starting introspection of node: d9685415-328f-4f37-aabe-f4edfb0eebaf
Starting introspection of node: 702b4804-a9d9-4d3b-895c-019377501022
Waiting for introspection to finish...
Introspection for UUID d9685415-328f-4f37-aabe-f4edfb0eebaf finished
with error: The following required parameters are missing:
['local_gb']
Introspection for UUID 702b4804-a9d9-4d3b-895c-019377501022 finished
with error: The following required parameters are missing:
['local_gb']
Setting manageable nodes to available...
Introspection completed with errors:
d9685415-328f-4f37-aabe-f4edfb0eebaf: The following required
parameters are missing: ['local_gb']
702b4804-a9d9-4d3b-895c-019377501022: The following required
parameters are missing: ['local_gb']
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
12:33 a.m.
On Tue, Jun 21, 2016 at 9:35 AM, Gerard Braad <me(a)gbraad.nl> wrote:
On Tue, Jun 21, 2016 at 2:55 AM, Ronelle Landy
<rlandy(a)redhat.com> wrote:
> If you are deploying with Mitaka
Yes, I am deploying Mitaka...
Currently deploying from a fresh installation of Mitaka quickstart...
I have not changed my setup (using the same deployment scripts and parameters)
2016-06-21 02:39:18 [Controller]: CREATE_COMPLETE state changed
2016-06-21 02:39:19 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:19 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:20 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:21 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:21 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:23 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:23 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:25 [ManagementPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:26 [ExternalPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:26 [TenantPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:26 [StoragePort]: CREATE_COMPLETE state changed
2016-06-21 02:39:27 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:27 [InternalApiPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:27 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:28 [NetIpSubnetMap]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:29 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:31 [NetworkConfig]: CREATE_COMPLETE state changed
2016-06-21 02:39:31 [NetIpMap]: CREATE_COMPLETE state changed
2016-06-21 02:39:32 [NetIpSubnetMap]: CREATE_COMPLETE state changed
2016-06-21 02:39:32 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
And this seems to pass the Nova scheduler filter check... and deploys.
But now the undercloud lost connectivity again. (and deployment hangs)
[gerard@server-124 ~]$ !407
ssh -F /home/gerard/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added 'server-124.local,10.0.106.124' (ECDSA) to
the list of known hosts.
channel 0: open failed: connect failed: No route to host
ssh_exchange_identification: Connection closed by remote host
This disconnection is occurring on different machines, all with base
7.2 and the quickstart.
Seems I have to shave this yak...
regards,
Gerard
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
9:56 a.m.
From: "Gerard Braad" <me(a)gbraad.nl>
To: "Ronelle Landy" <rlandy(a)redhat.com>
Cc: "John Trowbridge" <trown(a)redhat.com>, "rdo-list"
<rdo-list(a)redhat.com>
Sent: Tuesday, June 21, 2016 1:33:09 AM
Subject: Re: [rdo-list] [tripleo] [oooq] Deployment to baremetal fails; "No valid
host was found"
On Tue, Jun 21, 2016 at 9:35 AM, Gerard Braad <me(a)gbraad.nl> wrote:
> On Tue, Jun 21, 2016 at 2:55 AM, Ronelle Landy <rlandy(a)redhat.com> wrote:
>> If you are deploying with Mitaka
> Yes, I am deploying Mitaka...
Currently deploying from a fresh installation of Mitaka quickstart...
I have not changed my setup (using the same deployment scripts and
parameters)
2016-06-21 02:39:18 [Controller]: CREATE_COMPLETE state changed
2016-06-21 02:39:19 [ExternalPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:19 [TenantPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:20 [StoragePort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:21 [ManagementPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:21 [UpdateDeployment]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:23 [StorageMgmtPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:23 [InternalApiPort]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:25 [ManagementPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:26 [ExternalPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:26 [TenantPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:26 [StoragePort]: CREATE_COMPLETE state changed
2016-06-21 02:39:27 [StorageMgmtPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:27 [InternalApiPort]: CREATE_COMPLETE state changed
2016-06-21 02:39:27 [NetworkConfig]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:28 [NetIpSubnetMap]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:29 [NetIpMap]: CREATE_IN_PROGRESS state changed
2016-06-21 02:39:31 [NetworkConfig]: CREATE_COMPLETE state changed
2016-06-21 02:39:31 [NetIpMap]: CREATE_COMPLETE state changed
2016-06-21 02:39:32 [NetIpSubnetMap]: CREATE_COMPLETE state changed
2016-06-21 02:39:32 [NetworkDeployment]: CREATE_IN_PROGRESS state changed
And this seems to pass the Nova scheduler filter check... and deploys.
Ok - so your previous issue of 'no hosts' seems to be resolved with latest
Mitaka.
But now the undercloud lost connectivity again. (and deployment hangs)
[gerard@server-124 ~]$ !407
ssh -F /home/gerard/.quickstart/ssh.config.ansible undercloud
Warning: Permanently added 'server-124.local,10.0.106.124' (ECDSA) to
the list of known hosts.
channel 0: open failed: connect failed: No route to host
ssh_exchange_identification: Connection closed by remote host
This disconnection is occurring on different machines, all with base
7.2 and the quickstart.
I have seen a similar loss of connection working on Newton during deploy/introspection:
fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg":
"Failed to connect to the host via ssh.", "unreachable": true}
It is possibly an environment configuration issue - I am looking into it more.
Seems I have to shave this yak...
regards,
Gerard
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
7:51 p.m.
Hi,
Had several successful and unsuccessful (due to unreachable
undercloud) deployments. From both I learned a lot, especially related
to the Ironic association and provisioning states.
On Tue, Jun 21, 2016 at 10:56 PM, Ronelle Landy <rlandy(a)redhat.com> wrote:
> From: "Gerard Braad" <me(a)gbraad.nl>
> Currently deploying from a fresh installation of Mitaka quickstart...
> I have not changed my setup (using the same deployment scripts and
> parameters)
Ok - so your previous issue of 'no hosts' seems to be resolved with latest
Mitaka.
It seems. There is however a consistent issue remaining. I need to do a
systemctl restart openstack-ironic-conductor
after each reboot of the undercloud. The service refuses to start correctly:
[stack@undercloud ~]$ openstack baremetal introspection bulk start
No valid host was found. Reason: No conductor service registered which
supports driver pxe_ipmitool. (HTTP 400)
[stack@undercloud ~]$ sudo systemctl restart openstack-ironic-conductor
[stack@undercloud ~]$ openstack baremetal introspection bulk start
I have seen a similar loss of connection working on Newton during
deploy/introspection:
fatal: [undercloud]: UNREACHABLE! => {"changed": false, "msg":
"Failed to connect to the host via ssh.", "unreachable": true}
It is possibly an environment configuration issue - I am looking into it more.
I have seen this behaviour with liberty and mitaka, and more
consistent during deployment. When I did virtual deployments it wasn't
as obvious, but now during the physical deployments it starts biting.
Any suggestions what I should look at?
regards,
Gerard
--
Gerard Braad | http://gbraad.nl
[ Doing Open Source Matters ]
3105
days inactive
3107
days old
10 comments
4 participants
participants (4)
-
Gerard Braad -
John Fulton -
John Trowbridge -
Ronelle Landy