<div dir="ltr"><div>So gave it a few more hours, on heat resource nothing is failed only create_complete and some init_complete. <br></div><br>Nova show <br>| 61aaed37-4993-4165-93a7-3c9bf6b10a21 | overcloud-controller-0 | ACTIVE | - | Running | ctlplane=192.0.2.8 |<br>| 7f9f4f52-3ee6-42d9-9275-ff88582dd6e7 | overcloud-novacompute-0 | BUILD | spawning | NOSTATE | ctlplane=192.0.2.9 |<br><br><br><div>nova show 7f9f4f52-3ee6-42d9-9275-ff88582dd6e7<br>+--------------------------------------+----------------------------------------------------------+<br>| Property | Value |<br>+--------------------------------------+----------------------------------------------------------+<br>| OS-DCF:diskConfig | MANUAL |<br>| OS-EXT-AZ:availability_zone | nova |<br>| OS-EXT-SRV-ATTR:host | instack.localdomain |<br>| OS-EXT-SRV-ATTR:hypervisor_hostname | 4626bf90-7f95-4bd7-8bee-5f5b0a0981c6 |<br>| OS-EXT-SRV-ATTR:instance_name | instance-00000002 |<br>| OS-EXT-STS:power_state | 0 |<br>| OS-EXT-STS:task_state | spawning |<br>| OS-EXT-STS:vm_state | building |<br><br></div><div>Checking nova log this is what I see: <br></div><br><div>nova-compute.log:{"nodes": [{"target_power_state": null, "links": [{"href": "<a href="http://192.0.2.1:6385/v1/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6">http://192.0.2.1:6385/v1/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6</a>", "rel": "self"}, {"href": "<a href="http://192.0.2.1:6385/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6">http://192.0.2.1:6385/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6</a>", "rel": "bookmark"}], "extra": {}, "last_error": "<b>Failed to change power state to 'power on'. Error: Failed to execute command via SSH</b>: LC_ALL=C /usr/bin/virsh --connect qemu:///system start baremetalbrbm_1.", "updated_at": "2015-10-12T14:36:08+00:00", "maintenance_reason": null, "provision_state": "deploying", "clean_step": {}, "uuid": "4626bf90-7f95-4bd7-8bee-5f5b0a0981c6", "console_enabled": false, "target_provision_state": "active", "provision_updated_at": "2015-10-12T14:35:18+00:00", "power_state": "power off", "inspection_started_at": null, "inspection_finished_at": null, "maintenance": false, "driver": "pxe_ssh", "reservation": null, "properties": {"memory_mb": "4096", "cpu_arch": "x86_64", "local_gb": "40", "cpus": "1", "capabilities": "boot_option:local"}, "instance_uuid": "7f9f4f52-3ee6-42d9-9275-ff88582dd6e7", "name": null, "driver_info": {"ssh_username": "root", "deploy_kernel": "94cc528d-d91f-4ca7-876e-2d8cbec66f1b", "deploy_ramdisk": "057d3b42-002a-4c24-bb3f-2032b8086108", "ssh_key_contents": "-----BEGIN( I removed key..)END RSA PRIVATE KEY-----", "ssh_virt_type": "virsh", "ssh_address": "192.168.122.1"}, "created_at": "2015-10-12T14:26:30+00:00", "ports": [{"href": "<a href="http://192.0.2.1:6385/v1/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6/ports">http://192.0.2.1:6385/v1/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6/ports</a>", "rel": "self"}, {"href": "<a href="http://192.0.2.1:6385/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6/ports">http://192.0.2.1:6385/nodes/4626bf90-7f95-4bd7-8bee-5f5b0a0981c6/ports</a>", "rel": "bookmark"}], "driver_internal_info": {"clean_steps": null, "root_uuid_or_disk_id": "9ff90423-9d18-4dd1-ae96-a4466b52d9d9", "is_whole_disk_image": false}, "instance_info": {"ramdisk": "82639516-289d-4603-bf0e-8131fa75ec46", "kernel": "665ffcb0-2afe-4e04-8910-45b92826e328", "root_gb": "40", "display_name": "overcloud-novacompute-0", "image_source": "d99f460e-c6d9-4803-99e4-51347413f348", "capabilities": "{\"boot_option\": \"local\"}", "memory_mb": "4096", "vcpus": "1", "deploy_key": "BI0FRWDTD4VGHII9JK2BYDDFR8WB1WUG", "local_gb": "40", "configdrive": "H4sICGDEG1YC/3RtcHpwcWlpZQDt3WuT29iZ2HH02Bl7Fe/G5UxSqS3vLtyesaSl2CR4p1zyhk2Ct+ateScdVxcIgiR4A5sAr95xxa/iVOUz7EfJx8m7rXyE5IDslro1mpbGox15Zv6/lrpJ4AAHN/LBwXMIShIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADhJpvx+5UQq5EqNtvzldGs+MIfewJeNv53f/7n354F6xT/3v/TjH0v/chz0L5+8Gv2f3V+n0s+Pz34u/dj982PJfvSTvxFVfXQ7vfyBlRfGvOZo+kQuWWtNVgJn/jO/d6kHzvrGWlHOjGn0TDfmjmXL30kZtZSrlXPFREaVxQM5Hon4fdl0TU7nCmqtU6urRTlZVRP1clV+knwqK/F4UFbPOuVGKZNKFNTbgVFvwO+PyPmzipqo1solX/6slszmCuKozBzKuKPdMlE5ma<br></div><div><br><br></div><div>Any ideas on how to resolve a stuck spawning compute node, it's stuck hasn't changed for a few hours now.<br><br></div><div>Tzach <br></div><div><br></div><div>Tzach<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 12, 2015 at 11:25 PM, Dan Sneddon <span dir="ltr"><<a href="mailto:dsneddon@redhat.com" target="_blank">dsneddon@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 10/12/2015 08:10 AM, Tzach Shefi wrote:<br>
> Hi,<br>
><br>
> Server running centos 7.1, vm running for undercloud got up to<br>
> overcloud deploy stage.<br>
> It looks like its stuck nothing advancing for a while.<br>
> Ideas, what to check?<br>
><br>
> [stack@instack ~]$ openstack overcloud deploy --templates<br>
> Deploying templates in the directory<br>
> /usr/share/openstack-tripleo-heat-templates<br>
> [91665.696658] device vnet2 entered promiscuous mode<br>
> [91665.781346] device vnet3 entered promiscuous mode<br>
> [91675.260324] kvm [71183]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff<br>
> [91675.291232] kvm [71200]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff<br>
> [91767.799404] kvm: zapping shadow pages for mmio generation wraparound<br>
> [91767.880480] kvm: zapping shadow pages for mmio generation wraparound<br>
> [91768.957761] device vnet2 left promiscuous mode<br>
> [91769.799446] device vnet3 left promiscuous mode<br>
> [91771.223273] device vnet3 entered promiscuous mode<br>
> [91771.232996] device vnet2 entered promiscuous mode<br>
> [91773.733967] kvm [72245]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff<br>
> [91801.270510] device vnet2 left promiscuous mode<br>
><br>
><br>
> Thanks<br>
> Tzach<br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Rdo-list mailing list<br>
> <a href="mailto:Rdo-list@redhat.com">Rdo-list@redhat.com</a><br>
> <a href="https://www.redhat.com/mailman/listinfo/rdo-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
><br>
> To unsubscribe: <a href="mailto:rdo-list-unsubscribe@redhat.com">rdo-list-unsubscribe@redhat.com</a><br>
><br>
<br>
You're going to need a more complete command line than "openstack<br>
overcloud deploy --templates". For instance, if you are using VMs for<br>
your overcloud nodes, you will need to include "--libvirt-type qemu".<br>
There are probably a couple of other parameters that you will need.<br>
<br>
You can watch the deployment using this command, which will show you<br>
the progress:<br>
<br>
watch "heat resource-list -n 5 | grep -v COMPLETE"<br>
<br>
You can also explore which resources have failed:<br>
<br>
heat resource-list [-n 5]| grep FAILED<br>
<br>
And then look more closely at the failed resources:<br>
<br>
heat resource-show overcloud <resource><br>
<br>
There are some more complete troubleshooting instructions here:<br>
<br>
<a href="http://docs.openstack.org/developer/tripleo-docs/troubleshooting/troubleshooting-overcloud.html" rel="noreferrer" target="_blank">http://docs.openstack.org/developer/tripleo-docs/troubleshooting/troubleshooting-overcloud.html</a><br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Dan Sneddon | Principal OpenStack Engineer<br>
<a href="mailto:dsneddon@redhat.com">dsneddon@redhat.com</a> | <a href="http://redhat.com/openstack" rel="noreferrer" target="_blank">redhat.com/openstack</a><br>
<a href="tel:650.254.4025" value="+16502544025">650.254.4025</a> | dsneddon:irc @dxs:twitter<br>
<br>
_______________________________________________<br>
Rdo-list mailing list<br>
<a href="mailto:Rdo-list@redhat.com">Rdo-list@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/rdo-list" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
<br>
To unsubscribe: <a href="mailto:rdo-list-unsubscribe@redhat.com">rdo-list-unsubscribe@redhat.com</a><br>
</font></span></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><font size="4"><b><span>Tzach Shefi</span></b></font><br>Quality Engineer, Redhat OSP<br><span><a href="callto:+972-52-4534729" target="_blank">+972-54-4701080</a></span></div></div>
</div>