<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><div><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Well in the early stage of the introspection I can see Client IP of nodes (screenshot attached). But then I see continuous ironic-python-agent errors (screenshot-2 attached). Errors repeat after time out.. And the nodes are not powered off.</span></div><div><br></div><div><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Seems like I am stuck in introspection stage.. </span></div><div></div><div><br></div><div><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">I can use ipmitool command to successfully power on/off the nodes</span></div><div><br></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-size: small;">[stack@undercloud ~]$ ipmitool -I lanplus -H 192.168.0.19 -L ADMINISTRATOR -U root -R 3 -N 5 -P<span style="font-size: small;" data-mce-style="font-size: small;"> <password></span> power status</span><br><span style="font-size: small;">Chassis Power is <span style="font-size: small;" data-mce-style="font-size: small;">on</span></span></p></div><div><p style="margin: 0px;" data-mce-style="margin: 0px;"><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">[stack@undercloud ~]$ ipmitool -H 192.168.0.18 -I lanplus -U root -P <password> chassis power status</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Chassis Power is on</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">[stack@undercloud ~]$ ipmitool -H 192.168.0.18 -I lanplus -U root -P <password> chassis power off</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Chassis Power Control: Down/Off</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">[stack@undercloud ~]$ ipmitool -H 192.168.0.18 -I lanplus -U root -P <password> chassis power status</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Chassis Power is off</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">[stack@undercloud ~]$ ipmitool -H 192.168.0.18 -I lanplus -U root -P <span style="font-size: small;" data-mce-style="font-size: small;"><password></span> chassis power on</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Chassis Power Control: Up/On</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">[stack@undercloud ~]$ ipmitool -H 192.168.0.18 -I lanplus -U root -P <span style="font-size: small;" data-mce-style="font-size: small;"><password></span> chassis power status</span><br><span style="font-size: small; font-family: 'times new roman', 'new york', times, serif;">Chassis Power is on</span></p></div><div><br></div><div><br></div><div><span name="x"></span><div></div>
<div style="font-family: Tahoma,Geneva,sans-serif; color: #cc0000; font-size: 9px;"><strong>Esra ÇELİK</strong></div><div style="font-family: Tahoma,Geneva,sans-serif; color: #404040; font-size: 9px;">
</div>
<div style="font-family: Tahoma,Geneva,sans-serif; color: #404040; font-size: 9px;">TÜBİTAK BİLGEM</div>
<div style="margin-top: 4px; font-family: Tahoma,Geneva,sans-serif; color: #404040; font-size: 9px;"><span class="Object" id="OBJ_PREFIX_DWT517_com_zimbra_url"><a href="http://www.bilgem.tubitak.gov.tr">www.bilgem.tubitak.gov.tr</a></span></div>
<div style="margin-top: 4px; font-family: Tahoma,Geneva,sans-serif; color: #404040; font-size: 9px;"><span class="Object"><a href="mailto:celik.esra@tubitak.gov.tr">celik.esra@tubitak.gov.tr</a></span></div><div style="margin-top: 4px; font-family: Tahoma,Geneva,sans-serif; color: #404040; font-size: 9px;"><span class="Object"><br></span></div><div style="margin-top: 4px; font-family: Tahoma,Geneva,sans-serif; color: #404040; font-size: 9px;"><span class="Object"><br></span></div></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>Kimden: </b>"Marius Cornea" <mcornea@redhat.com><br><b>Kime: </b>"Esra Celik" <celik.esra@tubitak.gov.tr><br><b>Kk: </b>"Ignacio Bravo" <ibravo@ltgfederal.com>, rdo-list@redhat.com<br><b>Gönderilenler: </b>14 Ekim Çarşamba 2015 14:59:30<br><b>Konu: </b>Re: [Rdo-list] OverCloud deploy fails with error "No valid host was found"<br><div><br></div><br>----- Original Message -----<br>> From: "Esra Celik" <celik.esra@tubitak.gov.tr><br>> To: "Marius Cornea" <mcornea@redhat.com><br>> Cc: "Ignacio Bravo" <ibravo@ltgfederal.com>, rdo-list@redhat.com<br>> Sent: Wednesday, October 14, 2015 10:49:01 AM<br>> Subject: Re: [Rdo-list] OverCloud deploy fails with error "No valid host was found"<br>> <br>> <br>> Well today I started with re-installing the OS and nothing seems wrong with<br>> undercloud installation, then;<br>> <br>> <br>> <br>> <br>> <br>> <br>> I see an error during image build<br>> <br>> <br>> [stack@undercloud ~]$ openstack overcloud image build --all<br>> ...<br>> a lot of log<br>> ...<br>> ++ cat /etc/dib_dracut_drivers<br>> + dracut -N --install ' curl partprobe lsblk targetcli tail head awk ifconfig<br>> cut expr route ping nc wget tftp grep' --kernel-cmdline 'rd.shell rd.debug<br>> rd.neednet=1 rd.driver.pre=ahci' --include /var/tmp/image.YVhwuArQ/mnt/ /<br>> --kver 3.10.0-229.14.1.el7.x86_64 --add-drivers ' virtio virtio_net<br>> virtio_blk target_core_mod iscsi_target_mod target_core_iblock<br>> target_core_file target_core_pscsi configfs' -o 'dash plymouth' /tmp/ramdisk<br>> cat: write error: Broken pipe<br>> + cp /boot/vmlinuz-3.10.0-229.14.1.el7.x86_64 /tmp/kernel<br>> + chmod o+r /tmp/kernel<br>> + trap EXIT<br>> + target_tag=99-build-dracut-ramdisk<br>> + date +%s.%N<br>> + output '99-build-dracut-ramdisk completed'<br>> ...<br>> a lot of log<br>> ...<br><div><br></div>You can ignore that afaik, if you end up having all the required images it should be ok.<br><div><br></div>> <br>> Then, during introspection stage I see ironic-python-agent errors on nodes<br>> (screenshot attached) and the following warnings<br>> <br><div><br></div>That looks odd. Is it showing up in the early stage of the introspection? At some point it should receive an address by DHCP and the Network is unreachable error should disappear. Does the introspection complete and the nodes are turned off? <br><div><br></div>> <br>> <br>> [root@localhost ~]# journalctl -fl -u openstack-ironic-conductor.service |<br>> grep -i "warning\|error"<br>> Oct 14 10:30:12 undercloud.rdo ironic-conductor[619]: 2015-10-14 10:30:12.119<br>> 619 WARNING oslo_config.cfg [req-eccf8cb5-6e93-4d8f-9a05-0e8c8d2aab7b ]<br>> Option "http_url" from group "pxe" is deprecated. Use option "http_url" from<br>> group "deploy".<br>> Oct 14 10:30:12 undercloud.rdo ironic-conductor[619]: 2015-10-14 10:30:12.119<br>> 619 WARNING oslo_config.cfg [req-eccf8cb5-6e93-4d8f-9a05-0e8c8d2aab7b ]<br>> Option "http_root" from group "pxe" is deprecated. Use option "http_root"<br>> from group "deploy".<br>> <br>> <br>> Before deployment ironic node-list:<br>> <br><div><br></div>This is odd too as I'm expecting the nodes to be powered off before running deployment. <br><div><br></div>> <br>> <br>> [stack@undercloud ~]$ ironic node-list<br>> +--------------------------------------+------+---------------+-------------+--------------------+-------------+<br>> | UUID | Name | Instance UUID | Power State | Provisioning State |<br>> | Maintenance |<br>> +--------------------------------------+------+---------------+-------------+--------------------+-------------+<br>> | acfc1bb4-469d-479a-af70-c0bdd669b32d | None | None | power on | available |<br>> | False |<br>> | b5811c06-d5d1-41f1-87b3-2fd55ae63553 | None | None | power on | available |<br>> | False |<br>> +--------------------------------------+------+---------------+-------------+--------------------+-------------+<br>> <br>> During deployment I get following errors<br>> <br>> [root@localhost ~]# journalctl -fl -u openstack-ironic-conductor.service |<br>> grep -i "warning\|error"<br>> Oct 14 11:29:01 undercloud.rdo ironic-conductor[619]: 2015-10-14 11:29:01.739<br>> 619 ERROR ironic.drivers.modules.ipmitool [-] IPMI Error while attempting<br>> "ipmitool -I lanplus -H 192.168.0.19 -L ADMINISTRATOR -U root -R 3 -N 5 -f<br>> /tmp/tmpSCKHIv power status"for node b5811c06-d5d1-41f1-87b3-2fd55ae63553.<br>> Error: Unexpected error while running command.<br>> Oct 14 11:29:01 undercloud.rdo ironic-conductor[619]: 2015-10-14 11:29:01.739<br>> 619 WARNING ironic.drivers.modules.ipmitool [-] IPMI power status failed for<br>> node b5811c06-d5d1-41f1-87b3-2fd55ae63553 with error: Unexpected error while<br>> running command.<br>> Oct 14 11:29:01 undercloud.rdo ironic-conductor[619]: 2015-10-14 11:29:01.740<br>> 619 WARNING ironic.conductor.manager [-] During sync_power_state, could not<br>> get power state for node b5811c06-d5d1-41f1-87b3-2fd55ae63553, attempt 1 of<br>> 3. Error: IPMI call failed: power status..<br>> <br><div><br></div>This looks like an ipmi error, can you try to manually run commands using the ipmitool and see if you get any success? It's also worth filing a bug with details such as the ipmitool version, server model, drac firmware version. <br><div><br></div>> <br>> <br>> <br>> <br>> <br>> Thanks a lot<br>> <br>> <br>> <br>> ----- Orijinal Mesaj -----<br>> <br>> Kimden: "Marius Cornea" <mcornea@redhat.com><br>> Kime: "Esra Celik" <celik.esra@tubitak.gov.tr><br>> Kk: "Ignacio Bravo" <ibravo@ltgfederal.com>, rdo-list@redhat.com<br>> Gönderilenler: 13 Ekim Salı 2015 21:16:14<br>> Konu: Re: Yan: Re: [Rdo-list] OverCloud deploy fails with error "No valid<br>> host was found"<br>> <br>> <br>> ----- Original Message -----<br>> > From: "Esra Celik" <celik.esra@tubitak.gov.tr><br>> > To: "Marius Cornea" <mcornea@redhat.com><br>> > Cc: "Ignacio Bravo" <ibravo@ltgfederal.com>, rdo-list@redhat.com<br>> > Sent: Tuesday, October 13, 2015 5:02:09 PM<br>> > Subject: Yan: Re: [Rdo-list] OverCloud deploy fails with error "No valid<br>> > host was found"<br>> > <br>> > During deployment they are powering on and deploying the images. I see lot<br>> > of<br>> > connection error messages about ironic-python-agent but ignore them as<br>> > mentioned here<br>> > (https://www.redhat.com/archives/rdo-list/2015-October/msg00052.html)<br>> <br>> That was referring to the introspection stage. From what I can tell you are<br>> experiencing issues during deployment as it fails to provision the nova<br>> instances, can you check if during that stage the nodes get powered on?<br>> <br>> Make sure that before overcloud deploy the ironic nodes are available for<br>> provisioning (ironic node-list and check the provisioning state column).<br>> Also check that you didn't miss any step in the docs in regards to kernel<br>> and ramdisk assignment, introspection, flavor creation(so it matches the<br>> nodes resources)<br>> https://repos.fedorapeople.org/repos/openstack-m/rdo-manager-docs/liberty/basic_deployment/basic_deployment_cli.html<br>> <br>> <br>> > In instackenv.json file I do not need to add the undercloud node, or do I?<br>> <br>> No, the nodes details should be enough.<br>> <br>> > And which log files should I watch during deployment?<br>> <br>> You can check the openstack-ironic-conductor logs(journalctl -fl -u<br>> openstack-ironic-conductor.service) and the logs in /var/log/nova.<br>> <br>> > Thanks<br>> > Esra<br>> > <br>> > <br>> > ----- Orijinal Mesaj -----Kimden: Marius Cornea <mcornea@redhat.com>Kime:<br>> > Esra Celik <celik.esra@tubitak.gov.tr>Kk: Ignacio Bravo<br>> > <ibravo@ltgfederal.com>, rdo-list@redhat.comGönderilenler: Tue, 13 Oct<br>> > 2015 17:25:00 +0300 (EEST)Konu: Re: [Rdo-list] OverCloud deploy fails with<br>> > error "No valid host was found"<br>> > <br>> > ----- Original Message -----> From: "Esra Celik"<br>> > <celik.esra@tubitak.gov.tr>><br>> > To: "Ignacio Bravo" <ibravo@ltgfederal.com>> Cc: rdo-list@redhat.com> Sent:<br>> > Tuesday, October 13, 2015 3:47:57 PM> Subject: Re: [Rdo-list] OverCloud<br>> > deploy fails with error "No valid host was found"> > > > Actually I<br>> > re-installed the OS for Undercloud before deploying. However I did> not<br>> > re-install the OS in Compute and Controller nodes.. I will reinstall> basic<br>> > OS for them too, and retry..<br>> > <br>> > You don't need to reinstall the OS on the controller and compute, they will<br>> > get the image served by the undercloud. I'd recommend that during<br>> > deployment<br>> > you watch the servers console and make sure they get powered on, pxe boot,<br>> > and actually get the image deployed.<br>> > <br>> > Thanks<br>> > <br>> > > Thanks> > > > Esra ÇELİK> TÜBİTAK BİLGEM><br>> > > www.bilgem.tubitak.gov.tr> celik.esra@tubitak.gov.tr> > > Kimden:<br>> > > "Ignacio<br>> > > Bravo" <ibravo@ltgfederal.com>> Kime: "Esra Celik"<br>> > > <celik.esra@tubitak.gov.tr>> Kk: rdo-list@redhat.com> Gönderilenler:<br>> > > 13 Ekim Salı 2015 16:36:06> Konu: Re: [Rdo-list] OverCloud deploy fails<br>> > > with error "No valid host was> found"> > Esra,> > I encountered the same<br>> > > problem after deleting the stack and re-deploying.> > It turns out that<br>> > > 'heat stack-delete overcloud’ does remove the nodes from><br>> > > ‘nova list’ and one would assume that the baremetal servers<br>> > > are now ready to> be used for the next stack, but when redeploying, I get<br>> > > the same message of> not enough hosts available.> > You can look into the<br>> > > nova logs and it mentions something about ‘node xxx is> already<br>> > > associated with UUID yyyy’ and ‘I tried 3 times and I’m<br>> > > giving up’.> The issue is that the UUID yyyy belonged to a prior<br>> > > unsuccessful deployment.> > I’m now redeploying the basic OS to<br>> > > start from scratch again.> > IB> > __> Ignacio Bravo> LTG Federal, Inc><br>> > > www.ltgfederal.com> Office: (703) 951-7760> > > > On Oct 13, 2015, at<br>> > > 9:25<br>> > > AM, Esra Celik < celik.esra@tubitak.gov.tr > wrote:> > > Hi all,> ><br>> > > OverCloud deploy fails with error "No valid host was found"> ><br>> > > [stack@undercloud ~]$ openstack overcloud deploy --templates> Deploying<br>> > > templates in the directory> /usr/share/openstack-tripleo-heat-templates><br>> > > Stack failed with status: Resource CREATE failed: resources.Compute:><br>> > > ResourceInError: resources[0].resources.NovaCompute: Went to status<br>> > > ERROR><br>> > > due to "Message: No valid host was found. There are not enough hosts><br>> > > available., Code: 500"> Heat Stack create failed.> > Here are some logs:><br>> > > > Every 2.0s: heat resource-list -n 5 overcloud | grep -v COMPLETE Tue<br>> > > > Oct<br>> > > 13> 16:18:17 2015> ><br>> > > +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------+><br>> > > | resource_name | physical_resource_id | resource_type | resource_status<br>> > > |> | updated_time | stack_name |><br>> > > +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------+><br>> > > | Compute | e33b6b1e-8740-4ded-ad7f-720617a03393 |<br>> > > | OS::Heat::ResourceGroup<br>> > > |> | CREATE_FAILED | 2015-10-13T10:20:36 | overcloud |> | Controller |<br>> > > 116c57ff-debb-4c12-92e1-e4163b67dc17 | OS::Heat::ResourceGroup> | |<br>> > > CREATE_FAILED | 2015-10-13T10:20:36 | overcloud |> | 0 |<br>> > > 342a9023-de8f-4b5b-b3ec-498d99b56dc4 | OS::TripleO::Controller |> |<br>> > > CREATE_IN_PROGRESS | 2015-10-13T10:20:52 |> |<br>> > > overcloud-Controller-45bbw24xxhxs |> | 0 |<br>> > > e420a7bd-86f8-4cc1-b6a0-5ba8d1412453 | OS::TripleO::Compute |> |<br>> > > CREATE_FAILED | 2015-10-13T10:20:54 | overcloud-Compute-vqk632ysg64r |> |<br>> > > Controller | 2e9ac712-0566-49b5-958f-c3e151bb24d7 | OS::Nova::Server |> |<br>> > > CREATE_IN_PROGRESS | 2015-10-13T10:20:54 |> |<br>> > > overcloud-Controller-45bbw24xxhxs-0-3vyhjiak2rsk |> | NovaCompute |<br>> > > 96efee56-81cb-46af-beef-84f4a3af761a | OS::Nova::Server |> |<br>> > > CREATE_FAILED<br>> > > | 2015-10-13T10:20:56 |> | overcloud-Compute-vqk632ysg64r-0-32nalzkofmef<br>> > > |> <br>> > > +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+---------------------------------------------------------------------------------+><br>> > > > > [stack@undercloud ~]$ heat resource-show overcloud Compute><br>> > > +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+><br>> > > | Property | Value |><br>> > > +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+><br>> > > | attributes | { |> | | "attributes": null, |> | | "refs": null |> | | }<br>> > > |> | creation_time | 2015-10-13T10:20:36 |> | description | |> | links |><br>> > > | <br>> > > http://192.0.2.1:8004/v1/3c5942d315bc445c8f1f7b32bd445de5/stacks/overcloud/620ada83-fb7d-4769-b1e8-431cfbb37d70/resources/Compute><br>> > > | (self) |> | |<br>> > > http://192.0.2.1:8004/v1/3c5942d315bc445c8f1f7b32bd445de5/stacks/overcloud/620ada83-fb7d-4769-b1e8-431cfbb37d70><br>> > > | | (stack) |> | |<br>> > > http://192.0.2.1:8004/v1/3c5942d315bc445c8f1f7b32bd445de5/stacks/overcloud-Compute-vqk632ysg64r/e33b6b1e-8740-4ded-ad7f-720617a03393><br>> > > | | (nested) |> | logical_resource_id | Compute |> | physical_resource_id<br>> > > | e33b6b1e-8740-4ded-ad7f-720617a03393 |> | required_by |<br>> > > ComputeAllNodesDeployment |> | | ComputeNodesPostDeployment |> | |<br>> > > ComputeCephDeployment |> | | ComputeAllNodesValidationDeployment |> | |<br>> > > AllNodesExtraConfig |> | | allNodesConfig |> | resource_name | Compute |><br>> > > | resource_status | CREATE_FAILED |> | resource_status_reason |<br>> > > resources.Compute: ResourceInError:> |<br>> > > resources[0].resources.NovaCompute:<br>> > > Went to status ERROR due to "Message:> | No valid host was found. There<br>> > > are not enough hosts available., Code: 500"> | |> | resource_type |<br>> > > OS::Heat::ResourceGroup |> | updated_time | 2015-10-13T10:20:36 |><br>> > > +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+><br>> > > > > > This is my instackenv.json for 1 compute and 1 control node to be<br>> > > deployed.> > {> "nodes": [> {> "pm_type":"pxe_ipmitool",> "mac":[><br>> > > "08:9E:01:58:CC:A1"> ],> "cpu":"4",> "memory":"8192",> "disk":"10",><br>> > > "arch":"x86_64",> "pm_user":"root",> "pm_password":"calvin",><br>> > > "pm_addr":"192.168.0.18"> },> {> "pm_type":"pxe_ipmitool",> "mac":[><br>> > > "08:9E:01:58:D0:3D"> ],> "cpu":"4",> "memory":"8192",> "disk":"100",><br>> > > "arch":"x86_64",> "pm_user":"root",> "pm_password":"calvin",><br>> > > "pm_addr":"192.168.0.19"> }> ]> }> > > Any ideas? Thanks in advance> > ><br>> > > Esra ÇELİK> TÜBİTAK BİLGEM> www.bilgem.tubitak.gov.tr><br>> > > celik.esra@tubitak.gov.tr> ><br>> > > _______________________________________________> Rdo-list mailing list><br>> > > Rdo-list@redhat.com> https://www.redhat.com/mailman/listinfo/rdo-list> ><br>> > > To unsubscribe: rdo-list-unsubscribe@redhat.com> > > ><br>> > > _______________________________________________> Rdo-list mailing list><br>> > > Rdo-list@redhat.com> https://www.redhat.com/mailman/listinfo/rdo-list> ><br>> > > To unsubscribe: rdo-list-unsubscribe@redhat.com<br>> > <br>> <br>><br></div><div><br></div></div></body></html>