<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><br>
</p>
<br>
<br>
<div style="color: rgb(49, 55, 57);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> rdo-list-bounces@redhat.com <rdo-list-bounces@redhat.com> on behalf of Dan Sneddon <dsneddon@redhat.com><br>
<b>Sent:</b> Wednesday, June 29, 2016 1:46 PM<br>
<b>To:</b> rdo-list@redhat.com<br>
<b>Subject:</b> Re: [rdo-list] HA overcloud-deploy.sh crashes again ( ControllerOvercloudServicesDeployment_Step4 )</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">On 06/29/2016 10:42 AM, Dan Sneddon wrote:<br>
> On 06/29/2016 07:03 AM, Boris Derzhavets wrote:<br>
>> Boris Derzhavets has shared a OneDrive file with you. To view it, click<br>
>> the link below.<br>
>><br>
>> <<a id="LPlnk647728" href="https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk">https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk</a>>
<div style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;" id="LPBorder_GT_14672339797940.4052245162730551">
<table style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);" id="LPContainer_14672339797830.8302608259292806" cellspacing="0">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td colspan="1" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;" id="ImageCell_14672339797850.6407607656991819">
<div style="background-color: rgb(255, 255, 255); height: 96px; position: relative; margin: auto; display: table; width: 68px;" id="LPImageContainer_14672339797880.48865833291800365">
<a target="_blank" href="https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk" style="display: table-cell; text-align: center;" id="LPImageAnchor_14672339797890.9869942114606172"><img id="LPThumbnailImageID_14672339797890.20084199092994237" aria-label="Preview image with link selected. Double-tap to open the link." style="display: inline-block; max-width: 250px; max-height: 250px; height: 96px; width: 68px; border-width: 0px; vertical-align: bottom;" height="96" width="68" src="https://p.sfx.ms/icons/v2/Large/Default.png"></a></div>
</td>
<td colspan="2" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;" id="TextCell_14672339797900.07214788170673303">
<div id="LPRemovePreviewContainer_14672339797900.1606177933041295"></div>
<div style="top: 0px; color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: "wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; line-height: 21px;" id="LPTitle_14672339797900.7667021114791747">
<a target="_blank" href="https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk" style="text-decoration: none;" id="LPUrlAnchor_14672339797920.43482432657408165">HeatCrash2.txt 1.gz</a></div>
<div style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 14px;" id="LPMetadata_14672339797920.36828304621754515">
1drv.ms</div>
<div style="display: block; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 20px; max-height: 100px; overflow: hidden;" id="LPDescription_14672339797930.634126963848486">
GZ File</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
>> <br>
>> HeatCrash2.txt 1.gz <<a href="https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk">https://1drv.ms/u/s!AqjiDzRpwaKogSHAekH8ZluOaclk</a>><br>
>> [HeatCrash2.txt 1.gz]<br>
>><br>
>> Reattach gzip archive via One Drive<br>
>><br>
>><br>
>><br>
>> -----------------------------------------------------------------------<br>
>> *From:* rdo-list-bounces@redhat.com <rdo-list-bounces@redhat.com> on<br>
>> behalf of Boris Derzhavets <bderzhavets@hotmail.com><br>
>> *Sent:* Wednesday, June 29, 2016 9:36 AM<br>
>> *To:* John Trowbridge; shardy@redhat.com<br>
>> *Cc:* rdo-list@redhat.com<br>
>> *Subject:* [rdo-list] HA overcloud-deploy.sh crashes again (<br>
>> ControllerOvercloudServicesDeployment_Step4 )<br>
>> <br>
>><br>
>> Attempt to follow steps suggested<br>
>> in <a href="http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html">
http://hardysteven.blogspot.ru/2016/06/tripleo-partial-stack-updates.html</a><br>
>><br>
>><br>
>> ./deploy-overstack crashes<br>
>><br>
>><br>
>> 2016-06-29 12:42:41<br>
>> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk-ControllerOvercloudServicesDeployment_Step4-nzdoizlgrmx2]:<br>
>> CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment<br>
>> to server failed: deploy_status_code : Deployment exited with non-zero<br>
>> status code: 6<br>
>> 2016-06-29 12:42:42 [ControllerOvercloudServicesDeployment_Step4]:<br>
>> CREATE_FAILED Error:<br>
>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]:<br>
>> Deployment to server failed: deploy_status_code: Deployment exited with<br>
>> non-zero status code: 6<br>
>> 2016-06-29 12:42:43<br>
>> [overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk]: CREATE_FAILED<br>
>> Resource CREATE failed: Error:<br>
>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]:<br>
>> Deployment to server failed: deploy_status_code: Deployment exited with<br>
>> non-zero status code: 6<br>
>> 2016-06-29 12:42:44 [ControllerNodesPostDeployment]: CREATE_FAILED<br>
>> Error:<br>
>> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]:<br>
>> Deployment to server failed: deploy_status_code: Deployment exited with<br>
>> non-zero status code: 6<br>
>> 2016-06-29 12:42:44 [2]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:45 [2]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:46 [overcloud]: CREATE_FAILED Resource CREATE failed:<br>
>> Error:<br>
>> resources.ControllerNodesPostDeployment.resources.ControllerOvercloudServicesDeployment_Step4.resources[0]:<br>
>> Deployment to server failed: deploy_status_code: Deployment exited with<br>
>> non-zero status code: 6<br>
>> 2016-06-29 12:42:46 [2]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:47 [2]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:47 [ControllerDeployment]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:48 [NetworkDeployment]: SIGNAL_COMPLETE Unknown<br>
>> 2016-06-29 12:42:48 [2]: SIGNAL_COMPLETE Unknown<br>
>> Stack overcloud CREATE_FAILED<br>
>> Deployment failed: Heat Stack create failed.<br>
>> + heat stack-list<br>
>> + grep -q CREATE_FAILED<br>
>> + deploy_status=1<br>
>> ++ heat resource-list --nested-depth 5 overcloud<br>
>> ++ grep FAILED<br>
>> ++ grep 'StructuredDeployment '<br>
>> ++ cut -d '|' -f3<br>
>> + for failed in '$(heat resource-list --nested-depth 5<br>
>> overcloud | grep FAILED |<br>
>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)'<br>
>> + heat deployment-show 655c77fc-6a78-4cca-b4b7-a153a3f4ad52<br>
>> + for failed in '$(heat resource-list --nested-depth 5<br>
>> overcloud | grep FAILED |<br>
>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)'<br>
>> + heat deployment-show 1fe5153c-e017-4ee5-823a-3d1524430c1d<br>
>> + for failed in '$(heat resource-list --nested-depth 5<br>
>> overcloud | grep FAILED |<br>
>> grep '\''StructuredDeployment '\'' | cut -d '\''|'\'' -f3)'<br>
>> + heat deployment-show bf6f25f4-d812-41e9-a7a8-122de619a624<br>
>> + exit 1<br>
>><br>
>> *****************************<br>
>> Troubleshooting steps :-<br>
>> *****************************<br>
>><br>
>> [stack@undercloud ~]$ . stackrc<br>
>> [stack@undercloud ~]$ heat resource-list overcloud | grep<br>
>> ControllerNodesPost<br>
>> | ControllerNodesPostDeployment |<br>
>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 |<br>
>> OS::TripleO::ControllerPostDeployment | CREATE_FAILED |<br>
>> 2016-06-29T12:11:21 |<br>
>><br>
>><br>
>> [stack@undercloud ~]$ heat stack-list -n | grep "^|<br>
>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3"<br>
>> | f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3 |<br>
>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk
<br>
>> | CREATE_FAILED | 2016-06-29T12:31:11 | None |<br>
>> 17f82f6e-e0ca-44c6-9058-de82c00d4f79 |<br>
>><br>
>><br>
>><br>
>> [stack@undercloud ~]$ heat event-list -m<br>
>> f1d6a474-c946-46bf-ab0c-2fdaeb55d0b3<br>
>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk<br>
>><br>
>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+<br>
>> | resource_name |<br>
>> id |<br>
>> resource_status_reason
<br>
>> | resource_status | event_time |<br>
>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+<br>
>> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk |<br>
>> 10ec0cf9-b3c9-4191-9966-3f4d47f27e2a | Stack CREATE started <br>
>> . . . . . . . . . . . . . . . . .<br>
>> Step1,2,3 succeeded<br>
>> . . . . . . . . . . . . . . . . .<br>
>>
<br>
>> | CREATE_IN_PROGRESS | 2016-06-29T12:31:14 |<br>
>> | ControllerPuppetConfig |<br>
>> a2a1df33-5106-425c-b16d-8d2df709b19f | state<br>
>> changed
<br>
>> | CREATE_COMPLETE | 2016-06-29T12:35:02 |<br>
>> | ControllerOvercloudServicesDeployment_Step4 |<br>
>> 1e151333-4de5-4e7b-907c-ea0f42d31a47 | state<br>
>> changed
<br>
>> | CREATE_IN_PROGRESS | 2016-06-29T12:35:03 |<br>
>> | ControllerOvercloudServicesDeployment_Step4 |<br>
>> 7bf36334-3d92-4554-b6c0-41294a072ab6 | Error:<br>
>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]:<br>
>> Deployment to server failed: deploy_status_code: Deployment exited with<br>
>> non-zero status code: 6 | CREATE_FAILED |<br>
>> 2016-06-29T12:42:42 |<br>
>> | overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk<br>
>> | e72fb6f4-c2aa-4fe8-9bd1-5f5ad152685c | Resource CREATE failed:<br>
>> Error:<br>
>> resources.ControllerOvercloudServicesDeployment_Step4.resources[0]:<br>
>> Deployment to server failed: deploy_status_code: Deployment exited with<br>
>> non-zero status code: 6 | CREATE_FAILED | 2016-06-29T12:42:43 |<br>
>> +------------------------------------------------------+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------+---------------------+<br>
>><br>
>> [stack@undercloud ~]$ heat stack-show<br>
>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk | grep<br>
>> NodeConfigIdentifiers<br>
>> | | "NodeConfigIdentifiers":<br>
>> "{u'deployment_identifier': 1467202276, u'controller_config': {u'1':<br>
>> u'os-apply-config deployment 796df02a-7550-414b-a084-8b591a13e6db<br>
>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,',<br>
>> u'0': u'os-apply-config deployment 613ec889-d852-470a-8e4c-6e243e1d2033<br>
>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,',<br>
>> u'2': u'os-apply-config deployment c8b099d0-3af4-4ba0-a056-a0ce60f40e2d<br>
>> completed,Root CA cert injection not enabled.,TLS not enabled.,None,'},<br>
>> u'allnodes_extra': u'none'}" |<br>
>><br>
>> However, when stack creating crashed update wouldn't help.<br>
>><br>
>> [stack@undercloud ~]$ heat stack-update -x<br>
>> overcloud-ControllerNodesPostDeployment-2r4tlv5icaxk -e update_env.yaml<br>
>> ERROR: PATCH update to non-COMPLETE stack is not supported.<br>
>><br>
>> DUE TO :-<br>
>><br>
>> [stack@undercloud ~]$ heat stack-list<br>
>> +--------------------------------------+------------+---------------+---------------------+--------------+<br>
>> | id | stack_name | stack_status |<br>
>> creation_time | updated_time |<br>
>> +--------------------------------------+------------+---------------+---------------------+--------------+<br>
>> | 17f82f6e-e0ca-44c6-9058-de82c00d4f79 | overcloud | CREATE_FAILED |<br>
>> 2016-06-29T12:11:20 | None |<br>
>> +--------------------------------------+------------+---------------+---------------------+------<br>
>><br>
>><br>
>> Complete error file `heat deployment-show<br>
>> 655c77fc-6a78-4cca-b4b7-a153a3f4ad52` is attached a gzip archive.<br>
>><br>
>><br>
>> Thanks.<br>
>><br>
>> Boris.<br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> rdo-list mailing list<br>
>> rdo-list@redhat.com<br>
>> <a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
>><br>
>> To unsubscribe: rdo-list-unsubscribe@redhat.com<br>
>><br>
> <br>
> The failure occurred during the post-deployment, which means that the<br>
> initial deployment succeeded, but then the steps that are done to the<br>
> completed overcloud failed.<br>
> <br>
> This is most commonly attributable to network problems between the<br>
> Undercloud and the Overcloud Public API. The Undercloud needs to reach<br>
> the Public API in order to do some of the post-configuration steps. If<br>
> this API isn't reachable, you end up with the error you saw above.<br>
> <br>
> You can test this connectivity by pinging the Public API VIP from the<br>
> Undercloud. Starting with the failed deployment, run "neutron<br>
> port-list" against the Underlcloud and look for the IP on the port<br>
> named "public_virtual_ip". You should be able to ping this address from<br>
> the Undercloud. If you can't reach that IP, then you need to check the<br>
> connectivity/routing between the Undercloud and the External network on<br>
> the Overcloud.<br>
> <br>
<br>
I should also mention common causes of this problem:<br>
<br>
* Incorrect value for ExternalInterfaceDefaultRoute in the network<br>
environment file.<br>
* Controllers do not have the default route on the External network in<br>
the NIC config templates (required for reachability from remote subnets).<br>
* Incorrect subnet mask on the ExternalNetCidr in the network environment.<br>
* Incorrect ExternalAllocationPools values in the network environment.<br>
* Incorrect Ethernet switch config for the Controllers.<br>
<br>
Issue has been reproduced with exactly same error 4 times<br>
starting since 06/25/16 on daily basis with exactly same error at Step4<br>
of <font size="2"><span style="font-size:10pt;">overcloud-ControllerNodesPostDeployment</span></font>.<br>
In meantime I cannot reproduce the error. <br>
Config 3xNode HA Controller + 1xCompute works .<br>
There was one more issue <font size="2"><span style="font-size:10pt;">3xNode HA Controller + 2xCompute</span></font><br>
failed immediately when overcloud-deploy.sh started due to<br>
only 4 nodes could be introspected. I will test it tomorrow morning.<br>
<br>
Thanks a lot.<br>
Boris.<br>
<br>
-- <br>
Dan Sneddon | Principal OpenStack Engineer<br>
dsneddon@redhat.com | redhat.com/openstack<br>
650.254.4025 | dsneddon:irc @dxs:twitter<br>
<br>
_______________________________________________<br>
rdo-list mailing list<br>
rdo-list@redhat.com<br>
<a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
<br>
To unsubscribe: rdo-list-unsubscribe@redhat.com</div>
</span></font></div>
</div>
</body>
</html>