<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><br>
</p>
<br>
<br>
<div style="color: rgb(49, 55, 57);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Raoul Scarazzini <rasca@redhat.com><br>
<b>Sent:</b> Tuesday, August 23, 2016 11:29 AM<br>
<b>To:</b> Boris Derzhavets; Wesley Hayutin; Attila Darazs<br>
<b>Cc:</b> rdo-list<br>
<b>Subject:</b> Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in regards of managing HA PCS/Corosync cluster via pcs CLI</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hi Boris,<br>
so, for what I see the pcs commands that stops and starts the cluster on<br>
the rebooted node should not be used. It can happen that a service fails<br>
to start but we need to investigate why from the logs.<br>
<br>
Remember that cleaning up resources can be useful if we know what<br>
happened, but using it repeatedly makes no sense. In addition remember<br>
that you can use just "pcs resource cleanup" to cleanup the entire<br>
cluster status and in some way "start from the beginning".<br>
<br>
Now, about this specific problem we need to understand what is happening<br>
here. Correct me if I'm wrong:<br>
<br>
1) We have a clean env in which we reboot a node;<br>
That is correct<br>
<br>
2) The nodes comes up, but some resources fails;<br>
All resources fail<br>
<br>
3) After some cleanups the env becomes clean again;<br>
<br>
a) If VENV is setup by instack-virt-setup ( official guide )<br>
Mentioned script start.sh works right a way . It comes as well from official guide.<br>
<br>
b) if VENV is setup by Tripleo QuickStart ( where undecloud.qcow2 gets uploaded<br>
to libvirt pool already having overcloud images integrated per Jon's Video explanation<br>
QuickStart CI vs Tripleo CI )<br>
then ( via my experience ) before attempting start.sh I MUST restart PCS Cluster<br>
on bounced Controller-X , then invoke `. ./start.sh` ( not simply ./start.sh )<br>
Pretty often second run start.sh is required from another controller-Y.<br>
Some times I cannot fix it in script mode and have manually run commands giving<br>
delay more the 10 sec. So finally ( about 25 tests passed) I get `pcs status` OK.<br>
In other words all service are up and running on every controller-X,Y,Z<br>
<br>
Details :-<br>
<a id="LPlnk524321" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" class="OWAAutoLink">http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>
<div style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;" id="LPBorder_GT_14719731314380.31606882127687197">
<table style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);" id="LPContainer_14719731314300.648135248232508" cellspacing="0">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td colspan="1" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;" id="ImageCell_14719731314320.4161534724734727">
<div style="background-color: rgb(255, 255, 255); height: 131px; position: relative; margin: auto; display: table; width: 250px;" id="LPImageContainer_14719731314320.301290047438314">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" style="display: table-cell; text-align: center;" id="LPImageAnchor_14719731314330.9673155414124305"><img id="LPThumbnailImageID_14719731314330.5521081642568477" aria-label="Preview image with link selected. Double-tap to open the link." style="display: inline-block; max-width: 250px; max-height: 250px; height: 131px; width: 250px; border-width: 0px; vertical-align: bottom;" height="131" width="250" src="https://3.bp.blogspot.com/-xtnWVVrV2cs/V7nQssWM9pI/AAAAAAAAHBw/DrYHJeCNEO4nTCigqZpgt4P7iwgmKekhQCLcB/w1200-h630-p-nu/Screenshot%2Bfrom%2B2016-08-21%2B19-00-24.png"></a></div>
</td>
<td colspan="2" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;" id="TextCell_14719731314340.45140479604018613">
<div id="LPRemovePreviewContainer_14719731314340.7880241717011708"></div>
<div style="top: 0px; color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: "wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; line-height: 21px;" id="LPTitle_14719731314340.20986011246100922">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" style="text-decoration: none;" id="LPUrlAnchor_14719731314360.5302641593475634">Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA
Controller's Cluster failover</a></div>
<div style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 14px;" id="LPMetadata_14719731314370.9982619187801508">
bderzhavets.blogspot.ru</div>
</td>
</tr>
</tbody>
</table>
</div>
Is this the sequence of operations you are using? Is the problem<br>
systematic and can we reproduce it? <br>
><br>
YES<br>
><br>
Can we grab sosreports from the<br>
machine involved?<br>
><br>
Instruct me how to do this ?<br>
><br>
Most important question: which OpenStack version are<br>
you testing?<br>
><br>
Mitaka stable :-<br>
<br>
[tripleo-quickstart@stack] $ bash quickstart --config ./ha.yml $VIRTHOST<br>
By default , no --release specified Mitaka Delorean trunks get selected<br>
Just check<code> /etc/yum.repos.d/</code> for delorean.repos<br>
quickstart places on undercloud when it exits asking to to connect to undercloud<br>
><br>
Boris<br>
-- <br>
Raoul Scarazzini<br>
rasca@redhat.com<br>
<br>
<br>
On 22/08/2016 13:49, Boris Derzhavets wrote:<br>
> <br>
> Sorry , for my English<br>
> <br>
> I was also keeping (not kept ) track on Galera DB via `clustercheck`<br>
> <br>
> either I just kept.<br>
> <br>
> <br>
> Boris<br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> *From:* rdo-list-bounces@redhat.com <rdo-list-bounces@redhat.com> on<br>
> behalf of Boris Derzhavets <bderzhavets@hotmail.com><br>
> *Sent:* Monday, August 22, 2016 7:29 AM<br>
> *To:* Raoul Scarazzini; Wesley Hayutin; Attila Darazs<br>
> *Cc:* rdo-list<br>
> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in<br>
> regards of managing HA PCS/Corosync cluster via pcs CLI<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> *From:* Raoul Scarazzini <rasca@redhat.com><br>
> *Sent:* Monday, August 22, 2016 3:51 AM<br>
> *To:* Wesley Hayutin; Boris Derzhavets; Attila Darazs<br>
> *Cc:* David Moreau Simard; rdo-list<br>
> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in<br>
> regards of managing HA PCS/Corosync cluster via pcs CLI<br>
> <br>
> Hi everybody,<br>
> sorry for the late response but I was on PTO. I don't understand the<br>
> meaning of the cleanup commands, but maybe it's just because I'm not<br>
> getting the whole picture.<br>
> <br>
>><br>
> I have to confirm that fault was mine PCS CLI is working on TripeO<br>
> QuickStart<br>
> but requires pcs cluster restart on particular node which went down<br>
> via ` nova stop controller-X` and was brought up via `nova start<br>
> controller-X`<br>
> Details here :-<br>
> <br>
> <a id="LPlnk824369" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">
http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>
<div style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;" id="LPBorder_GT_14719722027030.39036588545413775">
<table style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);" id="LPContainer_14719722026950.7081156321473006" cellspacing="0">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td colspan="1" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;" id="ImageCell_14719722026970.6820412551806004">
<div style="background-color: rgb(255, 255, 255); height: 131px; position: relative; margin: auto; display: table; width: 250px;" id="LPImageContainer_14719722026980.38002613829793863">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" style="display: table-cell; text-align: center;" id="LPImageAnchor_14719722026990.9638026497350308"><img id="LPThumbnailImageID_14719722026990.3526848129593727" aria-label="Preview image with link selected. Double-tap to open the link." style="display: inline-block; max-width: 250px; max-height: 250px; height: 131px; width: 250px; border-width: 0px; vertical-align: bottom;" height="131" width="250" src="https://3.bp.blogspot.com/-xtnWVVrV2cs/V7nQssWM9pI/AAAAAAAAHBw/DrYHJeCNEO4nTCigqZpgt4P7iwgmKekhQCLcB/w1200-h630-p-nu/Screenshot%2Bfrom%2B2016-08-21%2B19-00-24.png"></a></div>
</td>
<td colspan="2" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;" id="TextCell_14719722027000.8112282511692364">
<div id="LPRemovePreviewContainer_14719722027000.05801690646445912"></div>
<div style="top: 0px; color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: "wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; line-height: 21px;" id="LPTitle_14719722027000.5740491435280448">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" style="text-decoration: none;" id="LPUrlAnchor_14719722027010.5910107256287941">Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA
Controller's Cluster failover</a></div>
<div style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 14px;" id="LPMetadata_14719722027020.7100427281506573">
bderzhavets.blogspot.ru</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
> <br>
> VENV been set up with instack-virt-setup doesn't require ( on bounced<br>
> Controller node )<br>
> <br>
> # pcs cluster stop<br>
> # pcs cluster start<br>
> <br>
> Before issuing start.sh<br>
> <br>
> #!/bash -x<br>
> pcs resource cleanup rabbitmq-clone ;<br>
> sleep 10<br>
> pcs resource cleanup neutron-server-clone ;<br>
> sleep 10<br>
> pcs resource cleanup openstack-nova-api-clone ;<br>
> sleep 10<br>
> pcs resource cleanup openstack-nova-consoleauth-clone ;<br>
> sleep 10<br>
> pcs resource cleanup openstack-heat-engine-clone ;<br>
> sleep 10<br>
> pcs resource cleanup openstack-cinder-api-clone ;<br>
> sleep 10<br>
> pcs resource cleanup openstack-glance-registry-clone ;<br>
> sleep 10<br>
> pcs resource cleanup httpd-clone ;<br>
> <br>
> # . ./start.sh <br>
> <br>
> In worse case scenario I have to issue start.sh twice from different<br>
> Controllers<br>
> pcs resource cleanup openstack-nova-api-clone attempts to start<br>
> corresponding<br>
> service , which is down at the moment. In fact two cleanups above start all<br>
> Nova Services && one neutron cleanup starts all neutron agents as well.<br>
> I was also kept track of Galera DB via `clustercheck`<br>
> <br>
> Thanks.<br>
> Boris<br>
>><br>
> <br>
> <br>
> I guess we're hitting a version problem here: if you deploy the actual<br>
> master (i.e. with quickstart) you'll get the environment with the<br>
> constraints limited to the core services because of [1] and [2] (so none<br>
> of the mentioned services exists in the cluster configuration).<br>
> <br>
> Hope this helps,<br>
> <br>
> [1] <a href="https://review.openstack.org/#/c/314208/">https://review.openstack.org/#/c/314208/</a><br>
> [2] <a href="https://review.openstack.org/#/c/342650/">https://review.openstack.org/#/c/342650/</a><br>
> <br>
> -- <br>
> Raoul Scarazzini<br>
> rasca@redhat.com<br>
> <br>
> On 08/08/2016 14:43, Wesley Hayutin wrote:<br>
>> Attila, Raoul<br>
>> Can you please investigate this issue.<br>
>> <br>
>> Thanks!<br>
>> <br>
>> On Sun, Aug 7, 2016 at 3:52 AM, Boris Derzhavets<br>
>> <bderzhavets@hotmail.com <<a href="mailto:bderzhavets@hotmail.com">mailto:bderzhavets@hotmail.com</a>>> wrote:<br>
>> <br>
>> TripleO HA Controller been installed via instack-virt-setup has PCS<br>
>> CLI like :-<br>
>> <br>
>> pcs resource cleanup neutron-server-clone<br>
>> pcs resource cleanup openstack-nova-api-clone<br>
>> pcs resource cleanup openstack-nova-consoleauth-clone<br>
>> pcs resource cleanup openstack-heat-engine-clone<br>
>> pcs resource cleanup openstack-cinder-api-clone<br>
>> pcs resource cleanup openstack-glance-registry-clone<br>
>> pcs resource cleanup httpd-clone<br>
>> <br>
>> been working as expected on bare metal<br>
>> <br>
>> <br>
>> Same cluster been setup via QuickStart (Virtual ENV) after bouncing<br>
>> one of controllers<br>
>> <br>
>> included in cluster ignores PCS CLI at least via my experience (<br>
>> which is obviously limited<br>
>> <br>
>> either format of particular commands is wrong for QuickStart )<br>
>> <br>
>> I believe that dropping (complete replacing ) instack-virt-setup is<br>
>> not a good idea in general. Personally, I believe that like in case<br>
>> with packstack it is always good<br>
>> <br>
>> to have VENV configuration been tested before going to bare metal<br>
>> deployment.<br>
>> <br>
>> My major concern is maintenance and disaster recovery tests , rather<br>
>> then deployment itself . What good is for me TripleO Quickstart<br>
>> running on bare metal if I cannot replace<br>
>> <br>
>> crashed VM Controller just been limited to Services HA ( all 3<br>
>> Cluster VMs running on single<br>
>> <br>
>> bare metal node )<br>
>> <br>
>> <br>
>> Thanks<br>
>> <br>
>> Boris.<br>
>> <br>
>> <br>
>> <br>
>> <br>
>> <br>
>> ------------------------------------------------------------------------<br>
>> <br>
>> <br>
>> <br>
>> _______________________________________________<br>
>> rdo-list mailing list<br>
>> rdo-list@redhat.com <<a href="mailto:rdo-list@redhat.com">mailto:rdo-list@redhat.com</a>><br>
>> <a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
>> <<a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a>><br>
>> <br>
>> To unsubscribe: rdo-list-unsubscribe@redhat.com<br>
>> <<a href="mailto:rdo-list-unsubscribe@redhat.com">mailto:rdo-list-unsubscribe@redhat.com</a>><br>
>> <br>
>> <br>
</div>
</span></font></div>
</div>
</body>
</html>