<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><br>
</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Raoul Scarazzini <rasca@redhat.com><br>
<b>Sent:</b> Wednesday, August 24, 2016 2:56 AM<br>
<b>To:</b> Boris Derzhavets; Wesley Hayutin; Attila Darazs<br>
<b>Cc:</b> rdo-list<br>
<b>Subject:</b> Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in regards of managing HA PCS/Corosync cluster via pcs CLI</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">What I can say for sure at the moment is that it's really hard to follow<br>
your quoting :)<br>
<br>
Still some info are missing:<br>
<br>
- Which version are you using? The latest? So are you installing from<br>
master?<br>
><br>
No idea. My actions :-<br>
<br>
<div>$ rm -fr .ansible .quickstart tripleo-quickstart<br>
$ git clone https://github.com/openstack/tripleo-quickstart <br>
$ cd tripleo*<br>
$ ssh root@$VIRTHOST uname -a<br>
$ vi <font size="2"><span style="font-size:10pt;"><span><span>/config/general_config/ha.yml</span></span></span></font> ==> to tune && save<br>
$ <span>bash quickstart.sh --config <span>./config/general_config/ha.yml</span> $VIRTHOST</span><br>
<br>
When I get prompt to login to undercloud <br>
/etc/yum.repos.d/ contains delorean.repo ( on fresh undercloud QS build )<br>
files pointing to Mitaka Delorean trunks.<br>
Just check for yourself.<br>
><br>
</div>
- Are the two versions the same while using quickstart and<br>
instack-virt-setup?<br>
><br>
No the versions are different ( points in delorean.repos differ )<br>
<br>
Details here :-<br>
<a id="LPlnk392982" href="http://bderzhavets.blogspot.ru/2016/07/stable-mitaka-ha-instack-virt-setup.html" class="OWAAutoLink">http://bderzhavets.blogspot.ru/2016/07/stable-mitaka-ha-instack-virt-setup.html</a>
<div style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;" id="LPBorder_GT_14720229788800.4162516028986234">
<table style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);" id="LPContainer_14720229788710.5747381726716515" cellspacing="0">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td colspan="1" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;" id="ImageCell_14720229788730.4572050912145239">
<div style="background-color: rgb(255, 255, 255); height: 131px; position: relative; margin: auto; display: table; width: 250px;" id="LPImageContainer_14720229788730.5917884154203209">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/07/stable-mitaka-ha-instack-virt-setup.html" style="display: table-cell; text-align: center;" id="LPImageAnchor_14720229788750.27879940542280646"><img id="LPThumbnailImageID_14720229788760.7144904325088448" aria-label="Preview image with link selected. Double-tap to open the link." style="display: inline-block; max-width: 250px; max-height: 250px; height: 131px; width: 250px; border-width: 0px; vertical-align: bottom;" width="250" height="131" src="https://1.bp.blogspot.com/-0sbpr2HVRsQ/V523UacdA9I/AAAAAAAAG_E/5FJ3IMWNl9ImdAYnYlHXGzWGMHps-Q-5wCEw/w1200-h630-p-nu/Screenshot%2Bfrom%2B2016-07-31%2B10-38-16.png"></a></div>
</td>
<td colspan="2" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;" id="TextCell_14720229788770.9596017729005892">
<div id="LPRemovePreviewContainer_14720229788770.025838411689453555"></div>
<div style="top: 0px; color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: "wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; line-height: 21px;" id="LPTitle_14720229788770.17282890853991817">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/07/stable-mitaka-ha-instack-virt-setup.html" style="text-decoration: none;" id="LPUrlAnchor_14720229788780.22196213770260453">Xen Virtualization on Linux and Solaris: Stable Mitaka HA instack-virt-setup
on CentOS 7.2 VIRTHOST</a></div>
<div style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 14px;" id="LPMetadata_14720229788790.026903205419389176">
bderzhavets.blogspot.ru</div>
</td>
</tr>
</tbody>
</table>
</div>
- Do we have some logs to look at?<br>
<br>
><br>
Sorry, it was not my concern.<br>
My primary target was identify sequence of steps<br>
which brings PCS Cluster to proper state with 100% warranty.<br>
<br>
Again, I think here we're hitting a version issure.<br>
><br>
I tested QuickStart ( obtained just at the same time on different box)<br>
delorean.repo files in instack-virt-setup build of VENV<br>
It doesn't change much final overcloud cluster behavior.<br>
I am aware of this link is outdated ( due to Wesley Hayutin respond )<br>
<a id="LPlnk379116" href="https://bluejeans.com/s/a5ua/" class="OWAAutoLink">https://bluejeans.com/s/a5ua/</a><br>
However , my current understanding of QuickStart CI ( not TripleO CI ) is based on this video<br>
<br>
I believe the core reason is the way how Tripleo QS assembles undercloud for VENV build.<br>
It makes HA Controllers cluster pretty stable and always recoverable (VENV case)<br>
<br>
It is different from official way at least in meantime.<br>
I've already wrote TripleO QS nowhere issues :-<br>
$ openstack overcloud images build --all ===> that is supposed to be removed<br>
as far as I understand TripleO Core team intend.<br>
><br>
Boris.<br>
<br>
<br>
<br>
Thanks,<br>
<br>
-- <br>
Raoul Scarazzini<br>
rasca@redhat.com<br>
<br>
On 23/08/2016 19:42, Boris Derzhavets wrote:<br>
> <br>
> <br>
> <br>
> ------------------------------------------------------------------------<br>
> *From:* Raoul Scarazzini <rasca@redhat.com><br>
> *Sent:* Tuesday, August 23, 2016 11:29 AM<br>
> *To:* Boris Derzhavets; Wesley Hayutin; Attila Darazs<br>
> *Cc:* rdo-list<br>
> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in<br>
> regards of managing HA PCS/Corosync cluster via pcs CLI<br>
> <br>
> Hi Boris,<br>
> so, for what I see the pcs commands that stops and starts the cluster on<br>
> the rebooted node should not be used. It can happen that a service fails<br>
> to start but we need to investigate why from the logs.<br>
> <br>
> Remember that cleaning up resources can be useful if we know what<br>
> happened, but using it repeatedly makes no sense. In addition remember<br>
> that you can use just "pcs resource cleanup" to cleanup the entire<br>
> cluster status and in some way "start from the beginning".<br>
> <br>
> Now, about this specific problem we need to understand what is happening<br>
> here. Correct me if I'm wrong:<br>
> <br>
> 1) We have a clean env in which we reboot a node;<br>
> That is correct<br>
> <br>
> 2) The nodes comes up, but some resources fails;<br>
> All resources fail<br>
> <br>
> 3) After some cleanups the env becomes clean again;<br>
> <br>
> a) If VENV is setup by instack-virt-setup ( official guide )<br>
> Mentioned script start.sh works right a way . It comes as well<br>
> from official guide.<br>
> <br>
> b) if VENV is setup by Tripleo QuickStart ( where undecloud.qcow2<br>
> gets uploaded<br>
> to libvirt pool already having overcloud images integrated per<br>
> Jon's Video explanation<br>
> QuickStart CI vs Tripleo CI )<br>
> then ( via my experience ) before attempting start.sh I<br>
> MUST restart PCS Cluster<br>
> on bounced Controller-X , then invoke `. ./start.sh` ( not<br>
> simply ./start.sh )<br>
> Pretty often second run start.sh is required from another<br>
> controller-Y.<br>
> Some times I cannot fix it in script mode and have manually<br>
> run commands giving<br>
> delay more the 10 sec. So finally ( about 25 tests passed) I<br>
> get `pcs status` OK.<br>
> In other words all service are up and running on every<br>
> controller-X,Y,Z<br>
> <br>
> Details :-<br>
> <a id="LPlnk183059" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">
http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>
<div style="margin-bottom: 20px; overflow: auto; width: 100%; text-indent: 0px;" id="LPBorder_GT_14720223875830.5450655836267964">
<table style="width: 90%; background-color: rgb(255, 255, 255); position: relative; overflow: auto; padding-top: 20px; padding-bottom: 20px; margin-top: 20px; border-top: 1px dotted rgb(200, 200, 200); border-bottom: 1px dotted rgb(200, 200, 200);" id="LPContainer_14720223875770.2997540848264658" cellspacing="0">
<tbody>
<tr style="border-spacing: 0px;" valign="top">
<td colspan="1" style="width: 250px; position: relative; display: table-cell; padding-right: 20px;" id="ImageCell_14720223875780.3058818120016743">
<div style="background-color: rgb(255, 255, 255); height: 131px; position: relative; margin: auto; display: table; width: 250px;" id="LPImageContainer_14720223875790.4892824921786657">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" style="display: table-cell; text-align: center;" id="LPImageAnchor_14720223875800.5550680064726158"><img id="LPThumbnailImageID_14720223875800.1618366559272163" aria-label="Preview image with link selected. Double-tap to open the link." style="display: inline-block; max-width: 250px; max-height: 250px; height: 131px; width: 250px; border-width: 0px; vertical-align: bottom;" width="250" height="131" src="https://3.bp.blogspot.com/-xtnWVVrV2cs/V7nQssWM9pI/AAAAAAAAHBw/DrYHJeCNEO4nTCigqZpgt4P7iwgmKekhQCLcB/w1200-h630-p-nu/Screenshot%2Bfrom%2B2016-08-21%2B19-00-24.png"></a></div>
</td>
<td colspan="2" style="vertical-align: top; position: relative; padding: 0px; display: table-cell;" id="TextCell_14720223875800.8829051978018659">
<div id="LPRemovePreviewContainer_14720223875810.9591593947425198"></div>
<div style="top: 0px; color: rgb(0, 120, 215); font-weight: 400; font-size: 21px; font-family: "wf_segoe-ui_light","Segoe UI Light","Segoe WP Light","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; line-height: 21px;" id="LPTitle_14720223875810.5454903132939386">
<a target="_blank" href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html" style="text-decoration: none;" id="LPUrlAnchor_14720223875810.2640950021032483">Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA
Controller's Cluster failover</a></div>
<div style="margin: 10px 0px 16px; color: rgb(102, 102, 102); font-weight: 400; font-family: "wf_segoe-ui_normal","Segoe UI","Segoe WP",Tahoma,Arial,sans-serif; font-size: 14px; line-height: 14px;" id="LPMetadata_14720223875820.25556394246962233">
bderzhavets.blogspot.ru</div>
</td>
</tr>
</tbody>
</table>
</div>
<br>
> <br>
> <<a href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>><br>
> <br>
> Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA<br>
> Controller's Cluster failover<br>
> <<a href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>><br>
> bderzhavets.blogspot.ru<br>
> <br>
> Is this the sequence of operations you are using? Is the problem<br>
> systematic and can we reproduce it?<br>
>><br>
> YES<br>
>><br>
> Can we grab sosreports from the<br>
> machine involved?<br>
>><br>
> Instruct me how to do this ?<br>
>><br>
> Most important question: which OpenStack version are<br>
> you testing?<br>
>><br>
> Mitaka stable :-<br>
> <br>
> [tripleo-quickstart@stack] $ bash quickstart --config ./ha.yml $VIRTHOST<br>
> By default , no --release specified Mitaka Delorean trunks get selected<br>
> Just check|/etc/yum.repos.d/| for delorean.repos<br>
> quickstart places on undercloud when it exits asking to to connect to<br>
> undercloud<br>
>><br>
> Boris<br>
> -- <br>
> Raoul Scarazzini<br>
> rasca@redhat.com<br>
> <br>
> <br>
> On 22/08/2016 13:49, Boris Derzhavets wrote:<br>
>> <br>
>> Sorry , for my English<br>
>> <br>
>> I was also keeping (not kept ) track on Galera DB via `clustercheck`<br>
>> <br>
>> either I just kept.<br>
>> <br>
>> <br>
>> Boris<br>
>> <br>
>> <br>
>> ------------------------------------------------------------------------<br>
>> *From:* rdo-list-bounces@redhat.com <rdo-list-bounces@redhat.com> on<br>
>> behalf of Boris Derzhavets <bderzhavets@hotmail.com><br>
>> *Sent:* Monday, August 22, 2016 7:29 AM<br>
>> *To:* Raoul Scarazzini; Wesley Hayutin; Attila Darazs<br>
>> *Cc:* rdo-list<br>
>> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in<br>
>> regards of managing HA PCS/Corosync cluster via pcs CLI<br>
>> <br>
>> <br>
>> <br>
>> <br>
>> <br>
>> ------------------------------------------------------------------------<br>
>> *From:* Raoul Scarazzini <rasca@redhat.com><br>
>> *Sent:* Monday, August 22, 2016 3:51 AM<br>
>> *To:* Wesley Hayutin; Boris Derzhavets; Attila Darazs<br>
>> *Cc:* David Moreau Simard; rdo-list<br>
>> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in<br>
>> regards of managing HA PCS/Corosync cluster via pcs CLI<br>
>> <br>
>> Hi everybody,<br>
>> sorry for the late response but I was on PTO. I don't understand the<br>
>> meaning of the cleanup commands, but maybe it's just because I'm not<br>
>> getting the whole picture.<br>
>> <br>
>>><br>
>> I have to confirm that fault was mine PCS CLI is working on TripeO<br>
>> QuickStart<br>
>> but requires pcs cluster restart on particular node which went down<br>
>> via ` nova stop controller-X` and was brought up via `nova start<br>
>> controller-X`<br>
>> Details here :-<br>
>> <br>
>> <a href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">
http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a><br>
> <br>
> <<a href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>><br>
> <br>
> Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA<br>
> Controller's Cluster failover<br>
> <<a href="http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html">http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html</a>><br>
> bderzhavets.blogspot.ru<br>
> <br>
> <br>
>> <br>
>> VENV been set up with instack-virt-setup doesn't require ( on bounced<br>
>> Controller node )<br>
>> <br>
>> # pcs cluster stop<br>
>> # pcs cluster start<br>
>> <br>
>> Before issuing start.sh<br>
>> <br>
>> #!/bash -x<br>
>> pcs resource cleanup rabbitmq-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup neutron-server-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup openstack-nova-api-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup openstack-nova-consoleauth-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup openstack-heat-engine-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup openstack-cinder-api-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup openstack-glance-registry-clone ;<br>
>> sleep 10<br>
>> pcs resource cleanup httpd-clone ;<br>
>> <br>
>> # . ./start.sh <br>
>> <br>
>> In worse case scenario I have to issue start.sh twice from different<br>
>> Controllers<br>
>> pcs resource cleanup openstack-nova-api-clone attempts to start<br>
>> corresponding<br>
>> service , which is down at the moment. In fact two cleanups above start all<br>
>> Nova Services && one neutron cleanup starts all neutron agents as well.<br>
>> I was also kept track of Galera DB via `clustercheck`<br>
>> <br>
>> Thanks.<br>
>> Boris<br>
>>><br>
>> <br>
>> <br>
>> I guess we're hitting a version problem here: if you deploy the actual<br>
>> master (i.e. with quickstart) you'll get the environment with the<br>
>> constraints limited to the core services because of [1] and [2] (so none<br>
>> of the mentioned services exists in the cluster configuration).<br>
>> <br>
>> Hope this helps,<br>
>> <br>
>> [1] <a href="https://review.openstack.org/#/c/314208/">https://review.openstack.org/#/c/314208/</a><br>
>> [2] <a href="https://review.openstack.org/#/c/342650/">https://review.openstack.org/#/c/342650/</a><br>
>> <br>
>> -- <br>
>> Raoul Scarazzini<br>
>> rasca@redhat.com<br>
>> <br>
>> On 08/08/2016 14:43, Wesley Hayutin wrote:<br>
>>> Attila, Raoul<br>
>>> Can you please investigate this issue.<br>
>>> <br>
>>> Thanks!<br>
>>> <br>
>>> On Sun, Aug 7, 2016 at 3:52 AM, Boris Derzhavets<br>
>>> <bderzhavets@hotmail.com <<a href="mailto:bderzhavets@hotmail.com">mailto:bderzhavets@hotmail.com</a>>> wrote:<br>
>>> <br>
>>> TripleO HA Controller been installed via instack-virt-setup has PCS<br>
>>> CLI like :-<br>
>>> <br>
>>> pcs resource cleanup neutron-server-clone<br>
>>> pcs resource cleanup openstack-nova-api-clone<br>
>>> pcs resource cleanup openstack-nova-consoleauth-clone<br>
>>> pcs resource cleanup openstack-heat-engine-clone<br>
>>> pcs resource cleanup openstack-cinder-api-clone<br>
>>> pcs resource cleanup openstack-glance-registry-clone<br>
>>> pcs resource cleanup httpd-clone<br>
>>> <br>
>>> been working as expected on bare metal<br>
>>> <br>
>>> <br>
>>> Same cluster been setup via QuickStart (Virtual ENV) after bouncing<br>
>>> one of controllers<br>
>>> <br>
>>> included in cluster ignores PCS CLI at least via my experience (<br>
>>> which is obviously limited<br>
>>> <br>
>>> either format of particular commands is wrong for QuickStart )<br>
>>> <br>
>>> I believe that dropping (complete replacing ) instack-virt-setup is<br>
>>> not a good idea in general. Personally, I believe that like in case<br>
>>> with packstack it is always good<br>
>>> <br>
>>> to have VENV configuration been tested before going to bare metal<br>
>>> deployment.<br>
>>> <br>
>>> My major concern is maintenance and disaster recovery tests , rather<br>
>>> then deployment itself . What good is for me TripleO Quickstart<br>
>>> running on bare metal if I cannot replace<br>
>>> <br>
>>> crashed VM Controller just been limited to Services HA ( all 3<br>
>>> Cluster VMs running on single<br>
>>> <br>
>>> bare metal node )<br>
>>> <br>
>>> <br>
>>> Thanks<br>
>>> <br>
>>> Boris.<br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> ------------------------------------------------------------------------<br>
>>> <br>
>>> <br>
>>> <br>
>>> _______________________________________________<br>
>>> rdo-list mailing list<br>
>>> rdo-list@redhat.com <<a href="mailto:rdo-list@redhat.com">mailto:rdo-list@redhat.com</a>><br>
>>> <a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a><br>
>>> <<a href="https://www.redhat.com/mailman/listinfo/rdo-list">https://www.redhat.com/mailman/listinfo/rdo-list</a>><br>
>>> <br>
>>> To unsubscribe: rdo-list-unsubscribe@redhat.com<br>
>>> <<a href="mailto:rdo-list-unsubscribe@redhat.com">mailto:rdo-list-unsubscribe@redhat.com</a>><br>
>>> <br>
>>> <br>
</div>
</span></font></div>
</div>
</body>
</html>