[rdo-list] Instack-virt-setup vs TripleO QuickStart in regards of managing HA PCS/Corosync cluster via pcs CLI

Raoul Scarazzini rasca at redhat.com
Wed Aug 24 06:56:01 UTC 2016


What I can say for sure at the moment is that it's really hard to follow
your quoting :)

Still some info are missing:

- Which version are you using? The latest? So are you installing from
master?

- Are the two versions the same while using quickstart and
instack-virt-setup?

- Do we have some logs to look at?

Again, I think here we're hitting a version issure.

Thanks,

-- 
Raoul Scarazzini
rasca at redhat.com

On 23/08/2016 19:42, Boris Derzhavets wrote:
> 
> 
> 
> ------------------------------------------------------------------------
> *From:* Raoul Scarazzini <rasca at redhat.com>
> *Sent:* Tuesday, August 23, 2016 11:29 AM
> *To:* Boris Derzhavets; Wesley Hayutin; Attila Darazs
> *Cc:* rdo-list
> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in
> regards of managing HA PCS/Corosync cluster via pcs CLI
>  
> Hi Boris,
> so, for what I see the pcs commands that stops and starts the cluster on
> the rebooted node should not be used. It can happen that a service fails
> to start but we need to investigate why from the logs.
> 
> Remember that cleaning up resources can be useful if we know what
> happened, but using it repeatedly makes no sense. In addition remember
> that you can use just "pcs resource cleanup" to cleanup the entire
> cluster status and in some way "start from the beginning".
> 
> Now, about this specific problem we need to understand what is happening
> here. Correct me if I'm wrong:
> 
> 1) We have a clean env in which we reboot a node;
>     That is correct
> 
> 2) The nodes comes up, but some resources fails;
>     All resources fail
> 
> 3) After some cleanups the env becomes clean again;
> 
>     a)   If VENV is setup by instack-virt-setup ( official guide )
>           Mentioned script start.sh works right a way . It comes as well
> from official guide.
> 
>      b) if VENV is setup by Tripleo QuickStart ( where undecloud.qcow2
> gets uploaded
>           to libvirt pool already having overcloud images integrated per
> Jon's Video explanation
>           QuickStart CI  vs Tripleo CI  )
>           then ( via my experience )   before attempting start.sh I
> MUST  restart PCS Cluster
>           on bounced Controller-X , then invoke `. ./start.sh`  ( not
> simply ./start.sh )
>               Pretty often second run start.sh is required from another
> controller-Y.
>           Some times I cannot fix it in script mode and have manually
> run commands giving
>           delay more the 10 sec.  So finally ( about 25 tests passed) I
> get `pcs status` OK.
>           In other words all service are up and running on every
> controller-X,Y,Z
> 
>   Details :-
>   http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html
>      
> <http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html>
> 	
> Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA
> Controller's Cluster failover
> <http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html>
> bderzhavets.blogspot.ru
> 
> Is this the sequence of operations you are using? Is the problem
> systematic and can we reproduce it?
>>
> YES
>>
> Can we grab sosreports from the
> machine involved?
>>
> Instruct me how to do this ?
>>
> Most important question: which OpenStack version are
> you testing?
>>
> Mitaka stable  :-
> 
> [tripleo-quickstart at stack] $ bash quickstart --config ./ha.yml $VIRTHOST
> By default , no --release specified Mitaka Delorean trunks get selected
> Just check|/etc/yum.repos.d/| for delorean.repos
> quickstart places on   undercloud  when it exits asking to to connect to
> undercloud
>>
> Boris
> -- 
> Raoul Scarazzini
> rasca at redhat.com
> 
> 
> On 22/08/2016 13:49, Boris Derzhavets wrote:
>> 
>> Sorry , for my English
>> 
>> I was also keeping (not kept ) track on Galera DB via `clustercheck`
>> 
>> either I just kept.
>> 
>> 
>> Boris
>> 
>> 
>> ------------------------------------------------------------------------
>> *From:* rdo-list-bounces at redhat.com <rdo-list-bounces at redhat.com> on
>> behalf of Boris Derzhavets <bderzhavets at hotmail.com>
>> *Sent:* Monday, August 22, 2016 7:29 AM
>> *To:* Raoul Scarazzini; Wesley Hayutin; Attila Darazs
>> *Cc:* rdo-list
>> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in
>> regards of managing HA PCS/Corosync cluster via pcs CLI
>>  
>> 
>> 
>> 
>> 
>> ------------------------------------------------------------------------
>> *From:* Raoul Scarazzini <rasca at redhat.com>
>> *Sent:* Monday, August 22, 2016 3:51 AM
>> *To:* Wesley Hayutin; Boris Derzhavets; Attila Darazs
>> *Cc:* David Moreau Simard; rdo-list
>> *Subject:* Re: [rdo-list] Instack-virt-setup vs TripleO QuickStart in
>> regards of managing HA PCS/Corosync cluster via pcs CLI
>>  
>> Hi everybody,
>> sorry for the late response but I was on PTO. I don't understand the
>> meaning of the cleanup commands, but maybe it's just because I'm not
>> getting the whole picture.
>> 
>>>
>> I have to confirm that fault was mine PCS CLI is working on TripeO
>> QuickStart
>> but requires pcs cluster restart on particular node which  went down
>> via ` nova stop controller-X`  and was brought up via `nova start
>> controller-X`
>> Details here :-
>> 
>> http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html
> 
> <http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html>
> 	
> Xen Virtualization on Linux and Solaris: Emulation Triple0 QuickStart HA
> Controller's Cluster failover
> <http://bderzhavets.blogspot.ru/2016/08/emulation-rdo-triple0-quickstart-ha.html>
> bderzhavets.blogspot.ru
> 
> 
>> 
>> VENV been set up with instack-virt-setup doesn't require ( on bounced
>> Controller node )
>> 
>> # pcs cluster stop
>> # pcs cluster start
>> 
>> Before issuing start.sh
>> 
>> #!/bash -x
>> pcs resource cleanup rabbitmq-clone ;
>> sleep 10
>> pcs resource cleanup neutron-server-clone ;
>> sleep 10
>> pcs resource cleanup openstack-nova-api-clone ;
>> sleep 10
>> pcs resource cleanup openstack-nova-consoleauth-clone ;
>> sleep 10
>> pcs resource cleanup openstack-heat-engine-clone ;
>> sleep 10
>> pcs resource cleanup openstack-cinder-api-clone ;
>> sleep 10
>> pcs resource cleanup openstack-glance-registry-clone ;
>> sleep 10
>> pcs resource cleanup httpd-clone ;
>> 
>> # .  ./start.sh  
>> 
>> In worse case scenario I have to issue start.sh   twice from different
>> Controllers
>> pcs resource cleanup openstack-nova-api-clone  attempts to start
>> corresponding
>> service , which is down at the moment.  In fact two cleanups above start all
>> Nova Services   && one neutron cleanup starts all neutron agents as well.
>> I was also kept   track of Galera DB via `clustercheck`
>> 
>> Thanks.
>> Boris
>>>
>> 
>> 
>> I guess we're hitting a version problem here: if you deploy the actual
>> master (i.e. with quickstart) you'll get the environment with the
>> constraints limited to the core services because of [1] and [2] (so none
>> of the mentioned services exists in the cluster configuration).
>> 
>> Hope this helps,
>> 
>> [1] https://review.openstack.org/#/c/314208/
>> [2] https://review.openstack.org/#/c/342650/
>> 
>> -- 
>> Raoul Scarazzini
>> rasca at redhat.com
>> 
>> On 08/08/2016 14:43, Wesley Hayutin wrote:
>>> Attila, Raoul
>>> Can you please investigate this issue.
>>> 
>>> Thanks!
>>> 
>>> On Sun, Aug 7, 2016 at 3:52 AM, Boris Derzhavets
>>> <bderzhavets at hotmail.com <mailto:bderzhavets at hotmail.com>> wrote:
>>> 
>>>     TripleO HA Controller been installed via instack-virt-setup  has PCS
>>>     CLI like :-
>>> 
>>>     pcs resource cleanup neutron-server-clone
>>>     pcs resource cleanup openstack-nova-api-clone
>>>     pcs resource cleanup openstack-nova-consoleauth-clone
>>>     pcs resource cleanup openstack-heat-engine-clone
>>>     pcs resource cleanup openstack-cinder-api-clone
>>>     pcs resource cleanup openstack-glance-registry-clone
>>>     pcs resource cleanup httpd-clone
>>> 
>>>     been working  as expected on bare metal
>>> 
>>> 
>>>     Same cluster been setup via QuickStart  (Virtual ENV) after bouncing
>>>     one of controllers
>>> 
>>>     included in cluster ignores PCS CLI at least via my experience (
>>>     which is obviously limited
>>> 
>>>     either format of particular commands is wrong for QuickStart )
>>> 
>>>     I believe that dropping (complete replacing ) instack-virt-setup is
>>>     not a good idea in general. Personally, I believe that like in case
>>>     with packstack it is always good
>>> 
>>>     to have VENV configuration been tested before going to bare metal
>>>     deployment.
>>> 
>>>     My major concern is maintenance and disaster recovery tests , rather
>>>     then deployment itself . What good is for me TripleO Quickstart
>>>     running on bare metal if I cannot replace
>>> 
>>>     crashed VM Controller just been limited to Services HA ( all 3
>>>     Cluster VMs running on single
>>> 
>>>     bare metal node )
>>> 
>>> 
>>>     Thanks
>>> 
>>>     Boris.
>>> 
>>> 
>>> 
>>> 
>>> 
>>>     ------------------------------------------------------------------------
>>> 
>>> 
>>> 
>>>     _______________________________________________
>>>     rdo-list mailing list
>>>     rdo-list at redhat.com <mailto:rdo-list at redhat.com>
>>>     https://www.redhat.com/mailman/listinfo/rdo-list
>>>     <https://www.redhat.com/mailman/listinfo/rdo-list>
>>> 
>>>     To unsubscribe: rdo-list-unsubscribe at redhat.com
>>>     <mailto:rdo-list-unsubscribe at redhat.com>
>>> 
>>> 




More information about the dev mailing list