Re: [rdo-list] Newton HA galera-ready dependency error

Hi, We noticed something that may be the reason for this patch not working which may be related to the way the Undercloud built the image? - This is difference between the puppet files in the undercloud and puppet in the image: [stack@hh-extcl05-undercloud ~]$ grep mysql_short_node_names /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp $galera_node_names_lookup = hiera('mysql_short_node_names', hiera('mysql_node_names', $::hostname)) [root@overcloud-controller-1 puppet]# grep mysql_short_node_names /etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp (nothing found) On 12/10/2016 09:09, Marius Cornea wrote: > > That's odd. I encountered the same issue and it was caused by missing > this patch. What do you get if you do sudo hiera > mysql_short_node_names on the controller node? > > On Wed, Oct 12, 2016 at 12:48 AM, Charles Short <cems(a)ebi.ac.uk&gt; wrote: >> >> Hi, >> >> puppet-tripleo-5.2.0-0.20161007035759.f32e484.el7.centos.noarch >> >> The patch seems to be already present - >> >> grep short >> >> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >> >> # short name which is already registered in pacemaker until we get >> around >> $galera_node_names_lookup = hiera('mysql_short_node_names', >> hiera('mysql_node_names', $::hostname)) >> >> Charles >> >> On 11/10/2016 19:39, Marius Cornea wrote: >>> >>> I think the issue is caused by the addresses in wsrep_cluster_address >>> not matching the pacemaker node names: >>> >>> >>> wsrep_cluster_address=gcomm://overcloud-controller-0.internalapi.localdomain,overcloud-controller-1.internalapi.localdomain,overcloud-controller-2.internalapi.localdomain >>> >>> Could you please confirm what version of puppet-tripleo you've got >>> installed on the overcloud nodes and if it contains the following >>> patch: >>> >>> https://review.openstack.org/#/c/382883/1/manifests/profile/pacemaker/dat... >>> >>> Thanks, >>> Marius >>> >>> On Tue, Oct 11, 2016 at 7:42 PM, Charles Short <cems(a)ebi.ac.uk&gt; wrote: >>>> >>>> Ok install finished with same error >>>> The latest pcs status etc >>>> >>>> http://pastebin.com/ZK683gZe >>>> >>>> >>>> On 11/10/2016 17:35, Charles Short wrote: >>>>> >>>>> Deployment almost finished...so >>>>> >>>>> >>>>> http://pastebin.com/zE9B19XB >>>>> >>>>> This shows the pcs status as the deployment nears the end, and pcs >>>>> resource show galera >>>>> >>>>> Charles >>>>> >>>>> On 11/10/2016 16:59, Marius Cornea wrote: >>>>>> >>>>>> Great, thanks for checking this. >>>>>> >>>>>> On Tue, Oct 11, 2016 at 5:58 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>> wrote: >>>>>>> >>>>>>> Currently having more generic deployment issues (no valid host found >>>>>>> etc). >>>>>>> I can work around/solve these. >>>>>>> I don't yet have another stack to analyse, but will do soon. >>>>>>> >>>>>>> Charles >>>>>>> >>>>>>> >>>>>>> On 11/10/2016 16:35, Marius Cornea wrote: >>>>>>>> >>>>>>>> Did it succeed in bringing the Galera nodes to Master? You can ssh >>>>>>>> to >>>>>>>> the nodes and run 'pcs resource show galera' even though the >>>>>>>> deployment hasn't finished. I'm interested to see how the >>>>>>>> wsrep_cluster_address is set to see if it's affected by the >>>>>>>> resource >>>>>>>> agent issue described in >>>>>>>> https://bugs.launchpad.net/tripleo/+bug/1628521 >>>>>>>> >>>>>>>> On Tue, Oct 11, 2016 at 5:18 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Looks similar to this bug (still waiting on deployment to finish) >>>>>>>>> >>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1368214 >>>>>>>>> >>>>>>>>> >>>>>>>>> On 11/10/2016 15:25, Charles Short wrote: >>>>>>>>>> >>>>>>>>>> Sorry for the delay. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Just redeploying to make sure I can repeat the same error. Should >>>>>>>>>> not >>>>>>>>>> be >>>>>>>>>> long. >>>>>>>>>> >>>>>>>>>> Charles >>>>>>>>>> >>>>>>>>>> On 11/10/2016 14:24, Marius Cornea wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Could you also please paste the output for 'pcs resource show >>>>>>>>>>> galera', >>>>>>>>>>> it looks that all the galera nodes show up as slaves? >>>>>>>>>>> >>>>>>>>>>> Master/Slave Set: galera-master [galera] >>>>>>>>>>> Slaves: [ overcloud-controller-0 >>>>>>>>>>> overcloud-controller-1 >>>>>>>>>>> overcloud-controller-2 ] >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 11, 2016 at 2:16 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Here you are - >>>>>>>>>>>> >>>>>>>>>>>> - Heat stack error - http://pastebin.com/E8KZa2vE >>>>>>>>>>>> - PCS status - http://pastebin.com/z34gSLq6 >>>>>>>>>>>> - mariadb.log - http://pastebin.com/APFXPBLc >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> >>>>>>>>>>>> Charles >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 11/10/2016 12:07, Marius Cornea wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Charles, >>>>>>>>>>>>> >>>>>>>>>>>>> Could you please paste the output of 'pcs status' ? The log in >>>>>>>>>>>>> /var/log/mariadb/mariadb.log might also be a good indicator. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Oct 11, 2016 at 11:16 AM, Charles Short >>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> To add I built my own image from >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> as the images in >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/d... >>>>>>>>>>>>>> caused sporadic ramdisk loading errors (hung at x% loaded on >>>>>>>>>>>>>> boot) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Does my image now need to be customised in any way for HA to >>>>>>>>>>>>>> work? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Charles >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11/10/2016 09:55, Charles Short wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am installing Newton with TripleO on baremetal HP blades. >>>>>>>>>>>>>>> I can deploy a single controller stack overcloud no problem, >>>>>>>>>>>>>>> however >>>>>>>>>>>>>>> when >>>>>>>>>>>>>>> I choose three controllers the deployment fails (including >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml) >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The heat stack error first complains "Dependency >>>>>>>>>>>>>>> Exec[galera-ready] >>>>>>>>>>>>>>> has >>>>>>>>>>>>>>> failures" which in turn causes lots of other errors. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have deployed Liberty and Mitaka successfully in the past >>>>>>>>>>>>>>> on >>>>>>>>>>>>>>> baremetal >>>>>>>>>>>>>>> with three controllers, and this is the first time I have >>>>>>>>>>>>>>> seen >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> error. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> rdo-list mailing list >>>>>>>>>>>>>> rdo-list(a)redhat.com >>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>>>>>>>>> >>>>>>>>>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Charles Short >>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Charles Short >>>>>>>>> Cloud Engineer >>>>>>>>> Virtualization and Cloud Team >>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>> >>>>>>> -- >>>>>>> Charles Short >>>>>>> Cloud Engineer >>>>>>> Virtualization and Cloud Team >>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>> Tel: +44 (0)1223 494205 >>>>>>> >>>> -- >>>> Charles Short >>>> Cloud Engineer >>>> Virtualization and Cloud Team >>>> European Bioinformatics Institute (EMBL-EBI) >>>> Tel: +44 (0)1223 494205 >>>> >> -- >> Charles Short >> Cloud Engineer >> Virtualization and Cloud Team >> European Bioinformatics Institute (EMBL-EBI) >> Tel: +44 (0)1223 494205 >> -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205