Re: [rdo-list] Newton HA galera-ready dependency error

Hi, Ok, I will rebuild using the undercloud repo and report back. Thanks for your help Charles On 12/10/2016 09:44, Marius Cornea wrote: > Oh, that explains it. It looks that the overcloud image doesn't > contain the patch. > > I'm not familiar with the image build process but according to the > docs[1] I think the packages get installed from the repo specified by > export DELOREAN_TRUNK_REPO so maybe you should try to rebuild the > image and use the same repo as the one set on the undercloud. > > [1] > http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_d... > > On Wed, Oct 12, 2016 at 10:30 AM, Charles Short <cems(a)ebi.ac.uk&gt; wrote: >> Hi, >> >> We noticed something that may be the reason for this patch not >> working which >> may be related to the way the Undercloud built the image? - >> >> This is difference between the puppet files in the undercloud and >> puppet in the image: >> >> [stack@hh-extcl05-undercloud ~]$ grep mysql_short_node_names >> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >> >> >> $galera_node_names_lookup = hiera('mysql_short_node_names', >> hiera('mysql_node_names', $::hostname)) >> >> [root@overcloud-controller-1 puppet]# grep mysql_short_node_names >> /etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >> >> >> (nothing found) >> >> >> On 12/10/2016 09:09, Marius Cornea wrote: >>> That's odd. I encountered the same issue and it was caused by missing >>> this patch. What do you get if you do sudo hiera >>> mysql_short_node_names on the controller node? >>> >>> On Wed, Oct 12, 2016 at 12:48 AM, Charles Short <cems(a)ebi.ac.uk&gt; >>> wrote: >>>> Hi, >>>> >>>> puppet-tripleo-5.2.0-0.20161007035759.f32e484.el7.centos.noarch >>>> >>>> The patch seems to be already present - >>>> >>>> grep short >>>> >>>> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>>> >>>> >>>> # short name which is already registered in pacemaker until we >>>> get >>>> around >>>> $galera_node_names_lookup = hiera('mysql_short_node_names', >>>> hiera('mysql_node_names', $::hostname)) >>>> >>>> Charles >>>> >>>> On 11/10/2016 19:39, Marius Cornea wrote: >>>>> I think the issue is caused by the addresses in >>>>> wsrep_cluster_address >>>>> not matching the pacemaker node names: >>>>> >>>>> >>>>> wsrep_cluster_address=gcomm://overcloud-controller-0.internalapi.localdomain,overcloud-controller-1.internalapi.localdomain,overcloud-controller-2.internalapi.localdomain >>>>> >>>>> >>>>> Could you please confirm what version of puppet-tripleo you've got >>>>> installed on the overcloud nodes and if it contains the following >>>>> patch: >>>>> >>>>> https://review.openstack.org/#/c/382883/1/manifests/profile/pacemaker/dat... >>>>> >>>>> >>>>> Thanks, >>>>> Marius >>>>> >>>>> On Tue, Oct 11, 2016 at 7:42 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>> wrote: >>>>>> Ok install finished with same error >>>>>> The latest pcs status etc >>>>>> >>>>>> http://pastebin.com/ZK683gZe >>>>>> >>>>>> >>>>>> On 11/10/2016 17:35, Charles Short wrote: >>>>>>> Deployment almost finished...so >>>>>>> >>>>>>> >>>>>>> http://pastebin.com/zE9B19XB >>>>>>> >>>>>>> This shows the pcs status as the deployment nears the end, and pcs >>>>>>> resource show galera >>>>>>> >>>>>>> Charles >>>>>>> >>>>>>> On 11/10/2016 16:59, Marius Cornea wrote: >>>>>>>> Great, thanks for checking this. >>>>>>>> >>>>>>>> On Tue, Oct 11, 2016 at 5:58 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>> wrote: >>>>>>>>> Currently having more generic deployment issues (no valid >>>>>>>>> host found >>>>>>>>> etc). >>>>>>>>> I can work around/solve these. >>>>>>>>> I don't yet have another stack to analyse, but will do soon. >>>>>>>>> >>>>>>>>> Charles >>>>>>>>> >>>>>>>>> >>>>>>>>> On 11/10/2016 16:35, Marius Cornea wrote: >>>>>>>>>> Did it succeed in bringing the Galera nodes to Master? You >>>>>>>>>> can ssh >>>>>>>>>> to >>>>>>>>>> the nodes and run 'pcs resource show galera' even though the >>>>>>>>>> deployment hasn't finished. I'm interested to see how the >>>>>>>>>> wsrep_cluster_address is set to see if it's affected by the >>>>>>>>>> resource >>>>>>>>>> agent issue described in >>>>>>>>>> https://bugs.launchpad.net/tripleo/+bug/1628521 >>>>>>>>>> >>>>>>>>>> On Tue, Oct 11, 2016 at 5:18 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>>>> wrote: >>>>>>>>>>> Looks similar to this bug (still waiting on deployment to >>>>>>>>>>> finish) >>>>>>>>>>> >>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1368214 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 11/10/2016 15:25, Charles Short wrote: >>>>>>>>>>>> Sorry for the delay. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Just redeploying to make sure I can repeat the same error. >>>>>>>>>>>> Should >>>>>>>>>>>> not >>>>>>>>>>>> be >>>>>>>>>>>> long. >>>>>>>>>>>> >>>>>>>>>>>> Charles >>>>>>>>>>>> >>>>>>>>>>>> On 11/10/2016 14:24, Marius Cornea wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Could you also please paste the output for 'pcs resource >>>>>>>>>>>>> show >>>>>>>>>>>>> galera', >>>>>>>>>>>>> it looks that all the galera nodes show up as slaves? >>>>>>>>>>>>> >>>>>>>>>>>>> Master/Slave Set: galera-master [galera] >>>>>>>>>>>>> Slaves: [ overcloud-controller-0 >>>>>>>>>>>>> overcloud-controller-1 >>>>>>>>>>>>> overcloud-controller-2 ] >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Oct 11, 2016 at 2:16 PM, Charles Short >>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here you are - >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Heat stack error - http://pastebin.com/E8KZa2vE >>>>>>>>>>>>>> - PCS status - http://pastebin.com/z34gSLq6 >>>>>>>>>>>>>> - mariadb.log - http://pastebin.com/APFXPBLc >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> Charles >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11/10/2016 12:07, Marius Cornea wrote: >>>>>>>>>>>>>>> Hi Charles, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Could you please paste the output of 'pcs status' ? The >>>>>>>>>>>>>>> log in >>>>>>>>>>>>>>> /var/log/mariadb/mariadb.log might also be a good >>>>>>>>>>>>>>> indicator. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Oct 11, 2016 at 11:16 AM, Charles Short >>>>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> To add I built my own image from >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> as the images in >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/d... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> caused sporadic ramdisk loading errors (hung at x% >>>>>>>>>>>>>>>> loaded on >>>>>>>>>>>>>>>> boot) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Does my image now need to be customised in any way for >>>>>>>>>>>>>>>> HA to >>>>>>>>>>>>>>>> work? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 11/10/2016 09:55, Charles Short wrote: >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am installing Newton with TripleO on baremetal HP >>>>>>>>>>>>>>>>> blades. >>>>>>>>>>>>>>>>> I can deploy a single controller stack overcloud no >>>>>>>>>>>>>>>>> problem, >>>>>>>>>>>>>>>>> however >>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>> I choose three controllers the deployment fails >>>>>>>>>>>>>>>>> (including >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The heat stack error first complains "Dependency >>>>>>>>>>>>>>>>> Exec[galera-ready] >>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>> failures" which in turn causes lots of other errors. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have deployed Liberty and Mitaka successfully in >>>>>>>>>>>>>>>>> the past >>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>> baremetal >>>>>>>>>>>>>>>>> with three controllers, and this is the first time I >>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>> seen >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> error. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> rdo-list mailing list >>>>>>>>>>>>>>>> rdo-list(a)redhat.com >>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Charles Short >>>>>>>>>>> Cloud Engineer >>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Charles Short >>>>>>>>> Cloud Engineer >>>>>>>>> Virtualization and Cloud Team >>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>> >>>>>> -- >>>>>> Charles Short >>>>>> Cloud Engineer >>>>>> Virtualization and Cloud Team >>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>> Tel: +44 (0)1223 494205 >>>>>> >>>> -- >>>> Charles Short >>>> Cloud Engineer >>>> Virtualization and Cloud Team >>>> European Bioinformatics Institute (EMBL-EBI) >>>> Tel: +44 (0)1223 494205 >>>> >> -- >> Charles Short >> Cloud Engineer >> Virtualization and Cloud Team >> European Bioinformatics Institute (EMBL-EBI) >> Tel: +44 (0)1223 494205 >>