Re: [rdo-list] Newton HA galera-ready dependency error

Just checked what I did to build the image previously. I used export DELOREAN_TRUNK_REPO="http://trunk.rdoproject.org/centos7-newton/curr... Which is the same repo I used to install the Undercloud. Maybe I need to do a yum update in the image with libguestfs-tools prior to building the image? I will rebuild again anyway in case I made an error C On 12/10/2016 10:28, Charles Short wrote: > > Hi, > > Ok, I will rebuild using the undercloud repo and report back. > > Thanks for your help > > Charles > > On 12/10/2016 09:44, Marius Cornea wrote: >> >> Oh, that explains it. It looks that the overcloud image doesn't >> contain the patch. >> >> I'm not familiar with the image build process but according to the >> docs[1] I think the packages get installed from the repo specified by >> export DELOREAN_TRUNK_REPO so maybe you should try to rebuild the >> image and use the same repo as the one set on the undercloud. >> >> [1] >> http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_d... >> >> On Wed, Oct 12, 2016 at 10:30 AM, Charles Short <cems(a)ebi.ac.uk&gt; wrote: >>> >>> Hi, >>> >>> We noticed something that may be the reason for this patch not working >>> which >>> may be related to the way the Undercloud built the image? - >>> >>> This is difference between the puppet files in the undercloud and >>> puppet in the image: >>> >>> [stack@hh-extcl05-undercloud ~]$ grep mysql_short_node_names >>> >>> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>> >>> $galera_node_names_lookup = hiera('mysql_short_node_names', >>> hiera('mysql_node_names', $::hostname)) >>> >>> [root@overcloud-controller-1 puppet]# grep mysql_short_node_names >>> >>> /etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>> >>> (nothing found) >>> >>> >>> On 12/10/2016 09:09, Marius Cornea wrote: >>>> >>>> That's odd. I encountered the same issue and it was caused by missing >>>> this patch. What do you get if you do sudo hiera >>>> mysql_short_node_names on the controller node? >>>> >>>> On Wed, Oct 12, 2016 at 12:48 AM, Charles Short <cems(a)ebi.ac.uk&gt; wrote: >>>>> >>>>> Hi, >>>>> >>>>> puppet-tripleo-5.2.0-0.20161007035759.f32e484.el7.centos.noarch >>>>> >>>>> The patch seems to be already present - >>>>> >>>>> grep short >>>>> >>>>> >>>>> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>>>> >>>>> # short name which is already registered in pacemaker until we get >>>>> around >>>>> $galera_node_names_lookup = hiera('mysql_short_node_names', >>>>> hiera('mysql_node_names', $::hostname)) >>>>> >>>>> Charles >>>>> >>>>> On 11/10/2016 19:39, Marius Cornea wrote: >>>>>> >>>>>> I think the issue is caused by the addresses in wsrep_cluster_address >>>>>> not matching the pacemaker node names: >>>>>> >>>>>> >>>>>> >>>>>> wsrep_cluster_address=gcomm://overcloud-controller-0.internalapi.localdomain,overcloud-controller-1.internalapi.localdomain,overcloud-controller-2.internalapi.localdomain >>>>>> >>>>>> Could you please confirm what version of puppet-tripleo you've got >>>>>> installed on the overcloud nodes and if it contains the following >>>>>> patch: >>>>>> >>>>>> >>>>>> https://review.openstack.org/#/c/382883/1/manifests/profile/pacemaker/dat... >>>>>> >>>>>> Thanks, >>>>>> Marius >>>>>> >>>>>> On Tue, Oct 11, 2016 at 7:42 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>> wrote: >>>>>>> >>>>>>> Ok install finished with same error >>>>>>> The latest pcs status etc >>>>>>> >>>>>>> http://pastebin.com/ZK683gZe >>>>>>> >>>>>>> >>>>>>> On 11/10/2016 17:35, Charles Short wrote: >>>>>>>> >>>>>>>> Deployment almost finished...so >>>>>>>> >>>>>>>> >>>>>>>> http://pastebin.com/zE9B19XB >>>>>>>> >>>>>>>> This shows the pcs status as the deployment nears the end, and pcs >>>>>>>> resource show galera >>>>>>>> >>>>>>>> Charles >>>>>>>> >>>>>>>> On 11/10/2016 16:59, Marius Cornea wrote: >>>>>>>>> >>>>>>>>> Great, thanks for checking this. >>>>>>>>> >>>>>>>>> On Tue, Oct 11, 2016 at 5:58 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Currently having more generic deployment issues (no valid host >>>>>>>>>> found >>>>>>>>>> etc). >>>>>>>>>> I can work around/solve these. >>>>>>>>>> I don't yet have another stack to analyse, but will do soon. >>>>>>>>>> >>>>>>>>>> Charles >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 11/10/2016 16:35, Marius Cornea wrote: >>>>>>>>>>> >>>>>>>>>>> Did it succeed in bringing the Galera nodes to Master? You can >>>>>>>>>>> ssh >>>>>>>>>>> to >>>>>>>>>>> the nodes and run 'pcs resource show galera' even though the >>>>>>>>>>> deployment hasn't finished. I'm interested to see how the >>>>>>>>>>> wsrep_cluster_address is set to see if it's affected by the >>>>>>>>>>> resource >>>>>>>>>>> agent issue described in >>>>>>>>>>> https://bugs.launchpad.net/tripleo/+bug/1628521 >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 11, 2016 at 5:18 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Looks similar to this bug (still waiting on deployment to >>>>>>>>>>>> finish) >>>>>>>>>>>> >>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1368214 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 11/10/2016 15:25, Charles Short wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry for the delay. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Just redeploying to make sure I can repeat the same error. >>>>>>>>>>>>> Should >>>>>>>>>>>>> not >>>>>>>>>>>>> be >>>>>>>>>>>>> long. >>>>>>>>>>>>> >>>>>>>>>>>>> Charles >>>>>>>>>>>>> >>>>>>>>>>>>> On 11/10/2016 14:24, Marius Cornea wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Could you also please paste the output for 'pcs resource show >>>>>>>>>>>>>> galera', >>>>>>>>>>>>>> it looks that all the galera nodes show up as slaves? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Master/Slave Set: galera-master [galera] >>>>>>>>>>>>>> Slaves: [ overcloud-controller-0 >>>>>>>>>>>>>> overcloud-controller-1 >>>>>>>>>>>>>> overcloud-controller-2 ] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Oct 11, 2016 at 2:16 PM, Charles Short >>>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here you are - >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Heat stack error - http://pastebin.com/E8KZa2vE >>>>>>>>>>>>>>> - PCS status - http://pastebin.com/z34gSLq6 >>>>>>>>>>>>>>> - mariadb.log - http://pastebin.com/APFXPBLc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 11/10/2016 12:07, Marius Cornea wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Charles, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Could you please paste the output of 'pcs status' ? The log >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> /var/log/mariadb/mariadb.log might also be a good >>>>>>>>>>>>>>>> indicator. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Oct 11, 2016 at 11:16 AM, Charles Short >>>>>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To add I built my own image from >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> as the images in >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/d... >>>>>>>>>>>>>>>>> caused sporadic ramdisk loading errors (hung at x% loaded >>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>> boot) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Does my image now need to be customised in any way for HA >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> work? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 11/10/2016 09:55, Charles Short wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I am installing Newton with TripleO on baremetal HP >>>>>>>>>>>>>>>>>> blades. >>>>>>>>>>>>>>>>>> I can deploy a single controller stack overcloud no >>>>>>>>>>>>>>>>>> problem, >>>>>>>>>>>>>>>>>> however >>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>> I choose three controllers the deployment fails >>>>>>>>>>>>>>>>>> (including >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml) >>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The heat stack error first complains "Dependency >>>>>>>>>>>>>>>>>> Exec[galera-ready] >>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>> failures" which in turn causes lots of other errors. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have deployed Liberty and Mitaka successfully in the >>>>>>>>>>>>>>>>>> past >>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>> baremetal >>>>>>>>>>>>>>>>>> with three controllers, and this is the first time I have >>>>>>>>>>>>>>>>>> seen >>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> error. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> rdo-list mailing list >>>>>>>>>>>>>>>>> rdo-list(a)redhat.com >>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Charles Short >>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Charles Short >>>>>>>>>> Cloud Engineer >>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>> >>>>>>> -- >>>>>>> Charles Short >>>>>>> Cloud Engineer >>>>>>> Virtualization and Cloud Team >>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>> Tel: +44 (0)1223 494205 >>>>>>> >>>>> -- >>>>> Charles Short >>>>> Cloud Engineer >>>>> Virtualization and Cloud Team >>>>> European Bioinformatics Institute (EMBL-EBI) >>>>> Tel: +44 (0)1223 494205 >>>>> >>> -- >>> Charles Short >>> Cloud Engineer >>> Virtualization and Cloud Team >>> European Bioinformatics Institute (EMBL-EBI) >>> Tel: +44 (0)1223 494205 >>> > -- Charles Short Cloud Engineer Virtualization and Cloud Team European Bioinformatics Institute (EMBL-EBI) Tel: +44 (0)1223 494205