Re: [rdo-list] Newton HA galera-ready dependency error

Ok, I updated puppet-tripleo as suggested with virt-customize and verified that the patch was indeed updated (using guestmount ) I am now redeploying Charles On 12/10/2016 10:56, Marius Cornea wrote: > I see that in that repo puppet-tripleo got updated on Friday, > 2016-10-07. If the repo file is present inside the image you could > update puppet-tripleo with virt-customize, then upload the image the > the undercloud Glance with 'openstack overcloud image upload > --update-existing' and redeploy. > > On Wed, Oct 12, 2016 at 11:45 AM, Charles Short <cems(a)ebi.ac.uk&gt; wrote: >> Just checked what I did to build the image previously. >> >> I used >> export >> DELOREAN_TRUNK_REPO="http://trunk.rdoproject.org/centos7-newton/curr... >> >> >> Which is the same repo I used to install the Undercloud. >> Maybe I need to do a yum update in the image with libguestfs-tools >> prior to >> building the image? >> >> I will rebuild again anyway in case I made an error >> >> C >> >> >> >> On 12/10/2016 10:28, Charles Short wrote: >>> Hi, >>> >>> Ok, I will rebuild using the undercloud repo and report back. >>> >>> Thanks for your help >>> >>> Charles >>> >>> On 12/10/2016 09:44, Marius Cornea wrote: >>>> Oh, that explains it. It looks that the overcloud image doesn't >>>> contain the patch. >>>> >>>> I'm not familiar with the image build process but according to the >>>> docs[1] I think the packages get installed from the repo specified by >>>> export DELOREAN_TRUNK_REPO so maybe you should try to rebuild the >>>> image and use the same repo as the one set on the undercloud. >>>> >>>> [1] >>>> http://docs.openstack.org/developer/tripleo-docs/basic_deployment/basic_d... >>>> >>>> >>>> On Wed, Oct 12, 2016 at 10:30 AM, Charles Short <cems(a)ebi.ac.uk&gt; >>>> wrote: >>>>> Hi, >>>>> >>>>> We noticed something that may be the reason for this patch not >>>>> working >>>>> which >>>>> may be related to the way the Undercloud built the image? - >>>>> >>>>> This is difference between the puppet files in the undercloud and >>>>> puppet in the image: >>>>> >>>>> [stack@hh-extcl05-undercloud ~]$ grep mysql_short_node_names >>>>> >>>>> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>>>> >>>>> >>>>> $galera_node_names_lookup = hiera('mysql_short_node_names', >>>>> hiera('mysql_node_names', $::hostname)) >>>>> >>>>> [root@overcloud-controller-1 puppet]# grep mysql_short_node_names >>>>> >>>>> /etc/puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>>>> >>>>> >>>>> (nothing found) >>>>> >>>>> >>>>> On 12/10/2016 09:09, Marius Cornea wrote: >>>>>> That's odd. I encountered the same issue and it was caused by >>>>>> missing >>>>>> this patch. What do you get if you do sudo hiera >>>>>> mysql_short_node_names on the controller node? >>>>>> >>>>>> On Wed, Oct 12, 2016 at 12:48 AM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>> wrote: >>>>>>> Hi, >>>>>>> >>>>>>> puppet-tripleo-5.2.0-0.20161007035759.f32e484.el7.centos.noarch >>>>>>> >>>>>>> The patch seems to be already present - >>>>>>> >>>>>>> grep short >>>>>>> >>>>>>> >>>>>>> /usr/share/openstack-puppet/modules/tripleo/manifests/profile/pacemaker/database/mysql.pp >>>>>>> >>>>>>> >>>>>>> # short name which is already registered in pacemaker >>>>>>> until we get >>>>>>> around >>>>>>> $galera_node_names_lookup = hiera('mysql_short_node_names', >>>>>>> hiera('mysql_node_names', $::hostname)) >>>>>>> >>>>>>> Charles >>>>>>> >>>>>>> On 11/10/2016 19:39, Marius Cornea wrote: >>>>>>>> I think the issue is caused by the addresses in >>>>>>>> wsrep_cluster_address >>>>>>>> not matching the pacemaker node names: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> wsrep_cluster_address=gcomm://overcloud-controller-0.internalapi.localdomain,overcloud-controller-1.internalapi.localdomain,overcloud-controller-2.internalapi.localdomain >>>>>>>> >>>>>>>> >>>>>>>> Could you please confirm what version of puppet-tripleo you've >>>>>>>> got >>>>>>>> installed on the overcloud nodes and if it contains the following >>>>>>>> patch: >>>>>>>> >>>>>>>> >>>>>>>> https://review.openstack.org/#/c/382883/1/manifests/profile/pacemaker/dat... >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Marius >>>>>>>> >>>>>>>> On Tue, Oct 11, 2016 at 7:42 PM, Charles Short <cems(a)ebi.ac.uk&gt; >>>>>>>> wrote: >>>>>>>>> Ok install finished with same error >>>>>>>>> The latest pcs status etc >>>>>>>>> >>>>>>>>> http://pastebin.com/ZK683gZe >>>>>>>>> >>>>>>>>> >>>>>>>>> On 11/10/2016 17:35, Charles Short wrote: >>>>>>>>>> Deployment almost finished...so >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> http://pastebin.com/zE9B19XB >>>>>>>>>> >>>>>>>>>> This shows the pcs status as the deployment nears the end, >>>>>>>>>> and pcs >>>>>>>>>> resource show galera >>>>>>>>>> >>>>>>>>>> Charles >>>>>>>>>> >>>>>>>>>> On 11/10/2016 16:59, Marius Cornea wrote: >>>>>>>>>>> Great, thanks for checking this. >>>>>>>>>>> >>>>>>>>>>> On Tue, Oct 11, 2016 at 5:58 PM, Charles Short >>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>> wrote: >>>>>>>>>>>> Currently having more generic deployment issues (no valid >>>>>>>>>>>> host >>>>>>>>>>>> found >>>>>>>>>>>> etc). >>>>>>>>>>>> I can work around/solve these. >>>>>>>>>>>> I don't yet have another stack to analyse, but will do soon. >>>>>>>>>>>> >>>>>>>>>>>> Charles >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 11/10/2016 16:35, Marius Cornea wrote: >>>>>>>>>>>>> Did it succeed in bringing the Galera nodes to Master? >>>>>>>>>>>>> You can >>>>>>>>>>>>> ssh >>>>>>>>>>>>> to >>>>>>>>>>>>> the nodes and run 'pcs resource show galera' even though the >>>>>>>>>>>>> deployment hasn't finished. I'm interested to see how the >>>>>>>>>>>>> wsrep_cluster_address is set to see if it's affected by the >>>>>>>>>>>>> resource >>>>>>>>>>>>> agent issue described in >>>>>>>>>>>>> https://bugs.launchpad.net/tripleo/+bug/1628521 >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Oct 11, 2016 at 5:18 PM, Charles Short >>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> Looks similar to this bug (still waiting on deployment to >>>>>>>>>>>>>> finish) >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1368214 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11/10/2016 15:25, Charles Short wrote: >>>>>>>>>>>>>>> Sorry for the delay. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just redeploying to make sure I can repeat the same error. >>>>>>>>>>>>>>> Should >>>>>>>>>>>>>>> not >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>> long. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 11/10/2016 14:24, Marius Cornea wrote: >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Could you also please paste the output for 'pcs >>>>>>>>>>>>>>>> resource show >>>>>>>>>>>>>>>> galera', >>>>>>>>>>>>>>>> it looks that all the galera nodes show up as slaves? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Master/Slave Set: galera-master [galera] >>>>>>>>>>>>>>>> Slaves: [ overcloud-controller-0 >>>>>>>>>>>>>>>> overcloud-controller-1 >>>>>>>>>>>>>>>> overcloud-controller-2 ] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Oct 11, 2016 at 2:16 PM, Charles Short >>>>>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Here you are - >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - Heat stack error - >>>>>>>>>>>>>>>>> http://pastebin.com/E8KZa2vE >>>>>>>>>>>>>>>>> - PCS status - http://pastebin.com/z34gSLq6 >>>>>>>>>>>>>>>>> - mariadb.log - http://pastebin.com/APFXPBLc >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 11/10/2016 12:07, Marius Cornea wrote: >>>>>>>>>>>>>>>>>> Hi Charles, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Could you please paste the output of 'pcs status' ? >>>>>>>>>>>>>>>>>> The log >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> /var/log/mariadb/mariadb.log might also be a good >>>>>>>>>>>>>>>>>> indicator. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Tue, Oct 11, 2016 at 11:16 AM, Charles Short >>>>>>>>>>>>>>>>>> <cems(a)ebi.ac.uk&gt; >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> To add I built my own image from >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> as the images in >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> http://buildlogs.centos.org/centos/7/cloud/x86_64/tripleo_images/newton/d... >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> caused sporadic ramdisk loading errors (hung at x% >>>>>>>>>>>>>>>>>>> loaded >>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>> boot) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Does my image now need to be customised in any way >>>>>>>>>>>>>>>>>>> for HA >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> work? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 11/10/2016 09:55, Charles Short wrote: >>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I am installing Newton with TripleO on baremetal HP >>>>>>>>>>>>>>>>>>>> blades. >>>>>>>>>>>>>>>>>>>> I can deploy a single controller stack overcloud no >>>>>>>>>>>>>>>>>>>> problem, >>>>>>>>>>>>>>>>>>>> however >>>>>>>>>>>>>>>>>>>> when >>>>>>>>>>>>>>>>>>>> I choose three controllers the deployment fails >>>>>>>>>>>>>>>>>>>> (including >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The heat stack error first complains "Dependency >>>>>>>>>>>>>>>>>>>> Exec[galera-ready] >>>>>>>>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>>> failures" which in turn causes lots of other errors. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I have deployed Liberty and Mitaka successfully in >>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>> past >>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>> baremetal >>>>>>>>>>>>>>>>>>>> with three controllers, and this is the first time >>>>>>>>>>>>>>>>>>>> I have >>>>>>>>>>>>>>>>>>>> seen >>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> error. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Charles >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> rdo-list mailing list >>>>>>>>>>>>>>>>>>> rdo-list(a)redhat.com >>>>>>>>>>>>>>>>>>> https://www.redhat.com/mailman/listinfo/rdo-list >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Charles Short >>>>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Charles Short >>>>>>>>>>>> Cloud Engineer >>>>>>>>>>>> Virtualization and Cloud Team >>>>>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Charles Short >>>>>>>>> Cloud Engineer >>>>>>>>> Virtualization and Cloud Team >>>>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>>>> Tel: +44 (0)1223 494205 >>>>>>>>> >>>>>>> -- >>>>>>> Charles Short >>>>>>> Cloud Engineer >>>>>>> Virtualization and Cloud Team >>>>>>> European Bioinformatics Institute (EMBL-EBI) >>>>>>> Tel: +44 (0)1223 494205 >>>>>>> >>>>> -- >>>>> Charles Short >>>>> Cloud Engineer >>>>> Virtualization and Cloud Team >>>>> European Bioinformatics Institute (EMBL-EBI) >>>>> Tel: +44 (0)1223 494205 >>>>> >> -- >> Charles Short >> Cloud Engineer >> Virtualization and Cloud Team >> European Bioinformatics Institute (EMBL-EBI) >> Tel: +44 (0)1223 494205 >>