<html><body><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000"><div>Hi,</div><div><br></div><div>When I am trying to create new projects and one network for each projects after 70 or 80 projects and networks in OC. Controller HA availablitliy failed with below error.</div><div><br></div><div> [stack@director LogTool_Python2]$ ssh heat-admin@192.168.100.28 "sudo pcs status"<br>Cluster name: tripleo_cluster<br>Stack: corosync<br>Current DC: overcloud-controller-1 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum<br>Last updated: Thu Jul 23 17:00:22 2020<br>Last change: Wed Jul 22 14:35:34 2020 by hacluster via crmd on overcloud-controller-2<br><br>12 nodes configured<br>37 resources configured<br><br>Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]<br>GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]<br><br>Full list of resources:<br><br> Docker container set: rabbitmq-bundle [192.168.100.1:8787/tripleorocky/centos-binary-rabbitmq:pcmklatest]<br> rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0<br> rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1<br> rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2<br> Docker container set: galera-bundle [192.168.100.1:8787/tripleorocky/centos-binary-mariadb:pcmklatest]<br> galera-bundle-0 (ocf::heartbeat:galera): Master overcloud-controller-0<br> galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-controller-1<br> galera-bundle-2 (ocf::heartbeat:galera): FAILED Master overcloud-controller-2 (blocked)<br> Docker container set: redis-bundle [192.168.100.1:8787/tripleorocky/centos-binary-redis:pcmklatest]<br> redis-bundle-0 (ocf::heartbeat:redis): Master overcloud-controller-0<br> redis-bundle-1 (ocf::heartbeat:redis): Slave overcloud-controller-1<br> redis-bundle-2 (ocf::heartbeat:redis): Slave overcloud-controller-2<br> ip-192.168.100.98 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0<br> ip-10.10.0.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1<br> ip-192.168.102.185 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2<br> ip-192.168.102.116 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0<br> ip-192.168.103.187 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1<br> ip-192.168.104.127 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2<br> Docker container set: haproxy-bundle [192.168.100.1:8787/tripleorocky/centos-binary-haproxy:pcmklatest]<br> haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0<br> haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started overcloud-controller-1<br> haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started overcloud-controller-2<br> Docker container: openstack-cinder-volume [192.168.100.1:8787/tripleorocky/centos-binary-cinder-volume:pcmklatest]<br> openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0<br><br>Failed Resource Actions:<br>* redis-bundle-docker-1_monitor_60000 on overcloud-controller-1 'unknown error' (1): call=132, status=Timed Out, exitreason='',<br> last-rc-change='Thu Jul 23 16:42:15 2020', queued=0ms, exec=0ms<br>* galera-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=41, status=Timed Out, exitreason='',<br> last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms<br>* redis-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=62, status=Timed Out, exitreason='',<br> last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms<br>* haproxy-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=106, status=Timed Out, exitreason='',<br> last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms<br>* rabbitmq-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=121, status=Timed Out, exitreason='',<br> last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms<br>* galera_promote_0 on galera-bundle-2 'unknown error' (1): call=43, status=complete, exitreason='MySQL server failed to start (pid=646) (rc=0), please check your installation',<br> last-rc-change='Thu Jul 23 16:49:14 2020', queued=0ms, exec=12193ms<br><br>Daemon Status:<br> corosync: active/enabled<br> pacemaker: active/enabled<br> pcsd: active/enabled<br></div><div><br><br></div><div>It happens all the time when OC total number of networks goes above 70+ . I am attaching error logs of overcloud also. <br></div><div><br></div><div><br></div><div><span name="x"></span><div>Regards<br><b>Rahul Pathak</b><br>i2k2 Networks (P) Ltd. | Spring Meadows Business Park<br>A61-B4 & 4A First Floor, Sector 63, Noida - 201 301<br>ISO/IEC 27001:2005 & ISO 9001:2008 Certified</div><span name="x"></span><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Alfredo Moralejo Alonso" <amoralej@redhat.com><br><b>To: </b>"Rahul Pathak" <rpathak@i2k2.com><br><b>Cc: </b>"RDO Developmen List" <dev@lists.rdoproject.org><br><b>Sent: </b>Thursday, July 23, 2020 3:34:45 PM<br><b>Subject: </b>Re: [rdo-dev] tripleo cluster failure<br><div><br></div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 22, 2020 at 3:23 PM Rahul Pathak <<a href="mailto:rpathak@i2k2.com" target="_blank">rpathak@i2k2.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:rgb(0,0,0)"><div>Hi,</div><div style="color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:rgb(0,0,0)"><p style="margin:0px;padding:0px 5px 5px 0px;font-size:14px;line-height:1.4;font-family:"Helvetica Neue",Arial,Helvetica,sans-serif;color:rgb(75,75,75);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I have installed tripleo openstack version rocky containerized with Undercloud in virtual platform and 3 controllers and 2 compute Baremetal.<br></p><p style="margin:0px;padding:0px 5px 5px 0px;font-size:14px;line-height:1.4;font-family:"Helvetica Neue",Arial,Helvetica,sans-serif;color:rgb(75,75,75);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">My whole setup is running on centos7.<br></p><p style="margin:0px;padding:0px 5px 5px 0px;font-size:14px;line-height:1.4;font-family:"Helvetica Neue",Arial,Helvetica,sans-serif;color:rgb(75,75,75);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Overcloud cluster start failing once number of networks in overcloud reach more than 70. Lots of resources failure issue shown there. I don't know why HA cluster failed after 70 networks in OC.</p></div></div></div></div></blockquote><div><br></div><div>What kind of errors are you seeing?, what "resource failures"?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:rgb(0,0,0)"><div style="color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:10pt;color:rgb(0,0,0)"><p style="margin:0px;padding:0px 5px 5px 0px;font-size:14px;line-height:1.4;font-family:"Helvetica Neue",Arial,Helvetica,sans-serif;color:rgb(75,75,75);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Is some kind of threshold in tripleo configuration? so it is restricted not to create more than 70 or 80 netwoks. How could i fix this?</p><p style="margin:0px;padding:0px 5px 0px 0px;border:medium none;font-size:14px;line-height:1.4;font-family:"Helvetica Neue",Arial,Helvetica,sans-serif;color:rgb(75,75,75);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">I did not see such issue when I am using redhat platform and it's repos. This issue coming in opensource repos on Centos 7 . Please help how to fix this issue so I can scale up my openstack upto 2000 vms in this situation it's not possible.</p><br><div><span></span><div>Regards<br><b>Rahul Pathak</b><br>i2k2 Networks (P) Ltd. | Spring Meadows Business Park<br>A61-B4 & 4A First Floor, Sector 63, Noida - 201 301<br>ISO/IEC 27001:2005 & ISO 9001:2008 Certified</div><span></span><br></div></div></div><div><br></div></div></div>_______________________________________________<br>
dev mailing list<br>
<a href="mailto:dev@lists.rdoproject.org" target="_blank">dev@lists.rdoproject.org</a><br>
<a href="http://lists.rdoproject.org/mailman/listinfo/dev" rel="noreferrer" target="_blank">http://lists.rdoproject.org/mailman/listinfo/dev</a><br>
<br>
To unsubscribe: <a href="mailto:dev-unsubscribe@lists.rdoproject.org" target="_blank">dev-unsubscribe@lists.rdoproject.org</a><br>
</blockquote></div></div>
</div><div><br></div></div></body></html>