Hi, When I am trying to create new projects and one network for each projects after 70 or 80 projects and networks in OC. Controller HA availablitliy failed with below error. [stack@director LogTool_Python2]$ ssh heat-admin(a)192.168.100.28 "sudo pcs status" Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-1 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum Last updated: Thu Jul 23 17:00:22 2020 Last change: Wed Jul 22 14:35:34 2020 by hacluster via crmd on overcloud-controller-2 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.100.1:8787/tripleorocky/centos-binary-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2 Docker container set: galera-bundle [192.168.100.1:8787/tripleorocky/centos-binary-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-controller-1 galera-bundle-2 (ocf::heartbeat:galera): FAILED Master overcloud-controller-2 (blocked) Docker container set: redis-bundle [192.168.100.1:8787/tripleorocky/centos-binary-redis:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave overcloud-controller-1 redis-bundle-2 (ocf::heartbeat:redis): Slave overcloud-controller-2 ip-192.168.100.98 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-10.10.0.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.102.185 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-192.168.102.116 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.103.187 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.104.127 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Docker container set: haproxy-bundle [192.168.100.1:8787/tripleorocky/centos-binary-haproxy:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started overcloud-controller-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started overcloud-controller-2 Docker container: openstack-cinder-volume [192.168.100.1:8787/tripleorocky/centos-binary-cinder-volume:pcmklatest] openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0 Failed Resource Actions: * redis-bundle-docker-1_monitor_60000 on overcloud-controller-1 'unknown error' (1): call=132, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:42:15 2020', queued=0ms, exec=0ms * galera-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=41, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * redis-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=62, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * haproxy-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=106, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * rabbitmq-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=121, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * galera_promote_0 on galera-bundle-2 'unknown error' (1): call=43, status=complete, exitreason='MySQL server failed to start (pid=646) (rc=0), please check your installation', last-rc-change='Thu Jul 23 16:49:14 2020', queued=0ms, exec=12193ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled It happens all the time when OC total number of networks goes above 70+ . I am attaching error logs of overcloud also. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified ----- Original Message ----- From: "Alfredo Moralejo Alonso" <amoralej(a)redhat.com&gt; To: "Rahul Pathak" <rpathak(a)i2k2.com&gt; Cc: "RDO Developmen List" <dev(a)lists.rdoproject.org&gt; Sent: Thursday, July 23, 2020 3:34:45 PM Subject: Re: [rdo-dev] tripleo cluster failure On Wed, Jul 22, 2020 at 3:23 PM Rahul Pathak < rpathak(a)i2k2.com > wrote: Hi, I have installed tripleo openstack version rocky containerized with Undercloud in virtual platform and 3 controllers and 2 compute Baremetal. My whole setup is running on centos7. Overcloud cluster start failing once number of networks in overcloud reach more than 70. Lots of resources failure issue shown there. I don't know why HA cluster failed after 70 networks in OC. What kind of errors are you seeing?, what "resource failures"? <blockquote> Is some kind of threshold in tripleo configuration? so it is restricted not to create more than 70 or 80 netwoks. How could i fix this? I did not see such issue when I am using redhat platform and it's repos. This issue coming in opensource repos on Centos 7 . Please help how to fix this issue so I can scale up my openstack upto 2000 vms in this situation it's not possible. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified _______________________________________________ dev mailing list dev(a)lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscribe(a)lists.rdoproject.org </blockquote>

On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler <ashtempl(a)redhat.com&gt; wrote: Yes, nothing interesting in the attached logs. I'd suggest to check rabbitmq and mariadb logs. AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers?

Reminder!! Please help. ------------------------------- Hi, Is there some kind of script that I can run to know the exact issue about what resource crunch is there ? Below is the memory and disk utilization of controller: [heat-admin@overcloud-controller-0 ~]$ free -m total used free shared buff/cache available Mem: 128722 25632 92042 70 11047 102377 Swap: 0 0 0 [heat-admin@overcloud-controller-0 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 39M 63G 1% /dev/shm tmpfs 63G 27M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda2 1.9T 12G 1.9T 1% / tmpfs 13G 0 13G 0% /run/user/0 tmpfs 13G 0 13G 0% /run/user/1000 Same memory and disk available on all three controllers. On the same environment when I installed overcloud with redhat repos and redhat overcloud images...I have not faced this issue. I have tested almost 500 projects and 500 networks (one network per project) on the same environment with redhat and it was working fine without any issue and cluster failure. But when I am using Centos-7 tripleo repo's its happening again n again. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified ----- Original Message ----- From: "Alfredo Moralejo Alonso" <amoralej(a)redhat.com&gt; To: "Arkady Shtempler" <ashtempl(a)redhat.com&gt; Cc: "Rahul Pathak" <rpathak(a)i2k2.com&gt;, "RDO Developmen List" <dev(a)lists.rdoproject.org&gt; Sent: Thursday, July 23, 2020 5:40:42 PM Subject: Re: [rdo-dev] tripleo cluster failure On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler < ashtempl(a)redhat.com > wrote: Hi all! Rahul - there is nothing relevant in the attached file, you've probably executed LogTool on "working environment", so there is nothing interesting in it. I think that you had to mention the Error we've detected in an already "crushed" environment, just as I was suggesting you to do. Yes, nothing interesting in the attached logs. <blockquote> Alfredo - this Error was logged almost on each OC node at the same time when the problems had started. * hyp-0 * ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ * IsTracebackBlock:False * UniqueCounter:1 * AnalyzedBlockLinesSize:18 * 26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! * 26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! * 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * 26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * 26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * ... * ... * ... * LogTool --> THIS BLOCK IS TOO LONG! * LogTool --> POTENTIAL BLOCK'S ISSUES: * 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_ rabbit [-] [2f214f0b-84d0-49d4-bcf4- 477565903585] AMQP server overcloud-controller-2.inter.. . * ection. Check login credentials: Socket closed: IOError: Socket closed * no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED * rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error> </blockquote> I'd suggest to check rabbitmq and mariadb logs. AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers? <blockquote> You can find more Error Blocks in the attached file. Thanks! </blockquote>