<html><body><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">Reminder!! Please help.<br></div><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">-------------------------------<br><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000"><div>Hi,</div><div><br></div><div>Is there some kind of script that I can run to know the exact issue about what resource crunch is there ?<br></div><div><br></div><div>Below is the memory and disk utilization of controller:<br></div><div><br></div><div>[heat-admin@overcloud-controller-0 ~]$ free -m<br> total used free shared buff/cache available<br>Mem: 128722 25632 92042 70 11047 102377<br>Swap: 0 0 0<br>[heat-admin@overcloud-controller-0 ~]$ df -h<br>Filesystem Size Used Avail Use% Mounted on<br>devtmpfs 63G 0 63G 0% /dev<br>tmpfs 63G 39M 63G 1% /dev/shm<br>tmpfs 63G 27M 63G 1% /run<br>tmpfs 63G 0 63G 0% /sys/fs/cgroup<br>/dev/sda2 1.9T 12G 1.9T 1% /<br>tmpfs 13G 0 13G 0% /run/user/0<br>tmpfs 13G 0 13G 0% /run/user/1000<br><div><br></div></div><div>Same memory and disk available on all three controllers.<br></div><div><br></div><div>On the same environment when I installed overcloud with redhat repos and redhat overcloud images...I have not faced this issue. I have tested almost 500 projects and 500 networks (one network per project) on the same environment with redhat and it was working fine without any issue and cluster failure. But when I am using Centos-7 tripleo repo's its happening again n again.<br></div><div><br></div><div><br></div><div><span></span><div>Regards<br><b>Rahul Pathak</b><br>i2k2 Networks (P) Ltd. | Spring Meadows Business Park<br>A61-B4 & 4A First Floor, Sector 63, Noida - 201 301<br>ISO/IEC 27001:2005 & ISO 9001:2008 Certified</div><span></span><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Alfredo Moralejo Alonso" <amoralej@redhat.com><br><b>To: </b>"Arkady Shtempler" <ashtempl@redhat.com><br><b>Cc: </b>"Rahul Pathak" <rpathak@i2k2.com>, "RDO Developmen List" <dev@lists.rdoproject.org><br><b>Sent: </b>Thursday, July 23, 2020 5:40:42 PM<br><b>Subject: </b>Re: [rdo-dev] tripleo cluster failure<br><div><br></div><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler <<a href="mailto:ashtempl@redhat.com" target="_blank">ashtempl@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi all!</div><div><br></div><div><b>Rahul </b>- there is nothing relevant in the attached file, you've probably executed LogTool on "working environment", so there is nothing interesting in it.<br></div><div>I think that you had to mention the Error we've detected in an already "crushed" environment, just as I was suggesting you to do.<br></div></div></blockquote><div><br></div><div>Yes, nothing interesting in the attached logs.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div><b>Alfredo </b>- this Error was logged almost on each OC node at the same time when the problems had started.<br></div><div>
<span style="color:rgb(255,0,0)"><span><ul><li style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top"><b><span style="font-size:large">hyp-0</span></b></div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">IsTracebackBlock:False</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">UniqueCounter:1</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">AnalyzedBlockLinesSize:18</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">...</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">...</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">...</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">LogTool --> THIS BLOCK IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top">LogTool --> POTENTIAL BLOCK'S ISSUES:</div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top"><span style="font-size: small;"><b>26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_</b></span><span style="font-size: small;"><b>rabbit [-] [2f214f0b-84d0-49d4-bcf4-</b></span><span style="font-size: small;"><b>477565903585] AMQP server overcloud-controller-2.inter..</b></span><span style="font-size: small;"><b>.</b></span></div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top"><span style="font-size: small;"><b>ection. Check login credentials: Socket closed: IOError: Socket closed</b></span></div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top"><span style="font-size: small;"><b>no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED</b></span></div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background: none repeat scroll 0% 0%;vertical-align:top"><span style="font-size: small;"><b><span style="background-color:rgb(255,255,0)"><a href="http://rnalapi.i2k2cloud02.com:5672" target="_blank">rnalapi.i2k2cloud02.com:5672</a>
is unreachable: <AMQPError: unknown error>. </span>Trying again in 1
seconds.: RecoverableConnectionError: <AMQPError: unknown error></b></span></div></li></ul></span><span style="font-size: small;"> </span></span><br></div></div></blockquote><div><br></div><div>I'd suggest to check rabbitmq and mariadb logs.</div><div><br></div><div>AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers?<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>You can find more Error Blocks in the attached file.</div><div><br></div><div>Thanks!</div><div id="gmail-m_-4775499539429953439m_432445082502786505DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><table style="border-top:1px solid rgb(211,212,222)" class="mceItemTable"><tbody><tr><td style="width:55px;padding-top:13px"></td><td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px"></td>
</tr>
</tbody></table><a href="#m_-4775499539429953439_m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"></a><br></div></div>
</blockquote></div></div>
</div><div><br></div></div></div><div><br></div></div></body></html>