<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler <<a href="mailto:ashtempl@redhat.com">ashtempl@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi all!</div><div><br></div><div><b>Rahul </b>- there is nothing relevant in the attached file, you've probably executed LogTool on "working environment", so there is nothing interesting in it.<br></div><div>I think that you had to mention the Error we've detected in an already "crushed" environment, just as I was suggesting you to do.<br></div></div></blockquote><div><br></div><div>Yes, nothing interesting in the attached logs.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div><b>Alfredo </b>- this Error was logged almost on each OC node at the same time when the problems had started.<br></div><div>
<span style="color:rgb(255,0,0)"><font size="-1"><ul><li style="background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top"><b><span style="font-size:large">hyp-0</span></b></div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">IsTracebackBlock:False</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">UniqueCounter:1</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">AnalyzedBlockLinesSize:18</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">...</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">...</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">...</div></li><li style="font-size:11px;background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">LogTool --> THIS BLOCK IS TOO LONG!</div></li><li style="font-size:11px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top">LogTool --> POTENTIAL BLOCK'S ISSUES:</div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top"><font size="2"><b>26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_</b></font><font size="2"><b>rabbit [-] [2f214f0b-84d0-49d4-bcf4-</b></font><font size="2"><b>477565903585] AMQP server overcloud-controller-2.inter..</b></font><font size="2"><b>.</b></font></div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top"><font size="2"><b>ection. Check login credentials: Socket closed: IOError: Socket closed</b></font></div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top"><font size="2"><b>no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED</b></font></div></li><li style="background:rgb(248,248,248) none repeat scroll 0% 0%"><div style="margin:0px;padding:0px;background:rgba(0,0,0,0) none repeat scroll 0% 0%;vertical-align:top"><font size="2"><b><span style="background-color:rgb(255,255,0)"><a href="http://rnalapi.i2k2cloud02.com:5672" target="_blank">rnalapi.i2k2cloud02.com:5672</a>
 is unreachable: <AMQPError: unknown error>. </span>Trying again in 1 
seconds.: RecoverableConnectionError: <AMQPError: unknown error></b></font></div></li></ul></font><font size="2"> </font></span><br></div></div></blockquote><div><br></div><div>I'd suggest to check rabbitmq and mariadb logs.</div><div><br></div><div>AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers?<br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>You can find more Error Blocks in the attached file.</div><div><br></div><div>Thanks!</div><div id="gmail-m_-4775499539429953439m_432445082502786505DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><table style="border-top:1px solid rgb(211,212,222)"><tbody><tr><td style="width:55px;padding-top:13px"></td><td style="width:470px;padding-top:12px;color:rgb(65,66,78);font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px"></td>
        </tr>
</tbody></table><a href="#m_-4775499539429953439_m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1" height="1"></a></div></div>
</blockquote></div></div>