[rdo-dev] tripleo cluster failure

[Alfredo Moralejo Alonso]

On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler <ashtempl at redhat.com>
wrote:

> Hi all!
>
> *Rahul *- there is nothing relevant in the attached file, you've probably
> executed LogTool on "working environment", so there is nothing interesting
> in it.
> I think that you had to mention the Error we've detected in an already
> "crushed" environment, just as I was suggesting you to do.
>

Yes, nothing interesting in the attached logs.


>
> *Alfredo *- this Error was logged almost on each OC node at the same time
> when the problems had started.
>
>    - *hyp-0*
>    - ------------------------------ LogPath:
>    /var/log/containers/neutron/openvswitch-agent.log.1
>    ------------------------------
>    - IsTracebackBlock:False
>    - UniqueCounter:1
>    - AnalyzedBlockLinesSize:18
>    - 26712-2020-07-21 20:07:54.604 54410 INFO
>    neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>    [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
>    - 26713-2020-07-21 20:07:54.605 54410 INFO
>    neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>    [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
>    - 26714:2020-07-21 20:07:56.363 54410 ERROR
>    oslo.messaging._drivers.impl_rabbit [-]
>    [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server
>    overcloud-control...<--LogTool-LINE IS TOO LONG!
>    - 26715:2020-07-21 20:07:56.364 54410 ERROR
>    oslo.messaging._drivers.impl_rabbit [-]
>    [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server
>    overcloud-control...<--LogTool-LINE IS TOO LONG!
>    - 26716:2020-07-21 20:07:56.364 54410 ERROR
>    oslo.messaging._drivers.impl_rabbit [-]
>    [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server
>    overcloud-control...<--LogTool-LINE IS TOO LONG!
>    - ...
>    - ...
>    - ...
>    - LogTool --> THIS BLOCK IS TOO LONG!
>    - LogTool --> POTENTIAL BLOCK'S ISSUES:
>    - *26714:2020-07-21 20:07:56.363 54410 ERROR
>    oslo.messaging._drivers.impl_**rabbit [-] [2f214f0b-84d0-49d4-bcf4-**477565903585]
>    AMQP server overcloud-controller-2.inter..**.*
>    - *ection. Check login credentials: Socket closed: IOError: Socket
>    closed*
>    - *no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno
>    111] ECONNREFUSED*
>    - *rnalapi.i2k2cloud02.com:5672 <http://rnalapi.i2k2cloud02.com:5672>
>    is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.:
>    RecoverableConnectionError: <AMQPError: unknown error>*
>
>
>
I'd suggest to check rabbitmq and mariadb logs.

AFAIK, there is not a configuration that may limit the number of networks
or project, but it may be hitting some resources scarcity that affect the
running services. What's the memory sizing and usage of the controllers?



>
> You can find more Error Blocks in the attached file.
>
> Thanks!
>
> <#m_-4775499539429953439_m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20200723/961c1067/attachment.html>