[rdo-dev] tripleo cluster failure

Arkady Shtempler ashtempl at redhat.com
Thu Jul 23 12:05:31 UTC 2020


Hi all!

*Rahul *- there is nothing relevant in the attached file, you've probably
executed LogTool on "working environment", so there is nothing interesting
in it.
I think that you had to mention the Error we've detected in an already
"crushed" environment, just as I was suggesting you to do.

*Alfredo *- this Error was logged almost on each OC node at the same time
when the problems had started.

   - *hyp-0*
   - ------------------------------ LogPath:
   /var/log/containers/neutron/openvswitch-agent.log.1
   ------------------------------
   - IsTracebackBlock:False
   - UniqueCounter:1
   - AnalyzedBlockLinesSize:18
   - 26712-2020-07-21 20:07:54.604 54410 INFO
   neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
   [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
   - 26713-2020-07-21 20:07:54.605 54410 INFO
   neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
   [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
   - 26714:2020-07-21 20:07:56.363 54410 ERROR
   oslo.messaging._drivers.impl_rabbit [-]
   [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server
   overcloud-control...<--LogTool-LINE IS TOO LONG!
   - 26715:2020-07-21 20:07:56.364 54410 ERROR
   oslo.messaging._drivers.impl_rabbit [-]
   [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server
   overcloud-control...<--LogTool-LINE IS TOO LONG!
   - 26716:2020-07-21 20:07:56.364 54410 ERROR
   oslo.messaging._drivers.impl_rabbit [-]
   [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server
   overcloud-control...<--LogTool-LINE IS TOO LONG!
   - ...
   - ...
   - ...
   - LogTool --> THIS BLOCK IS TOO LONG!
   - LogTool --> POTENTIAL BLOCK'S ISSUES:
   - *26714:2020-07-21 20:07:56.363 54410 ERROR
   oslo.messaging._drivers.impl_**rabbit [-]
[2f214f0b-84d0-49d4-bcf4-**477565903585]
   AMQP server overcloud-controller-2.inter..**.*
   - *ection. Check login credentials: Socket closed: IOError: Socket
   closed*
   - *no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111]
   ECONNREFUSED*
   - *rnalapi.i2k2cloud02.com:5672 <http://rnalapi.i2k2cloud02.com:5672> is
   unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.:
   RecoverableConnectionError: <AMQPError: unknown error>*



You can find more Error Blocks in the attached file.

Thanks!
<#m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20200723/e77dcac5/attachment-0001.html>
-------------- next part --------------
    hyp-0
    ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:18
    26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
    26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
    26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    ...
    ...
    ...
    LogTool --> THIS BLOCK IS TOO LONG!
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-controller-2.inter...
                                        ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    ------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:17
    692-2020-07-21 20:06:32.245 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    693-2020-07-21 20:07:32.328 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    694:2020-07-21 20:08:04.803 1 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOErro...<--LogTool-LINE IS TOO LONG!
    695:2020-07-21 20:08:19.844 1 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errn...<--LogTool-LINE IS TOO LONG!
    696-2020-07-21 20:08:34.439 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    697-2020-07-21 20:09:34.182 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    698-2020-07-21 20:10:34.274 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    699-2020-07-21 20:11:36.243 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    700-2020-07-21 20:12:38.198 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    701-2020-07-21 20:13:40.121 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
    702-2020-07-21 20:14:39.239 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^
    .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
     
     
     
     
    hyp-1
    ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:33
    26746-2020-07-21 20:07:55.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
    26747-2020-07-21 20:07:55.624 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
    26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26749:2020-07-21 20:07:56.370 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [36454cb1-0589-4cb6-9f63-7e2834d27056] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26750:2020-07-21 20:07:56.371 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [b6ab5be6-cc06-4d26-bc76-0f8161b10867] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26751:2020-07-21 20:07:56.372 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [db71cb6f-a426-4456-b56d-13562f98aa84] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26752:2020-07-21 20:07:56.373 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [9af86e0e-d8e0-4738-bb9c-92c544981c2b] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    26753-2020-07-21 20:07:57.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
    26754-2020-07-21 20:07:57.625 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
    26755:2020-07-21 20:07:57.710 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
    ...
    ---< BLOCK IS TOO LONG >---
    ...
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-controller-2.inter...
                                        ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
     
     
    ------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:25
    691-2020-07-21 20:06:13.016 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    692-2020-07-21 20:07:13.052 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2....<--LogTool-LINE IS TOO LONG!
    694:2020-07-21 20:07:57.715 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG!
    695:2020-07-21 20:07:58.744 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG!
    696-2020-07-21 20:07:59.767 1 INFO oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] Reconnected to AMQP server on overcl...<--LogTool-LINE IS TOO LONG!
    697:2020-07-21 20:08:01.107 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG!
    698:2020-07-21 20:08:02.182 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG!
    699-2020-07-21 20:08:03.205 1 INFO oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-4...<--LogTool-LINE IS TOO LONG!
    700-2020-07-21 20:08:17.601 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    701-2020-07-21 20:09:14.030 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    702-2020-07-21 20:10:16.027 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    703-2020-07-21 20:11:16.955 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    704-2020-07-21 20:12:17.038 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
    705-2020-07-21 20:13:16.986 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2.inter...
                                  ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
     
     
     
    controller-0
    ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:27
    29685-2020-07-21 20:07:55.226 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
    29686-2020-07-21 20:07:55.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
    29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29688:2020-07-21 20:07:56.375 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29689:2020-07-21 20:07:56.376 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29690:2020-07-21 20:07:56.377 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29691-2020-07-21 20:07:57.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
    29692-2020-07-21 20:07:57.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
    29693:2020-07-21 20:07:57.483 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29694:2020-07-21 20:07:57.706 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29695:2020-07-21 20:07:57.729 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29696:2020-07-21 20:07:57.733 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29697-2020-07-21 20:07:58.513 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29698-2020-07-21 20:07:58.765 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29699-2020-07-21 20:07:58.768 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29700-2020-07-21 20:07:58.812 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29701-2020-07-21 20:07:59.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
    29702-2020-07-21 20:07:59.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
    29703-2020-07-21 20:08:01.228 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-controller-2.inter...
                                         ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
     
     
     
    ------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:26
    1609-2020-07-21 20:07:16.952 177642 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG!
    1610-2020-07-21 20:07:17.108 177642 INFO neutron.agent.dhcp.agent [req-026ba197-47b8-4006-8ff3-155d45cb87a2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    1612:2020-07-21 20:07:57.777 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
    1613-2020-07-21 20:07:58.790 177642 INFO oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG!
    1614:2020-07-21 20:08:08.565 177642 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG!
    1615:2020-07-21 20:08:23.607 177642 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG!
    1616-2020-07-21 20:08:49.052 177642 INFO neutron.agent.dhcp.agent [req-46823ae5-be71-437f-adf4-c4db97fe28d7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1617-2020-07-21 20:08:49.336 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
    1618-2020-07-21 20:08:49.355 177642 INFO neutron.agent.dhcp.agent [req-c1e2fa2e-8376-417f-aa25-04989ec04ce7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1619-2020-07-21 20:08:49.406 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
    1620-2020-07-21 20:08:49.458 177642 INFO neutron.agent.dhcp.agent [req-d81a548c-f5ff-4ddc-bfbb-1a038ec6e334 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1621-2020-07-21 20:08:49.482 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
    1622-2020-07-21 20:08:49.561 177642 INFO neutron.agent.dhcp.agent [req-52d2cc2c-3cfa-4b00-9d1f-3612f7eb9d8e - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-controller-2.inter...
                                        ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^
    .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
                                                      ^^^^^
     
    ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:22
    38644-2020-07-21 20:07:55.818 38 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0009890
    38645-2020-07-21 20:07:55.942 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0009868
    38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    38647:2020-07-21 20:07:56.373 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [bdd4035e-b970-48aa-bd17-5f3a9fa3ee4d] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    38648:2020-07-21 20:07:56.374 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    ...
    ...
    ...
    LogTool --> THIS BLOCK IS TOO LONG!
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller-2.inter...
                                     ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
                                                      ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^
     
     
     
    ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:18
    38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38679-2020-07-21 20:07:57.699 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38680:2020-07-21 20:07:57.708 47 ERROR oslo.messaging._drivers.impl_rabbit [-] [7dfef257-02a9-447a-8be4-9149a980f37d] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38681-2020-07-21 20:07:57.709 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [42757d9d-ade4-4c52-9083-7fef8f3d23c0] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38682-2020-07-21 20:07:57.710 50 ERROR oslo.messaging._drivers.impl_rabbit [-] [98688dd7-c96a-4bee-a980-92cbca8e1541] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38683-2020-07-21 20:07:57.712 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [d836ff75-44af-4e1e-b985-2a3108a1f337] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38684-2020-07-21 20:07:57.712 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [efa6cb10-7f90-48f5-95a7-595d6a3d9c9a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38685-2020-07-21 20:07:57.712 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [52904842-0a17-4378-a279-82d24a91a95a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38686-2020-07-21 20:07:57.715 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [63bd5e71-06e1-45f9-bd66-ccc1fd087a1c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38687-2020-07-21 20:07:57.719 46 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6f11462-ac09-4098-b70c-43f4e99fe87c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-controller-2.in...
                                     ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
     
     
     
     
     
    controller-1
    ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:31
    29669-2020-07-21 20:07:55.494 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
    29670-2020-07-21 20:07:55.499 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
    29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29672:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [f4016ef6-75a6-405d-82cf-ae3f4557a3d8] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29673:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [eda2f576-314d-47fa-9348-0f9a5e5fe9c7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29674:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea7378e5-bd62-46a3-8cf0-6cf8000fdba7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29675:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29676:2020-07-21 20:07:56.505 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3d4af668-0b5f-4f30-8f03-9778c5650d1b] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29677-2020-07-21 20:07:57.502 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
    29678-2020-07-21 20:07:57.534 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
    ...
    ---< BLOCK IS TOO LONG >---
    ...
    29690-2020-07-21 20:07:58.926 153102 INFO oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29691-2020-07-21 20:07:59.503 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-controller-2.inter...
                                         ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    ection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
     
    ------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:2
    AnalyzedBlockLinesSize:24
    1682-2020-07-21 20:07:16.957 152406 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG!
    1683-2020-07-21 20:07:17.177 152406 INFO neutron.agent.dhcp.agent [req-2c406398-1696-43dc-b0f5-457a287a5e34 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
    1685:2020-07-21 20:07:57.699 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
    1686:2020-07-21 20:07:58.732 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
    1687-2020-07-21 20:07:59.765 152406 INFO oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG!
    1688:2020-07-21 20:08:08.015 152406 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG!
    1689:2020-07-21 20:08:23.058 152406 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG!
    1690-2020-07-21 20:09:01.075 152406 INFO neutron.agent.dhcp.agent [req-f1c75a55-aeff-4415-9fa3-824874ed72ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1691-2020-07-21 20:09:01.392 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
    1692-2020-07-21 20:09:01.479 152406 INFO neutron.agent.dhcp.agent [req-062d6247-fb9a-4a27-953c-1e3e15da83e2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1693-2020-07-21 20:09:01.492 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
    1694-2020-07-21 20:09:01.584 152406 INFO neutron.agent.dhcp.agent [req-b71f0f58-3857-47d7-8b7e-b19c52e196ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
    1695-2020-07-21 20:09:01.612 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
    1696-2020-07-21 20:09:01.689 152406 INFO neutron.agent.dhcp.agent [req-ea33f202-4d02-4a88-9905-8a250e1a47d5 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-controller-2.inter...
                                        ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^
    .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
                                                      ^^^^^
     
     
    ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:2
    AnalyzedBlockLinesSize:22
    38396-2020-07-21 20:07:55.925 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0011032
    38397-2020-07-21 20:07:56.030 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0008850
    38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    38399:2020-07-21 20:07:56.380 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [da0fafee-b840-44b7-9c5f-0afda51faab6] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    38400:2020-07-21 20:07:56.380 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [5997eee9-f4a0-4120-8c1a-143089d3fdc7] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    ...
    ...
    ...
    LogTool --> THIS BLOCK IS TOO LONG!
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller-2.inter...
                                     ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
     
     
     
    controller-2
    ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:2
    AnalyzedBlockLinesSize:20
    29367-2020-07-21 20:07:55.723 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
    29368-2020-07-21 20:07:55.730 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
    29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29370:2020-07-21 20:07:56.379 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29371:2020-07-21 20:07:56.382 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29372:2020-07-21 20:07:56.383 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29373:2020-07-21 20:07:56.385 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29374:2020-07-21 20:07:56.386 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
    29375:2020-07-21 20:07:57.716 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29376-2020-07-21 20:07:57.724 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
    29377-2020-07-21 20:07:57.729 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
    29378:2020-07-21 20:07:57.805 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29379:2020-07-21 20:07:57.814 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29380:2020-07-21 20:07:57.818 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29381:2020-07-21 20:07:57.821 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29382:2020-07-21 20:07:57.824 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29383:2020-07-21 20:07:58.781 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
    29384-2020-07-21 20:07:58.858 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29385-2020-07-21 20:07:58.859 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29386-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29387-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29388-2020-07-21 20:07:58.866 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
    29389-2020-07-21 20:07:59.725 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
    29390-2020-07-21 20:07:59.732 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-controller-2.inter...
                                         ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
     
     
     
     
    ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:20
    38505-2020-07-21 20:07:56.043 28 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0026181
    38506-2020-07-21 20:07:56.045 29 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0032661
    38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    38508:2020-07-21 20:07:56.379 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    38509:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
    ...
    ...
    ...
    LogTool --> THIS BLOCK IS TOO LONG!
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller-2.inter...
                                     ^^^^^
    ection. Check login credentials: Socket closed: IOError: Socket closed
                                                      ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^
     
     
     
     
    ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
    IsTracebackBlock:False
    UniqueCounter:1
    AnalyzedBlockLinesSize:29
    38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38525-2020-07-21 20:07:57.723 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38526:2020-07-21 20:07:57.724 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [6ac6e794-3561-4cc3-99a9-cc5e078dd130] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38527-2020-07-21 20:07:57.724 38 ERROR oslo.messaging._drivers.impl_rabbit [-] [12f38994-d07a-40df-bdb4-4f49c905b417] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38528-2020-07-21 20:07:57.732 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38529-2020-07-21 20:07:57.739 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38530-2020-07-21 20:07:57.740 39 ERROR oslo.messaging._drivers.impl_rabbit [-] [b237f335-3172-434d-88f5-f14d37bf485c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38531-2020-07-21 20:07:57.745 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [8246b7f8-db45-4681-8910-2bf85df8fafa] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38532-2020-07-21 20:07:57.745 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [93e8e483-ef8e-41f0-bf61-8f8c25fb2efb] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38533-2020-07-21 20:07:57.754 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [c11cfdee-53e0-4574-988e-e037adf8422b] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38534-2020-07-21 20:07:57.759 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [091e9689-2d34-46e5-a40c-8c4fe3b200c5] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38535:2020-07-21 20:07:57.958 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [2a4c7a49-5f9b-4b09-8636-16320c659a68] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
    38536-2020-07-21 20:07:58.047 33 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0022709
    38537-2020-07-21 20:07:58.049 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0022650
    38538-2020-07-21 20:07:58.470 32 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200  len: 252 time: 0.0023279
    38539-2020-07-21 20:07:58.513 43 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOE...<--LogTool-LINE IS TOO LONG!
    38540-2020-07-21 20:07:58.527 43 INFO oslo.messaging._drivers.impl_rabbit [-] [9f9ed2f1-0b23-40a4-99ac-4082ec985e21] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG!
    38541-2020-07-21 20:07:58.557 40 INFO oslo.messaging._drivers.impl_rabbit [-] [1e042b35-4e4d-4e0d-8aec-ebdc7efe19b7] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG!
    38542-2020-07-21 20:07:58.564 40 INFO oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG!
     
    LogTool --> POTENTIAL BLOCK'S ISSUES:
    38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-controller-2.in...
                                     ^^^^^
    no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
                                                      ^^^^^
    rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
                                                      ^^^^^          ^^^^^                                          ^^^^^^^^^^^^^^^       ^^^^^          ^^^^^
    slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
                                                      ^^^^^                                                    ^^^^^


More information about the dev mailing list