[rdo-dev] tripleo cluster failure
Arkady Shtempler
ashtempl at redhat.com
Thu Jul 23 12:05:31 UTC 2020
Hi all!
*Rahul *- there is nothing relevant in the attached file, you've probably
executed LogTool on "working environment", so there is nothing interesting
in it.
I think that you had to mention the Error we've detected in an already
"crushed" environment, just as I was suggesting you to do.
*Alfredo *- this Error was logged almost on each OC node at the same time
when the problems had started.
- *hyp-0*
- ------------------------------ LogPath:
/var/log/containers/neutron/openvswitch-agent.log.1
------------------------------
- IsTracebackBlock:False
- UniqueCounter:1
- AnalyzedBlockLinesSize:18
- 26712-2020-07-21 20:07:54.604 54410 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
- 26713-2020-07-21 20:07:54.605 54410 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
[req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
- 26714:2020-07-21 20:07:56.363 54410 ERROR
oslo.messaging._drivers.impl_rabbit [-]
[2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server
overcloud-control...<--LogTool-LINE IS TOO LONG!
- 26715:2020-07-21 20:07:56.364 54410 ERROR
oslo.messaging._drivers.impl_rabbit [-]
[664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server
overcloud-control...<--LogTool-LINE IS TOO LONG!
- 26716:2020-07-21 20:07:56.364 54410 ERROR
oslo.messaging._drivers.impl_rabbit [-]
[68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server
overcloud-control...<--LogTool-LINE IS TOO LONG!
- ...
- ...
- ...
- LogTool --> THIS BLOCK IS TOO LONG!
- LogTool --> POTENTIAL BLOCK'S ISSUES:
- *26714:2020-07-21 20:07:56.363 54410 ERROR
oslo.messaging._drivers.impl_**rabbit [-]
[2f214f0b-84d0-49d4-bcf4-**477565903585]
AMQP server overcloud-controller-2.inter..**.*
- *ection. Check login credentials: Socket closed: IOError: Socket
closed*
- *no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111]
ECONNREFUSED*
- *rnalapi.i2k2cloud02.com:5672 <http://rnalapi.i2k2cloud02.com:5672> is
unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.:
RecoverableConnectionError: <AMQPError: unknown error>*
You can find more Error Blocks in the attached file.
Thanks!
<#m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rdoproject.org/pipermail/dev/attachments/20200723/e77dcac5/attachment-0001.html>
-------------- next part --------------
hyp-0
------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:18
26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG!
26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
...
...
...
LogTool --> THIS BLOCK IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:17
692-2020-07-21 20:06:32.245 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
693-2020-07-21 20:07:32.328 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
694:2020-07-21 20:08:04.803 1 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOErro...<--LogTool-LINE IS TOO LONG!
695:2020-07-21 20:08:19.844 1 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errn...<--LogTool-LINE IS TOO LONG!
696-2020-07-21 20:08:34.439 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
697-2020-07-21 20:09:34.182 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
698-2020-07-21 20:10:34.274 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
699-2020-07-21 20:11:36.243 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
700-2020-07-21 20:12:38.198 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
701-2020-07-21 20:13:40.121 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
702-2020-07-21 20:14:39.239 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
hyp-1
------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:33
26746-2020-07-21 20:07:55.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
26747-2020-07-21 20:07:55.624 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26749:2020-07-21 20:07:56.370 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [36454cb1-0589-4cb6-9f63-7e2834d27056] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26750:2020-07-21 20:07:56.371 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [b6ab5be6-cc06-4d26-bc76-0f8161b10867] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26751:2020-07-21 20:07:56.372 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [db71cb6f-a426-4456-b56d-13562f98aa84] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26752:2020-07-21 20:07:56.373 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [9af86e0e-d8e0-4738-bb9c-92c544981c2b] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
26753-2020-07-21 20:07:57.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
26754-2020-07-21 20:07:57.625 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG!
26755:2020-07-21 20:07:57.710 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
...
---< BLOCK IS TOO LONG >---
...
LogTool --> POTENTIAL BLOCK'S ISSUES:
26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:25
691-2020-07-21 20:06:13.016 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
692-2020-07-21 20:07:13.052 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2....<--LogTool-LINE IS TOO LONG!
694:2020-07-21 20:07:57.715 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG!
695:2020-07-21 20:07:58.744 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG!
696-2020-07-21 20:07:59.767 1 INFO oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] Reconnected to AMQP server on overcl...<--LogTool-LINE IS TOO LONG!
697:2020-07-21 20:08:01.107 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG!
698:2020-07-21 20:08:02.182 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG!
699-2020-07-21 20:08:03.205 1 INFO oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-4...<--LogTool-LINE IS TOO LONG!
700-2020-07-21 20:08:17.601 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
701-2020-07-21 20:09:14.030 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
702-2020-07-21 20:10:16.027 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
703-2020-07-21 20:11:16.955 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
704-2020-07-21 20:12:17.038 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
705-2020-07-21 20:13:16.986 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
controller-0
------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:27
29685-2020-07-21 20:07:55.226 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
29686-2020-07-21 20:07:55.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29688:2020-07-21 20:07:56.375 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29689:2020-07-21 20:07:56.376 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29690:2020-07-21 20:07:56.377 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29691-2020-07-21 20:07:57.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
29692-2020-07-21 20:07:57.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
29693:2020-07-21 20:07:57.483 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29694:2020-07-21 20:07:57.706 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29695:2020-07-21 20:07:57.729 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29696:2020-07-21 20:07:57.733 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29697-2020-07-21 20:07:58.513 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29698-2020-07-21 20:07:58.765 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29699-2020-07-21 20:07:58.768 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29700-2020-07-21 20:07:58.812 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29701-2020-07-21 20:07:59.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
29702-2020-07-21 20:07:59.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
29703-2020-07-21 20:08:01.228 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:26
1609-2020-07-21 20:07:16.952 177642 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG!
1610-2020-07-21 20:07:17.108 177642 INFO neutron.agent.dhcp.agent [req-026ba197-47b8-4006-8ff3-155d45cb87a2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
1612:2020-07-21 20:07:57.777 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
1613-2020-07-21 20:07:58.790 177642 INFO oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG!
1614:2020-07-21 20:08:08.565 177642 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG!
1615:2020-07-21 20:08:23.607 177642 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG!
1616-2020-07-21 20:08:49.052 177642 INFO neutron.agent.dhcp.agent [req-46823ae5-be71-437f-adf4-c4db97fe28d7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1617-2020-07-21 20:08:49.336 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
1618-2020-07-21 20:08:49.355 177642 INFO neutron.agent.dhcp.agent [req-c1e2fa2e-8376-417f-aa25-04989ec04ce7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1619-2020-07-21 20:08:49.406 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
1620-2020-07-21 20:08:49.458 177642 INFO neutron.agent.dhcp.agent [req-d81a548c-f5ff-4ddc-bfbb-1a038ec6e334 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1621-2020-07-21 20:08:49.482 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
1622-2020-07-21 20:08:49.561 177642 INFO neutron.agent.dhcp.agent [req-52d2cc2c-3cfa-4b00-9d1f-3612f7eb9d8e - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
^^^^^
------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:22
38644-2020-07-21 20:07:55.818 38 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0009890
38645-2020-07-21 20:07:55.942 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0009868
38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
38647:2020-07-21 20:07:56.373 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [bdd4035e-b970-48aa-bd17-5f3a9fa3ee4d] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
38648:2020-07-21 20:07:56.374 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
...
...
...
LogTool --> THIS BLOCK IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:18
38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38679-2020-07-21 20:07:57.699 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38680:2020-07-21 20:07:57.708 47 ERROR oslo.messaging._drivers.impl_rabbit [-] [7dfef257-02a9-447a-8be4-9149a980f37d] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38681-2020-07-21 20:07:57.709 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [42757d9d-ade4-4c52-9083-7fef8f3d23c0] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38682-2020-07-21 20:07:57.710 50 ERROR oslo.messaging._drivers.impl_rabbit [-] [98688dd7-c96a-4bee-a980-92cbca8e1541] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38683-2020-07-21 20:07:57.712 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [d836ff75-44af-4e1e-b985-2a3108a1f337] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38684-2020-07-21 20:07:57.712 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [efa6cb10-7f90-48f5-95a7-595d6a3d9c9a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38685-2020-07-21 20:07:57.712 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [52904842-0a17-4378-a279-82d24a91a95a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38686-2020-07-21 20:07:57.715 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [63bd5e71-06e1-45f9-bd66-ccc1fd087a1c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38687-2020-07-21 20:07:57.719 46 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6f11462-ac09-4098-b70c-43f4e99fe87c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-controller-2.in...
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
controller-1
------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:31
29669-2020-07-21 20:07:55.494 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
29670-2020-07-21 20:07:55.499 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29672:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [f4016ef6-75a6-405d-82cf-ae3f4557a3d8] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29673:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [eda2f576-314d-47fa-9348-0f9a5e5fe9c7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29674:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea7378e5-bd62-46a3-8cf0-6cf8000fdba7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29675:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29676:2020-07-21 20:07:56.505 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3d4af668-0b5f-4f30-8f03-9778c5650d1b] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29677-2020-07-21 20:07:57.502 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
29678-2020-07-21 20:07:57.534 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
...
---< BLOCK IS TOO LONG >---
...
29690-2020-07-21 20:07:58.926 153102 INFO oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29691-2020-07-21 20:07:59.503 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
ection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer
------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:2
AnalyzedBlockLinesSize:24
1682-2020-07-21 20:07:16.957 152406 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG!
1683-2020-07-21 20:07:17.177 152406 INFO neutron.agent.dhcp.agent [req-2c406398-1696-43dc-b0f5-457a287a5e34 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG!
1685:2020-07-21 20:07:57.699 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
1686:2020-07-21 20:07:58.732 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG!
1687-2020-07-21 20:07:59.765 152406 INFO oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG!
1688:2020-07-21 20:08:08.015 152406 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG!
1689:2020-07-21 20:08:23.058 152406 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG!
1690-2020-07-21 20:09:01.075 152406 INFO neutron.agent.dhcp.agent [req-f1c75a55-aeff-4415-9fa3-824874ed72ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1691-2020-07-21 20:09:01.392 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
1692-2020-07-21 20:09:01.479 152406 INFO neutron.agent.dhcp.agent [req-062d6247-fb9a-4a27-953c-1e3e15da83e2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1693-2020-07-21 20:09:01.492 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
1694-2020-07-21 20:09:01.584 152406 INFO neutron.agent.dhcp.agent [req-b71f0f58-3857-47d7-8b7e-b19c52e196ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
1695-2020-07-21 20:09:01.612 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG!
1696-2020-07-21 20:09:01.689 152406 INFO neutron.agent.dhcp.agent [req-ea33f202-4d02-4a88-9905-8a250e1a47d5 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED
^^^^^
------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:2
AnalyzedBlockLinesSize:22
38396-2020-07-21 20:07:55.925 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0011032
38397-2020-07-21 20:07:56.030 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0008850
38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
38399:2020-07-21 20:07:56.380 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [da0fafee-b840-44b7-9c5f-0afda51faab6] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
38400:2020-07-21 20:07:56.380 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [5997eee9-f4a0-4120-8c1a-143089d3fdc7] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
...
...
...
LogTool --> THIS BLOCK IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
controller-2
------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:2
AnalyzedBlockLinesSize:20
29367-2020-07-21 20:07:55.723 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
29368-2020-07-21 20:07:55.730 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29370:2020-07-21 20:07:56.379 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29371:2020-07-21 20:07:56.382 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29372:2020-07-21 20:07:56.383 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29373:2020-07-21 20:07:56.385 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29374:2020-07-21 20:07:56.386 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG!
29375:2020-07-21 20:07:57.716 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29376-2020-07-21 20:07:57.724 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
29377-2020-07-21 20:07:57.729 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
29378:2020-07-21 20:07:57.805 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29379:2020-07-21 20:07:57.814 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29380:2020-07-21 20:07:57.818 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29381:2020-07-21 20:07:57.821 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29382:2020-07-21 20:07:57.824 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29383:2020-07-21 20:07:58.781 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG!
29384-2020-07-21 20:07:58.858 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29385-2020-07-21 20:07:58.859 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29386-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29387-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29388-2020-07-21 20:07:58.866 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG!
29389-2020-07-21 20:07:59.725 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
29390-2020-07-21 20:07:59.732 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:20
38505-2020-07-21 20:07:56.043 28 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0026181
38506-2020-07-21 20:07:56.045 29 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0032661
38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
38508:2020-07-21 20:07:56.379 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
38509:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG!
...
...
...
LogTool --> THIS BLOCK IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller-2.inter...
^^^^^
ection. Check login credentials: Socket closed: IOError: Socket closed
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------
IsTracebackBlock:False
UniqueCounter:1
AnalyzedBlockLinesSize:29
38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38525-2020-07-21 20:07:57.723 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38526:2020-07-21 20:07:57.724 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [6ac6e794-3561-4cc3-99a9-cc5e078dd130] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38527-2020-07-21 20:07:57.724 38 ERROR oslo.messaging._drivers.impl_rabbit [-] [12f38994-d07a-40df-bdb4-4f49c905b417] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38528-2020-07-21 20:07:57.732 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38529-2020-07-21 20:07:57.739 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38530-2020-07-21 20:07:57.740 39 ERROR oslo.messaging._drivers.impl_rabbit [-] [b237f335-3172-434d-88f5-f14d37bf485c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38531-2020-07-21 20:07:57.745 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [8246b7f8-db45-4681-8910-2bf85df8fafa] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38532-2020-07-21 20:07:57.745 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [93e8e483-ef8e-41f0-bf61-8f8c25fb2efb] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38533-2020-07-21 20:07:57.754 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [c11cfdee-53e0-4574-988e-e037adf8422b] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38534-2020-07-21 20:07:57.759 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [091e9689-2d34-46e5-a40c-8c4fe3b200c5] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38535:2020-07-21 20:07:57.958 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [2a4c7a49-5f9b-4b09-8636-16320c659a68] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG!
38536-2020-07-21 20:07:58.047 33 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0022709
38537-2020-07-21 20:07:58.049 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0022650
38538-2020-07-21 20:07:58.470 32 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0023279
38539-2020-07-21 20:07:58.513 43 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOE...<--LogTool-LINE IS TOO LONG!
38540-2020-07-21 20:07:58.527 43 INFO oslo.messaging._drivers.impl_rabbit [-] [9f9ed2f1-0b23-40a4-99ac-4082ec985e21] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG!
38541-2020-07-21 20:07:58.557 40 INFO oslo.messaging._drivers.impl_rabbit [-] [1e042b35-4e4d-4e0d-8aec-ebdc7efe19b7] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG!
38542-2020-07-21 20:07:58.564 40 INFO oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG!
LogTool --> POTENTIAL BLOCK'S ISSUES:
38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-controller-2.in...
^^^^^
no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED
^^^^^
rnalapi.i2k2cloud02.com:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds.: RecoverableConnectionError: <AMQPError: unknown error>
^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^
slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed
^^^^^ ^^^^^
More information about the dev
mailing list