hyp-0 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:18 26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! 26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ ------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:17 692-2020-07-21 20:06:32.245 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 693-2020-07-21 20:07:32.328 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 694:2020-07-21 20:08:04.803 1 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOErro...<--LogTool-LINE IS TOO LONG! 695:2020-07-21 20:08:19.844 1 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errn...<--LogTool-LINE IS TOO LONG! 696-2020-07-21 20:08:34.439 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 697-2020-07-21 20:09:34.182 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 698-2020-07-21 20:10:34.274 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 699-2020-07-21 20:11:36.243 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 700-2020-07-21 20:12:38.198 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 701-2020-07-21 20:13:40.121 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 702-2020-07-21 20:14:39.239 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED hyp-1 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:33 26746-2020-07-21 20:07:55.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26747-2020-07-21 20:07:55.624 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26749:2020-07-21 20:07:56.370 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [36454cb1-0589-4cb6-9f63-7e2834d27056] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26750:2020-07-21 20:07:56.371 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [b6ab5be6-cc06-4d26-bc76-0f8161b10867] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26751:2020-07-21 20:07:56.372 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [db71cb6f-a426-4456-b56d-13562f98aa84] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26752:2020-07-21 20:07:56.373 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [9af86e0e-d8e0-4738-bb9c-92c544981c2b] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26753-2020-07-21 20:07:57.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26754-2020-07-21 20:07:57.625 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26755:2020-07-21 20:07:57.710 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! ... ---< BLOCK IS TOO LONG >--- ... LogTool --> POTENTIAL BLOCK'S ISSUES: 26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:25 691-2020-07-21 20:06:13.016 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 692-2020-07-21 20:07:13.052 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2....<--LogTool-LINE IS TOO LONG! 694:2020-07-21 20:07:57.715 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG! 695:2020-07-21 20:07:58.744 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG! 696-2020-07-21 20:07:59.767 1 INFO oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] Reconnected to AMQP server on overcl...<--LogTool-LINE IS TOO LONG! 697:2020-07-21 20:08:01.107 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG! 698:2020-07-21 20:08:02.182 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG! 699-2020-07-21 20:08:03.205 1 INFO oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-4...<--LogTool-LINE IS TOO LONG! 700-2020-07-21 20:08:17.601 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 701-2020-07-21 20:09:14.030 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 702-2020-07-21 20:10:16.027 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 703-2020-07-21 20:11:16.955 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 704-2020-07-21 20:12:17.038 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 705-2020-07-21 20:13:16.986 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ controller-0 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:27 29685-2020-07-21 20:07:55.226 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29686-2020-07-21 20:07:55.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29688:2020-07-21 20:07:56.375 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29689:2020-07-21 20:07:56.376 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29690:2020-07-21 20:07:56.377 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29691-2020-07-21 20:07:57.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29692-2020-07-21 20:07:57.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29693:2020-07-21 20:07:57.483 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29694:2020-07-21 20:07:57.706 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29695:2020-07-21 20:07:57.729 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29696:2020-07-21 20:07:57.733 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29697-2020-07-21 20:07:58.513 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29698-2020-07-21 20:07:58.765 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29699-2020-07-21 20:07:58.768 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29700-2020-07-21 20:07:58.812 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29701-2020-07-21 20:07:59.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29702-2020-07-21 20:07:59.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29703-2020-07-21 20:08:01.228 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:26 1609-2020-07-21 20:07:16.952 177642 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG! 1610-2020-07-21 20:07:17.108 177642 INFO neutron.agent.dhcp.agent [req-026ba197-47b8-4006-8ff3-155d45cb87a2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 1612:2020-07-21 20:07:57.777 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! 1613-2020-07-21 20:07:58.790 177642 INFO oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG! 1614:2020-07-21 20:08:08.565 177642 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG! 1615:2020-07-21 20:08:23.607 177642 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG! 1616-2020-07-21 20:08:49.052 177642 INFO neutron.agent.dhcp.agent [req-46823ae5-be71-437f-adf4-c4db97fe28d7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1617-2020-07-21 20:08:49.336 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1618-2020-07-21 20:08:49.355 177642 INFO neutron.agent.dhcp.agent [req-c1e2fa2e-8376-417f-aa25-04989ec04ce7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1619-2020-07-21 20:08:49.406 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1620-2020-07-21 20:08:49.458 177642 INFO neutron.agent.dhcp.agent [req-d81a548c-f5ff-4ddc-bfbb-1a038ec6e334 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1621-2020-07-21 20:08:49.482 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1622-2020-07-21 20:08:49.561 177642 INFO neutron.agent.dhcp.agent [req-52d2cc2c-3cfa-4b00-9d1f-3612f7eb9d8e - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:22 38644-2020-07-21 20:07:55.818 38 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0009890 38645-2020-07-21 20:07:55.942 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0009868 38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38647:2020-07-21 20:07:56.373 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [bdd4035e-b970-48aa-bd17-5f3a9fa3ee4d] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38648:2020-07-21 20:07:56.374 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:18 38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38679-2020-07-21 20:07:57.699 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38680:2020-07-21 20:07:57.708 47 ERROR oslo.messaging._drivers.impl_rabbit [-] [7dfef257-02a9-447a-8be4-9149a980f37d] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38681-2020-07-21 20:07:57.709 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [42757d9d-ade4-4c52-9083-7fef8f3d23c0] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38682-2020-07-21 20:07:57.710 50 ERROR oslo.messaging._drivers.impl_rabbit [-] [98688dd7-c96a-4bee-a980-92cbca8e1541] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38683-2020-07-21 20:07:57.712 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [d836ff75-44af-4e1e-b985-2a3108a1f337] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38684-2020-07-21 20:07:57.712 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [efa6cb10-7f90-48f5-95a7-595d6a3d9c9a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38685-2020-07-21 20:07:57.712 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [52904842-0a17-4378-a279-82d24a91a95a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38686-2020-07-21 20:07:57.715 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [63bd5e71-06e1-45f9-bd66-ccc1fd087a1c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38687-2020-07-21 20:07:57.719 46 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6f11462-ac09-4098-b70c-43f4e99fe87c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-controller-2.in... ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ controller-1 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:31 29669-2020-07-21 20:07:55.494 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! 29670-2020-07-21 20:07:55.499 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! 29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29672:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [f4016ef6-75a6-405d-82cf-ae3f4557a3d8] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29673:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [eda2f576-314d-47fa-9348-0f9a5e5fe9c7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29674:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea7378e5-bd62-46a3-8cf0-6cf8000fdba7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29675:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29676:2020-07-21 20:07:56.505 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3d4af668-0b5f-4f30-8f03-9778c5650d1b] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29677-2020-07-21 20:07:57.502 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! 29678-2020-07-21 20:07:57.534 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! ... ---< BLOCK IS TOO LONG >--- ... 29690-2020-07-21 20:07:58.926 153102 INFO oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29691-2020-07-21 20:07:59.503 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ ection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer ------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:2 AnalyzedBlockLinesSize:24 1682-2020-07-21 20:07:16.957 152406 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG! 1683-2020-07-21 20:07:17.177 152406 INFO neutron.agent.dhcp.agent [req-2c406398-1696-43dc-b0f5-457a287a5e34 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 1685:2020-07-21 20:07:57.699 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! 1686:2020-07-21 20:07:58.732 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! 1687-2020-07-21 20:07:59.765 152406 INFO oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG! 1688:2020-07-21 20:08:08.015 152406 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG! 1689:2020-07-21 20:08:23.058 152406 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG! 1690-2020-07-21 20:09:01.075 152406 INFO neutron.agent.dhcp.agent [req-f1c75a55-aeff-4415-9fa3-824874ed72ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1691-2020-07-21 20:09:01.392 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1692-2020-07-21 20:09:01.479 152406 INFO neutron.agent.dhcp.agent [req-062d6247-fb9a-4a27-953c-1e3e15da83e2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1693-2020-07-21 20:09:01.492 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1694-2020-07-21 20:09:01.584 152406 INFO neutron.agent.dhcp.agent [req-b71f0f58-3857-47d7-8b7e-b19c52e196ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1695-2020-07-21 20:09:01.612 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1696-2020-07-21 20:09:01.689 152406 INFO neutron.agent.dhcp.agent [req-ea33f202-4d02-4a88-9905-8a250e1a47d5 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:2 AnalyzedBlockLinesSize:22 38396-2020-07-21 20:07:55.925 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0011032 38397-2020-07-21 20:07:56.030 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0008850 38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38399:2020-07-21 20:07:56.380 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [da0fafee-b840-44b7-9c5f-0afda51faab6] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38400:2020-07-21 20:07:56.380 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [5997eee9-f4a0-4120-8c1a-143089d3fdc7] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ controller-2 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:2 AnalyzedBlockLinesSize:20 29367-2020-07-21 20:07:55.723 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29368-2020-07-21 20:07:55.730 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29370:2020-07-21 20:07:56.379 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29371:2020-07-21 20:07:56.382 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29372:2020-07-21 20:07:56.383 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29373:2020-07-21 20:07:56.385 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29374:2020-07-21 20:07:56.386 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29375:2020-07-21 20:07:57.716 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29376-2020-07-21 20:07:57.724 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29377-2020-07-21 20:07:57.729 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29378:2020-07-21 20:07:57.805 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29379:2020-07-21 20:07:57.814 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29380:2020-07-21 20:07:57.818 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29381:2020-07-21 20:07:57.821 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29382:2020-07-21 20:07:57.824 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29383:2020-07-21 20:07:58.781 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29384-2020-07-21 20:07:58.858 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29385-2020-07-21 20:07:58.859 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29386-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29387-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29388-2020-07-21 20:07:58.866 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29389-2020-07-21 20:07:59.725 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29390-2020-07-21 20:07:59.732 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:20 38505-2020-07-21 20:07:56.043 28 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0026181 38506-2020-07-21 20:07:56.045 29 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0032661 38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38508:2020-07-21 20:07:56.379 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38509:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:29 38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38525-2020-07-21 20:07:57.723 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38526:2020-07-21 20:07:57.724 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [6ac6e794-3561-4cc3-99a9-cc5e078dd130] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38527-2020-07-21 20:07:57.724 38 ERROR oslo.messaging._drivers.impl_rabbit [-] [12f38994-d07a-40df-bdb4-4f49c905b417] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38528-2020-07-21 20:07:57.732 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38529-2020-07-21 20:07:57.739 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38530-2020-07-21 20:07:57.740 39 ERROR oslo.messaging._drivers.impl_rabbit [-] [b237f335-3172-434d-88f5-f14d37bf485c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38531-2020-07-21 20:07:57.745 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [8246b7f8-db45-4681-8910-2bf85df8fafa] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38532-2020-07-21 20:07:57.745 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [93e8e483-ef8e-41f0-bf61-8f8c25fb2efb] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38533-2020-07-21 20:07:57.754 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [c11cfdee-53e0-4574-988e-e037adf8422b] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38534-2020-07-21 20:07:57.759 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [091e9689-2d34-46e5-a40c-8c4fe3b200c5] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38535:2020-07-21 20:07:57.958 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [2a4c7a49-5f9b-4b09-8636-16320c659a68] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38536-2020-07-21 20:07:58.047 33 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0022709 38537-2020-07-21 20:07:58.049 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0022650 38538-2020-07-21 20:07:58.470 32 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0023279 38539-2020-07-21 20:07:58.513 43 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOE...<--LogTool-LINE IS TOO LONG! 38540-2020-07-21 20:07:58.527 43 INFO oslo.messaging._drivers.impl_rabbit [-] [9f9ed2f1-0b23-40a4-99ac-4082ec985e21] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG! 38541-2020-07-21 20:07:58.557 40 INFO oslo.messaging._drivers.impl_rabbit [-] [1e042b35-4e4d-4e0d-8aec-ebdc7efe19b7] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG! 38542-2020-07-21 20:07:58.564 40 INFO oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-controller-2.in... ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^