From jcapitao at redhat.com Wed Jul 1 15:14:07 2020 From: jcapitao at redhat.com (Joel Capitao) Date: Wed, 1 Jul 2020 17:14:07 +0200 Subject: [rdo-dev] [rdo-users] [Meeting] RDO meeting (2020-07-01) minutes Message-ID: ============================== #rdo: RDO meeting - 2020-07-01 ============================== Meeting started by jcapitao at 14:01:45 UTC. The full logs are available athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_07_01/2020/rdo_meeting___2020_07_01.2020-07-01-14.01.log.html . Meeting summary --------------- * roll call (jcapitao, 14:02:16) * reqcheck - override option (jcapitao, 14:08:05) * LINK: https://softwarefactory-project.io/r/#/c/18736/ (jcapitao, 14:08:38) * LINK: https://review.rdoproject.org/r/#/c/26165/ (jcapitao, 14:14:02) * Test Days and Doc Days events (jcapitao, 14:15:12) * LINK: http://rdoproject.org/events/docdays/ (spotz, 14:18:52) * ACTION: spotz talk to cshort about whether RDO can claim twitch time (spotz, 14:37:24) * ACTION: spotz - Can hopin.to do a hackfest? If not alternatives (spotz, 14:47:25) * chair for next week (jcapitao, 15:04:52) * ACTION: ykarel to chair next meeting (jcapitao, 15:05:28) Meeting ended at 15:07:00 UTC. Action items, by person ----------------------- * spotz * spotz talk to cshort about whether RDO can claim twitch time * spotz - Can hopin.to do a hackfest? If not alternatives * ykarel * ykarel to chair next meeting People present (lines said) --------------------------- * amoralej (68) * spotz (52) * jcapitao (42) * openstack (8) * jbrooks (8) * ykarel (6) * rdogerrit (4) * rh-jelabarre (1) Generated by `MeetBot`_ 0.1.4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Mon Jul 6 09:23:53 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Mon, 6 Jul 2020 11:23:53 +0200 Subject: [rdo-dev] [RDO] Weekly Status for 2020-07-03 Message-ID: Promotions * Latest promotions (TripleO CI): * Master: 02nd July * Ussuri: 02nd July * Train: 02nd July * Stein: 25th June * Known blocker stein/queens * https://bugs.launchpad.net/tripleo/+bug/1873892 * Known blocker train * https://bugs.launchpad.net/tripleo/+bug/1883106 Packages * Neutron-fwaas has been retired upstream and is being removed from RDO Victoria * Congress and paunch are retired upstream and are removed from RDO Victoria. * Ansible is updated to 2.9.10 in Train/Ussuri and Victoria. * openstack-selinux updated to 0.8.23 to fix some CentOS 8 issues. * Ovn is updated to 20.03.0-4 in Ussuri and Master * Urllib3 is updated to 1.25.7-3 in Ussuri/Train to handle upgrade of python3-six. Vexxhost * ci.centos weirdo jobs are migrated to vexxhost, Other RDO jobs are also being evaluated on vexxhost * https://review.rdoproject.org/r/#/q/topic:vexxhost Other * Update of OpenStack clients to Ussuri releases in Fedora is in progress * https://trello.com/c/xHGOdCU0/737-update-openstack-clients-to-ussuri-releases-in-fedora * networking-l2gw removal is being discussed upstream. Finally it will probably be moved to x namespace: * https://review.opendev.org/#/q/topic:complete-retirement-networking-l2gw * rabbitmq-server has been updated for CentOS 8.2, tested and pushed to CentOS mirrors including aarch64 builds * CentOS8 jobs for train release are being set up -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at osuosl.org Mon Jul 6 15:46:31 2020 From: lance at osuosl.org (Lance Albertson) Date: Mon, 6 Jul 2020 08:46:31 -0700 Subject: [rdo-dev] rdo-trunk-ussuri-tested repo not working on CentOS 8 Message-ID: Hi all, I've noticed that the rdo-trunk-ussuri-tested repo that is included with centos-release-openstack-ussuri package is not working. Specifically the URL https://trunk.rdoproject.org/centos8-ussuri/current-passed-ci/ does not exist. Is this the correct location or should it be something different? Thanks! -- Lance Albertson Director Oregon State University | Open Source Lab -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Mon Jul 6 15:57:42 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Mon, 6 Jul 2020 17:57:42 +0200 Subject: [rdo-dev] rdo-trunk-ussuri-tested repo not working on CentOS 8 In-Reply-To: References: Message-ID: Hello, On Mon, Jul 6, 2020 at 5:47 PM Lance Albertson wrote: > Hi all, > > I've noticed that the rdo-trunk-ussuri-tested repo that is included > with centos-release-openstack-ussuri package is not working. Specifically > the URL > https://trunk.rdoproject.org/centos8-ussuri/current-passed-ci/ does not > exist. Is this the correct location or should it be something different? > That's URL is not longer valid. With the componentization of the RDO Trunk repo in ussuri there is no single url that you can use. The best alternative is to copy file https://trunk.rdoproject.org/centos8-master/current-passed-ci/delorean.repo to /etc/yum.repos.d > Thanks! > > -- > Lance Albertson > Director > Oregon State University | Open Source Lab > _______________________________________________ > dev mailing list > dev at lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscribe at lists.rdoproject.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at osuosl.org Mon Jul 6 16:00:15 2020 From: lance at osuosl.org (Lance Albertson) Date: Mon, 6 Jul 2020 09:00:15 -0700 Subject: [rdo-dev] rdo-trunk-ussuri-tested repo not working on CentOS 8 In-Reply-To: References: Message-ID: On Mon, Jul 6, 2020 at 8:58 AM Alfredo Moralejo Alonso wrote: > Hello, > > On Mon, Jul 6, 2020 at 5:47 PM Lance Albertson wrote: > >> Hi all, >> >> I've noticed that the rdo-trunk-ussuri-tested repo that is included >> with centos-release-openstack-ussuri package is not working. Specifically >> the URL >> https://trunk.rdoproject.org/centos8-ussuri/current-passed-ci/ does not >> exist. Is this the correct location or should it be something different? >> > > That's URL is not longer valid. With the componentization of the RDO Trunk > repo in ussuri there is no single url that you can use. The best > alternative is to copy file > https://trunk.rdoproject.org/centos8-master/current-passed-ci/delorean.repo > to /etc/yum.repos.d > Should this repo just be removed from the centos-release-openstack-ussuri package then or is there another way to deal with this from a configuration management perspective? Thanks- -- Lance Albertson Director Oregon State University | Open Source Lab -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Mon Jul 6 15:59:55 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Mon, 6 Jul 2020 17:59:55 +0200 Subject: [rdo-dev] rdo-trunk-ussuri-tested repo not working on CentOS 8 In-Reply-To: References: Message-ID: On Mon, Jul 6, 2020 at 5:57 PM Alfredo Moralejo Alonso wrote: > Hello, > > On Mon, Jul 6, 2020 at 5:47 PM Lance Albertson wrote: > >> Hi all, >> >> I've noticed that the rdo-trunk-ussuri-tested repo that is included >> with centos-release-openstack-ussuri package is not working. Specifically >> the URL >> https://trunk.rdoproject.org/centos8-ussuri/current-passed-ci/ does not >> exist. Is this the correct location or should it be something different? >> > > That's URL is not longer valid. With the componentization of the RDO Trunk > repo in ussuri there is no single url that you can use. The best > alternative is to copy file > https://trunk.rdoproject.org/centos8-master/current-passed-ci/delorean.repo > to /etc/yum.repos.d > Sorry, that's master, not ussuri. For ussuri you can use https://trunk.rdoproject.org/centos8-ussuri/current-tripleo/delorean.repo > > >> Thanks! >> >> -- >> Lance Albertson >> Director >> Oregon State University | Open Source Lab >> _______________________________________________ >> dev mailing list >> dev at lists.rdoproject.org >> http://lists.rdoproject.org/mailman/listinfo/dev >> >> To unsubscribe: dev-unsubscribe at lists.rdoproject.org >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Tue Jul 7 08:26:27 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Tue, 7 Jul 2020 10:26:27 +0200 Subject: [rdo-dev] rdo-trunk-ussuri-tested repo not working on CentOS 8 In-Reply-To: References: Message-ID: On Mon, Jul 6, 2020 at 6:00 PM Lance Albertson wrote: > > > On Mon, Jul 6, 2020 at 8:58 AM Alfredo Moralejo Alonso < > amoralej at redhat.com> wrote: > >> Hello, >> >> On Mon, Jul 6, 2020 at 5:47 PM Lance Albertson wrote: >> >>> Hi all, >>> >>> I've noticed that the rdo-trunk-ussuri-tested repo that is included >>> with centos-release-openstack-ussuri package is not working. Specifically >>> the URL >>> https://trunk.rdoproject.org/centos8-ussuri/current-passed-ci/ does not >>> exist. Is this the correct location or should it be something different? >>> >> >> That's URL is not longer valid. With the componentization of the RDO >> Trunk repo in ussuri there is no single url that you can use. The best >> alternative is to copy file >> https://trunk.rdoproject.org/centos8-master/current-passed-ci/delorean.repo >> to /etc/yum.repos.d >> > > Should this repo just be removed from the centos-release-openstack-ussuri > package then or is there another way to deal with this from a configuration > management perspective? > I'll replace existing repo by a set of per-component repositories but it will take a while to have the new release rpm in the repos: https://review.rdoproject.org/r/28407 You'll need to enable all the rdo-trunk-ussuri-tested-component-* repositories. > > Thanks- > > -- > Lance Albertson > Director > Oregon State University | Open Source Lab > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lance at osuosl.org Tue Jul 7 16:25:13 2020 From: lance at osuosl.org (Lance Albertson) Date: Tue, 7 Jul 2020 09:25:13 -0700 Subject: [rdo-dev] rdo-trunk-ussuri-tested repo not working on CentOS 8 In-Reply-To: References: Message-ID: On Tue, Jul 7, 2020 at 1:27 AM Alfredo Moralejo Alonso wrote: > > > On Mon, Jul 6, 2020 at 6:00 PM Lance Albertson wrote: > >> >> >> On Mon, Jul 6, 2020 at 8:58 AM Alfredo Moralejo Alonso < >> amoralej at redhat.com> wrote: >> >>> Hello, >>> >>> On Mon, Jul 6, 2020 at 5:47 PM Lance Albertson wrote: >>> >>>> Hi all, >>>> >>>> I've noticed that the rdo-trunk-ussuri-tested repo that is included >>>> with centos-release-openstack-ussuri package is not working. Specifically >>>> the URL >>>> https://trunk.rdoproject.org/centos8-ussuri/current-passed-ci/ does >>>> not exist. Is this the correct location or should it be something >>>> different? >>>> >>> >>> That's URL is not longer valid. With the componentization of the RDO >>> Trunk repo in ussuri there is no single url that you can use. The best >>> alternative is to copy file >>> https://trunk.rdoproject.org/centos8-master/current-passed-ci/delorean.repo >>> to /etc/yum.repos.d >>> >> >> Should this repo just be removed from the centos-release-openstack-ussuri >> package then or is there another way to deal with this from a configuration >> management perspective? >> > > I'll replace existing repo by a set of per-component repositories but it > will take a while to have the new release rpm in the repos: > > https://review.rdoproject.org/r/28407 > > You'll need to enable all the rdo-trunk-ussuri-tested-component-* > repositories. > >> Excellent. Thanks for pointing that out! -- Lance Albertson Director Oregon State University | Open Source Lab -------------- next part -------------- An HTML attachment was scrubbed... URL: From apevec at redhat.com Wed Jul 8 11:10:43 2020 From: apevec at redhat.com (Alan Pevec) Date: Wed, 8 Jul 2020 13:10:43 +0200 Subject: [rdo-dev] RDO Cloud operations today Message-ID: Hi all, FYI RDO Cloud is undergoing scheduled movement of some of its racks, control plane and infra services (www, lists, CI pool) should stay up all the time. In case of unplanned outage we'll let you know in this thread and also announce when those operations are finished. At one point there will be reduced CI pool capacity, so expect to see longer queues in https://review.rdoproject.org/zuul/status during the day. Cheers, Alan From yatinkarel at gmail.com Wed Jul 8 15:03:30 2020 From: yatinkarel at gmail.com (YATIN KAREL) Date: Wed, 8 Jul 2020 20:33:30 +0530 Subject: [rdo-dev] [rdo-users] [Meeting] RDO meeting (2020-07-08) minutes Message-ID: ============================== #rdo: RDO meeting - 2020-07-08 ============================== Meeting started by ykarel at 14:02:50 UTC. The full logs are available athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_07_08/2020/rdo_meeting___2020_07_08.2020-07-08-14.02.log.html . Meeting summary --------------- * roll call (ykarel, 14:03:25) * Upcoming events (ykarel, 14:06:19) * ACTION: spotz mail lists about doc holes (spotz, 14:35:34) * ACTION: spotz make a basic plan for workshop and hackfest to share with the group next week (spotz, 14:36:28) * streaming and OpenShift COmmons AMA (7/27) (ykarel, 14:41:05) * review needed (spotz, 14:52:51) * LINK: https://review.rdoproject.org/r/#/c/28218/ (spotz, 14:53:03) * LINK: https://docs.openstack.org/placement/latest/install/verify.html is the page with the instructions (spotz, 14:56:24) * Next Week's chair (ykarel, 14:59:14) * ACTION: amoralej to chair next week (ykarel, 14:59:55) * Open Floor (ykarel, 15:00:01) Meeting ended at 15:00:34 UTC. Action items, by person ----------------------- * amoralej * amoralej to chair next week * spotz * spotz mail lists about doc holes * spotz make a basic plan for workshop and hackfest to share with the group next week People present (lines said) --------------------------- * spotz (57) * amoralej (44) * ykarel (34) * jcapitao (5) * openstack (4) Generated by `MeetBot`_ 0.1.4 ______________________________________________ users mailing list users at lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/users To unsubscribe: users-unsubscribe at lists.rdoproject.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From whayutin at redhat.com Thu Jul 9 12:14:57 2020 From: whayutin at redhat.com (Wesley Hayutin) Date: Thu, 9 Jul 2020 06:14:57 -0600 Subject: [rdo-dev] RDO Cloud operations today In-Reply-To: References: Message-ID: On Wed, Jul 8, 2020 at 5:11 AM Alan Pevec wrote: > Hi all, > > FYI RDO Cloud is undergoing scheduled movement of some of its racks, > control plane and infra services (www, lists, CI pool) should stay up > all the time. > In case of unplanned outage we'll let you know in this thread and also > announce when those operations are finished. > At one point there will be reduced CI pool capacity, so expect to see > longer queues in https://review.rdoproject.org/zuul/status during the > day. > > Cheers, > Alan > > Any updates on the status of the operations? > _______________________________________________ > dev mailing list > dev at lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscribe at lists.rdoproject.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From apevec at redhat.com Thu Jul 9 15:40:45 2020 From: apevec at redhat.com (Alan Pevec) Date: Thu, 9 Jul 2020 17:40:45 +0200 Subject: [rdo-dev] RDO Cloud operations today In-Reply-To: References: Message-ID: > Any updates on the status of the operations? I was giving updates in #rdo IRC since we had unstable networking and lists.r.o was not reachable, last update as of few hours ago was: rdocloud networking should be now stable, uplink is not redundant, IT will work on getting back failover during the day I'll update this thread when I get confirmation from ops that redundancy and full CI rack is back, CI pool is now reduced ~50% Cheers, Alan From apevec at redhat.com Fri Jul 10 07:46:04 2020 From: apevec at redhat.com (Alan Pevec) Date: Fri, 10 Jul 2020 09:46:04 +0200 Subject: [rdo-dev] RDO Cloud operations today In-Reply-To: References: Message-ID: > last update as of few hours ago was: rdocloud networking should be now > stable, uplink is not redundant, IT will work on getting back failover > during the day Update as of this morning: uplink redundancy was restored last night, restoring full CI pool is planned today. Cheers, Alan From whayutin at redhat.com Fri Jul 10 11:44:30 2020 From: whayutin at redhat.com (Wesley Hayutin) Date: Fri, 10 Jul 2020 05:44:30 -0600 Subject: [rdo-dev] RDO Cloud operations today In-Reply-To: References: Message-ID: On Fri, Jul 10, 2020 at 1:46 AM Alan Pevec wrote: > > last update as of few hours ago was: rdocloud networking should be now > > stable, uplink is not redundant, IT will work on getting back failover > > during the day > > Update as of this morning: > uplink redundancy was restored last night, > restoring full CI pool is planned today. > > Cheers, > Alan > Good news.. thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From yatinkarel at gmail.com Mon Jul 13 09:21:42 2020 From: yatinkarel at gmail.com (YATIN KAREL) Date: Mon, 13 Jul 2020 14:51:42 +0530 Subject: [rdo-dev] [rdo-users] [RDO] Weekly Status for 2020-07-10 Message-ID: Promotions * Latest promotions (TripleO CI): * Master: 08th July * Ussuri: 09th July * Train: 08th July * Stein: 08th July * Known blocker stein/queens - Random failures * https://bugs.launchpad.net/tripleo/+bug/1873892 * Known blocker master - Compute Component * https://bugs.launchpad.net/tripleo/+bug/1886862 Packages * Neutron-fwaas has been retired upstream and is being removed from RDO Victoria * Ansible is updated to 2.9.10 in Train CentOS7. Vexxhost * ci.centos weirdo jobs are migrated to vexxhost, Other RDO jobs are also being evaluated on vexxhost * https://review.rdoproject.org/r/#/q/topic:vexxhost Other * Update of OpenStack clients to Ussuri releases in Fedora is in progress * https://trello.com/c/xHGOdCU0/737-update-openstack-clients-to-ussuri-releases-in-fedora * networking-l2gw removal is being discussed upstream. Finally it will probably be moved to x namespace: * https://review.opendev.org/#/q/topic:complete-retirement-networking-l2gw * RPM specs are being updated to support python3.10 * https://trello.com/c/A9GZWxqR/739-make-rdo-support-python-version-with-2-digits * CentOS8 check and promotion jobs for train release are being set up. * Revival of NFV SIG is work in progress. New chair has been approved by CentOS board On behalf of RDO Team _______________________________________________ users mailing list users at lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/users To unsubscribe: users-unsubscribe at lists.rdoproject.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From amy at demarco.com Wed Jul 15 00:17:31 2020 From: amy at demarco.com (Amy Marrich) Date: Tue, 14 Jul 2020 19:17:31 -0500 Subject: [rdo-dev] RDO Documentation Message-ID: In preparation for holding an upcoming RDO Doc Days, we are looking for feedback from the community as to where the RDO docs need improvement. Is there documentation you've read that you noticed was out of date? Or maybe you went to look for something and the information was incomplete or not even there? Even if these RDO(RHEL/CentOS) related issues were on openstack.org vs rdoproject.org we want to know! Thanks, Amy Marrich (spotz) -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Wed Jul 15 15:06:52 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Wed, 15 Jul 2020 17:06:52 +0200 Subject: [rdo-dev] [Meeting] RDO meeting - 2020-07-15 Message-ID: =========================== #rdo: RDO meeting - 2020-07-15 ============================== Meeting started by amoralej at 14:00:56 UTC. The full logs are available athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_07_15/2020/rdo_meeting___2020_07_15.2020-07-15-14.00.log.html . Meeting summary --------------- * rall call (amoralej, 14:01:02) * Upcoming events - https://review.rdoproject.org/etherpad/p/rdo-events (amoralej, 14:05:59) * LINK: https://github.com/cjeanner/tripleo-lab (mwhahaha, 14:24:35) * restarting NFV SIG for more official OVS/OVN builds (amoralej, 14:39:35) * LINK: https://lists.centos.org/pipermail/centos-devel/2020-July/055933.html (amoralej, 14:39:50) * ACTION: everyone interested in NFV SIG please join us! (amoralej, 14:41:04) * LINK: scratch OVS FDP build https://cbs.centos.org/koji/taskinfo?taskID=1386673 (apevec, 14:44:44) * next week chair (amoralej, 14:49:10) * ACTION: spotz to chair next week (amoralej, 14:49:57) * open floor (amoralej, 14:50:08) Meeting ended at 15:02:59 UTC. Action items, by person ----------------------- * spotz * spotz to chair next week People present (lines said) --------------------------- * amoralej (91) * spotz (55) * ykarel (33) * apevec (11) * mwhahaha (9) * rdogerrit (6) * Tengu (6) * openstack (6) * jcapitao (2) * sshnaidm (2) * EmilienM (1) Generated by `MeetBot`_ 0.1.4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nhicher at redhat.com Wed Jul 15 15:38:32 2020 From: nhicher at redhat.com (Nicolas Hicher) Date: Wed, 15 Jul 2020 11:38:32 -0400 Subject: [rdo-dev] Planned outage of review.rdoproject.org: 2020-07-15 from 18:00 to 20:00 UTC Message-ID: Hello folks, Our cloud provider plans to do maintainance operation on 2020-07-15 from 18:00 to 20:00 UTC. Service interruption is expected, including: - Zuul CI not running jobs for gerrit, github or opendev. - RDO Trunk not building new packages. - DLRN API. - review.rdoproject.org and softwarefactory-project.io gerrit service. Regards, Nicolas, on behalf of the Software Factory Operation Team From amoralej at redhat.com Mon Jul 20 10:29:52 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Mon, 20 Jul 2020 12:29:52 +0200 Subject: [rdo-dev] [RDO] Weekly Status for 2020-07-10 Message-ID: Promotions * Latest promotions (TripleO CI): * Master: 15th July * Ussuri: 16th July * Train: 15th July * Stein: 15th July * Known blocker stein/queens - Random failures * https://bugs.launchpad.net/tripleo/+bug/1873892 * Known blocker master * https://bugs.launchpad.net/tripleo/+bug/1887856 Packages * Neutron-fwaas has been retired upstream and is removed from RDO Victoria * python-importlib-metadata and python-importlib-resources are being updated in Victoria release: * https://review.rdoproject.org/r/#/c/28604 Vexxhost * ci.centos weirdo jobs are migrated to vexxhost, Other RDO jobs are also being evaluated on vexxhost * https://review.rdoproject.org/r/#/q/topic:vexxhost Other * RPM specs are being updated to support python3.10 * https://trello.com/c/A9GZWxqR/739-make-rdo-support-python-version-with-2-digits * CentOS8 check and promotion jobs for train release are being set up. * neutron-fwaas has been removed from RDO Victoria release * Revival of NFV SIG is work in progress. New chair has been approved by CentOS board On behalf of RDO -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpathak at i2k2.com Wed Jul 22 13:23:39 2020 From: rpathak at i2k2.com (Rahul Pathak) Date: Wed, 22 Jul 2020 18:53:39 +0530 (IST) Subject: [rdo-dev] tripleo cluster failure In-Reply-To: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> Message-ID: <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> Hi, I have installed tripleo openstack version rocky containerized with Undercloud in virtual platform and 3 controllers and 2 compute Baremetal. My whole setup is running on centos7. Overcloud cluster start failing once number of networks in overcloud reach more than 70. Lots of resources failure issue shown there. I don't know why HA cluster failed after 70 networks in OC. Is some kind of threshold in tripleo configuration? so it is restricted not to create more than 70 or 80 netwoks. How could i fix this? I did not see such issue when I am using redhat platform and it's repos. This issue coming in opensource repos on Centos 7 . Please help how to fix this issue so I can scale up my openstack upto 2000 vms in this situation it's not possible. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified -------------- next part -------------- An HTML attachment was scrubbed... URL: From amy at demarco.com Wed Jul 22 15:03:11 2020 From: amy at demarco.com (Amy Marrich) Date: Wed, 22 Jul 2020 10:03:11 -0500 Subject: [rdo-dev] [Meeting] RDO meeting - 2020-07-22 Message-ID: ============================== #rdo: RDO meeting - 2020-07-22 ============================== Meeting started by spotz at 14:01:10 UTC. The full logs are available athttp://eavesdrop.openstack.org/meetings/rdo_meeting___2020_07_22/2020/rdo_meeting___2020_07_22.2020-07-22-14.01.log.html . Meeting summary --------------- * LINK: https://etherpad.opendev.org/p/RDO-Meeting (spotz, 14:04:25) * OpenDev (spotz, 14:05:23) * LINK: https://www.youtube.com/watch?v=qNh2eKpe7CU (spotz, 14:06:19) * LINK: https://opendev_hardwareautomation.eventbrite.com/?_ga=2.247588996.266113931.1593534561-1940861233.1584556918 (spotz, 14:07:35) * LINK: https://www.eventbrite.com/e/opendev-containers-in-production-registration-105020424918?_ga=2.42723366.266113931.1593534561-1940861233.1584556918 (spotz, 14:08:48) * RDO Events (spotz, 14:11:56) * LINK: https://review.rdoproject.org/etherpad/p/rdo-events (spotz, 14:12:03) * LINK: http://rdoproject.org/install/ (ykarel, 14:22:38) * Next Week's Chair (spotz, 14:51:21) * Open Floor (spotz, 14:52:51) Meeting ended at 14:54:25 UTC. People present (lines said) --------------------------- * spotz (67) * amoralej (38) * ykarel (15) * rdogerrit (5) * openstack (5) * jcapitao (4) * jpena (4) Generated by `MeetBot`_ 0.1.4 -------------- next part -------------- An HTML attachment was scrubbed... URL: From amoralej at redhat.com Thu Jul 23 10:04:45 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Thu, 23 Jul 2020 12:04:45 +0200 Subject: [rdo-dev] tripleo cluster failure In-Reply-To: <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> Message-ID: On Wed, Jul 22, 2020 at 3:23 PM Rahul Pathak wrote: > Hi, > > I have installed tripleo openstack version rocky containerized with > Undercloud in virtual platform and 3 controllers and 2 compute Baremetal. > > My whole setup is running on centos7. > > Overcloud cluster start failing once number of networks in overcloud reach > more than 70. Lots of resources failure issue shown there. I don't know why > HA cluster failed after 70 networks in OC. > What kind of errors are you seeing?, what "resource failures"? > Is some kind of threshold in tripleo configuration? so it is restricted > not to create more than 70 or 80 netwoks. How could i fix this? > > I did not see such issue when I am using redhat platform and it's repos. > This issue coming in opensource repos on Centos 7 . Please help how to fix > this issue so I can scale up my openstack upto 2000 vms in this situation > it's not possible. > > Regards > *Rahul Pathak* > i2k2 Networks (P) Ltd. | Spring Meadows Business Park > A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 > ISO/IEC 27001:2005 & ISO 9001:2008 Certified > > > _______________________________________________ > dev mailing list > dev at lists.rdoproject.org > http://lists.rdoproject.org/mailman/listinfo/dev > > To unsubscribe: dev-unsubscribe at lists.rdoproject.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpathak at i2k2.com Thu Jul 23 11:37:53 2020 From: rpathak at i2k2.com (Rahul Pathak) Date: Thu, 23 Jul 2020 17:07:53 +0530 (IST) Subject: [rdo-dev] tripleo cluster failure In-Reply-To: References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> Message-ID: <649846251.5620299.1595504273652.JavaMail.zimbra@i2k2.com> Hi, When I am trying to create new projects and one network for each projects after 70 or 80 projects and networks in OC. Controller HA availablitliy failed with below error. [stack at director LogTool_Python2]$ ssh heat-admin at 192.168.100.28 "sudo pcs status" Cluster name: tripleo_cluster Stack: corosync Current DC: overcloud-controller-1 (version 1.1.21-4.el7-f14e36fd43) - partition with quorum Last updated: Thu Jul 23 17:00:22 2020 Last change: Wed Jul 22 14:35:34 2020 by hacluster via crmd on overcloud-controller-2 12 nodes configured 37 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] GuestOnline: [ galera-bundle-0 at overcloud-controller-0 galera-bundle-1 at overcloud-controller-1 galera-bundle-2 at overcloud-controller-2 rabbitmq-bundle-0 at overcloud-controller-0 rabbitmq-bundle-1 at overcloud-controller-1 rabbitmq-bundle-2 at overcloud-controller-2 redis-bundle-0 at overcloud-controller-0 redis-bundle-1 at overcloud-controller-1 redis-bundle-2 at overcloud-controller-2 ] Full list of resources: Docker container set: rabbitmq-bundle [192.168.100.1:8787/tripleorocky/centos-binary-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2 Docker container set: galera-bundle [192.168.100.1:8787/tripleorocky/centos-binary-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master overcloud-controller-0 galera-bundle-1 (ocf::heartbeat:galera): Master overcloud-controller-1 galera-bundle-2 (ocf::heartbeat:galera): FAILED Master overcloud-controller-2 (blocked) Docker container set: redis-bundle [192.168.100.1:8787/tripleorocky/centos-binary-redis:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master overcloud-controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave overcloud-controller-1 redis-bundle-2 (ocf::heartbeat:redis): Slave overcloud-controller-2 ip-192.168.100.98 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-10.10.0.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.102.185 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-192.168.102.116 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.103.187 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.104.127 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Docker container set: haproxy-bundle [192.168.100.1:8787/tripleorocky/centos-binary-haproxy:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started overcloud-controller-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started overcloud-controller-2 Docker container: openstack-cinder-volume [192.168.100.1:8787/tripleorocky/centos-binary-cinder-volume:pcmklatest] openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started overcloud-controller-0 Failed Resource Actions: * redis-bundle-docker-1_monitor_60000 on overcloud-controller-1 'unknown error' (1): call=132, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:42:15 2020', queued=0ms, exec=0ms * galera-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=41, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * redis-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=62, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * haproxy-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=106, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * rabbitmq-bundle-docker-2_monitor_60000 on overcloud-controller-2 'unknown error' (1): call=121, status=Timed Out, exitreason='', last-rc-change='Thu Jul 23 16:48:39 2020', queued=0ms, exec=0ms * galera_promote_0 on galera-bundle-2 'unknown error' (1): call=43, status=complete, exitreason='MySQL server failed to start (pid=646) (rc=0), please check your installation', last-rc-change='Thu Jul 23 16:49:14 2020', queued=0ms, exec=12193ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled It happens all the time when OC total number of networks goes above 70+ . I am attaching error logs of overcloud also. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified ----- Original Message ----- From: "Alfredo Moralejo Alonso" To: "Rahul Pathak" Cc: "RDO Developmen List" Sent: Thursday, July 23, 2020 3:34:45 PM Subject: Re: [rdo-dev] tripleo cluster failure On Wed, Jul 22, 2020 at 3:23 PM Rahul Pathak < rpathak at i2k2.com > wrote: Hi, I have installed tripleo openstack version rocky containerized with Undercloud in virtual platform and 3 controllers and 2 compute Baremetal. My whole setup is running on centos7. Overcloud cluster start failing once number of networks in overcloud reach more than 70. Lots of resources failure issue shown there. I don't know why HA cluster failed after 70 networks in OC. What kind of errors are you seeing?, what "resource failures"?
Is some kind of threshold in tripleo configuration? so it is restricted not to create more than 70 or 80 netwoks. How could i fix this? I did not see such issue when I am using redhat platform and it's repos. This issue coming in opensource repos on Centos 7 . Please help how to fix this issue so I can scale up my openstack upto 2000 vms in this situation it's not possible. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified _______________________________________________ dev mailing list dev at lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/dev To unsubscribe: dev-unsubscribe at lists.rdoproject.org
-------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Overcloud_ERROR.rar Type: application/x-rar Size: 6285 bytes Desc: not available URL: From ashtempl at redhat.com Thu Jul 23 12:05:31 2020 From: ashtempl at redhat.com (Arkady Shtempler) Date: Thu, 23 Jul 2020 15:05:31 +0300 Subject: [rdo-dev] tripleo cluster failure In-Reply-To: <649846251.5620299.1595504273652.JavaMail.zimbra@i2k2.com> References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> <649846251.5620299.1595504273652.JavaMail.zimbra@i2k2.com> Message-ID: Hi all! *Rahul *- there is nothing relevant in the attached file, you've probably executed LogTool on "working environment", so there is nothing interesting in it. I think that you had to mention the Error we've detected in an already "crushed" environment, just as I was suggesting you to do. *Alfredo *- this Error was logged almost on each OC node at the same time when the problems had started. - *hyp-0* - ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ - IsTracebackBlock:False - UniqueCounter:1 - AnalyzedBlockLinesSize:18 - 26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! - 26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! - 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! - 26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! - 26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! - ... - ... - ... - LogTool --> THIS BLOCK IS TOO LONG! - LogTool --> POTENTIAL BLOCK'S ISSUES: - *26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_**rabbit [-] [2f214f0b-84d0-49d4-bcf4-**477565903585] AMQP server overcloud-controller-2.inter..**.* - *ection. Check login credentials: Socket closed: IOError: Socket closed* - *no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED* - *rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: * You can find more Error Blocks in the attached file. Thanks! <#m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- hyp-0 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:18 26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! 26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ ------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:17 692-2020-07-21 20:06:32.245 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 693-2020-07-21 20:07:32.328 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 694:2020-07-21 20:08:04.803 1 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOErro...<--LogTool-LINE IS TOO LONG! 695:2020-07-21 20:08:19.844 1 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errn...<--LogTool-LINE IS TOO LONG! 696-2020-07-21 20:08:34.439 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 697-2020-07-21 20:09:34.182 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 698-2020-07-21 20:10:34.274 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 699-2020-07-21 20:11:36.243 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 700-2020-07-21 20:12:38.198 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 701-2020-07-21 20:13:40.121 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! 702-2020-07-21 20:14:39.239 1 INFO nova.compute.resource_tracker [req-bc07bfb9-7c75-42f0-ae99-6dac2070d502 - - - - -] Final resource view: name=hyp-0....<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED hyp-1 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:33 26746-2020-07-21 20:07:55.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26747-2020-07-21 20:07:55.624 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26749:2020-07-21 20:07:56.370 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [36454cb1-0589-4cb6-9f63-7e2834d27056] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26750:2020-07-21 20:07:56.371 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [b6ab5be6-cc06-4d26-bc76-0f8161b10867] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26751:2020-07-21 20:07:56.372 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [db71cb6f-a426-4456-b56d-13562f98aa84] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26752:2020-07-21 20:07:56.373 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [9af86e0e-d8e0-4738-bb9c-92c544981c2b] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 26753-2020-07-21 20:07:57.621 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26754-2020-07-21 20:07:57.625 54519 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-29d10c6e-dfff-44b2-8c77-51a0f44012dd - -...<--LogTool-LINE IS TOO LONG! 26755:2020-07-21 20:07:57.710 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! ... ---< BLOCK IS TOO LONG >--- ... LogTool --> POTENTIAL BLOCK'S ISSUES: 26748:2020-07-21 20:07:56.369 54519 ERROR oslo.messaging._drivers.impl_rabbit [-] [92bab05d-4bcf-4209-b731-b8fda1fae3a1] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ------------------------------ LogPath: /var/log/containers/nova/nova-compute.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:25 691-2020-07-21 20:06:13.016 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 692-2020-07-21 20:07:13.052 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2....<--LogTool-LINE IS TOO LONG! 694:2020-07-21 20:07:57.715 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG! 695:2020-07-21 20:07:58.744 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server on overcloud-controller...<--LogTool-LINE IS TOO LONG! 696-2020-07-21 20:07:59.767 1 INFO oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] Reconnected to AMQP server on overcl...<--LogTool-LINE IS TOO LONG! 697:2020-07-21 20:08:01.107 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG! 698:2020-07-21 20:08:02.182 1 ERROR oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-...<--LogTool-LINE IS TOO LONG! 699-2020-07-21 20:08:03.205 1 INFO oslo.messaging._drivers.impl_rabbit [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] [c2a4c9a9-b6f4-45c0-97e4-4...<--LogTool-LINE IS TOO LONG! 700-2020-07-21 20:08:17.601 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 701-2020-07-21 20:09:14.030 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 702-2020-07-21 20:10:16.027 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 703-2020-07-21 20:11:16.955 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 704-2020-07-21 20:12:17.038 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! 705-2020-07-21 20:13:16.986 1 INFO nova.compute.resource_tracker [req-efc19468-35e3-4b4a-8216-7015aaebc41e - - - - -] Final resource view: name=hyp-1....<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 693:2020-07-21 20:07:56.371 1 ERROR oslo.messaging._drivers.impl_rabbit [-] [8f4a235d-da43-4494-bb76-57b825f08f43] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ controller-0 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:27 29685-2020-07-21 20:07:55.226 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29686-2020-07-21 20:07:55.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29688:2020-07-21 20:07:56.375 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29689:2020-07-21 20:07:56.376 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29690:2020-07-21 20:07:56.377 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29691-2020-07-21 20:07:57.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29692-2020-07-21 20:07:57.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29693:2020-07-21 20:07:57.483 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29694:2020-07-21 20:07:57.706 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29695:2020-07-21 20:07:57.729 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29696:2020-07-21 20:07:57.733 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29697-2020-07-21 20:07:58.513 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [3c474279-c41f-4abe-a9ac-109e3c5f71b5] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29698-2020-07-21 20:07:58.765 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [a847ea79-ec63-49cb-adce-e5536397194f] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29699-2020-07-21 20:07:58.768 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [f6d4f6a3-f3ea-4f82-b414-0c5a3bb96f50] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29700-2020-07-21 20:07:58.812 178838 INFO oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29701-2020-07-21 20:07:59.227 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29702-2020-07-21 20:07:59.229 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! 29703-2020-07-21 20:08:01.228 178838 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-42815204-1d49-4172-9aa9-be478da4d537 - ...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 29687:2020-07-21 20:07:56.374 178838 ERROR oslo.messaging._drivers.impl_rabbit [-] [41861c25-d6af-4550-80dc-3b1132746863] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:26 1609-2020-07-21 20:07:16.952 177642 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG! 1610-2020-07-21 20:07:17.108 177642 INFO neutron.agent.dhcp.agent [req-026ba197-47b8-4006-8ff3-155d45cb87a2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 1612:2020-07-21 20:07:57.777 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! 1613-2020-07-21 20:07:58.790 177642 INFO oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG! 1614:2020-07-21 20:08:08.565 177642 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG! 1615:2020-07-21 20:08:23.607 177642 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG! 1616-2020-07-21 20:08:49.052 177642 INFO neutron.agent.dhcp.agent [req-46823ae5-be71-437f-adf4-c4db97fe28d7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1617-2020-07-21 20:08:49.336 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1618-2020-07-21 20:08:49.355 177642 INFO neutron.agent.dhcp.agent [req-c1e2fa2e-8376-417f-aa25-04989ec04ce7 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1619-2020-07-21 20:08:49.406 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1620-2020-07-21 20:08:49.458 177642 INFO neutron.agent.dhcp.agent [req-d81a548c-f5ff-4ddc-bfbb-1a038ec6e334 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1621-2020-07-21 20:08:49.482 177642 INFO neutron.agent.dhcp.agent [req-d4952ca7-1f20-4949-9c36-bc4fa6029c83 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1622-2020-07-21 20:08:49.561 177642 INFO neutron.agent.dhcp.agent [req-52d2cc2c-3cfa-4b00-9d1f-3612f7eb9d8e - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 1611:2020-07-21 20:07:56.375 177642 ERROR oslo.messaging._drivers.impl_rabbit [-] [e56f8f16-4d7b-4dc1-a02f-2017d3cfedc7] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:22 38644-2020-07-21 20:07:55.818 38 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0009890 38645-2020-07-21 20:07:55.942 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0009868 38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38647:2020-07-21 20:07:56.373 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [bdd4035e-b970-48aa-bd17-5f3a9fa3ee4d] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38648:2020-07-21 20:07:56.374 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38646:2020-07-21 20:07:56.373 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:18 38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38679-2020-07-21 20:07:57.699 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [49d1ca2a-988d-46db-9325-4530a6cc075f] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38680:2020-07-21 20:07:57.708 47 ERROR oslo.messaging._drivers.impl_rabbit [-] [7dfef257-02a9-447a-8be4-9149a980f37d] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38681-2020-07-21 20:07:57.709 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [42757d9d-ade4-4c52-9083-7fef8f3d23c0] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38682-2020-07-21 20:07:57.710 50 ERROR oslo.messaging._drivers.impl_rabbit [-] [98688dd7-c96a-4bee-a980-92cbca8e1541] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38683-2020-07-21 20:07:57.712 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [d836ff75-44af-4e1e-b985-2a3108a1f337] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38684-2020-07-21 20:07:57.712 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [efa6cb10-7f90-48f5-95a7-595d6a3d9c9a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38685-2020-07-21 20:07:57.712 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [52904842-0a17-4378-a279-82d24a91a95a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38686-2020-07-21 20:07:57.715 48 ERROR oslo.messaging._drivers.impl_rabbit [-] [63bd5e71-06e1-45f9-bd66-ccc1fd087a1c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38687-2020-07-21 20:07:57.719 46 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6f11462-ac09-4098-b70c-43f4e99fe87c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38678-2020-07-21 20:07:57.697 49 ERROR oslo.messaging._drivers.impl_rabbit [-] [a6ebbf44-c679-4c6f-a9e1-c3e797bbd7a2] AMQP server on overcloud-controller-2.in... ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ controller-1 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:31 29669-2020-07-21 20:07:55.494 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! 29670-2020-07-21 20:07:55.499 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! 29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29672:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [f4016ef6-75a6-405d-82cf-ae3f4557a3d8] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29673:2020-07-21 20:07:56.384 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [eda2f576-314d-47fa-9348-0f9a5e5fe9c7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29674:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [ea7378e5-bd62-46a3-8cf0-6cf8000fdba7] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29675:2020-07-21 20:07:56.385 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29676:2020-07-21 20:07:56.505 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [3d4af668-0b5f-4f30-8f03-9778c5650d1b] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29677-2020-07-21 20:07:57.502 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! 29678-2020-07-21 20:07:57.534 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! ... ---< BLOCK IS TOO LONG >--- ... 29690-2020-07-21 20:07:58.926 153102 INFO oslo.messaging._drivers.impl_rabbit [-] [3aca6487-e396-494d-9e1e-39838a6d60f3] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29691-2020-07-21 20:07:59.503 153102 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-04672e43-e8fa-40b5-b9d6-4c219858e22e - ...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 29671:2020-07-21 20:07:56.383 153102 ERROR oslo.messaging._drivers.impl_rabbit [-] [fb1f47e0-ea09-426b-862c-56494db4c24f] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ ection reset by peer. Trying again in 1 seconds.: error: [Errno 104] Connection reset by peer ------------------------------ LogPath: /var/log/containers/neutron/dhcp-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:2 AnalyzedBlockLinesSize:24 1682-2020-07-21 20:07:16.957 152406 INFO neutron.agent.dhcp.agent [-] Trigger reload_allocations for port admin_state_up=True, allowed_address_pairs=[...<--LogTool-LINE IS TOO LONG! 1683-2020-07-21 20:07:17.177 152406 INFO neutron.agent.dhcp.agent [req-2c406398-1696-43dc-b0f5-457a287a5e34 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! 1685:2020-07-21 20:07:57.699 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! 1686:2020-07-21 20:07:58.732 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server on overcloud-cont...<--LogTool-LINE IS TOO LONG! 1687-2020-07-21 20:07:59.765 152406 INFO oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] Reconnected to AMQP server on ...<--LogTool-LINE IS TOO LONG! 1688:2020-07-21 20:08:08.015 152406 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: ...<--LogTool-LINE IS TOO LONG! 1689:2020-07-21 20:08:23.058 152406 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect:...<--LogTool-LINE IS TOO LONG! 1690-2020-07-21 20:09:01.075 152406 INFO neutron.agent.dhcp.agent [req-f1c75a55-aeff-4415-9fa3-824874ed72ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1691-2020-07-21 20:09:01.392 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1692-2020-07-21 20:09:01.479 152406 INFO neutron.agent.dhcp.agent [req-062d6247-fb9a-4a27-953c-1e3e15da83e2 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1693-2020-07-21 20:09:01.492 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1694-2020-07-21 20:09:01.584 152406 INFO neutron.agent.dhcp.agent [req-b71f0f58-3857-47d7-8b7e-b19c52e196ad - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! 1695-2020-07-21 20:09:01.612 152406 INFO neutron.agent.dhcp.agent [req-8159b1fe-a3a0-4d3b-bcd1-b3f8418cf281 - - - - -] Trigger reload_allocations for ...<--LogTool-LINE IS TOO LONG! 1696-2020-07-21 20:09:01.689 152406 INFO neutron.agent.dhcp.agent [req-ea33f202-4d02-4a88-9905-8a250e1a47d5 - - - - -] DHCP configuration for ports se...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 1684:2020-07-21 20:07:56.383 152406 ERROR oslo.messaging._drivers.impl_rabbit [-] [7cb63753-390d-4f7f-b5d6-3c9c7e087682] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ .impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:2 AnalyzedBlockLinesSize:22 38396-2020-07-21 20:07:55.925 30 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0011032 38397-2020-07-21 20:07:56.030 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0008850 38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38399:2020-07-21 20:07:56.380 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [da0fafee-b840-44b7-9c5f-0afda51faab6] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38400:2020-07-21 20:07:56.380 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [5997eee9-f4a0-4120-8c1a-143089d3fdc7] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38398:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [845c9d17-876b-411f-9a2e-62b849f6d639] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ controller-2 ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:2 AnalyzedBlockLinesSize:20 29367-2020-07-21 20:07:55.723 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29368-2020-07-21 20:07:55.730 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29370:2020-07-21 20:07:56.379 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29371:2020-07-21 20:07:56.382 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29372:2020-07-21 20:07:56.383 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29373:2020-07-21 20:07:56.385 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29374:2020-07-21 20:07:56.386 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server overcloud-contro...<--LogTool-LINE IS TOO LONG! 29375:2020-07-21 20:07:57.716 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29376-2020-07-21 20:07:57.724 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29377-2020-07-21 20:07:57.729 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29378:2020-07-21 20:07:57.805 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29379:2020-07-21 20:07:57.814 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29380:2020-07-21 20:07:57.818 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29381:2020-07-21 20:07:57.821 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29382:2020-07-21 20:07:57.824 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29383:2020-07-21 20:07:58.781 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server on overcloud-con...<--LogTool-LINE IS TOO LONG! 29384-2020-07-21 20:07:58.858 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [7f64e698-a3aa-4db0-8c67-075a9913da62] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29385-2020-07-21 20:07:58.859 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [c3015a2c-c21e-4907-b385-1bfdef7d92b6] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29386-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [9b6ae99b-9059-4a83-a103-245eedd31b3c] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29387-2020-07-21 20:07:58.865 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [87b232a3-8677-44b9-b6e8-c0ef86588eb9] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29388-2020-07-21 20:07:58.866 153506 INFO oslo.messaging._drivers.impl_rabbit [-] [ac374e20-7463-4847-b8e2-cd956d703b28] Reconnected to AMQP server on...<--LogTool-LINE IS TOO LONG! 29389-2020-07-21 20:07:59.725 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! 29390-2020-07-21 20:07:59.732 153506 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3b929914-2d8a-4ec2-9e5f-be87e0318176 - ...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 29369:2020-07-21 20:07:56.378 153506 ERROR oslo.messaging._drivers.impl_rabbit [-] [8d5b5c34-fec1-49fb-ad5b-9797773491e3] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:20 38505-2020-07-21 20:07:56.043 28 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0026181 38506-2020-07-21 20:07:56.045 29 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0032661 38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38508:2020-07-21 20:07:56.379 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! 38509:2020-07-21 20:07:56.380 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server overcloud-controller...<--LogTool-LINE IS TOO LONG! ... ... ... LogTool --> THIS BLOCK IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38507:2020-07-21 20:07:56.379 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server overcloud-controller-2.inter... ^^^^^ ection. Check login credentials: Socket closed: IOError: Socket closed ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ ------------------------------ LogPath: /var/log/containers/neutron/server.log.1 ------------------------------ IsTracebackBlock:False UniqueCounter:1 AnalyzedBlockLinesSize:29 38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38525-2020-07-21 20:07:57.723 42 ERROR oslo.messaging._drivers.impl_rabbit [-] [f6adb2d9-255b-4c0f-8515-8bdf45856137] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38526:2020-07-21 20:07:57.724 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [6ac6e794-3561-4cc3-99a9-cc5e078dd130] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38527-2020-07-21 20:07:57.724 38 ERROR oslo.messaging._drivers.impl_rabbit [-] [12f38994-d07a-40df-bdb4-4f49c905b417] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38528-2020-07-21 20:07:57.732 44 ERROR oslo.messaging._drivers.impl_rabbit [-] [9f5be651-69e5-490b-8d61-f4b0fe90f0ee] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38529-2020-07-21 20:07:57.739 45 ERROR oslo.messaging._drivers.impl_rabbit [-] [724677df-d0f6-4b7c-8717-22f0f076e769] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38530-2020-07-21 20:07:57.740 39 ERROR oslo.messaging._drivers.impl_rabbit [-] [b237f335-3172-434d-88f5-f14d37bf485c] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38531-2020-07-21 20:07:57.745 41 ERROR oslo.messaging._drivers.impl_rabbit [-] [8246b7f8-db45-4681-8910-2bf85df8fafa] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38532-2020-07-21 20:07:57.745 36 ERROR oslo.messaging._drivers.impl_rabbit [-] [93e8e483-ef8e-41f0-bf61-8f8c25fb2efb] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38533-2020-07-21 20:07:57.754 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [c11cfdee-53e0-4574-988e-e037adf8422b] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38534-2020-07-21 20:07:57.759 37 ERROR oslo.messaging._drivers.impl_rabbit [-] [091e9689-2d34-46e5-a40c-8c4fe3b200c5] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38535:2020-07-21 20:07:57.958 43 ERROR oslo.messaging._drivers.impl_rabbit [-] [2a4c7a49-5f9b-4b09-8636-16320c659a68] AMQP server on overcloud-control...<--LogTool-LINE IS TOO LONG! 38536-2020-07-21 20:07:58.047 33 INFO neutron.wsgi [-] 192.168.102.126 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0022709 38537-2020-07-21 20:07:58.049 30 INFO neutron.wsgi [-] 192.168.102.144 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0022650 38538-2020-07-21 20:07:58.470 32 INFO neutron.wsgi [-] 192.168.102.153 "OPTIONS / HTTP/1.0" status: 200 len: 252 time: 0.0023279 38539-2020-07-21 20:07:58.513 43 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOE...<--LogTool-LINE IS TOO LONG! 38540-2020-07-21 20:07:58.527 43 INFO oslo.messaging._drivers.impl_rabbit [-] [9f9ed2f1-0b23-40a4-99ac-4082ec985e21] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG! 38541-2020-07-21 20:07:58.557 40 INFO oslo.messaging._drivers.impl_rabbit [-] [1e042b35-4e4d-4e0d-8aec-ebdc7efe19b7] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG! 38542-2020-07-21 20:07:58.564 40 INFO oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] Reconnected to AMQP server on ove...<--LogTool-LINE IS TOO LONG! LogTool --> POTENTIAL BLOCK'S ISSUES: 38524-2020-07-21 20:07:57.512 40 ERROR oslo.messaging._drivers.impl_rabbit [-] [05f61b3a-b52a-4d34-bb50-6b842c60563a] AMQP server on overcloud-controller-2.in... ^^^^^ no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED ^^^^^ rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError: ^^^^^ ^^^^^ ^^^^^^^^^^^^^^^ ^^^^^ ^^^^^ slo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeat thread processing, retrying...: IOError: Socket closed ^^^^^ ^^^^^ From amoralej at redhat.com Thu Jul 23 12:10:42 2020 From: amoralej at redhat.com (Alfredo Moralejo Alonso) Date: Thu, 23 Jul 2020 14:10:42 +0200 Subject: [rdo-dev] tripleo cluster failure In-Reply-To: References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> <649846251.5620299.1595504273652.JavaMail.zimbra@i2k2.com> Message-ID: On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler wrote: > Hi all! > > *Rahul *- there is nothing relevant in the attached file, you've probably > executed LogTool on "working environment", so there is nothing interesting > in it. > I think that you had to mention the Error we've detected in an already > "crushed" environment, just as I was suggesting you to do. > Yes, nothing interesting in the attached logs. > > *Alfredo *- this Error was logged almost on each OC node at the same time > when the problems had started. > > - *hyp-0* > - ------------------------------ LogPath: > /var/log/containers/neutron/openvswitch-agent.log.1 > ------------------------------ > - IsTracebackBlock:False > - UniqueCounter:1 > - AnalyzedBlockLinesSize:18 > - 26712-2020-07-21 20:07:54.604 54410 INFO > neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent > [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! > - 26713-2020-07-21 20:07:54.605 54410 INFO > neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent > [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! > - 26714:2020-07-21 20:07:56.363 54410 ERROR > oslo.messaging._drivers.impl_rabbit [-] > [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server > overcloud-control...<--LogTool-LINE IS TOO LONG! > - 26715:2020-07-21 20:07:56.364 54410 ERROR > oslo.messaging._drivers.impl_rabbit [-] > [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server > overcloud-control...<--LogTool-LINE IS TOO LONG! > - 26716:2020-07-21 20:07:56.364 54410 ERROR > oslo.messaging._drivers.impl_rabbit [-] > [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server > overcloud-control...<--LogTool-LINE IS TOO LONG! > - ... > - ... > - ... > - LogTool --> THIS BLOCK IS TOO LONG! > - LogTool --> POTENTIAL BLOCK'S ISSUES: > - *26714:2020-07-21 20:07:56.363 54410 ERROR > oslo.messaging._drivers.impl_**rabbit [-] [2f214f0b-84d0-49d4-bcf4-**477565903585] > AMQP server overcloud-controller-2.inter..**.* > - *ection. Check login credentials: Socket closed: IOError: Socket > closed* > - *no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno > 111] ECONNREFUSED* > - *rnalapi.i2k2cloud02.com:5672 > is unreachable: . Trying again in 1 seconds.: > RecoverableConnectionError: * > > > I'd suggest to check rabbitmq and mariadb logs. AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers? > > You can find more Error Blocks in the attached file. > > Thanks! > > <#m_-4775499539429953439_m_432445082502786505_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpathak at i2k2.com Thu Jul 23 12:32:02 2020 From: rpathak at i2k2.com (Rahul Pathak) Date: Thu, 23 Jul 2020 18:02:02 +0530 (IST) Subject: [rdo-dev] tripleo cluster failure In-Reply-To: References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> <649846251.5620299.1595504273652.JavaMail.zimbra@i2k2.com> Message-ID: <223192519.5626374.1595507522941.JavaMail.zimbra@i2k2.com> Hi, Is there some kind of script that I can run to know the exact issue about what resource crunch is there ? Below is the memory and disk utilization of controller: [heat-admin at overcloud-controller-0 ~]$ free -m total used free shared buff/cache available Mem: 128722 25632 92042 70 11047 102377 Swap: 0 0 0 [heat-admin at overcloud-controller-0 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 39M 63G 1% /dev/shm tmpfs 63G 27M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda2 1.9T 12G 1.9T 1% / tmpfs 13G 0 13G 0% /run/user/0 tmpfs 13G 0 13G 0% /run/user/1000 Same memory and disk available on all three controllers. On the same environment when I installed overcloud with redhat repos and redhat overcloud images...I have not faced this issue. I have tested almost 500 projects and 500 networks (one network per project) on the same environment with redhat and it was working fine without any issue and cluster failure. But when I am using Centos-7 tripleo repo's its happening again n again. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified ----- Original Message ----- From: "Alfredo Moralejo Alonso" To: "Arkady Shtempler" Cc: "Rahul Pathak" , "RDO Developmen List" Sent: Thursday, July 23, 2020 5:40:42 PM Subject: Re: [rdo-dev] tripleo cluster failure On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler < ashtempl at redhat.com > wrote: Hi all! Rahul - there is nothing relevant in the attached file, you've probably executed LogTool on "working environment", so there is nothing interesting in it. I think that you had to mention the Error we've detected in an already "crushed" environment, just as I was suggesting you to do. Yes, nothing interesting in the attached logs.
Alfredo - this Error was logged almost on each OC node at the same time when the problems had started. * hyp-0 * ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ * IsTracebackBlock:False * UniqueCounter:1 * AnalyzedBlockLinesSize:18 * 26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! * 26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! * 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * 26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * 26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * ... * ... * ... * LogTool --> THIS BLOCK IS TOO LONG! * LogTool --> POTENTIAL BLOCK'S ISSUES: * 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_ rabbit [-] [2f214f0b-84d0-49d4-bcf4- 477565903585] AMQP server overcloud-controller-2.inter.. . * ection. Check login credentials: Socket closed: IOError: Socket closed * no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED * rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError:
I'd suggest to check rabbitmq and mariadb logs. AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers?
You can find more Error Blocks in the attached file. Thanks!
-------------- next part -------------- An HTML attachment was scrubbed... URL: From yatinkarel at gmail.com Mon Jul 27 09:57:58 2020 From: yatinkarel at gmail.com (YATIN KAREL) Date: Mon, 27 Jul 2020 15:27:58 +0530 Subject: [rdo-dev] [rdo-users] [RDO] Weekly Status for 2020-07-24 Message-ID: Promotions * Latest promotions (TripleO CI): * Master: 23rd July * Ussuri: 23rd July * Train: 15th July * Stein: 22nd July * Known blocker (random fails) * https://bugs.launchpad.net/tripleo/+bug/1885314 Packages * Python-shaker is removed from victoria * Ansible is being updated to 2.9.11 in Victoria * Python-packaging-20.4 is being added to Victoria repos as oslo.test needs >= 20.4(PowerTools repo provides 16.8) * Manila-tempest-plugin is being updated to 1.1.0 in queens/rocky/stein/train/ussuri. Vexxhost * ci.centos weirdo jobs are migrated to vexxhost, Other RDO jobs are also being evaluated on vexxhost * https://review.rdoproject.org/r/#/q/topic:vexxhost Other * Update of OpenStack clients to Ussuri releases in Fedora is in progress * https://trello.com/c/xHGOdCU0/737-update-openstack-clients-to-ussuri-releases-in-fedora * RPM specs are being updated to support python3.10 * https://trello.com/c/A9GZWxqR/739-make-rdo-support-python-version-with-2-digits * Rdoinfo gate job for victoria tags migrated to non kolla containers * https://trello.com/c/wo5F38Cw/740-migrate-rdoinfo-tripleo-001-gate-job-for-testing-promotion-to-non-kolla-based * CentOS8 check and promotion jobs for train release are being set up. * Revival of NFV SIG is work in progress. New chair has been approved by CentOS board * A new script is being created to compare upper-constraints with actual versions of python libraries in repos: * https://review.rdoproject.org/r/#/c/28618/ On behalf of RDO Team _______________________________________________ users mailing list users at lists.rdoproject.org http://lists.rdoproject.org/mailman/listinfo/users To unsubscribe: users-unsubscribe at lists.rdoproject.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpathak at i2k2.com Tue Jul 28 07:57:06 2020 From: rpathak at i2k2.com (Rahul Pathak) Date: Tue, 28 Jul 2020 13:27:06 +0530 (IST) Subject: [rdo-dev] tripleo cluster failure In-Reply-To: <223192519.5626374.1595507522941.JavaMail.zimbra@i2k2.com> References: <914846645.5457087.1595423328515.JavaMail.zimbra@i2k2.com> <1455669115.5458554.1595424219319.JavaMail.zimbra@i2k2.com> <649846251.5620299.1595504273652.JavaMail.zimbra@i2k2.com> <223192519.5626374.1595507522941.JavaMail.zimbra@i2k2.com> Message-ID: <1313293670.346580.1595923026318.JavaMail.zimbra@i2k2.com> Reminder!! Please help. ------------------------------- Hi, Is there some kind of script that I can run to know the exact issue about what resource crunch is there ? Below is the memory and disk utilization of controller: [heat-admin at overcloud-controller-0 ~]$ free -m total used free shared buff/cache available Mem: 128722 25632 92042 70 11047 102377 Swap: 0 0 0 [heat-admin at overcloud-controller-0 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 39M 63G 1% /dev/shm tmpfs 63G 27M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/sda2 1.9T 12G 1.9T 1% / tmpfs 13G 0 13G 0% /run/user/0 tmpfs 13G 0 13G 0% /run/user/1000 Same memory and disk available on all three controllers. On the same environment when I installed overcloud with redhat repos and redhat overcloud images...I have not faced this issue. I have tested almost 500 projects and 500 networks (one network per project) on the same environment with redhat and it was working fine without any issue and cluster failure. But when I am using Centos-7 tripleo repo's its happening again n again. Regards Rahul Pathak i2k2 Networks (P) Ltd. | Spring Meadows Business Park A61-B4 & 4A First Floor, Sector 63, Noida - 201 301 ISO/IEC 27001:2005 & ISO 9001:2008 Certified ----- Original Message ----- From: "Alfredo Moralejo Alonso" To: "Arkady Shtempler" Cc: "Rahul Pathak" , "RDO Developmen List" Sent: Thursday, July 23, 2020 5:40:42 PM Subject: Re: [rdo-dev] tripleo cluster failure On Thu, Jul 23, 2020 at 2:05 PM Arkady Shtempler < ashtempl at redhat.com > wrote: Hi all! Rahul - there is nothing relevant in the attached file, you've probably executed LogTool on "working environment", so there is nothing interesting in it. I think that you had to mention the Error we've detected in an already "crushed" environment, just as I was suggesting you to do. Yes, nothing interesting in the attached logs.
Alfredo - this Error was logged almost on each OC node at the same time when the problems had started. * hyp-0 * ------------------------------ LogPath: /var/log/containers/neutron/openvswitch-agent.log.1 ------------------------------ * IsTracebackBlock:False * UniqueCounter:1 * AnalyzedBlockLinesSize:18 * 26712-2020-07-21 20:07:54.604 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! * 26713-2020-07-21 20:07:54.605 54410 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-a2173b45-f18b-45f9-90d8-cb8ab1754332 - -...<--LogTool-LINE IS TOO LONG! * 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [2f214f0b-84d0-49d4-bcf4-477565903585] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * 26715:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [664475bc-3b39-4ce5-a60e-f010b8d5201d] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * 26716:2020-07-21 20:07:56.364 54410 ERROR oslo.messaging._drivers.impl_rabbit [-] [68a0ab43-a216-43fe-aa58-860d3dc5e69e] AMQP server overcloud-control...<--LogTool-LINE IS TOO LONG! * ... * ... * ... * LogTool --> THIS BLOCK IS TOO LONG! * LogTool --> POTENTIAL BLOCK'S ISSUES: * 26714:2020-07-21 20:07:56.363 54410 ERROR oslo.messaging._drivers.impl_ rabbit [-] [2f214f0b-84d0-49d4-bcf4- 477565903585] AMQP server overcloud-controller-2.inter.. . * ection. Check login credentials: Socket closed: IOError: Socket closed * no 111] ECONNREFUSED. Trying again in 1 seconds.: error: [Errno 111] ECONNREFUSED * rnalapi.i2k2cloud02.com:5672 is unreachable: . Trying again in 1 seconds.: RecoverableConnectionError:
I'd suggest to check rabbitmq and mariadb logs. AFAIK, there is not a configuration that may limit the number of networks or project, but it may be hitting some resources scarcity that affect the running services. What's the memory sizing and usage of the controllers?
You can find more Error Blocks in the attached file. Thanks!
-------------- next part -------------- An HTML attachment was scrubbed... URL: From jcapitao at redhat.com Wed Jul 29 15:00:43 2020 From: jcapitao at redhat.com (Joel Capitao) Date: Wed, 29 Jul 2020 17:00:43 +0200 Subject: [rdo-dev] [Meeting] RDO Meeting (2020-07-29) minutes Message-ID: <20200729150043.setoniq7bozhezt4@redhat> ============================== #rdo: RDO meeting - 2020-07-29 ============================== Meeting started by jcapitao at 14:01:26 UTC. The full logs are available at http://eavesdrop.openstack.org/meetings/rdo_meeting___2020_07_29/2020/rdo_meeting___2020_07_29.2020-07-29-14.01.log.html . Meeting summary --------------- * Roll Call (jcapitao, 14:01:57) * Events Update (spotz, 14:06:32) * LINK: https://www.twitch.tv/redhatopenshift (spotz, 14:12:14) * jpena and ykarel to live stream about Packstack day 08/03/20 at 16:00 UTC on the redhatopenshift channel (Twitch) (jcapitao, 14:14:40) * next week chair (jcapitao, 14:18:29) * ACTION: ykarel to chair next week meeting (jcapitao, 14:20:14) * open floor (jcapitao, 14:20:39) Meeting ended at 14:31:55 UTC. Action items, by person ----------------------- * ykarel * ykarel to chair next week meeting People present (lines said) --------------------------- * jcapitao (24) * spotz (12) * amoralej (6) * openstack (6) * ykarel (5) * rdogerrit (3) * jpena (1) * rh-jlabarre (1) Generated by `MeetBot`_ 0.1.4