[infra][tripleo-ci] Disk space usage in logs.rdoproject.org
by Javier Pena
Hi all,
For the last few days, I have been monitoring a spike in disk space utilization for logs.rdoproject.org. The current situation is:
- 94% of space used, with less than 140GB out of 2TB available.
- The log pruning script has been reclaiming less space than we are using for new logs during this week.
- I expect the situation to improve over the weekend, but we're definitely running out of space.
I have looked at a random job (https://review.opendev.org/639324, patch set 26), and found that each run is consuming 1.2 GB of disk space in logs. The worst offenders I have found are:
- atop.bin.gz files (one per job, 8 jobs per recheck), ranging between 15 and 40 MB each
- logs/undercloud/home/zuul/tempest/.stackviz directory on tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 jobs, which is a virtualenv eating up 81 MB.
As a temporary measure, I am reducing log retention from 21 days to 14, but we still need to reduce the rate at which we are uploading logs. Would it be possible to check the oooq-generated logs and see where we can reduce? These jobs are by far the ones consuming most space.
Thanks,
Javier
4 years, 8 months
Unscheduled updates and upgrade for registry.rdoproject.org
by David Moreau Simard
Hi,
In light of the recent unreliability of trunk.registry.rdoproject.org,
the instance was updated and resized with more RAM to account for the
increased amount of images.
There may have been outages earlier today between 15:00 UTC and 16:30
UTC while we were applying updates, rebooting and resizing.
Please let us know if there is any issues moving forward.
Thanks,
David Moreau Simard
dmsimard = [irc, github, twitter]
4 years, 10 months
[RDO] Weekly status for 2019-06-21
by YATIN KAREL
Promotions
* Latest promotion (TripleO CI) for Stein is 20th June while for Master
is from 18th June.
* RDO registry related issues(connection refused) are seen which
letting promotion jobs to fail. Some swap is added to registry server and
is being monitored how it goes
Deps Update
* Facter has been updated to 3.9.3 in master.
* Python-sqlalchemy-collectd is being transferred to OpsTools SIG.
* python-configparser and python-backports added to Train, needed by
ironic-prometheus-exporter
* Ansible-2.8 being pushed to stein
Packages
* Tripleo-ui package is removed from Train
* Neutron-lbaas and neutron-lbaas-dashboard have been removed from Train
CentOS8
* Build loop 0 for CentOS 8 is finished according to
https://wiki.centos.org/About/Building_8
* Preliminary analysis to find out the required dependencies to be
rebuilt for CentOS8 and analysis on potential issues and required ordering
for bootstrap is ongoing.
Pike EOL Preparation
* Pike has Entered Extended Maintenance Upstream, means there will be
no new releases, so we started cleanup in RDO, it will be done in phases.
* Tracking Pike EOL Taks at https://trello.com/c/6e0bVGzK
CI/Validation
* Move rdoinfo gating jobs to utilize CloudSIG repos instead of DLRN
ones
* https://review.rdoproject.org/r/#/q/topic:cloudsig-containers
* https://review.opendev.org/#/c/665959/
On behalf of RDO
_______________________________________________
dev mailing list
dev(a)lists.rdoproject.org
http://lists.rdoproject.org/mailman/listinfo/dev
To unsubscribe: dev-unsubscribe(a)lists.rdoproject.org
4 years, 10 months
RDO Pike going EOL
by Alfredo Moralejo Alonso
Hi,
OpenStack Pike was moved to Extended Maintenance upstream [1]. This means
that no more releases will be produced for Pike series and there is no
commitment on validation. Also, some projects may decide to not maintain
the branches and go to EOL.
>From the RDO perspective, this means that we are setting RDO CloudSIG Pike
repo as EOL and the repositories will be dropped from CentOS official
mirrors on next oportunity (when CentOS 7.7 is published).
For users that still want to keep using packages from pike branches tip, we
will keep the RDO Trunk Pike repo running in [2] chasing any new commit
being merged, but note that we will probably stop running CI on it at some
point in the near future.
Best regards,
Alfredo
[1]
https://docs.openstack.org/project-team-guide/stable-branches.html#extend...
[2] https://trunk.rdoproject.org/centos7-pike/report.html
4 years, 10 months
[Meeting] RDO Meeting (2019-06-12) minutes
by Javier Pena
==============================
#rdo: RDO meeting - 2019-06-12
==============================
Meeting started by jpena at 15:00:29 UTC. The full logs are available
at
http://eavesdrop.openstack.org/meetings/rdo_meeting___2019_06_12/2019/rdo...
.
Meeting summary
---------------
* roll call (jpena, 15:00:33)
* Pike is now extended maintenance, time for EOL in RDO (jpena,
15:03:58)
* ACTION: amoralej to send a mail about Pike CloudSIG repos being EOL
(amoralej, 15:09:54)
* tracking pike removal in
https://trello.com/c/6e0bVGzK/712-pike-is-em-removal-from-rdo
(amoralej, 15:13:09)
* altarch container discussion (jpena, 15:14:02)
* New release of python-sqlalchemy-collectd requires collectd subpackage
which is included in opstools (jpena, 15:27:40)
* LINK: https://review.opendev.org/#/c/659678/ (amoralej, 15:31:25)
* LINK: https://review.opendev.org/#/c/659678/ is running with centos7
so we need to provide it (amoralej, 15:40:41)
* ACTION: amoralej to follow-up with mrunge about the topic
(amoralej, 15:45:03)
* Next week's chair (jpena, 15:45:58)
* ACTION: ykarel to chair the next meeting (jpena, 15:47:29)
* open floor (jpena, 15:47:36)
Meeting ended at 15:54:56 UTC.
Action items, by person
-----------------------
* amoralej
* amoralej to send a mail about Pike CloudSIG repos being EOL
* amoralej to follow-up with mrunge about the topic
* mrunge
* amoralej to follow-up with mrunge about the topic
* ykarel
* ykarel to chair the next meeting
People present (lines said)
---------------------------
* amoralej (76)
* jpena (31)
* mjturek (27)
* ykarel (19)
* mrunge (12)
* zzzeek (12)
* openstack (5)
* baha (2)
* gfidente (2)
* mriosfer (1)
* apevec (1)
Generated by `MeetBot`_ 0.1.4
4 years, 10 months
Scheduled maintenance of review.rdoproject.org: 2019-06-05 23:00 UTC
by Nicolas Hicher
Hello folks,
We plan to upgrade review.rdoproject.org on 2019-06-05 23:00 UTC. We
don't expect any downtime during the upgrade process.
First we have to upgrade softwarefactory-project.io to upgrade zuul to
3.8.1, then upgrade review.rdoproject.organd
ansible.softwarefactory-project.io.
Zuul 3.8supports ansible 2.5, 2.6 and 2.7, and after the upgrade, all
job will be executed with ansible 2.7, but you could
force ansible version on your job by adding in the job definition (note
ansible 2.5 will be deprecated soon):
- job:
ansible-version: 2.5
Let us know if you prefer a different date.
-Nicolas, on behalf of the ProdChainInfra Team
4 years, 10 months
OpenInfrastructure Summit Shanghai
by Rain Leander
Hello RDO Community!
Call for Presentations is open for the Open Infrastructure Summit [0] to be
held in Shanghai, China 4-6 November 2019!
** Deadline to submit your presentations, panels, and workshops [1] is July
2, 2019 at 11:59pm PT (July 3 at 6:59 UTC).
Here are the Summit Tracks [2] that span Cloud, Ansible, Ceph, Kubernetes,
Edge, etc., to give you a sense of the types of topics you can propose.
** NOTE: Before submitting a session, please ensure that you have approval
to attend from your leadership, especially as this event is in China and
you may need a visa.
If you have any questions, please don't hesitate to reach out!
[0] https://www.openstack.org/summit/shanghai-2019
[1] https://cfp.openstack.org/
[2] https://www.openstack.org/summit/shanghai-2019/summit-categories
--
K Rain Leander
OpenStack Community Liaison
Open Source Program Office
https://www.rdoproject.org/
http://community.redhat.com
4 years, 10 months
