<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">2TB seems very little for a data hoarder like me, I happen to have ~30TB of storage at home. <div class=""><br class=""></div><div class="">10-14TB hard drives are not really so expensive.<br class=""><div><br class=""></div><div>While I totally agree that we should control/limit better what we collect, I wonder if we should not aim to get a setup where we do not struggle with disk space, where we can keep logs for ~60 days without having to think too much about running out of disk space.</div><div><br class=""></div><div>Another approach which I used in the past was a cleanup script that was removing old builds based on their age as long the free disk space was under a specific value (10%?). That means a dynamic retention period.</div><div><br class=""></div><div>Thanks</div><div>Sorin</div><div><br class=""><blockquote type="cite" class=""><div class="">On 13 Jun 2019, at 15:58, Wesley Hayutin <<a href="mailto:whayutin@redhat.com" class="">whayutin@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 13, 2019 at 8:55 AM Javier Pena <<a href="mailto:jpena@redhat.com" class="">jpena@redhat.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div style="font-family: "times new roman", "new york", times, serif; font-size: 12pt;" class=""><div class=""><br class=""></div><div class=""><br class=""></div><hr id="gmail-m_1699110453803735921zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 13, 2019 at 8:22 AM Javier Pena <<a href="mailto:jpena@redhat.com" target="_blank" class="">jpena@redhat.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">Hi all,<br class=""><br class="">For the last few days, I have been monitoring a spike in disk space utilization for<span class="Apple-converted-space"> </span><a href="http://logs.rdoproject.org/" rel="noreferrer" target="_blank" class="">logs.rdoproject.org</a>. The current situation is:<br class=""><br class="">- 94% of space used, with less than 140GB out of 2TB available.<br class="">- The log pruning script has been reclaiming less space than we are using for new logs during this week.<br class="">- I expect the situation to improve over the weekend, but we're definitely running out of space.<br class=""><br class="">I have looked at a random job (<a href="https://review.opendev.org/639324" rel="noreferrer" target="_blank" class="">https://review.opendev.org/639324</a>, patch set 26), and found that each run is consuming 1.2 GB of disk space in logs. The worst offenders I have found are:<br class=""><br class="">- atop.bin.gz files (one per job, 8 jobs per recheck), ranging between 15 and 40 MB each<br class="">- logs/undercloud/home/zuul/tempest/.stackviz directory on tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 jobs, which is a virtualenv eating up 81 MB.<br class=""></blockquote><div class=""><br class=""></div><div class="">Can we sync up w/ how you are calculating these results as they do not match our results.</div><div class="">I see each job consuming about 215M of space, we are close on stackviz being 83M. Oddly I don't see atop.bin.gz in our calculations so I'll have to look into that.</div></div></div></blockquote><div class="">I've checked it directly using du on the logserver. By 1.2 GB I meant the aggregate of the 8 jobs running for a single patchset. PS26 is currently using 2.5 GB and had one recheck.<br class=""></div><div class=""><br class=""></div><div class="">About the atop.bin.gz file:<br class=""></div><div class=""><br class=""></div><div class=""># find . -name atop.bin.gz -exec du -sh {} \;<br class="">16M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/042cb8f/logs/undercloud/var/log/atop.bin.gz<br class="">16M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/e4171d7/logs/undercloud/var/log/atop.bin.gz<br class="">28M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/ffd4de9/logs/undercloud/var/log/atop.bin.gz<br class="">26M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/34d44bf/logs/undercloud/var/log/atop.bin.gz<br class="">25M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/b89761d/logs/undercloud/var/log/atop.bin.gz<br class="">24M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/9ade834/logs/undercloud/var/log/atop.bin.gz<br class="">29M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/a10447d/logs/undercloud/var/log/atop.bin.gz<br class="">44M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053/99a5f9f/logs/undercloud/var/log/atop.bin.gz<br class="">15M ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/c8a8c60/logs/subnode-2/var/log/atop.bin.gz<br class="">33M ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/c8a8c60/logs/undercloud/var/log/atop.bin.gz<br class="">16M ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/73ef532/logs/subnode-2/var/log/atop.bin.gz<br class="">33M ./tripleo-ci-centos-7-multinode-1ctlr-featureset010/73ef532/logs/undercloud/var/log/atop.bin.gz<br class="">40M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/109d5ae/logs/undercloud/var/log/atop.bin.gz<br class="">45M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/c2ebeae/logs/undercloud/var/log/atop.bin.gz<br class="">39M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/7fe5bbb/logs/undercloud/var/log/atop.bin.gz<br class="">16M ./tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/5e6cb0f/logs/undercloud/var/log/atop.bin.gz<br class="">40M ./tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/c6bf5ea/logs/undercloud/var/log/atop.bin.gz<br class="">40M ./tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039/6ec5ac6/logs/undercloud/var/log/atop.bin.gz<br class=""><br class=""></div><div class="">Can I safely delete all .stackviz directories? I guess that would give us some breathing room.<br class=""></div></div></div></blockquote><div class=""><br class=""></div><div class="">Yup, go for it</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div style="font-family: "times new roman", "new york", times, serif; font-size: 12pt;" class=""><div class=""></div><div class=""><br class=""></div><div class="">Regards,<br class=""></div><div class="">Javier<br class=""></div><div class=""><br class=""></div><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><div dir="ltr" class=""><div class="gmail_quote"><div class="">Each job reports the size of the logs e.g. [1]</div><div class=""><a href="http://logs.rdoproject.org/24/639324/26/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/9ade834/logs/quickstart_files/log-size.txt" target="_blank" class="">http://logs.rdoproject.org/24/639324/26/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/9ade834/logs/quickstart_files/log-size.txt</a><br class=""></div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><br class="">As a temporary measure, I am reducing log retention from 21 days to 14, but we still need to reduce the rate at which we are uploading logs. Would it be possible to check the oooq-generated logs and see where we can reduce? These jobs are by far the ones consuming most space.<br class=""><br class="">Thanks,<br class="">Javier<br class="">_______________________________________________<br class="">dev mailing list<br class=""><a href="mailto:dev@lists.rdoproject.org" target="_blank" class="">dev@lists.rdoproject.org</a><br class=""><a href="http://lists.rdoproject.org/mailman/listinfo/dev" rel="noreferrer" target="_blank" class="">http://lists.rdoproject.org/mailman/listinfo/dev</a><br class=""><br class="">To unsubscribe:<span class="Apple-converted-space"> </span><a href="mailto:dev-unsubscribe@lists.rdoproject.org" target="_blank" class="">dev-unsubscribe@lists.rdoproject.org</a><br class=""></blockquote></div></div></blockquote><div class=""><br class=""></div></div></div></blockquote></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">dev mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="mailto:dev@lists.rdoproject.org" class="">dev@lists.rdoproject.org</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="http://lists.rdoproject.org/mailman/listinfo/dev" class="">http://lists.rdoproject.org/mailman/listinfo/dev</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">To unsubscribe: <a href="mailto:dev-unsubscribe@lists.rdoproject.org" class="">dev-unsubscribe@lists.rdoproject.org</a></span></div></blockquote></div><br class=""></div></body></html>