The proposed improvement to get more CI promotions is, while we wait for the fixes to be ready, to get the oldest commit that is currently causing an issue from the current analysis and to try the previous commits in reverse order to promote before the issues appear. With the database of DLRN we have all the information to be able to implement this backward tries and have more chances to promote.

On Sat, Aug 13, 2016 at 11:21 AM, Frédéric Lepied <frederic.lepied(a)redhat.com&gt; wrote: > Hi, > > Our CI promotion system is as if we were running downhill: we cannot > stop and the only way to fix issues is to move forward by getting new > fixed packages. And it usually takes days before we can get the fixes > and by the time we get them we can have other issues appearing and so > never being able to have a promotion. > > I would like to propose an improved way of working to get more CI > promotions. When we have failures in our tests, we do like we do > today: debug the issues and find the commits that are causing the > regression and working with upstream or fixing packaging issues to > solve the regression. > > The proposed improvement to get more CI promotions is, while we wait > for the fixes to be ready, to get the oldest commit that is currently > causing an issue from the current analysis and to try the previous > commits in reverse order to promote before the issues appear. With the > database of DLRN we have all the information to be able to implement > this backward tries and have more chances to promote. > > I'm in holidays next week but I wanted to bring this idea before been > completely off. WDYT? In the past we already applied this approach a couple of times for upstream issues that took longer that desired to be fixed (these cases lead us to move to the current u-c pinning for libraries). IMO, this has some drawbacks: - Breaks the consistency principle we try to enforce among packages in RDO repos. - Requires manual definition of the versions or commits for each package.

- When a package is pinned to a specific commit or version, we stop packaging and testing any new commit, so we are delaying and piling the detection of new issues that may appear in the package. - Currently, DLRN doesn't manage properly moving back to a previous commit whenever a new one has been built. It requires manual error-prone tasks to be performed on the dlrn instance (anyway it'd be probably a good idea to implement this us. IMO, we shouldn't use this as a systematic approach for managing versions in our repos to enforce promotions but only as last resort for exceptional cases. Anyway, it's probably a good idea to improve dlrn to implement this use case in a easier and cleaner way.

> -- > Fred - May the Source be with you > > > _______________________________________________ > rdo-list mailing list > rdo-list(a)redhat.com > https://www.redhat.com/mailman/listinfo/rdo-list > > To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On Tue, Aug 16, 2016 at 12:26 PM, Javier Pena <javier.pena(a)redhat.com&gt; wrote: > > > ----- Original Message ----- >> On Sat, Aug 13, 2016 at 11:21 AM, Frédéric Lepied >> <frederic.lepied(a)redhat.com&gt; wrote: >> > Hi, >> > >> > Our CI promotion system is as if we were running downhill: we cannot >> > stop and the only way to fix issues is to move forward by getting new >> > fixed packages. And it usually takes days before we can get the fixes >> > and by the time we get them we can have other issues appearing and so >> > never being able to have a promotion. >> > >> > I would like to propose an improved way of working to get more CI >> > promotions. When we have failures in our tests, we do like we do >> > today: debug the issues and find the commits that are causing the >> > regression and working with upstream or fixing packaging issues to >> > solve the regression. >> > >> > The proposed improvement to get more CI promotions is, while we wait >> > for the fixes to be ready, to get the oldest commit that is currently >> > causing an issue from the current analysis and to try the previous >> > commits in reverse order to promote before the issues appear. With the >> > database of DLRN we have all the information to be able to implement >> > this backward tries and have more chances to promote. >> > >> > I'm in holidays next week but I wanted to bring this idea before been >> > completely off. WDYT? >> >> In the past we already applied this approach a couple of times for >> upstream issues that took longer that desired to be fixed (these cases >> lead us to move to the current u-c pinning for libraries). IMO, this >> has some drawbacks: >> - Breaks the consistency principle we try to enforce among packages in >> RDO repos. >> - Requires manual definition of the versions or commits for each package. > > Well, if I understood Frédéric's proposal correctly, that's not what we'd > do, just go back to the list of previous commits and start trying > backwards. They don't even have to be for the same package. > > Using an example, let's say we have the following commits, from newest to > oldest: > > - Commit 2 to openstack-nova (consistent) > - Commit 2 to openstack-cinder (consistent) > - Commit 1 to openstack-cinder (consistent) > - Commit 1 to openstack-keystone (consistent, previous promoted commit) > > Just by luck, when the promotion job is started, it tests commit 2 to > openstack-nova, and fails due to a new issue. If we can go back just one > commit, to "commit 2 to openstack-cinder", and it is passes CI, we could > still have a consistent repo promoted, just not the very last one at the > time of running the CI job. > > Actually, this makes sense to me. By doing this, we could promote the last > known good+consistent repo, while we fix the latest issue. > OK, i understood "try the previous commits in reverse order", he was referring to the same package but re-reading the mail, you are probably right about the proposal. This approach is easier and better, as we don't break consistency and don't need any trick in dlrn side. However, it will not work if issues of different packages overlap (as we had last week, in fact we were discussing on irc about setting a specific hash but at that time there was not "sane" repo), but probably may help in some cases.

>> - When a package is pinned to a specific commit or version, we stop >> packaging and testing any new commit, so we are delaying and piling >> the detection of new issues that may appear in the package. >> - Currently, DLRN doesn't manage properly moving back to a previous >> commit whenever a new one has been built. It requires manual >> error-prone tasks to be performed on the dlrn instance (anyway it'd be >> probably a good idea to implement this us. >> >> IMO, we shouldn't use this as a systematic approach for managing >> versions in our repos to enforce promotions but only as last resort >> for exceptional cases. Anyway, it's probably a good idea to improve >> dlrn to implement this use case in a easier and cleaner way. >> > > This requires some good thinking. Right now, the only way I know is to > fiddle with the DLRN database, which is quite scary and can lead to > inconsistencies. Maybe we can come up with a better approach. > Yeap, it's probably not an easy change but i think it's still worthy to do it as I'm pretty sure we'll find scenarios were we'll need it, don't you think so?

> Regards, > Javier > >> >> > -- >> > Fred - May the Source be with you >> > >> > >> > _______________________________________________ >> > rdo-list mailing list >> > rdo-list(a)redhat.com >> > https://www.redhat.com/mailman/listinfo/rdo-list >> > >> > To unsubscribe: rdo-list-unsubscribe(a)redhat.com >> >> _______________________________________________ >> rdo-list mailing list >> rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com _______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

> If we can go back just one commit, to "commit 2 to openstack-cinder", and it is passes CI, we could still have a consistent repo promoted, just not the very last one at the time of running the CI job. > > Actually, this makes sense to me. By doing this, we could promote the last known good+consistent repo, while we fix the latest issue. Yes that was the intent. Let's see how we could implement this.

wrote:

On 08/24/2016 12:45 PM, Alan Pevec wrote: >>> If we can go back just one commit, to "commit 2 to openstack-cinder", and it is passes CI, we could still have a consistent repo promoted, just not the very last one at the time of running the CI job. >>> >>> Actually, this makes sense to me. By doing this, we could promote the last known good+consistent repo, while we fix the latest issue. >> Yes that was the intent. Let's see how we could implement this. > Hold on, haven't we previously discussed to switch to staged promotion > i.e. puppet CI promotes RDO Trunk consistent first, then tripleo CI > tries to promote that and finally RDO CI uses what tripleo CI > promoted? Now we have all three starting from different random latest > "consistent" (at the time corresponding promotion runs) and diverging. > But in any case, first step is to have a database of "consistent" > hashes recorded somewhere, it would be just input to the puppet CI > promotion proposal script, right? The staged promotion is a different story that we need to continue. For me that's a parallel effort we need to pursue. But even with a staged promotion, we could still want to try on a failure a previous consistent repo that we could promote. Fred

_______________________________________________ rdo-list mailing list rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com