[rdo-dev] State of distgit jobs for rdoproject

[Paul Belanger]

On Mon, Jul 09, 2018 at 02:33:23PM +0000, Tristan Cacqueray wrote:
> Hello,
> 
> On July 4, 2018 2:26 pm, Paul Belanger wrote:
> > On Tue, Jul 03, 2018 at 11:42:09PM +0200, Alan Pevec wrote:
> > > > With the zuulv3 migration wrapping up, I wanted to start a thread about projects
> > > > that use the package-distgit-check-jobs template. These are projects like:
> > > >
> > > >   openstack/cloudkittyclient-distgit
> > > >
> > > > I wanted to raise the idea of maybe pushing these projects directly upstream into
> > > > git.openstack.org. The main reason would be to leverage the upstream testing
> > > > infrastructure upstream, and maybe increase adoption of rpm with other
> > > > openstack teams. It is also one less this we as RDO have to manage on our own.
> > > 
> > > 
> > > > From a governance POV these could be under an rdoproject team, tripleo or some
> > > > other.
> > > 
> > > Ideally we would not have one core team but keep them synced with
> > > maintainers listed in rdoinfo,
> > > how could that work with distgit in review.o.o ?
> > > In review.rdo we're using SF resource manager to create gerrit ACL, is
> > > that manual operation in openstack gerrit?
> > > 
> > Yes, we manage ACLs via code review too, so we can create one per distgit or
> > have a core group.  We bootstrap each ACL with the PTL (or person who created
> > the projects) then they manage users vi gerrit.
> > 
> The SF resource manager to update gerrit ACL is different from using the
> gerrit interface as it is fully managed through code review in Software Factory.
> See this example of an ACL update: https://review.rdoproject.org/r/10152
> 
> We could contribute the resource manager to jeepby, but that's a critical
> workflow modification. Do you think the openstack-infra would accept this?
> Note that branch creation and suppression are also managed by the
> resource manager.
> 
> For tracability reasons the resource manager is a very important feature, but
> perhaps we could adopt the openstack-infra way of doing this instead.
> 
I would reach out to infra team on Tuesday and maybe start a spec to contribute
this feature upstream. Give our love of all things yaml and code review, this
might be a welcome feature.

> > > > To me the main question is around publishing of RPM and DLRN. Given secrets are
> > > > now part of zuulv3, is there any external services running that we'd need to
> > > > worry about?  Could the publishing process be run from openstack-infra service?
> > > 
> > > I would not be comfortable putting CBS Koji certs in openstack-infra.
> > > Could we instead trigger CBS Koji builds via 3rd party CI in review.rdo ?
> > > 
> > Why not comfortable? It the same process as RDO zuul for upstream, no root would
> > be able to access them.  However, yes, we can setup rdo zuul as 3pci and trigger
> > CBS builds from a post pipeline if needed.
> > 
> Beside the above concerns, I'd like to note a few other things we need to
> be careful about undertaking such a migration.
> 
> Using review.openstack.org requires an UbuntuOne account...
> review.rdoproject.org is currently configured to use GitHub identity,
> but the next version of Software Factory supports openid
> (e.g. fedora account system) and SAML2 (e.g. keycloack).
> 
Been on the table for a long while to migrate off UbuntuOne for SSO. I believe
this will get active again with winterscale effort happening.

> Tripleo-ci users are currently using a config repository to manage
> check and gate jobs that need secrets as well as to update the
> openstack-periodic pipeline timer to re-schedule the promotion jobs.
> Could those users have +3 permissions on a config repository upstream?
> 
We don't allow untrusted jobs to have secrets in check / gate pipelines today,
I don't believe we'd give +3 just to use secrets.  We'd likely want to work with
upstream here and better understand why those jobs need it.  Projects are using
secrets in tree, but more for post / artifact publishers.

> We have also been working on new Zuul features[0] to support enqueue and
> abort from the REST api[1], Unfortunately the change needed in Zuul
> doesn't get much attention. Do you think we could enable them in
> openstack-infra zuul? Or are we going to be relying on infra-core
> to stop and re-enqueue RDO promotion jobs when needed?
> 
My first reaction is no, we don't want random users doing enqueue / abort on
zuul.o.o. I can see places where this would be exploited. Give how little we
actually enqueue changes, I still see this being an infra-root only task.

However, I think there is some confusion here, we wouldn't move promotion jobs
upstream, they are specific to tripleo and require OVB (single cloud). We moved
them downstream to give tripleo folks more freedom to manage them, so pushing
them upstream now, isn't the first goal.  distgit packages seem to be generic
enough to run on any cloud and do not have these special requirements.

> There are lots of pros to doing such migration, but the blockers are
> very important. Is there an etherpad we could list and comment on the
> pros/cons of this migration?
> 
https://etherpad.openstack.org/p/rdo-distgit-upstream

I'll start collecting thoughts there.


> Best regards,
> -Tristan
> 
> [0] https://review.openstack.org/#/c/562321/
> [0] https://tree.taiga.io/project/morucci-software-factory/us/960