[rdo-dev] zuulv3 migration update

[Paul Belanger]

On Thu, Jun 28, 2018 at 10:16:02PM -0400, Paul Belanger wrote:
> On Mon, Jun 25, 2018 at 03:33:20PM -0400, Paul Belanger wrote:
> > On Fri, Jun 22, 2018 at 01:00:14PM -0400, Paul Belanger wrote:
> > > Greetings,
> > > 
> > > Wanted to give an update to the zuulv3 migration process. As it stands right
> > > now, all projects specific to the rdoproject.org have been fully migrated from
> > > jenkins to zuulv3.  Hopefully this process was transparent to you, for the most
> > > part we were able to fully test everything before flipping the switch on a
> > > project.  However we did have a few issues with jobs, but thanks to jpena,
> > > amoralej and number80, to name a few, we managed to land fixes quickly.
> > > 
> > > At this point, we are still waiting to complete the migration of tripleo-ci
> > > jobs, this include OVB and various 3rd party CI jobs.  We've been blocked by a
> > > nodepool migration of upstream-centos-7 nodes, but I'm told that will be
> > > finished today.  Meaning, we are aiming for Monday morning to flip the switch on
> > > the rest of the tripleo-ci specific projects.
> > > 
> > > Due to this, we are still converting JJB changes into ansible playbooks using
> > > the zuul-migrate command, so I please hold of on moving forward with changes to
> > > the new rdo-jobs project.  If everything goes well on Monday, we'll remove the
> > > freeze on config / rdo-jobs for people to start contributing to again.
> > > 
> > > Thanks for your patience during these last 3 weeks, and hopefully we'll be
> > > wrapped up Monday.  If you have any questions or problems, please reply here or
> > > #rdo on freenode.
> > > 
> > Next update, today we successfully tested zuulv3 base jobs for devstack based
> > tripleo jobs.  This basically means we are ready to migrate the remaining
> > tripleo in rdoproject.org from jenkins to zuulv3. We still need to test OVB
> > jobs, but confident they will work.
> > 
> > Nodepool capacity has been addressed however we are running into scale issues
> > with zuulv3 now. We have an IRC message out to the SoftwareFactory team to see
> > about bringing onlne more zuul-executors / zuul-mergers, hopefully this will not
> > take long.  Once that happens, I finally think we'll be ready to migrate the
> > left over tripleo jobs from jenkins to zuulv3.
> > 
> Hopefully the last update before we make zuul migration complete.
> 
> We've been able to validate tripleo OVB jobs are working correctly under zuulv3.
> Since it is now Friday, we won't migrate them from jenkins until early next
> week.
> 
> Thanks again for the patience and look forward to sending the all clear next
> week.
> 
We seen some scale issues today with zuulv3 migration, this will account for
some POST_FAILURE results in jobs. We are increasing the capacity of our
zuul-executors and hopefully addng a 4th.

This has delayed the migration of promotion jobs, to help ensure everything is
working well before we move them.  I'll send out another update tomorrow.

- Paul