[rdo-dev] [infra][outage] trunk.rdoproject.org down

Javier Pena jpena at redhat.com
Wed Sep 4 13:19:28 UTC 2019


> 
> > > Hi all,
> > > 
> > > Due to an issue in the cloud provider, trunk.rdoproject.org has been down
> > > during the weekend.
> > > 
> > > While we are still trying to recover it, we have switched to our
> > > secondary
> > > server inside RDO Cloud. Depending on the DNS propagation time, it may
> > > still
> > > take a few minutes to be visible, but normal operations will be resumed
> > > soon.
> > > 
> > > We will send another email as soon as everything is back to normal.
> > > Please
> > > do
> > > not hesitate to contact me if you need additional information.
> > > 
> > 
> > Hi,
> > 
> > Apparently, the volumes where the RDO Trunk data was stored in
> > trunk.rdoproject.org are not recoverable. Thus, we have set up a new
> > trunk.rdo machine, and will follow this procedure to recover data:
> > 
> > - We have stopped new builds on RDO Trunk to sync the data to the new
> > machine.
> > - Tomorrow morning, once all data has been synced, we will switch the
> > trunk.rdo DNS entry to the new machine and resume normal operations.
> > 
> > All repositories will remain accessible during the process, with only a
> > brief
> > outage expected tomorrow while the DNS is propagating.
> > 
> 
> Hi,
> 
> The DNS entries have been updated. I am currently monitoring DNS propagation,
> please be aware that there may be some flakiness until it is completed. For
> example, some Google DNS servers are currently showing the new IP, while
> some others are not.
> 
> Once DNS propagation is stabilized, we will resume normal operations.
> 

Hi,

Everything seems to be back to normal since this morning. If you find anything unusual, please let me know.

Regards,
Javier

> Regards,
> Javier
> 
> > Regards,
> > Javier
> > 
> > 
> > > Regards,
> > > Javier
> > > _______________________________________________
> > > dev mailing list
> > > dev at lists.rdoproject.org
> > > http://lists.rdoproject.org/mailman/listinfo/dev
> > > 
> > > To unsubscribe: dev-unsubscribe at lists.rdoproject.org
> > > 
> > 
> _______________________________________________
> dev mailing list
> dev at lists.rdoproject.org
> http://lists.rdoproject.org/mailman/listinfo/dev
> 
> To unsubscribe: dev-unsubscribe at lists.rdoproject.org
> 


More information about the dev mailing list