Re: [rdo-list] Replacing the tripleo-quickstart HA job with a single controller pacemaker job

On 30/06/2016 20:46, John Trowbridge wrote: > Howdy folks, > > Just wanted to give a heads up that I plan to replace the > "high-availability" tripleo-quickstart job in the CI promotion > pipeline[1], with a job with a lower footprint. In CI, we get a virthost > with 32G of RAM and a mediocre CPU. It is really hard to fit 5 really > active VMs on that, and we have never had the HA job stable enough to > use as a gate for that reason. > > Instead, we will test the pacemaker code path in tripleo by using a > single controller setup with pacemaker enabled. We were never actually > testing HA (ie failover scenarios) in the current job, so this should be > a pretty minimal loss in coverage. > > Since this allows us to drop two CPU intensive nodes from the deploy, we > can add a ceph node to that job. This will end up with more code > coverage then the current HA job, and will hopefully will end up being > stable enough to use as a gate as well. > > Longer term, it would be good to restore an actual HA job, maybe even > adding some failure scenario tests to the job. I have a couple of ideas > about how we could do this, but none are feasible in the short term. > > 1. Use pre-existing servers for deploying[2] > > This would allow running the HA job against any cloud, where we could > size the nodes appropriately to make the job stable. > > 2. Use an OVB cloud for the HA job. > > Soon we should have an OVB (openstack virtual baremetal) cloud to run > tests in. OVB would have all of the benefits of the solution above > (unrestricted VM size), and would also provide us a way to test Ironic > in a more realistic way since it mocks IPMI rather than our current > method of using a fake ironic driver (which just does virsh commands > over SSH). > > 3. Add a feature to tripleo-quickstart to bridge multiple virthosts > > If we could deploy our virtual machines across 2 different hosts, we > would then have much more room to deploy the HA job. > > > If anyone has some better ideas, they are very welcome! Hi John, No better ideas here, I just want to add that all the work I've done in the last months about roles (ansible-role-tripleo-baremetal-undercloud) was made for having a physical environment in which test HA scenarios (using ansible-role-tripleo-overcloud-validate-ha). Not much of this is in place at the moment (in particular for the HA validation), since I got a lot of connected patches merged just yesterday, but I think we are on a good track.