I've scaled down the amount of total threads we had after being
notified by the
team that our usage had gone
significantly up [1].
We had 16 threads before and with the new slaves we were up to 34 threads.
I've scaled down this number back to 24 for the time being to see if
that improves the situation.
[1]
David Moreau Simard
Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
On Wed, Aug 3, 2016 at 2:31 PM, David Moreau Simard <dms(a)redhat.com> wrote:
We noticed that the promotion jobs relied on a property file created
locally on the slave.
Since promotion jobs rely on that property file to share parameters
(such as which trunk repository jobs should be testing) and jobs could
be running on any of the four slaves, this was problematic.
For the time being, promotion jobs have been pinned to a single slave
but we are looking at a way to remove this limitation to benefit from
the redundancy and increased capacity.
David Moreau Simard
Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
On Wed, Aug 3, 2016 at 12:40 PM, David Moreau Simard <dms(a)redhat.com> wrote:
> Now that's a lot of running jobs [1] :)
>
> [1]:
http://i.imgur.com/s7Cq53M.png
>
> David Moreau Simard
> Senior Software Engineer | Openstack RDO
>
> dmsimard = [irc, github, twitter]
>
>
> On Wed, Aug 3, 2016 at 12:14 PM, David Moreau Simard <dms(a)redhat.com> wrote:
>> Hi,
>>
>> We've tested successfully three new slaves out of the "beta"
OpenStack
>> cloud on
ci.centos.org.
>> We're going to be lowering the amount of threads on our existing slave
>> and spread the load evenly across the 4 slaves.
>>
>> The objective is two-fold:
>> - Spread load evenly across four slaves rather than one: redundancy
>> and additional capacity/concurrency
>> - Test real workloads on the
ci.centos.org OpenStack cloud before it
>> is opened up to additional tenants
>>
>> I will be monitoring closely (moreso than usual) the jobs but
>> everything /should/ work.
>> You can tell on which slave a particular job was run from at the very
>> beginning of the console output, it looks like this:
>> "Building remotely on rdo-ci-cloudslave01 (rdo) in workspace [...]"
>>
>> If you notice anything odd about jobs running on the new cloudslaves
>> machines, please let me know directly.
>>
>> Thanks !
>>
>> David Moreau Simard
>> Senior Software Engineer | Openstack RDO
>>
>> dmsimard = [irc, github, twitter]