[Rdo-list] [RDO-Manager] Solving "nova is locked" errors from Ironic

[Imre Farkas]

On 06/23/2015 01:26 PM, Dmitry Tantsur wrote:
> Hi all,
>
> So, Ironic task manager is attacking us again:
> https://bugzilla.redhat.com/show_bug.cgi?id=1233452
>
> Previously we already had
> https://bugzilla.redhat.com/show_bug.cgi?id=1212134
> I've implemented retries upstream in ironicclient and backported to our
> packages. Later I had to bump the retry timeout in instack-undercloud to
> 1 minute:
> https://github.com/rdo-management/instack-undercloud/blob/master/scripts/instack-ironic-deployment#L72-L74
>
>
> Now we have the same problem in another place and I wonder how to fix
> it. I have 2 obvious idea:
> 1. Patch ironicclient to have longer default timeout (2 mins?)
> 2. Update stackrc to carry longer IRONIC_MAX_RETRIES and
> IRONIC_RETRY_INTERVAL
>
> I'd prefer the latter, as it does not touch ironicclient package, only
> undercloud installation tool. It can also be changed more easily in run
> time. WDYT?

Hi Dmitry,

These env variables work with the unified CLI too, not just with the 
instack script, right? I assume the answer is yes (as it seems the 
script is using the CLI), so #2 seems to be a good option. However, I 
was wondering whether others will also hit this issue outside RDO, which 
would make #1 a better option. Considering that, we can try to fix it in 
ironicclient and if there's any push-back, we can still go with updating 
stackrc.
Also, as an improvement to #1, can we make the timeout configurable in 
ironicclient eg. by passing a --timeout flag?

Imre


>
> I wonder what the root cause is as well. I suspect some very slow BMC's
> take too much time to do power actions or power syncs. It's possible
> we'll make a wrong guess, and the problem will persist despite becoming
> rare.
>
> Thanks,
> Dmitry
>
> _______________________________________________
> Rdo-list mailing list
> Rdo-list at redhat.com
> https://www.redhat.com/mailman/listinfo/rdo-list
>
> To unsubscribe: rdo-list-unsubscribe at redhat.com