Hi all, So, Ironic task manager is attacking us again: https://bugzilla.redhat.com/show_bug.cgi?id=1233452 Previously we already had https://bugzilla.redhat.com/show_bug.cgi?id=1212134 I've implemented retries upstream in ironicclient and backported to our packages. Later I had to bump the retry timeout in instack-undercloud to 1 minute: https://github.com/rdo-management/instack-undercloud/blob/master/scripts/... Now we have the same problem in another place and I wonder how to fix it. I have 2 obvious idea: 1. Patch ironicclient to have longer default timeout (2 mins?) 2. Update stackrc to carry longer IRONIC_MAX_RETRIES and IRONIC_RETRY_INTERVAL I'd prefer the latter, as it does not touch ironicclient package, only undercloud installation tool. It can also be changed more easily in run time. WDYT?

I wonder what the root cause is as well. I suspect some very slow BMC's take too much time to do power actions or power syncs. It's possible we'll make a wrong guess, and the problem will persist despite becoming rare. Thanks, Dmitry _______________________________________________ Rdo-list mailing list Rdo-list(a)redhat.com https://www.redhat.com/mailman/listinfo/rdo-list To unsubscribe: rdo-list-unsubscribe(a)redhat.com

On 06/23/2015 05:39 PM, Imre Farkas wrote: > On 06/23/2015 01:26 PM, Dmitry Tantsur wrote: >> Hi all, >> >> So, Ironic task manager is attacking us again: >> https://bugzilla.redhat.com/show_bug.cgi?id=1233452 >> >> Previously we already had >> https://bugzilla.redhat.com/show_bug.cgi?id=1212134 >> I've implemented retries upstream in ironicclient and backported to our >> packages. Later I had to bump the retry timeout in instack-undercloud to >> 1 minute: >> https://github.com/rdo-management/instack-undercloud/blob/master/scripts/... >> >> >> >> >> Now we have the same problem in another place and I wonder how to fix >> it. I have 2 obvious idea: >> 1. Patch ironicclient to have longer default timeout (2 mins?) >> 2. Update stackrc to carry longer IRONIC_MAX_RETRIES and >> IRONIC_RETRY_INTERVAL >> >> I'd prefer the latter, as it does not touch ironicclient package, only >> undercloud installation tool. It can also be changed more easily in run >> time. WDYT? > > Hi Dmitry, > > These env variables work with the unified CLI too, not just with the > instack script, right? I assume the answer is yes (as it seems the > script is using the CLI), so #2 seems to be a good option. However, I > was wondering whether others will also hit this issue outside RDO, which > would make #1 a better option. Considering that, we can try to fix it in > ironicclient and if there's any push-back, we can still go with updating > stackrc. > Also, as an improvement to #1, can we make the timeout configurable in > ironicclient eg. by passing a --timeout flag? It is configurable, that's why I'm not that worried about other users of ironicclient :)

> > Imre > > >> >> I wonder what the root cause is as well. I suspect some very slow BMC's >> take too much time to do power actions or power syncs. It's possible >> we'll make a wrong guess, and the problem will persist despite becoming >> rare. >> >> Thanks, >> Dmitry >> >> _______________________________________________ >> Rdo-list mailing list >> Rdo-list(a)redhat.com >> https://www.redhat.com/mailman/listinfo/rdo-list >> >> To unsubscribe: rdo-list-unsubscribe(a)redhat.com >