Konstantin,
On 7/17/14, 7:58 AM, Konstantin Kolinko wrote:
> 2014-07-15 23:43 GMT+04:00 Christopher Schultz <ch...@christopherschultz.net>:
>> Mark,
>>
>> (...)
>>
>> We have had situations /in production/ where a "single thread" has
>> triggered an OOME .. in a particular case, that thread was trying to
>> pull an unbounded set of results back from a database query and cache
>> them in a local data structure. The OOME killed the operation, the GC
>> kicked-in, trashed that huge structure, and the system stabilized.
>>
>> We didn't want to bounce the server in the middle of the day and
>> interrupt people if we didn't have to. We watched things closely and
>> were convinced that it was in fact stable. We waited for the weekend and
>> bounced the JVM -- just to be "safe".
>>
>> Had the background thread died in that situation, we would have had to
>> bounce the JVM within about 30 minutes given our user load since the
>> sessions would never die. We also have to meet certain "security
>> requirements" about expiration of idle connections, and without the
>> background thread, we cannot enforce those rules. (Note that, when the
>> background thread is dead, you can't even use JMX to ask that expired
>> sessions be expired: the 'backgroundProcess' operation seems to try to
>> nudge the background processor which .. isn't around anymore).
>>
>> This discussion has become even more academic since Konstantin has begun
>> a bit of work already to help alleviate this. Thanks, K.
>>
>
> It wouldn't be committed unless there is some consensus. Real-life
> examples are important to evaluate the feature.
>
> Well, at least the logging part of the patch can be useful.
>
> Note that ContainerBase#threadStart() protects itself from repeated
> calls with "if (thread!=null)" check.
>
> If you add try{}finally{ thread=null; } to
> ContainerBackgroundProcessor.run() then it will be possible to
> explicitly call threadStart() to restart the thread.It's probably worth doing, even if we don't expose a "restart" method just yet. > It may be possible to expose threadStart() to JMX. (Though maybe > better to create a separate public method. E.g. one that walks all > children and tries to start their threads). It needs some testing > whether that is feasible. > > An issue with automated restart of background thread is that the > restarted thread is likely to perform the same work. As thus, it is > likely to meet the same stackoverflowexception a few seconds after it > has been started. Agreed, but this is something that wouldn't necessarily be automatic: an admin can make the determination of whether to restart the thread or not and observe the results. > Note also that you can configure a separate background thread in each > container. You may configure to have separate thread in Host (doing > autodeployment) and separate threads in Contexts (doing session > expiration work). I would be happy if the Manager's "backgroundProcess" JMX operation would simply run the background process one single time, synchronously or not. Right now, it ties into the background-processing lifecycle such that it won't necessarily expire any sessions on a given execution. Perhaps what I'm really looking for is direct access to the processExpires() method. -chris
signature.asc
Description: OpenPGP digital signature
