Looking at POOL-350, I realized that we don't really have a coherent
strategy for handling liveness issues in GKOP. We have been playing
whack-a-mole with problems resulting from two facts about how GKOP works:
1. There are two capacity constraints that bind on individual keyed
pools at different times: total instance count and total per key.
2. Borrowers waiting on keyed pools have no way to know when capacity
exists to create new instances. They have to wait for new instances
to be added to the LinkedBlockingDeques that they are waiting on.
To (partially) work around 2, we added two methods. First, clearOldest,
called by borrowObject when no instances are available for the given
key, creates make capacity by clearing the oldest idle instances across
all of the keyed pools. This makes it less likely that a borrower will
park waiting when overall capacity exists to create an instance under
the desired key. Second, reuseCapacity, called by returnObject and
invalidateObject, tries to create an instance in the most heavily loaded
pool that can add an instance. These both help reduce the incidence of
borrowers waiting when instances could be created to serve them, but
they don't handle all cases and as reported in POOL-350, when there are
a lot of pools, they can cause performance problems.
I think we need to agree on some principles before continuing to patch
liveness issues in GKOP. I hope we can lazily agree with:
0) Users should be able to turn off all optimizations that look across
keyed pools (e.g. reuseCapacity, clearOdest). With these things turned
off, liveness at the individual keyed pool level should work like GOP
(failed validations, invalidate, etc trigger creates under the same key).
1) New borrowers should not take precedence over waiting threads when
capacity exists to create new instances.
2) When configured to globally optimize resources (new config option),
GKOP should decide on each return, invalidate, evict (anything that
passivates or destroys an instance) where to allocate the newly
available capacity. How exactly this works should be configurable.
None of these are true in the code today. Note that 2) means that when
instances are returned, they may not be added to their returning idle
instance pools. That is basically what clearOldest is doing ad hoc only
for new borrowers.
Please anyone interested - especially users - let me know what you think
of 0)-2). If we can agree on them, we can refactor to make the code
consistent with the principles.
Phil
