1.) Does anyone know of a way to do a rollback where the put is already reflected in the region? If that is the desired behavior, then perhaps we will have to live with the current (leaving the region and indexes in a bad state, wan and other callbacks that occur after index maintenance will not occur for the one operation but the put has made it into the region) until someone can figure out how to roll a put back and revert the update to all the indexes.
How should this affect putAll, if at all? Any callbacks that occur before index update have already been called (cache writers?). I am not sure how those should be affected by a rollback... 2.) So the index behavior changes if they are marked for sync/async. In sync the index would reject the put, but in async they would just be marked as invalid. On Sat, Sep 9, 2017 at 6:48 AM John Blum <jb...@pivotal.io> wrote: > +1 to both of Anil's points. > > On Fri, Sep 8, 2017 at 3:04 PM, Anilkumar Gingade <aging...@pivotal.io> > wrote: > > > Indexes are critical for querying; most of the databases doesn't allow > > insert/update if there is any failure with index maintenance... > > > > As Geode OQL supports two ways (sync and async) to maintain the indexes, > we > > need be careful about the error handling in both cases... > > > > My take is: > > 1. For synchronous index maintenance: > > If there is any failure in updating any index (security/auth or logical > > error) on the region; throw an exception and rollback the cache update/op > > (index management id done under region.entry lock - we should be able to > > revert the op). If index or cache is left in bad state, then its a bug > that > > needs to be addressed. > > > > Most of the time, If there is any logical error in index, it will be > > detected as soon as index is created (on existing data) or when first > > update is done to the cache. > > > > 2. For Asynchronous index maintenance: > > As this is async (assuming) user has good understanding of the risk > > involved with async, any error with index maintenance, the index should > be > > invalidated... > > > > About the security/auth, the user permission with region read/write > needs > > to be applied for index updates, there should not be different permission > > on index. > > > > -Anil. > > > > > > > > On Fri, Sep 8, 2017 at 2:01 PM, Nabarun Nag <n...@pivotal.io> wrote: > > > > > Hi Mike, > > > > > > Please do find our answers below: > > > *Question:* What if there were multiple indices that were in flight and > > > only the third > > > one errors out, will they all be marked invalid? > > > > > > *Answer:* Only the third will be marked invalid and only the third one > > will > > > not be used for query execution. > > > > > > *Question/Statement:* If anything goes wrong with the put it should > > > probably still throw back to > > > the caller. Silent invalidation of the index is probably not desirable. > > > > > > *Answer: * > > > In our current design this the flow of execution of a put operation: > > > entry put into region -> update index -> other wan related executions / > > > callbacks etc. > > > > > > If an exception happens while updating the index, the cache gets into a > > bad > > > state, and we may end up getting different results depending on the > index > > > we are using. As the failure happens half way in a put operation, the > > > regions / cache are now in a bad state. > > > -------------------------- > > > We are thinking that if index is created over a method invocation in > an > > > empty region and then we do puts, but method invocation is not allowed > as > > > per security policies. The puts will now be successful but the index > will > > > be rendered invalid. Previously the puts will fail with exception and > put > > > the entire cache in a bad state. > > > > > > > > > > > > Regards > > > Nabarun > > > > > > > > > > > > > > > > > > On Fri, Sep 8, 2017 at 10:43 AM Michael Stolz <mst...@pivotal.io> > wrote: > > > > > > > Just to help me understand, the index is corrupted in a way beyond > just > > > the > > > > field that errors out? > > > > What if there were multiple indices that were in flight and only the > > > third > > > > one errors out, will they all be marked invalid? > > > > If anything goes wrong with the put it should probably still throw > back > > > to > > > > the caller. Silent invalidation of the index is probably not > desirable. > > > > > > > > -- > > > > Mike Stolz > > > > Principal Engineer, GemFire Product Manager > > > > Mobile: +1-631-835-4771 <(631)%20835-4771> <(631)%20835-4771> > > > > > > > > On Fri, Sep 8, 2017 at 12:34 PM, Dan Smith <dsm...@pivotal.io> > wrote: > > > > > > > > > +1 > > > > > > > > > > -Dan > > > > > > > > > > On Thu, Sep 7, 2017 at 9:14 PM, Nabarun Nag <n...@apache.org> > wrote: > > > > > > > > > > > *Proposal:* > > > > > > * Index interface will include an API - isValid() which will > return > > > > true > > > > > if > > > > > > the index is still valid / uncorrupted, else will return false if > > it > > > > > > corrupted / invalid. > > > > > > * gfsh command "list index" will have one more column "Is Valid" > > > which > > > > > will > > > > > > state the status as "true" or "false". > > > > > > * Invalid indexes will not be used during query executions. > > > > > > * In case the index is found to be invalid, the user will be able > > to > > > > > > remove/destroy the index. > > > > > > * When a put operation corrupts an index, it will be logged. > > > > > > > > > > > > *Reasoning:* > > > > > > * Currently if a put operation raises an exception while updating > > the > > > > > > index, the put operation fails with an exception to the putter. > > > > > > * For example, if an index is created on an object method, and > that > > > > > method > > > > > > causes an exception while updating the index as a part of a put > > > > > operation, > > > > > > then the put operation fails for that particular entry and the > > index > > > is > > > > > > left in a bad state. > > > > > > * This may occur also due to lack of permission to update index > but > > > > have > > > > > > permission to do puts. > > > > > > * We are proposing that in the above mentioned scenarios, the put > > > > > succeeds > > > > > > in putting the entry in the region but the index which it was > > trying > > > to > > > > > > update will be marked invalid and will not be used for query > > > > executions. > > > > > > * This can be justified because the corrupted index will not > > > correctly > > > > > > represent the region entries. > > > > > > > > > > > > *Status Quo:* > > > > > > * Index creation will still fail if we are trying to create an > > index > > > > > over a > > > > > > region containing an entry/entries which will cause an exception > > > while > > > > > > loading the entry into the index. > > > > > > > > > > > > Regards > > > > > > Nabarun Nag > > > > > > > > > > > > > > > > > > > > > > > > -- > -John > john.blum10101 (skype) >
