Great, that's exactly the behavior I would expect. Thanks.
-- Mike Stolz Principal Engineer, GemFire Product Manager Mobile: +1-631-835-4771 On Mon, Sep 11, 2017 at 4:34 PM, Jason Huynh <jhu...@pivotal.io> wrote: > Hi Mike, I think the concern was less about the security portion but rather > if any exception occurs during index update, right now, the region gets > updated and the rest of the system (index/wan/callbacks) may or may not be > updated. I think Naba just tried to provide an example where this might > occur, but that specific scenario is invalid. > > I believe Nabarun has opened a ticket for rolling back the put operation > when an index exception occurs. GEODE-3589. It can probably be modified to > state any exception instead of index exceptions. > > To summarize my understanding: > -Someone will need to implement the rollback for GEODE-3589. This means > that if any exception occurs during a put, geode it will propagate back to > the user and it is expected the rollback mechanism will clean up any > partial put. > > GEODE-3520 should be modified to: > -Add the isValid() api to index interface > -Mark an index as invalid during async index updates but not for > synchronous index updates. The synchronous index updates will rely on a > rollback mechanism > > > > > On Mon, Sep 11, 2017 at 1:23 PM Michael Stolz <mst...@pivotal.io> wrote: > > > I think there was an intention of having CREATION of an index require a > > higher privilege than DATA:WRITE, but it shouldn't affect applying the > > index on either of put or get operations. > > > > If we are requiring something like CLUSTER:MANAGE for put on an indexed > > region, that is an incorrect requirement. Only DATA:WRITE should be > > required to put an entry and have it be indexed if an index is present. > > > > -- > > Mike Stolz > > Principal Engineer, GemFire Product Manager > > Mobile: +1-631-835-4771 <(631)%20835-4771> > > > > On Fri, Sep 8, 2017 at 6:04 PM, Anilkumar Gingade <aging...@pivotal.io> > > wrote: > > > > > Indexes are critical for querying; most of the databases doesn't allow > > > insert/update if there is any failure with index maintenance... > > > > > > As Geode OQL supports two ways (sync and async) to maintain the > indexes, > > we > > > need be careful about the error handling in both cases... > > > > > > My take is: > > > 1. For synchronous index maintenance: > > > If there is any failure in updating any index (security/auth or logical > > > error) on the region; throw an exception and rollback the cache > update/op > > > (index management id done under region.entry lock - we should be able > to > > > revert the op). If index or cache is left in bad state, then its a bug > > that > > > needs to be addressed. > > > > > > Most of the time, If there is any logical error in index, it will be > > > detected as soon as index is created (on existing data) or when first > > > update is done to the cache. > > > > > > 2. For Asynchronous index maintenance: > > > As this is async (assuming) user has good understanding of the risk > > > involved with async, any error with index maintenance, the index should > > be > > > invalidated... > > > > > > About the security/auth, the user permission with region read/write > > needs > > > to be applied for index updates, there should not be different > permission > > > on index. > > > > > > -Anil. > > > > > > > > > > > > On Fri, Sep 8, 2017 at 2:01 PM, Nabarun Nag <n...@pivotal.io> wrote: > > > > > > > Hi Mike, > > > > > > > > Please do find our answers below: > > > > *Question:* What if there were multiple indices that were in flight > and > > > > only the third > > > > one errors out, will they all be marked invalid? > > > > > > > > *Answer:* Only the third will be marked invalid and only the third > one > > > will > > > > not be used for query execution. > > > > > > > > *Question/Statement:* If anything goes wrong with the put it should > > > > probably still throw back to > > > > the caller. Silent invalidation of the index is probably not > desirable. > > > > > > > > *Answer: * > > > > In our current design this the flow of execution of a put operation: > > > > entry put into region -> update index -> other wan related > executions / > > > > callbacks etc. > > > > > > > > If an exception happens while updating the index, the cache gets > into a > > > bad > > > > state, and we may end up getting different results depending on the > > index > > > > we are using. As the failure happens half way in a put operation, the > > > > regions / cache are now in a bad state. > > > > -------------------------- > > > > We are thinking that if index is created over a method invocation in > > an > > > > empty region and then we do puts, but method invocation is not > allowed > > as > > > > per security policies. The puts will now be successful but the index > > will > > > > be rendered invalid. Previously the puts will fail with exception and > > put > > > > the entire cache in a bad state. > > > > > > > > > > > > > > > > Regards > > > > Nabarun > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 8, 2017 at 10:43 AM Michael Stolz <mst...@pivotal.io> > > wrote: > > > > > > > > > Just to help me understand, the index is corrupted in a way beyond > > just > > > > the > > > > > field that errors out? > > > > > What if there were multiple indices that were in flight and only > the > > > > third > > > > > one errors out, will they all be marked invalid? > > > > > If anything goes wrong with the put it should probably still throw > > back > > > > to > > > > > the caller. Silent invalidation of the index is probably not > > desirable. > > > > > > > > > > -- > > > > > Mike Stolz > > > > > Principal Engineer, GemFire Product Manager > > > > > Mobile: +1-631-835-4771 <(631)%20835-4771> <(631)%20835-4771> > > > > > > > > > > On Fri, Sep 8, 2017 at 12:34 PM, Dan Smith <dsm...@pivotal.io> > > wrote: > > > > > > > > > > > +1 > > > > > > > > > > > > -Dan > > > > > > > > > > > > On Thu, Sep 7, 2017 at 9:14 PM, Nabarun Nag <n...@apache.org> > > wrote: > > > > > > > > > > > > > *Proposal:* > > > > > > > * Index interface will include an API - isValid() which will > > return > > > > > true > > > > > > if > > > > > > > the index is still valid / uncorrupted, else will return false > if > > > it > > > > > > > corrupted / invalid. > > > > > > > * gfsh command "list index" will have one more column "Is > Valid" > > > > which > > > > > > will > > > > > > > state the status as "true" or "false". > > > > > > > * Invalid indexes will not be used during query executions. > > > > > > > * In case the index is found to be invalid, the user will be > able > > > to > > > > > > > remove/destroy the index. > > > > > > > * When a put operation corrupts an index, it will be logged. > > > > > > > > > > > > > > *Reasoning:* > > > > > > > * Currently if a put operation raises an exception while > updating > > > the > > > > > > > index, the put operation fails with an exception to the putter. > > > > > > > * For example, if an index is created on an object method, and > > that > > > > > > method > > > > > > > causes an exception while updating the index as a part of a put > > > > > > operation, > > > > > > > then the put operation fails for that particular entry and the > > > index > > > > is > > > > > > > left in a bad state. > > > > > > > * This may occur also due to lack of permission to update index > > but > > > > > have > > > > > > > permission to do puts. > > > > > > > * We are proposing that in the above mentioned scenarios, the > put > > > > > > succeeds > > > > > > > in putting the entry in the region but the index which it was > > > trying > > > > to > > > > > > > update will be marked invalid and will not be used for query > > > > > executions. > > > > > > > * This can be justified because the corrupted index will not > > > > correctly > > > > > > > represent the region entries. > > > > > > > > > > > > > > *Status Quo:* > > > > > > > * Index creation will still fail if we are trying to create an > > > index > > > > > > over a > > > > > > > region containing an entry/entries which will cause an > exception > > > > while > > > > > > > loading the entry into the index. > > > > > > > > > > > > > > Regards > > > > > > > Nabarun Nag > > > > > > > > > > > > > > > > > > > > > > > > > > > >
