Re: Unique key constraint and optimistic locking (versioning)

2012-02-29 Thread Per Steffensen
Created SOLR-3178 covering the versioning/optimistic-locking part. In combination SOLR-3173 and SOLR-3178 should provide the features I am missing, and that I believe lots of other SOLR users will be able to benefit from. Please help shape by commenting on the Jira issues. Thanks. Per Steffens

Re: Unique key constraint and optimistic locking (versioning)

2012-02-28 Thread Per Steffensen
Created SOLR-3173 on the part about making insert fail if document (with same uniqueKey) already exists. SOLR-3173 also includes to make "update" not insert document if not already exists - just for consistency with normal RDBMS behaviour. So basically the feature allowes you to turn on this be

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Mark Miller
On Feb 24, 2012, at 6:55 AM, Em wrote: > You need a log for failover. There is a transaction log. - Mark Miller lucidimagination.com

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
Yonik, thanks for sharing deeper details about how SolrCloud is going to work. Do you plan to release any wiki-updates about the small details, so that other developers are able to get in touch with what you've already done there? I think small guides and the mentioning of class-names and their r

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
Hi Per, if you are evaluating with your ProductOwner whether he/she wants to contribute back: Try to not see it only as a gift to the community for a highly usefull product, but also see it as a protection of your investment. What you are going to customize will be deeply integrated in Solr - in

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Yonik Seeley
On Fri, Feb 24, 2012 at 8:59 AM, Per Steffensen wrote: > We might make it "outside" Solr/Lucene but I > hope to be able to convince my ProductOwner to make it as a Solr-feature > contributing it back - especiallly if the Solr community agrees that it > would be a nice and commonly usable feature.

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Per Steffensen skrev: Em skrev: This is a really cool feature! Thanks for pointing us in that direction! A feature where you can flag your "index" operation to provide "create sematics" would be cool. When setting the "create-semantics" flag, an "index" operation will fail if a document wit

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Yonik Seeley skrev: On Fri, Feb 24, 2012 at 9:04 AM, Per Steffensen wrote: Cool. We have a test doing exactly that - indexing 2000 documents into Solr, kill-9'ing Solr in the middle of the process, starting Solr again and checking that 2000 documents will eventually be searchable. It lights

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Yonik Seeley
On Fri, Feb 24, 2012 at 9:04 AM, Per Steffensen wrote: > Cool. We have a test doing exactly that - indexing 2000 documents into Solr, > kill-9'ing Solr in the middle of the process, starting Solr again and > checking that 2000 documents will eventually be searchable. It lights red as > it is right

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Yonik Seeley skrev: On Fri, Feb 24, 2012 at 6:55 AM, Em wrote: However, regarding a versioning-system, one always has to keep in mind that an uncommited document is not guaranteed to be persisted in the index. We now have durability via an update log. With a recent nightly trunk build

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Em skrev: This is a really cool feature! Thanks for pointing us in that direction! A feature where you can flag your "index" operation to provide "create sematics" would be cool. When setting the "create-semantics" flag, an "index" operation will fail if a document with simular id (or whatev

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Yonik Seeley
On Fri, Feb 24, 2012 at 6:55 AM, Em wrote: > However, regarding a versioning-system, one always has to keep in mind > that an uncommited document is not guaranteed to be persisted in the index. We now have durability via an update log. With a recent nightly trunk build, you can send a document to

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
This is a really cool feature! Thanks for pointing us in that direction! As the "Quick Start" says, a document does not need a commit nor a soft-commit or anything else to be available via RealTimeGet. However, regarding a versioning-system, one always has to keep in mind that an uncommited docum

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Em
Hi Per, > Can you give a code-pointer to where I can find the pending-set stuff? > Does solr use this pending-set for query responses, so that solr deliver > 100% real-time search results? As of Solr 3.5 it can be found within the DirectUpdateHandler and DirectUpdateHandler2-classes. I am currentl

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Sami Siren
On Fri, Feb 24, 2012 at 12:06 PM, Per Steffensen wrote: > Sami Siren skrev: > Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - upd

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Sami Siren skrev: Given that you've set a uniqueKey-field and there already exists a document with that uniqueKey, it will delete the old one and insert the new one. There is really no difference between the semantics - updates do not exist. To create a UNIQUE-constraint as you know it from a dat

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Sami Siren
>> Given that you've set a uniqueKey-field and there already exists a >> document with that uniqueKey, it will delete the old one and insert the >> new one. There is really no difference between the semantics - updates >> do not exist. >> To create a UNIQUE-constraint as you know it from a database

Re: Unique key constraint and optimistic locking (versioning)

2012-02-24 Thread Per Steffensen
Em skrev: Hi Per, I want an error to occur if a document with the same id already exists, when my intent is to INSERT a new document. When my intent is to UPDATE a document in solr/lucene I want the old document already in solr/lucene deleted and the new version of this document added (exact

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Em
Hi Per, > I want an error to occur if a document with the same id already > exists, when my intent is to INSERT a new document. When my intent is > to UPDATE a document in solr/lucene I want the old document already > in solr/lucene deleted and the new version of this document added > (exactly as

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Erick Erickson
Per: Yep, you've got it. You could write a custom update handler that queried (via TermDocs or something) for the ID when your intent was to INSERT, but it'll have to be custom work. I suppose you could query with a divide-and-conquer approach, that is query for id:(1 2 58 90... all your insert ID

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Per Steffensen
Em skrev: Hi Per, well, Solr has no "Update"-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. Yes I understand. But it is not always what I want to acheive. I want an error

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Em
Hi Per, well, Solr has no "Update"-Method like a RDBMS. It is a re-insert of the whole document. Therefore a document with an existing UniqueKey marks the old document as deleted and inserts the new one. However this is not the whole story, since this "constraint" only works per index/SolrCore/Sha

Re: Unique key constraint and optimistic locking (versioning)

2012-02-23 Thread Per Steffensen
Em skrev: Hi Per, Solr provides the so called "UniqueKey"-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey Belive the uniqueKey does not enforce a "unique key constraint", so that you are not allowed to create a document with an id's when an document with the sa

Re: Unique key constraint and optimistic locking (versioning)

2012-02-22 Thread Per Steffensen
Per Steffensen skrev: Thanks a lot. We will use the UniqueKey feature and build versioning ourselves. Do you think it would be a good idea if we built a versioning feature into Solr/Lucene instead of doing it outside, so that others can benefit from the feature as well? Guess contributions wil

Re: Unique key constraint and optimistic locking (versioning)

2012-02-22 Thread Per Steffensen
Thanks a lot. We will use the UniqueKey feature and build versioning ourselves. Do you think it would be a good idea if we built a versioning feature into Solr/Lucene instead of doing it outside, so that others can benefit from the feature as well? Guess contributions will be made according to

Re: Unique key constraint and optimistic locking (versioning)

2012-02-21 Thread Em
Hi Per, Solr provides the so called "UniqueKey"-field. Refer to the Wiki to learn more: http://wiki.apache.org/solr/UniqueKey > Optimistic locking (versioning) ... is not provided by Solr out of the box. If you add a new document with the same UniqueKey it replaces the old one. You have to do the