You're right. I misunderstood. I thought that you wanted to optimize the "finding by id" path which is typically done for comparing versions during inserts in Solr.
Yes, it won't help with the case where the ID does not exist. On Wed, Jul 30, 2014 at 6:14 PM, Per Steffensen <st...@designware.dk> wrote: > Hi > > I am not sure exactly what LUCENE-5675 does, but reading the description > it seems to me that it would help finding out that there is no document > (having an id-field) where version-field is less than <some-version>. As > far as I can see this will not help finding out if a document with > id=<some-id> exists. We want to ask "does a document with id <some-id> > exist", without knowing the value of its version-field (if it actually > exists). You do not know if it ever existed, either. > > Please elaborate. Thanks! > > Regarding " The only other choice today is bloom filters, which use up > huge amounts of memory", I guess a bloom filter only takes as much space > (disk or memory) as you want it to. The more space you allow it to use the > more it gives you a false positive (saying "this doc might exist" in cases > where the doc actually does not exist). So the space you need to use for > the bloom filter depends on how frequently you can live with false > positives (where you have to actually look it up in the real index). > > Regards, Per Steffensen > > > On 30/07/14 10:05, Shalin Shekhar Mangar wrote: > >> Hi Per, >> >> There's LUCENE-5675 which has added a new postings format for IDs. Trying >> it out in Solr is in my todo list but maybe you can get to it before me. >> >> https://issues.apache.org/jira/browse/LUCENE-5675 >> >> >> On Wed, Jul 30, 2014 at 12:57 PM, Per Steffensen <st...@designware.dk> >> wrote: >> >> On 30/07/14 08:55, jim ferenczi wrote: >>> >>> Hi Per, >>>> First of all the BloomFilter implementation in Lucene is not exactly a >>>> bloom filter. It uses only one hash function and you cannot set the >>>> false >>>> positive ratio beforehand. ElasticSearch has its own bloom filter >>>> implementation (using "guava like" BloomFilter), you should take a look >>>> at >>>> their implementation if you really need this feature. >>>> >>>> Yes, I am looking into what Lucene can do and how to use it through >>> Solr. >>> If it does not fit our needs I will enhance it - potentially with >>> inspiration from ES implementation. Thanks >>> >>> What is your use-case ? If your index fits in RAM the bloom filter >>> won't >>> >>>> help (and it may have a negative impact if you have a lot of segments). >>>> In >>>> fact the only use case where the bloom filter can help is when your term >>>> dictionary does not fit in RAM which is rarely the case. >>>> >>>> We have so many documents that it will never fit in memory. We use >>> optimistic locking (our own implementation) to do correct concurrent >>> assembly of documents and to do duplicate control. This require a lot of >>> finding docs from their id, and most of the time the document is not >>> there, >>> but to be sure we need to check both transactionlog and the actual index >>> (UpdateLog). We would like to use Bloom Filter to quickly tell that a >>> document with a particular id is NOT present. >>> >>> Regards, >>>> Jim >>>> >>>> Regards, Per Steffensen >>> >>> >> >> > -- Regards, Shalin Shekhar Mangar.
