Sorry - I meant indexed. I don't store the fields. --dallan
> -----Original Message----- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, January 22, 2010 9:30 AM > To: solr-user@lucene.apache.org > Subject: Re: How to store a dense field value efficiently > > Oops, that's a Lucene bit (got confused which list I was on). > > You can still control storing the raw text in SOLR, so my > question is still relevant, but the solution may be > different. Do you store the fields? > > Erick > > On Fri, Jan 22, 2010 at 10:27 AM, Erick Erickson > <erickerick...@gmail.com>wrote: > > > I'm surprised by a 30% increase. The approach of adding a special > > token for "not present" is one of the standard ones.... > > > > So just to check, when you say "stored", are you really storing the > > missing value? As in Field.Store.YES? As opposed to > Field.Index.###? > > Because theres no need to Store this value. > > > > Erick > > > > On Thu, Jan 21, 2010 at 11:22 PM, Dallan Quass > <dal...@quass.org> wrote: > > > >> Hi, > >> > >> I want to issue queries where queried fields have a > specified value > >> or are "missing". I know that I can query missing values using a > >> negated full-range query, but it doesn't seem like that's very > >> efficient (the fields in question have a lot of possible > values). So > >> I've opted to store special "missing" value for each field > that isn't > >> found in a document, and issue queries like "+(field1:value > >> field1:missing) +(field2:value field2:missing)". > >> > >> The issue is that storing the missing values increases the size of > >> the index by 30%, because a lot of documents don't have values for > >> all fields. I'd like to keep the index as small as possible so it > >> can be cached in memory. > >> > >> Any ideas on an alternative approach? Is there a way to convince > >> lucene to store the doc-id list for the "missing" field value as a > >> bitmap? What if I added some boolean fields to my schema; e.g., > >> field1_missing and field2_missing and stored a true in > those fields > >> for documents that were missing the corresponding fields? Does > >> lucene store BoolField's as bitmaps? > >> > >> -dallan > >> > >> > > >
