Re: Post Processing Solr Results

2011-09-01 Thread Jamie Johnson
Ok, think I got it. Basically the issue was that I can't modify the offset and start params when the search is a distributed one, otherwise the correct offset and max are lost, a simple check in prepare fixed this. On Thu, Sep 1, 2011 at 11:10 AM, Jamie Johnson wrote: > Ok, so I feel like I'm 90

Re: Post Processing Solr Results

2011-09-01 Thread Jamie Johnson
Ok, so I feel like I'm 90% of the way there. For standard queries things work fine, but for distributed queries I'm running into a bit of an issue. Right now the queries run fine but when doing distributed queries (using SolrCloud) the numFound is always getting set to the number of requested row

Re: Post Processing Solr Results

2011-08-30 Thread Jamie Johnson
This might work in conjunction with what POST processing to help to pair down the results, but the logic for the actual access to the data is too complex to have entirely in solr. On Mon, Aug 29, 2011 at 2:02 PM, Erick Erickson wrote: > It's reasonable, but post-filtering is often difficult, you

Re: Post Processing Solr Results

2011-08-29 Thread Erik Hatcher
Sounds like you're looking for https://issues.apache.org/jira/browse/SOLR-2429 which has been committed to trunk and also the 3_x branch (after the release of 3.3). Erik On Aug 29, 2011, at 11:46 , Jamie Johnson wrote: > Thanks guys, perhaps I am just going about this the wrong way. S

Re: Post Processing Solr Results

2011-08-29 Thread Erick Erickson
It's reasonable, but post-filtering is often difficult, you have too many documents to wade through. If you can see any way at all to just include a clause in the query, you'll save a world of effort... Is there any way you can include a value in some kind of "permissions" field? Let's say you hav

Re: Post Processing Solr Results

2011-08-29 Thread Jamie Johnson
Thanks guys, perhaps I am just going about this the wrong way. So let me explain my problem and perhaps there is a more appropriate solution. What I need to do is basically hide certain results based on some passed in user parameter (say their service tier for instance). What I'd like to do is h

Re: Post Processing Solr Results

2011-08-29 Thread Erik Hatcher
I haven't followed the details, but what I'm guessing you want here is Lucene's FieldCache. Perhaps something along the lines of how faceting uses it (in SimpleFacets.java) - FieldCache.DocTermsIndex si = FieldCache.DEFAULT.getTermsIndex(searcher.getIndexReader(), fieldName); Erik

Re: Post Processing Solr Results

2011-08-29 Thread Erick Erickson
If you're asking whether there's a way to find, say, all the values for the "auth" field associated with a document... no. The nature of an inverted index makes this hard (think of finding all the definitions in a dictionary where the word "earth" was in the definition). Best Erick On Mon, Aug 29

Re: Post Processing Solr Results

2011-08-29 Thread Jamie Johnson
Thanks Erick, if I did not know the token up front that could be in the index is there not an efficient way to get the field for a specific document and do some custom processing on it? On Mon, Aug 29, 2011 at 8:34 AM, Erick Erickson wrote: > Start here I think: > > http://lucene.apache.org/java/

Re: Post Processing Solr Results

2011-08-29 Thread Erick Erickson
Start here I think: http://lucene.apache.org/java/3_0_2/api/core/index.html?org/apache/lucene/index/TermDocs.html Best Erick On Mon, Aug 29, 2011 at 8:24 AM, Jamie Johnson wrote: > Thanks for the reply.  The fields I want are indexed, but how would I > go directly at the fields I wanted? > > In

Re: Post Processing Solr Results

2011-08-29 Thread Jamie Johnson
Thanks for the reply. The fields I want are indexed, but how would I go directly at the fields I wanted? In regards to indexing the auth tokens I've thought about this and am trying to get confirmation if that is reasonable given our constraints. On Mon, Aug 29, 2011 at 8:20 AM, Erick Erickson

Re: Post Processing Solr Results

2011-08-29 Thread Erick Erickson
Yeah, loading the document inside a Collector is a definite no-no. Have you tried going directly at the fields you want (assuming they're indexed)? That *should* be much faster, but whether it'll be fast enough is a good question. I'm thinking some of the Terms methods here. You *might* get some jo

Re: Post Processing Solr Results

2011-08-28 Thread Jamie Johnson
Just a bit more information. Inside my class which extends FilteredDocIdSet all of the time seems to be getting spent in retrieving the document from the readerCtx, doing this Document doc = readerCtx.reader.document(docid); If I comment out this and just return true things fly along as I expect

Post Processing Solr Results

2011-08-28 Thread Jamie Johnson
I have a need to post process Solr results based on some access controls which are setup outside of Solr, currently we've written something that extends SearchComponent and in the prepare method I'm doing something like this QueryWrapperFilter qwf = new QueryWrapperFilter(rb.ge