Digging a bit more, I think I have answered my own questions. Can someone please say if this sounds right?
http://wiki.apache.org/solr/LotsOfCores looks like a pretty good solution. If I give each user his own shard, each query can be run in only one shard. The effect of the filter query will basically be to find that shard. The requirements listed on the wiki suggest that performance will be good. But in Solr 3.x, this won't scale with the # users/shards. Prepending a user id to indexed keywords using an analyzer will break wildcard search. If there is a wildcard, the query analyzer doesn't run filters, so it won't prepend the user id. I could prepend the user id myself before calling Solr, but that seems... bad. Scott > -----Original Message----- > From: Scott Schneider [mailto:scott_schnei...@symantec.com] > Sent: Thursday, November 07, 2013 2:03 PM > To: solr-user@lucene.apache.org > Subject: RE: fq efficiency > > Thanks, that link is very helpful, especially the section, "Leapfrog, > anyone?" This actually seems quite slow for my use case. Suppose we > have 10,000 users and 1,000,000 documents. We search for "hello" for a > particular user and let's assume that the fq set for the user is > cached. "hello" is a common word and perhaps 10,000 documents will > match. If the user has 100 documents, then finding the intersection > requires checking each list ~100 times. If the user has 1,000 > documents, we check each list ~1,000 times. That doesn't scale well. > > My searches are usually in one user's data. How can I take advantage > of that? I could have a separate index for each user, but loading so > many indexes at once seems infeasible; and dynamically loading & > unloading indexes is a pain. > > Or I could create a filter that takes tokens and prepends them with the > user id. That seems like a good solution, since my keyword searches > always include a user id (and usually just 1 user id). Though I wonder > if there is a downside I haven't thought of. > > Thanks, > Scott > > > > -----Original Message----- > > From: Shawn Heisey [mailto:s...@elyograg.org] > > Sent: Tuesday, November 05, 2013 4:35 PM > > To: solr-user@lucene.apache.org > > Subject: Re: fq efficiency > > > > On 11/5/2013 3:36 PM, Scott Schneider wrote: > > > I'm wondering if filter queries are efficient enough for my use > > cases. I have lots and lots of users in a big, multi-tenant, sharded > > index. To run a search, I can use an fq on the user id and pass in > the > > search terms. Does this scale well with the # users? I suppose > that, > > since user id is indexed, generating the filter data (which is > cached) > > will be fast. And looking up search terms is fast, of course. But > if > > the search term is a common one that many users have in their > > documents, then Solr may have to perform an intersection between two > > large sets: docs from all users with the search term and all of the > > current user's docs. > > > > > > Also, how about auto-complete and searching with a trailing > wildcard? > > As I understand it, these work well in a single-tenant index because > > keywords are sorted in the index, so it's easy to get all the search > > terms that match "foo*". In a multi-tenant index, all users' > keywords > > are stored together. So if Lucene were to look at all the keywords > > from "foo" to "foozzzzz" (I'm not sure if it actually does this), it > > would skip over a large majority of keywords that don't belong to > this > > user. > > > > From what I understand, there's not really a whole lot of difference > > between queries and filter queries when they are NOT cached, except > > that > > the main query and the filter queries are executed in parallel, which > > can save time. > > > > When filter queries are found in the filterCache, it's a different > > story. They get applied *before* the main query, which means that > the > > main query won't have to work as hard. The filterCache stores > > information about which documents in the entire index match the > filter. > > By storing it as a bitset, the amount of space required is relatively > > low. Applying filterCache results is very efficient. > > > > There are also advanced techniques, like assigning a cost to each > > filter > > and creating postfilters: > > > > http://yonik.com/posts/advanced-filter-caching-in-solr/ > > > > Thanks, > > Shawn
