Hi Otis,
I have fixed it by assigning the value to rb same as assigned to sreq:
rb.shards = shards.toString().split(",");
not tested that fully yet, but distributed faceting works at least on my pc
_3 shards 1 router_ setup.
Dmitry
On Thu, Jun 16, 2011 at 4:53 PM, Dmitry Kan <[email protected]> wrote:
> Hi Otis,
>
> I followed your recommendation and decided to implement the
> SearchComponent::modifyRequest(ResponseBuilder rb, SearchComponent who,
> ShardRequest sreq) method, where the query routing happens. So far it is
> working OK for the non-facet search, this is good news. The bad news is that
> it fails on the facet search.
>
> This is how request modification happens:
>
> [code_snippet, SearchComponent::modifyRequest]
> SolrQueryRequest req_routed = rb.req;
> req_routed = routeRequest(req_routed);
> rb.req = req_routed;
> sreq.shards = shards.toString().split(",");
> [/code_snippet]
>
> where shards is StringBuilder, that accumulates the shards the request
> should go to. req_routed also contains the target shards. Those are set like
> this:
>
>
> [code_snippet, my function routeRequest(SolrQueryRequest req)]
> // could not find clone(), used ref reassignment
> SolrQueryRequest req_local = req;
> ModifiableSolrParams params = new
> ModifiableSolrParams(req_local.getParams());
> ...
> params.remove(ShardParams.SHARDS);
> params.set(ShardParams.SHARDS, getShardsParams(yearToQuarterMap));
> params.remove(ShardParams.IS_SHARD);
> params.set(ShardParams.IS_SHARD, true);
> req_local.setParams(params);
> ...
> return req_local;
> [/code_snippet]
>
> The NPE happens down the road during the facet search, in the
> FacetComponent::countFacets(), the cause of which is that OpenBitSet obs is
> null for shardNum=0.
>
> Do you have any idea why this happens, should some other field
> of ResponseBuilder, SearchComponent or ShardRequest be changed?
>
> BTW, I have tried to call FacetInfo::parse method inside
> FacetComponent::modifyRequest() and countFacets(). Where do
> the fi.facets.values() get initiated, is there some method to call?
>
> Thanks,
> Dmitry
>
> On Fri, Jun 3, 2011 at 8:00 PM, Otis Gospodnetic <
> [email protected]> wrote:
>
>> Nah, if you can quickly figure out which shard a given query maps to, then
>> all
>> this component needs to do is stick the appropriate shards param value in
>> the
>> request and let the request pass through to the other SearchComponents in
>> the
>> chain, including QueryComponent, which will know what to do with the
>> shards
>> param.
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>
>>
>> ----- Original Message ----
>> > From: Dmitry Kan <[email protected]>
>> > To: [email protected]
>> > Sent: Fri, June 3, 2011 12:56:15 PM
>> > Subject: Re: query routing with shards
>> >
>> > Hi Otis,
>> >
>> > Thanks! This sounds promising. This custom implementation, will it hurt
>> in
>> > any way the stability of the front end SOLR? After implementing it, can
>> I
>> > run some tests to verify the stability / performance?
>> >
>> > Dmitry
>> > On Fri, Jun 3, 2011 at 4:49 PM, Otis Gospodnetic <
>> [email protected]
>> > > wrote:
>> >
>> > > Hi Dmitry,
>> > >
>> > > Yes, you could also implement your own custom SearchComponent. In
>> this
>> > > component you could grab the query param, examine the query value,
>> and
>> > > based on
>> > > that add the shards URL param with appropriate value, so that when
>> the
>> > > regular
>> > > QueryComponent grabs stuff from the request, it has the correct shard
>> in
>> > > there
>> > > already.
>> > >
>> > > Otis
>> > > ----
>> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> > > Lucene ecosystem search :: http://search-lucene.com/
>> > >
>> > >
>> > >
>> > > ----- Original Message ----
>> > > > From: Dmitry Kan <[email protected]>
>> > > > To: [email protected]
>> > > > Sent: Fri, June 3, 2011 2:47:00 AM
>> > > > Subject: Re: query routing with shards
>> > > >
>> > > > Hi Otis,
>> > > >
>> > > > I merely followed on the gmail's suggestion to include other
>> people
>> into
>> > > the
>> > > > recipients list, Yonik was the first one :) I won't do it next
>> time.
>> > > >
>> > > > Thanks for a rapid reply. The reason for doing this query routing
>> is
>> > > that we
>> > > > abstract the distributed SOLR from the client code for security
>> reasons
>> > > > (that is, we don't want to expose the entire shard farm to the
>> world,
>> > > but
>> > > > only the frontend SOLR) and for better decoupling.
>> > > >
>> > > > Is it possible to implement a plugin to SOLR that would map
>> queries to
>> > > > shards?
>> > > >
>> > > > We have other choices too, they'll take quite some time, that's
>> why I
>> > > > decided to quickly ask, if I was missing something from the SOLR
>> main
>> > > > components design and configuration.
>> > > >
>> > > > Dmitry
>> > > >
>> > > > On Fri, Jun 3, 2011 at 8:25 AM, Otis Gospodnetic <
>> > > [email protected]
>> > > > > wrote:
>> > > >
>> > > > > Hi Dmitry (you may not want to additionally copy Yonik, he's
>> > > subscribed to
>> > > > > this
>> > > > > list, too)
>> > > > >
>> > > > >
>> > > > > It sounds like you have the knowledge of which query maps to
>> which
>> > > shard.
>> > > > > If
>> > > > > so, why not control/change the value of "shards" param in the
>> request
>> > > to
>> > > > > your
>> > > > > front-end Solr (aka distributed request dispatcher) within your
>> app,
>> > > which
>> > > > > is
>> > > > > the one calling Solr?
>> > > > >
>> > > > > Otis
>> > > > > ----
>> > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> > > > > Lucene ecosystem search :: http://search-lucene.com/
>> > > > >
>> > > > >
>> > > > >
>> > > > > ----- Original Message ----
>> > > > > > From: Dmitry Kan <[email protected]>
>> > > > > > To: [email protected]; [email protected]
>> > > > > > Sent: Thu, June 2, 2011 7:00:53 AM
>> > > > > > Subject: query routing with shards
>> > > > > >
>> > > > > > Hello all,
>> > > > > >
>> > > > > > We have currently several pretty fat logically isolated shards
>> with
>> > > the
>> > > > > same
>> > > > > > schema / solrconfig (indices are separate). We currently have
>> one
>> > > single
>> > > > > > front end SOLR (1.4) for the client code calls. Since a
>> client
>> code
>> > > > > query
>> > > > > > usually hits only one shard, we are considering making a smart
>> > > routing
>> > > > > of
>> > > > > > queries to the shards they map to. Can you please give some
>> > > pointers as
>> > > > > to
>> > > > > > what would be an optimal way to achieve such a routing inside
>> the
>> > > front
>> > > > > end
>> > > > > > solr? Is there a way to configure mapping inside the
>> solrconfig?
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > --
>> > > > > > Regards,
>> > > > > >
>> > > > > > Dmitry Kan
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Regards,
>> > > >
>> > > > Dmitry Kan
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Regards,
>> >
>> > Dmitry Kan
>> >
>>
>
>
>
--
Regards,
Dmitry Kan
