Nutch has two ways to make a distributed query - through HDFS(hadoop file
system) or RPC call that is in
"org.apache.nutch.searcher.DistributedSearch" class.
But I think these are both not good enough.
If we use HDFS to service the user's query. Stability is a problem. We must
all do the crawl , index , query on HDFS and use mapreduce. Can we trust in
hadoop all the time?:)
If we use the RPC call in nutch . Manually separate the index is required .
We will receive reduplicate result if there is reduplicate index document on
different servers. And also the data updating and single server's error is
hard to deal with.
Thanks,
Jarvis
-----Original Message-----
From: Stu Hood [mailto:[EMAIL PROTECTED]
Sent: Wednesday, September 19, 2007 10:37 PM
To: solr-user@lucene.apache.org
Subject: Re: How can i make a distribute search on Solr?
Nutch implements federated search separately from their index generation.
My understanding is that MapReduce jobs generate the indexes (Nutch calls
them segments) from raw data that has been downloaded, and then makes them
available to be searched via remote procedure calls. Queries never pass
through MapReduce in any shape or form, only the raw data and indexes.
If you take a look at the "org.apache.nutch.searcher.DistributedSearch"
class, specifically the #Client.search method, you can see how they handle
the actual federation of results.
Thanks,
Stu
-----Original Message-----
From: Norberto Meijome
Sent: Wednesday, September 19, 2007 10:23am
To: solr-user@lucene.apache.org
Cc: [EMAIL PROTECTED]
Subject: Re: How can i make a distribute search on Solr?
On Wed, 19 Sep 2007 01:46:53 -0400
Ryan McKinley wrote:
> Stu is referring to Federated Search - where each index has some of the
> data and results are combined before they are returned. This is not yet
> supported out of the "box"
Maybe this is related. How does this compare to the map-reduce functionality
in Nutch/Hadoop ?
cheers,
B
_________________________
{Beto|Norberto|Numard} Meijome
"With sufficient thrust, pigs fly just fine. However, this is not
necessarily a good idea.
It is hard to be sure where they are going to land, and it could be
dangerous sitting under them as they fly overhead."
[RFC1925 - section 2, subsection 3]
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
