Mahout includes a file reader for Lucene indexes. It will read from HDFS or local disks.
On Thu, Jul 26, 2012 at 6:57 PM, Darren Govoni <dar...@ontrenet.com> wrote: > You raise an interesting possibility. A map/reduce solr handler over > solrcloud....... > > On Thu, 2012-07-26 at 18:52 -0700, Trung Pham wrote: > >> I think the performance should be close to Hadoop running on HDFS, if >> somehow Hadoop job can directly read the Solr Index file while executing >> the job on the local solr node. >> >> Kindna like how HBase and Cassadra integrate with Hadoop. >> >> Plus, we can run the map reduce job on a standby Solr4 cluster. >> >> This way, the documents in Solr will be our primary source of truth. And we >> have the ability to run near real time search queries and analytics on it. >> No need to export data around. >> >> Solr4 is becoming a very interesting solution to many web scale problems. >> Just missing the map/reduce component. :) >> >> On Thu, Jul 26, 2012 at 3:01 PM, Darren Govoni <dar...@ontrenet.com> wrote: >> >> > Of course you can do it, but the question is whether this will produce >> > the performance results you expect. >> > I've seen talk about this in other forums, so you might find some prior >> > work here. >> > >> > Solr and HDFS serve somewhat different purposes. The key issue would be >> > if your map and reduce code >> > overloads the Solr endpoint. Even using SolrCloud, I believe all >> > requests will have to go through a single >> > URL (to be routed), so if you have thousands of map/reduce jobs all >> > running simultaneously, the question is whether >> > your Solr is architected to handle that amount of throughput. >> > >> > >> > On Thu, 2012-07-26 at 14:55 -0700, Trung Pham wrote: >> > >> > > Is it possible to run map reduce jobs directly on Solr4? >> > > >> > > I'm asking this because I want to use Solr4 as the primary storage >> > engine. >> > > And I want to be able to run near real time analytics against it as well. >> > > Rather than export solr4 data out to a hadoop cluster. >> > >> > >> > > > -- Lance Norskog goks...@gmail.com
