well, i have 8 m1.large ec2 having 2 core 7gb ram and 1tb ebs attached to each server for index.
in my case i dont expect index to be store in ram neither a quick reply as its not a real time application, i just want fault tolerance in application and availability of full data. Is it good to use HDFS over normal solr cloud? Best, Amey --- Original Message --- From: "Michael Della Bitta" <michael.della.bi...@appinions.com> Sent: September 15, 2014 9:26 PM To: solr-user@lucene.apache.org Subject: Re: Moving to HDFS, How to merge indices from 8 servers ? There's not much about Solr Cloud or HDFS indexes that suggests you should only have one logical shard. If your goal is better uptime with a sharded index, you should add more replicas. If your collection is small enough that one machine can serve one query with acceptable performance, but you want to scale to many queries, then just adding mirrors of a single-sharded collection is fine. But that's a big "if." Switching to HDFS is an option if you have enough RAM for your whole collection, and have a lot of existing storage devoted to HDFS, or if you want to batch create indexes. It's not really aimed at preserving uptime as far as I know. Michael Della Bitta Applications Developer o: +1 646 532 3062 appinions inc. “The Science of Influence Marketing” 18 East 41st Street New York, NY 10017 t: @appinions <https://twitter.com/Appinions> | g+: plus.google.com/appinions <https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts> w: appinions.com <http://www.appinions.com/> On Mon, Sep 15, 2014 at 11:23 AM, Amey Jadiye <ameyjad...@codeinventory.com> wrote: > Thanks for reply Erik, > I think i have some misconfusion about how SOLR works with HDFS, and > solution i am thinking could be reorganised by user community :) > Here is the actual solution/situation which is implemented by me > *Usecase* : I need a google like search engine which should be work in > distributed and fault tolerant mode, we are collecting the health related > URLs from a third party system in large amount, approx 1Million/hour. we > want to build an inventory which contains all of there detail. now i am > fetching that URL data breaking it in H1, P, Div like tags with help of > Jsoup lib and putting in Solr as a documents with different boost to > different fields. > Now after the putting this data, i have a custom program with which we > categorise all the data Example. All the cancer related pages, i am > querying the SOLR and fetching all URL related to cancer with CursorMark > and putting in a file for further use of our system. > *Old Solution* : For this i have build the 8 SOLR servers with 3 > zookeepers on the individual AWS Ec2 instances with one collection:8 shards > problem with this solution is whenever any instance go down i am loosing > that data for a moment. link of current solution > http://postimg.org/image/luli3ybtj/ > *New _OR_ could be faulty solution* : I am thinking that if i use HDFS > which is virtually only one file system is better so if my server go down > that data is available through another server, below is steps i am thinking > to do. > 1 > I will merge all the 8 server indices somewhere in to one.2 > Make > setting for HDFS on same 8 servers.3 > Put the merged index folder in HDFS > so it will be distributed in 8 servers physically it self.4 > Restart 8 > servers pointing to HDFS on each instance.5 > and now i am ready to go for > putting data on 8 servers and fetching through any one of SOLR , if that is > down choose another so it will be guaranteed to get all the data. > So is this solution sounds good, OR you guys suggest me another better > solution ? > Regards,Amey > > > > Date: Thu, 11 Sep 2014 14:41:48 -0700 > > Subject: Re: Moving to HDFS, How to merge indices from 8 servers ? > > From: erickerick...@gmail.com > > To: solr-user@lucene.apache.org > > > > Um, I really think this is pretty likely to not be a great solution. > > When you say "merge indexes", I'm thinking you want to go from 8 > > shards to 1 shard. Now, this can be done with the "merge indexes" core > > admin API, see: > > https://wiki.apache.org/solr/MergingSolrIndexes > > > > BUT. > > 1> This will break all things SolrCloud-ish assuming you created your > > 8 shards under SolrCloud. > > 2> Solr is usually limited by memory, so trying to fit enough of your > > single huge index into memory may be problematical. > > > > This feels like an XY problem, _why_ are you asking about this? What > > is the use-case you want to handle by this? > > > > Best, > > Erick > > > > On Thu, Sep 11, 2014 at 7:44 AM, Amey Jadiye > > <ameyjad...@codeinventory.com> wrote: > > > FYI, I searched the google for this problem but didn't find any > satisfactory answer.Here is the current situation : I have the 8 shards in > my solr cloud backed up with 3 zookeeper all are setup on AWS EC2 > instances, all 8 are leader with no replicas.I have only 1 collection say > collection1 divided in 8 shards, i have configured the index and tlog > folder on each server pointing into 1TB EBS disk attached to each servers, > all 8 servers are having around 100GB for index folder each. so total index > files i have is ~800Gb.Now, i want to move all the data to HDFS, so I am > going to setup the HDFS on all 8 serversMerge all the indexes from 8 > serversPut in HDFS.Stop and Start my all solr servers on HDFS to access > that common index data with setting below cp parameter and few > more.-Dsolr.directoryFactory=HdfsDirectoryFactory > -Dsolr.lock.type=hdfs -Dsolr.data.dir=hdfs://host:port/path > -Dsolr.updatelog=hdfs://host:port/path -jarNow could you tell me is this > correct approach? if yes how can i merge all indices from 8 server > ?Regards,Amey > >
