Liz, I've built terrabyte (1-2 TB) test Lucene indexes, but have not reached to the petabyte level, so I am not sure. Certainly there is overhead in using the http and xml marshaling/de-marshaling, which may or may not be a critical factor for you.
Could you give more information with respect to your application, i.e. the nature of your data loading (i.e. many PB at once or GB per hour/day/week accumulating to PB or MB per second/minute/hour eventually accumulating to PB...;) searching ( i.e. the number of fields indexed & the query complexity; if you are using facets, etc), number of queries per second expected... Lucene has a limit on the number of documents (in a single index) that might impact your application: http://lucene.apache.org/java/3_0_2/api/core/org/apache/lucene/index/IndexWriter.html#numDocs%28%29 of a 32bit int, 2 147 483 648. -glen On 24 August 2010 12:29, Liz Sommers <lizswo...@gmail.com> wrote: > I was worried that it wouldn't scale. We are going to be indexing petabytes > of data. Does the httpserver solution scale? > > Thanks > > Liz Sommers > lizswo...@gmail.com > > On Tue, Aug 24, 2010 at 12:23 PM, Thomas Joiner > <thomas.b.joi...@gmail.com>wrote: > >> Is there any reason you aren't using http://wiki.apache.org/solr/Solrj to >> interact with Solr? >> >> On Tue, Aug 24, 2010 at 11:12 AM, Liz Sommers <lizswo...@gmail.com> wrote: >> >> > I am very new to the solr/lucene world. I am using solr 1.4.0 and cannot >> > move to 1.4.1. >> > >> > I have to index about 50 fields for each document, these fields are >> already >> > in key/value pairs by the time I get to my index methods. I was able to >> > index them with lucene without any problem, but found that I could not >> then >> > read the indexes with solr/admin. So, I decided to use Solr for my >> > indexing. >> > >> > The error I am currently getting is >> > java.lang.RuntimeException: Can't find resource 'synonyms.txt' in >> classpath >> > or 'solr/conf'/' >> > >> > This exception is being thrown by SolrResourceLoader.openResource(line >> > 260). >> > which is called by IndexSchema<init> (line 102) >> > >> > My code that leads up to this follows: >> > >> > <code> >> > String path = "c:/swdev/apache-solr-1.4.0/IDW" >> > SolrConfig cfg new SolrConfig(path + "/solr/conf/solrconfig.xml"); >> > schema = new IndexSchema(cfg,path + "/solr/conf/schema.xml",null); >> > >> > </code> >> > >> > This also fails if I use >> > schema = new IndexSchema(cfg,"schema.xml",null); >> > >> > >> > Any help would be greatly appreciated. >> > >> > Thank you >> > >> > Liz Sommers >> > lizswo...@gmail.com >> > >> > -- -
