Upgrading solr from 3.3 to 3.4
Hi all, Does upgrading solr from 3.3 to 3.4 requires reindexing of all the cores or we can directly copy the data folders to the new solr ? -- Thanks & Regards, Isan Fulia.
Re: Upgrading solr from 3.3 to 3.4
Hi , Ya we need to upgrade but my question is whether reindexing of all cores is required or we can directly use already indexed data folders of solr 3.3 to solr 3.4. Thanks, Isan Fulia. On 19 September 2011 11:03, Wyhw Whon wrote: > If you are already using Apache Lucene 3.1, 3.2 or 3.3, we strongly > recommend you upgrade to 3.4.0 because of the index corruption bug on > OS or computer crash or power loss (LUCENE-3418), now fixed in 3.4.0. > > 2011/9/19 Isan Fulia > > > Hi all, > > > > Does upgrading solr from 3.3 to 3.4 requires reindexing of all the cores > or > > we can directly copy the data folders to > > the new solr ? > > > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
Re: Upgrading solr from 3.3 to 3.4
Thanks Erick. On 19 September 2011 15:10, Erik Hatcher wrote: > Reindexing is not necessary. Drop in 3.4 and go. > > For this sort of scenario, it's easy enough to try using a copy of your > directory with an instance of the newest release of Solr. If > the release notes don't say a reindex is necessary, then it's not, but > always a good idea to try it and run any tests you have handy. > >Erik > > > > On Sep 19, 2011, at 00:02 , Isan Fulia wrote: > > > Hi , > > > > Ya we need to upgrade but my question is whether reindexing of all cores > is > > required > > or > > we can directly use already indexed data folders of solr 3.3 to solr 3.4. > > > > Thanks, > > Isan Fulia. > > > > > > > > > > > > On 19 September 2011 11:03, Wyhw Whon wrote: > > > >> If you are already using Apache Lucene 3.1, 3.2 or 3.3, we strongly > >> recommend you upgrade to 3.4.0 because of the index corruption bug on > >> OS or computer crash or power loss (LUCENE-3418), now fixed in 3.4.0. > >> > >> 2011/9/19 Isan Fulia > >> > >>> Hi all, > >>> > >>> Does upgrading solr from 3.3 to 3.4 requires reindexing of all the > cores > >> or > >>> we can directly copy the data folders to > >>> the new solr ? > >>> > >>> > >>> -- > >>> Thanks & Regards, > >>> Isan Fulia. > >>> > >> > > > > > > > > -- > > Thanks & Regards, > > Isan Fulia. > > -- Thanks & Regards, Isan Fulia.
Solr stopword problem in Query
Hi all,
I have a text field named* textForQuery* .
Following content has been indexed into solr in field textForQuery
*Coke Studio at MTV*
when i fired the query as
*textForQuery:("coke studio at mtv")* the results showed 0 documents
After runing the same query in debugMode i got the following results
textForQuery:("coke studio at mtv")
textForQuery:("coke studio at mtv")
PhraseQuery(textForQuery:"coke studio ? mtv")
textForQuery:"coke studio *? *mtv"
Why the query did not matched any document even when there is a document
with value of textForQuery as *Coke Studio at MTV*?
Is this because of the stopword *at* present in stopwordList?
--
Thanks & Regards,
Isan Fulia.
Re: Solr stopword problem in Query
Hi Rahul,
I also tried searching "Coke Studio MTV" but no documents were returned.
Here is the snippet of my schema file.
*
*
Thanks,
Isan Fulia.
On 26 September 2011 21:19, Rahul Warawdekar wrote:
> Hi Isan,
>
> Does your search return any documents when you remove the 'at' keyword and
> just search for "Coke studio MTV" ?
> Also, can you please provide the snippet of schema.xml file where you have
> mentioned this field name and its "type" description ?
>
> On Mon, Sep 26, 2011 at 6:09 AM, Isan Fulia >wrote:
>
> > Hi all,
> >
> > I have a text field named* textForQuery* .
> > Following content has been indexed into solr in field textForQuery
> > *Coke Studio at MTV*
> >
> > when i fired the query as
> > *textForQuery:("coke studio at mtv")* the results showed 0 documents
> >
> > After runing the same query in debugMode i got the following results
> >
> >
> >
> > textForQuery:("coke studio at mtv")
> > textForQuery:("coke studio at mtv")
> > PhraseQuery(textForQuery:"coke studio ?
> mtv")
> > textForQuery:"coke studio *? *mtv"
> >
> > Why the query did not matched any document even when there is a document
> > with value of textForQuery as *Coke Studio at MTV*?
> > Is this because of the stopword *at* present in stopwordList?
> >
> >
> >
> > --
> > Thanks & Regards,
> > Isan Fulia.
> >
>
>
>
> --
> Thanks and Regards
> Rahul A. Warawdekar
>
--
Thanks & Regards,
Isan Fulia.
Query failing because of omitTermFreqAndPositions
Hi All, My schema consisted of field textForQuery which was defined as After indexing 10 lakhs of documents I changed the field to So documents that were indexed after that omiited the position information of the terms. As a result I was not able to search the text which rely on position information for eg. "coke studio at mtv" even though its present in some documents. So I again changed the field textForQuery to But now even for new documents added the query requiring positon information is still failing. For example i reindexed certain documents that consisted of "coke studio at mtv" but still the query is not returning any documents when searched for *textForQuery:"coke studio at mtv"* Can anyone please help me out why this is happening -- Thanks & Regards, Isan Fulia.
Re: Solr stopword problem in Query
Thanks Erick.
On 29 September 2011 18:31, Erick Erickson wrote:
> I think your problem is that you've set
>
> omitTermFreqAndPositions="true"
>
> It's not real clear from the Wiki page, but
> the tricky little phrase
>
> "Queries that rely on position that are issued
> on a field with this option will silently fail to
> find documents."
>
> And phrase queries rely on position information
>
> Best
> Erick
>
> On Tue, Sep 27, 2011 at 11:00 AM, Rahul Warawdekar
> wrote:
> > Hi Isan,
> >
> > The schema.xml seems OK to me.
> >
> > Is "textForQuery" the only field you are searching in ?
> > Are you also searching on any other non text based fields ? If yes,
> please
> > provide schema description for those fields also.
> > Also, provide your solrconfig.xml file.
> >
> >
> > On Tue, Sep 27, 2011 at 1:12 AM, Isan Fulia >wrote:
> >
> >> Hi Rahul,
> >>
> >> I also tried searching "Coke Studio MTV" but no documents were returned.
> >>
> >> Here is the snippet of my schema file.
> >>
> >> >> positionIncrementGap="100" autoGeneratePhraseQueries="true">
> >>
> >>
> >>
> >>
> >> >>ignoreCase="true"
> >>
> >>words="stopwords_en.txt"
> >>enablePositionIncrements="true"
> >>
> >>/>
> >> >> generateWordParts="1" generateNumberParts="1" catenateWords="1"
> >> catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
> >>
> >>
> >>
> >> >> protected="protwords.txt"/>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> >> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> >>
> >> >>ignoreCase="true"
> >>
> >>words="stopwords_en.txt"
> >>enablePositionIncrements="true"
> >>
> >>/>
> >> >> generateWordParts="1" generateNumberParts="1" catenateWords="0"
> >> catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
> >>
> >>
> >>
> >> >> protected="protwords.txt"/>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> * >> multiValued="false"/>
> >> >> multiValued="false"/>
> >>
> >> ** >> multiValued="true" omitTermFreqAndPositions="true"/>**
> >>
> >>
> >> *
> >>
> >>
> >> Thanks,
> >> Isan Fulia.
> >>
> >>
> >> On 26 September 2011 21:19, Rahul Warawdekar <
> rahul.warawde...@gmail.com
> >> >wrote:
> >>
> >> > Hi Isan,
> >> >
> >> > Does your search return any documents when you remove the 'at' keyword
> >> and
> >> > just search for "Coke studio MTV" ?
> >> > Also, can you please provide the snippet of schema.xml file where you
> >> have
> >> > mentioned this field name and its "type" description ?
> >> >
> >> > On Mon, Sep 26, 2011 at 6:09 AM, Isan Fulia >> > >wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I have a text field named* textForQuery* .
> >> > > Following content has been indexed into solr in field textForQuery
> >> > > *Coke Studio at MTV*
> >> > >
> >> > > when i fired the query as
> >> > > *textForQuery:("coke studio at mtv")* the results showed 0 documents
> >> > >
> >> > > After runing the same query in debugMode i got the following results
> >> > >
> >> > >
> >> > >
> >> > > textForQuery:("coke studio at mtv")
> >> > > textForQuery:("coke studio at mtv")
> >> > > PhraseQuery(textForQuery:"coke studio ?
> >> > mtv")
> >> > > textForQuery:"coke studio *?
> >> *mtv"
> >> > >
> >> > > Why the query did not matched any document even when there is a
> >> document
> >> > > with value of textForQuery as *Coke Studio at MTV*?
> >> > > Is this because of the stopword *at* present in stopwordList?
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Thanks & Regards,
> >> > > Isan Fulia.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Thanks and Regards
> >> > Rahul A. Warawdekar
> >> >
> >>
> >>
> >>
> >> --
> >> Thanks & Regards,
> >> Isan Fulia.
> >>
> >
> >
> >
> > --
> > Thanks and Regards
> > Rahul A. Warawdekar
> >
>
--
Thanks & Regards,
Isan Fulia.
Re: Query failing because of omitTermFreqAndPositions
Hi Mike, Thanks for the information.But why is it that once omiited positions in the past , it will always omit positions even if omitPositions is made false. Thanks, Isan Fulia. On 29 September 2011 17:49, Michael McCandless wrote: > Once a given field has omitted positions in the past, even for just > one document, it "sticks" and that field will forever omit positions. > > Try creating a new index, never omitting positions from that field? > > Mike McCandless > > http://blog.mikemccandless.com > > On Thu, Sep 29, 2011 at 1:14 AM, Isan Fulia > wrote: > > Hi All, > > > > My schema consisted of field textForQuery which was defined as > > > multiValued="true"/> > > > > After indexing 10 lakhs of documents I changed the field to > > > multiValued="true" *omitTermFreqAndPositions="true"*/> > > > > So documents that were indexed after that omiited the position > information > > of the terms. > > As a result I was not able to search the text which rely on position > > information for eg. "coke studio at mtv" even though its present in some > > documents. > > > > So I again changed the field textForQuery to > > > multiValued="true"/> > > > > But now even for new documents added the query requiring positon > > information is still failing. > > For example i reindexed certain documents that consisted of "coke studio > at > > mtv" but still the query is not returning any documents when searched for > > *textForQuery:"coke studio at mtv"* > > > > Can anyone please help me out why this is happening > > > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
Re: Using solr during optimization
Hi Mark, In the above case , what if the index is optimized partly ie. by specifying the max no of segments we want. It has been observed that after optimizing(even partly optimization), the indexing as well as searching had been faster than in case of an unoptimized one. Decreasing the merge factor will affect the performance as it will increase the indexing time due to the frequent merges. So is it good that we optimize partly(let say once in a month), rather than decreasing the merge factor and affect the indexing speed.Also since we will be sharding, that 100 GB index will be divided in different shards. Thanks, Isan Fulia. On 14 November 2011 11:28, Kalika Mishra wrote: > Hi Mark, > > Thanks for your reply. > > What you saying is interesting; so are you suggesting that optimizations > should be done usually when there not many updates. Also can you please > point out further under what conditions optimizations might be beneficial. > > Thanks. > > On 11 November 2011 20:30, Mark Miller wrote: > > > I would not optimize - it's very expensive. With 11,000 updates a day, I > > think it makes sense to completely avoid optimizing. > > > > That should be your default move in any case. If you notice performance > > suffers more than is acceptable (good chance you won't), then I'd use a > > lower merge factor. It defaults to 10 - lower numbers will lower the > number > > of segments in your index, and essentially amortize the cost of an > optimize. > > > > Optimize is generally only useful when you will have a mostly static > index. > > > > - Mark Miller > > lucidimagination.com > > > > > > On Nov 11, 2011, at 9:12 AM, Kalika Mishra wrote: > > > > > Hi Mark, > > > > > > We are performing almost 11,000 updates a day, we have around 50 > million > > > docs in the index (i understand we will need to shard) the core seg > will > > > get fragmented over a period of time. We will need to do optimize every > > few > > > days or once in a month; do you have any reason not to optimize the > core. > > > Please let me know. > > > > > > Thanks. > > > > > > On 11 November 2011 18:51, Mark Miller wrote: > > > > > >> Do a you have something forcing you to optimize, or are you just doing > > it > > >> for the heck of it? > > >> > > >> On Nov 11, 2011, at 7:50 AM, Kalika Mishra wrote: > > >> > > >>> Hi, > > >>> > > >>> I would like to optimize solr core which is in Reader Writer mode. > > Since > > >>> the Solr cores are huge in size (above 100 GB) the optimization takes > > >> hours > > >>> to complete. > > >>> > > >>> When the optimization is going on say. on the Writer core, the > > >> application > > >>> wants to continue using the indexes for both query and write > purposes. > > >> What > > >>> is the best approach to do this. > > >>> > > >>> I was thinking of using a temporary index (empty core) to write the > > >>> documents and use the same Reader to read the documents. (Please note > > >> that > > >>> temp index and the Reader cannot be made Reader Writer as Reader is > > >> already > > >>> setup for the Writer on which optimization is taking place) But there > > >> could > > >>> be some updates to the temp index which I would like to get reflected > > in > > >>> the Reader. Whats the best setup to support this. > > >>> > > >>> Thanks, > > >>> Kalika > > >> > > >> - Mark Miller > > >> lucidimagination.com > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > > > > > > > > -- > > > Thanks & Regards, > > > Kalika > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Thanks & Regards, > Kalika > -- Thanks & Regards, Isan Fulia.
Re: Using solr during optimization
Hi Mark, Thanks for the reply. You are right.We need to test first by decreasing the mergefactor and see the indexing as well as searching performance and have some numbers in hand. Also after partial optimize with the same mergefactor how long the performance lasts(both searching and indexing) by continuously adding more documents. Thanks, Isan Fulia,. On 14 November 2011 19:41, Mark Miller wrote: > > On Nov 14, 2011, at 8:27 AM, Isan Fulia wrote: > > > Hi Mark, > > > > In the above case , what if the index is optimized partly ie. by > > specifying the max no of segments we want. > > It has been observed that after optimizing(even partly optimization), the > > indexing as well as searching had been faster than in case of an > > unoptimized one. > > Yes, this remains true - searching against fewer segments is faster than > searching against many segments. Unless you have a really high merge > factor, this is just generally not a big deal IMO. > > It tends to be something like, a given query is say 10-30% slower. If you > have good performance though, this should often be something like a 50ms > query goes to 80 or 90ms. You really have to decide/test if there is a > practical difference to your users. > > You should also pay attention to how long that perf improvement lasts > while you are continuously adding more documents. Is it a super high cost > for a short perf boost? > > > Decreasing the merge factor will affect the performance as it will > > increase the indexing time due to the frequent merges. > > True - it will essentially amortize the cost of reducing segments. Have > you tested lower merge factors though? Does it really slow down indexing to > the point where you find it unacceptable? I've been surprised in the past. > Usually you can find a pretty nice balance. > > > So is it good that we optimize partly(let say once in a month), rather > than > > decreasing the merge factor and affect the indexing speed.Also since we > > will be sharding, that 100 GB index will be divided in different shards. > > Partial optimize is a good option, and optimize is an option. They both > exist for a reason ;) Many people pay the price because they assume they > have to though, when they really have no practical need. > > Generally, the best way to manage the number of segments in your index is > through the merge policy IMO - not necessarily optimize calls. > > I'm pretty sure optimize also blocks adds in previous version of Solr as > well - it grabs the commit lock. It won't do that in Solr 4, but that is > another reason I wouldn't recommend it under normal circumstances. > > I look at optimize as a last option, or when creating a static index > personally. > > > > > Thanks, > > Isan Fulia. > > > > > > > > On 14 November 2011 11:28, Kalika Mishra >wrote: > > > >> Hi Mark, > >> > >> Thanks for your reply. > >> > >> What you saying is interesting; so are you suggesting that optimizations > >> should be done usually when there not many updates. Also can you please > >> point out further under what conditions optimizations might be > beneficial. > >> > >> Thanks. > >> > >> On 11 November 2011 20:30, Mark Miller wrote: > >> > >>> I would not optimize - it's very expensive. With 11,000 updates a day, > I > >>> think it makes sense to completely avoid optimizing. > >>> > >>> That should be your default move in any case. If you notice performance > >>> suffers more than is acceptable (good chance you won't), then I'd use a > >>> lower merge factor. It defaults to 10 - lower numbers will lower the > >> number > >>> of segments in your index, and essentially amortize the cost of an > >> optimize. > >>> > >>> Optimize is generally only useful when you will have a mostly static > >> index. > >>> > >>> - Mark Miller > >>> lucidimagination.com > >>> > >>> > >>> On Nov 11, 2011, at 9:12 AM, Kalika Mishra wrote: > >>> > >>>> Hi Mark, > >>>> > >>>> We are performing almost 11,000 updates a day, we have around 50 > >> million > >>>> docs in the index (i understand we will need to shard) the core seg > >> will > >>>> get fragmented over a period of time. We will need to do optimize > every > >>> few > >>>> days or once in a month; do you have any reason not to optimize the > >>
Autocommit woes
Hi All, My autocommit settings are max docs - 1000 max time - 86 secs We have put newrelic agent so as to monitor our solr performance. In that we see a continous curve for autocommit. It is as good as autocommit is continuously being fired. Is it that if Autocommit for certain documents takes some time and in that time say some new documents are added than will it add that documents in the ongoing autocommit operation or it will immediately start a new autocommit for the new documents added(say max timeout occured) once the ongoing autocommit is done. -- Thanks & Regards, Isan Fulia.
Querying on dynamic field
Hi, I hava a dynamic field E_* I want to seach for E_abc*:something Is there any way i can do this in solr. If not possible in Solr 3.4 , does Solr 4.0 includes wildcard query on dynamic field. -- Thanks & Regards, Isan Fulia.
Solr Out of Memory Error
Hi all, I got the following error on solr with m/c configuration 4GB RAM and Intel Dual Core Processor.Can you please help me out. java.lang.OutOfMemoryError: Java heap space 2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a a value smaller than the current chunk size (2147483647)||java.lang. OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a a value smaller than the current chunk size (2147483647)|?at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161)|?at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139)|?at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285)|?at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160)|?at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)|?at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)|?at org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)|?at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|?at org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:120)|?at org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:86)|?at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|?at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:133)|?at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:221)|?at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|?at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46)|?at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:321)|?at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:253)|?at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)|?at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|?at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)|?at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|?at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)|?at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)|?at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?at org.mortbay.jetty.Server.handle(Server.java:326)|?at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)|?at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)|Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded| 2011-01-18 18:00:27.656:WARN::/solr/ProdContentIndex/select java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1024) at org.mortbay.jetty.Response.sendError(Response.java:240) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:361) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:271) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollect
Re: Solr Out of Memory Error
Hi markus, We dont have any XMX memory settings as such .Our java version is 1.6.0_19 and solr version is 1.4 developer version. Can u plz help us out. Thanks, Isan. On 18 January 2011 19:54, Markus Jelsma wrote: > Hi > > I haven't seen one like this before. Please provide JVM settings and Solr > version. > > Cheers > > On Tuesday 18 January 2011 15:08:35 Isan Fulia wrote: > > Hi all, > > I got the following error on solr with m/c configuration 4GB RAM and > > Intel Dual Core Processor.Can you please help me out. > > > > java.lang.OutOfMemoryError: Java heap space > > 2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError > likely > > caused by the Sun VM Bug described in > > https://issues.apache.org/jira/browse/LUCENE-1566; try calling > > FSDirectory.setReadChunkSize with a a value smaller than the current > chunk > > size (2147483647)||java.lang. > > OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug > > described in https://issues.apache.org/jira/browse/LUCENE-1566; try > > calling > > FSDirectory.setReadChunkSize with a a value smaller than the current > chunk > > size (2147483647)|?at > > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDi > > rectory.java:161)|?at > > > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.ja > > va:139)|?at > > > org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Compo > > undFileReader.java:285)|?at > > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: > > 160)|?at > > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav > > a:39)|?at > org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at > > org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at > > > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at > > > org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)| > > ?at > org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at > > > org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)| > > ?at > > > org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|? > > at > > > org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(Binary > > ResponseWriter.java:120)|?at > > > org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryRespo > > nseWriter.java:86)|?at > > > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|? > > at > > > org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java: > > 133)|?at > > > org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java: > > 221)|?at > > > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|? > > at > > > org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at > > > org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.j > > ava:46)|?at > > > org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilte > > r.java:321)|?at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav > > a:253)|?at > > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl > > er.java:1157)|?at > > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|? > > at > > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216 > > )|?at > > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|? > > at > > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|? > > at > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at > > > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCo > > llection.java:230)|?at > > > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java: > > 114)|?at > > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|? > > at org.mortbay.jetty.Server.handle(Server.java:326)|?at > > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at > > > org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java > > :938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at > > org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at > > org.mortbay.je
Re: Solr Out of Memory Error
Hi Grijesh,all, We are having only single master and are using multicore environment with size of various indexes as 675MB ,516 MB , 3GB , 25GB. Number of documents with 3GB index are roughly around 14 lakhs and with 25 GB are roughly around 7 lakh Queries are fired very frequently. ramBufferSize and indexing are all default settings. Thanks , Isan. On 19 January 2011 10:41, Grijesh wrote: > > On which server [master/slave] Out of Memory ocuur > What is your index in size[GB]? > How many documents you have? > What is query per second? > How you are indexing? > What is you ramBufferSize? > > - > Thanx: > Grijesh > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285392.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Thanks & Regards, Isan Fulia.
Re: Solr Out of Memory Error
Hi all, By adding more servers do u mean sharding of index.And after sharding , how my query performance will be affected . Will the query execution time increase. Thanks, Isan Fulia. On 19 January 2011 12:52, Grijesh wrote: > > Hi Isan, > > It seems your index size 25GB si much more compared to you have total Ram > size is 4GB. > You have to do 2 things to avoid Out Of Memory Problem. > 1-Buy more Ram ,add at least 12 GB of more ram. > 2-Increase the Memory allocated to solr by setting XMX values.at least 12 > GB > allocate to solr. > > But if your all index will fit into the Cache memory it will give you the > better result. > > Also add more servers to load balance as your QPS is high. > Your 7 Laks data makes 25 GB of index its looking quite high.Try to lower > the index size > What are you indexing in your 25GB of index? > > - > Thanx: > Grijesh > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Solr-Out-of-Memory-Error-tp2280037p2285779.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Thanks & Regards, Isan Fulia.
DismaxParser Query
Hi all, The query for standard request handler is as follows field1:(keyword1 OR keyword2) OR field2:(keyword1 OR keyword2) OR field3:(keyword1 OR keyword2) AND field4:(keyword3 OR keyword4) AND field5:(keyword5) How the same above query can be written for dismax request handler -- Thanks & Regards, Isan Fulia.
Re: DismaxParser Query
but q="keyword1 keyword2" does AND operation not OR On 27 January 2011 16:22, lee carroll wrote: > use dismax q for first three fields and a filter query for the 4th and 5th > fields > so > q="keyword1 keyword 2" > qf = field1,feild2,field3 > pf = field1,feild2,field3 > mm=something sensible for you > defType=dismax > fq=" field4:(keyword3 OR keyword4) AND field5:(keyword5)" > > take a look at the dismax docs for extra params > > > > On 27 January 2011 08:52, Isan Fulia wrote: > > > Hi all, > > The query for standard request handler is as follows > > field1:(keyword1 OR keyword2) OR field2:(keyword1 OR keyword2) OR > > field3:(keyword1 OR keyword2) AND field4:(keyword3 OR keyword4) AND > > field5:(keyword5) > > > > > > How the same above query can be written for dismax request handler > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
Re: DismaxParser Query
It worked by making mm=0 (it acted as OR operator) but how to handle this field1:((keyword1 AND keyword2) OR (keyword3 AND keyword4)) OR field2:((keyword1 AND keyword2) OR (keyword3 AND keyword4)) OR field3:((keyword1 AND keyword2) OR (keyword3 AND keyword4)) On 27 January 2011 17:06, lee carroll wrote: > sorry ignore that - we are on dismax here - look at mm param in the docs > you can set this to achieve what you need > > On 27 January 2011 11:34, lee carroll > wrote: > > > the default operation can be set in your config to be "or" or on the > query > > something like q.op=OR > > > > > > > > On 27 January 2011 11:26, Isan Fulia wrote: > > > >> but q="keyword1 keyword2" does AND operation not OR > >> > >> On 27 January 2011 16:22, lee carroll > >> wrote: > >> > >> > use dismax q for first three fields and a filter query for the 4th and > >> 5th > >> > fields > >> > so > >> > q="keyword1 keyword 2" > >> > qf = field1,feild2,field3 > >> > pf = field1,feild2,field3 > >> > mm=something sensible for you > >> > defType=dismax > >> > fq=" field4:(keyword3 OR keyword4) AND field5:(keyword5)" > >> > > >> > take a look at the dismax docs for extra params > >> > > >> > > >> > > >> > On 27 January 2011 08:52, Isan Fulia > wrote: > >> > > >> > > Hi all, > >> > > The query for standard request handler is as follows > >> > > field1:(keyword1 OR keyword2) OR field2:(keyword1 OR keyword2) OR > >> > > field3:(keyword1 OR keyword2) AND field4:(keyword3 OR keyword4) AND > >> > > field5:(keyword5) > >> > > > >> > > > >> > > How the same above query can be written for dismax request handler > >> > > > >> > > -- > >> > > Thanks & Regards, > >> > > Isan Fulia. > >> > > > >> > > >> > >> > >> > >> -- > >> Thanks & Regards, > >> Isan Fulia. > >> > > > > > -- Thanks & Regards, Isan Fulia.
Re: DismaxParser Query
Hi all,
I am currently using solr1.4.1 .Do I need to apply patch for extended
dismax parser.
On 28 January 2011 03:42, Erick Erickson wrote:
> In general, patches are applied to the source tree and it's re-compiled.
> See: http://wiki.apache.org/solr/HowToContribute#Working_With_Patches
>
> This is pretty easy, and I do know that "some people" have applied the
> eDismax
> patch to the 1.4 code line, but I haven't done it myself.
>
> Best
> Erick
>
> On Thu, Jan 27, 2011 at 10:27 AM, Jonathan Rochkind >wrote:
>
> > Yes, I think nested queries are the only way to do that, and yes, nested
> > queries like Daniel's example work (I've done it myself). I haven't
> really
> > tried to get into understanding/demonstrating _exactly_ how the relevance
> > ends up working on the overall master query in such a situation, but it
> sort
> > of works.
> >
> > (Just note that Daniel's example isn't quite right, I think you need
> double
> > quotes for the nested _query_, just check the wiki page/blog post on
> nested
> > queries).
> >
> > Does eDismax handle parens for order of operation too? If so, eDismax is
> > probably the best/easiest solution, especially if you're trying to parse
> an
> > incoming query from some OTHER format and translate it to something that
> can
> > be sent to Solr, which is what I often do.
> >
> > I haven't messed with eDismax myself yet. Does anyone know if there's
> any
> > easy (easy!) way to get eDismax in a Solr 1.4? Any easy way to compile
> an
> > eDismax query parser on it's own that works with Solr 1.4, and then just
> > drop it into your local lib/ for use with an existing Solr 1.4?
> >
> > Jonathan
> >
> >
> > From: Daniel Pötzinger [daniel.poetzin...@aoemedia.de]
> > Sent: Thursday, January 27, 2011 9:26 AM
> > To: solr-user@lucene.apache.org
> > Subject: AW: DismaxParser Query
> >
> > It may also be an option to mix the query parsers?
> > Something like this (not tested):
> >
> > q={!lucene}field1:test OR field2:test2 _query_:{!dismax qf=fields}+my
> > dismax -bad
> >
> > So you have the benefits of lucene and dismax parser
> >
> > -Ursprüngliche Nachricht-
> > Von: Erick Erickson [mailto:erickerick...@gmail.com]
> > Gesendet: Donnerstag, 27. Januar 2011 15:15
> > An: solr-user@lucene.apache.org
> > Betreff: Re: DismaxParser Query
> >
> > What version of Solr are you using, and could you consider either 3x or
> > applying a patch to 1.4.1? Because eDismax (extended dismax) handles the
> > full Lucene query language and probably works here. See the Solr
> > JIRA 1553 at https://issues.apache.org/jira/browse/SOLR-1553
> >
> > Best
> > Erick
> >
> > On Thu, Jan 27, 2011 at 8:32 AM, Isan Fulia > >wrote:
> >
> > > It worked by making mm=0 (it acted as OR operator)
> > > but how to handle this
> > >
> > > field1:((keyword1 AND keyword2) OR (keyword3 AND keyword4)) OR
> > > field2:((keyword1 AND keyword2) OR (keyword3 AND keyword4)) OR
> > > field3:((keyword1 AND keyword2) OR (keyword3 AND keyword4))
> > >
> > >
> > >
> > >
> > > On 27 January 2011 17:06, lee carroll
> > > wrote:
> > >
> > > > sorry ignore that - we are on dismax here - look at mm param in the
> > docs
> > > > you can set this to achieve what you need
> > > >
> > > > On 27 January 2011 11:34, lee carroll
> > > > wrote:
> > > >
> > > > > the default operation can be set in your config to be "or" or on
> the
> > > > query
> > > > > something like q.op=OR
> > > > >
> > > > >
> > > > >
> > > > > On 27 January 2011 11:26, Isan Fulia
> > wrote:
> > > > >
> > > > >> but q="keyword1 keyword2" does AND operation not OR
> > > > >>
> > > > >> On 27 January 2011 16:22, lee carroll <
> lee.a.carr...@googlemail.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > use dismax q for first three fields and a filter query for the
> 4th
> > > and
> > > > >> 5th
> > > > >> > fields
> > > > >> > so
> > > > >> > q="keyword1 keyword 2"
> > > > >> &
Patch for edismax Query Parser
Hi all, I want to know how to apply patch for extended dismax query parser on solr 1.4.1. -- Thanks & Regards, Isan Fulia.
Re: Patch for edismax Query Parser
specifically for edismax patch On 31 January 2011 18:22, Erick Erickson wrote: > Do you know how to apply patches in general? Or is this specifically > about the edismax patch? > > Quick response for the general "how to apply a patch" question: > 1> get the source code for Solr > 2> get to the point you can run "ant clean test" successfully. > 3> apply the source patch > 4> execute "ant dist". > > You should now have a war file in your /dist > > See: http://wiki.apache.org/solr/HowToContribute#Working_With_Patches > > NOTE: I haven't applied that specific patch to 1.4.1, so I don't know what > gremlins > are hanging around. > > Best > Erick > > On Mon, Jan 31, 2011 at 7:12 AM, Isan Fulia >wrote: > > > Hi all, > > I want to know how to apply patch for extended dismax query parser on > solr > > 1.4.1. > > > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
facet.mincount
Hi all, Even after making facet.mincount=1 , it is showing the results with count = 0. Does anyone know why this is happening. -- Thanks & Regards, Isan Fulia.
Re: facet.mincount
Any query followed by &facet=on&facet.date=aUpdDt&facet.date.start=2011-01-02T08:00:00.000Z&facet.date.end=2011-02-03T08:00:00.000Z&facet.date.gap=%2B1HOUR&facet.mincount=1 On 3 February 2011 15:14, Savvas-Andreas Moysidis < savvas.andreas.moysi...@googlemail.com> wrote: > could you post the query you are submitting to Solr? > > On 3 February 2011 09:33, Isan Fulia wrote: > > > Hi all, > > Even after making facet.mincount=1 , it is showing the results with count > = > > 0. > > Does anyone know why this is happening. > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
Re: facet.mincount
I am using solr1.4.1 release version I got the following error while using facet.mincount java.lang.IllegalStateException: STREAM at org.mortbay.jetty.Response.getWriter(Response.java:571) at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:158) at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:151) at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:208) at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:144) at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:95) at org.apache.jsp.admin.index_jsp._jspService(org.apache.jsp.admin.index_jsp:397) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:80) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:367) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:268) at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126) at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:431) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1098) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:286) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) at org.mortbay.jetty.Server.handle(Server.java:285) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) On 3 February 2011 16:17, dan sutton wrote: > I don't think facet.mincount works with date faceting, see here: > > http://wiki.apache.org/solr/SimpleFacetParameters > > Dan > > On Thu, Feb 3, 2011 at 10:11 AM, Isan Fulia > wrote: > > Any query followed by > > > > > &facet=on&facet.date=aUpdDt&facet.date.start=2011-01-02T08:00:00.000Z&facet.date.end=2011-02-03T08:00:00.000Z&facet.date.gap=%2B1HOUR&facet.mincount=1 > > > > On 3 February 2011 15:14, Savvas-Andreas Moysidis < > > savvas.andreas.moysi...@googlemail.com> wrote: > > > >> could you post the query you are submitting to Solr? > >> > >> On 3 February 2011 09:33, Isan Fulia wrote: > >> > >> > Hi all, > >> > Even after making facet.mincount=1 , it is showing the results with > count > >> = > >> > 0. > >> > Does anyone know why this is happening. > >> > > >> > -- > >> > Thanks & Regards, > >> > Isan Fulia. > >> > > >> > > > > > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
Re: facet.mincount
Thanks to all On 3 February 2011 20:21, Grijesh wrote: > > Hi > > facet.mincount not works with facet.date option afaik. > There is an issue for it as solr-343, but resolved. > Try apply patch, provided as a solution in this issue may solve the > problem. > Fix version for this may be 1.5 > > - > Thanx: > Grijesh > http://lucidimagination.com > -- > View this message in context: > http://lucene.472066.n3.nabble.com/facet-mincount-tp2411930p2414232.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Thanks & Regards, Isan Fulia.
Separating Index Reader and Writer
Hi all, I have setup two indexes one for reading(R) and other for writing(W).Index R refers to the same data dir of W (defined in solrconfig via ). To make sure the R index sees the indexed documents of W , i am firing an empty commit on R. With this , I am getting performance improvement as compared to using the same index for reading and writing . Can anyone help me in knowing why this performance improvement is taking place even though both the indexeses are pointing to the same data directory. -- Thanks & Regards, Isan Fulia.
Re: Separating Index Reader and Writer
Hi peter , Can you elaborate a little on how performance gain is in cache warming.I am getting a good improvement on search time. On 6 February 2011 23:29, Peter Sturge wrote: > Hi, > > We use this scenario in production where we have one write-only Solr > instance and 1 read-only, pointing to the same data. > We do this so we can optimize caching/etc. for each instance for > write/read. The main performance gain is in cache warming and > associated parameters. > For your Index W, it's worth turning off cache warming altogether, so > commits aren't slowed down by warming. > > Peter > > > On Sun, Feb 6, 2011 at 3:25 PM, Isan Fulia > wrote: > > Hi all, > > I have setup two indexes one for reading(R) and other for > writing(W).Index R > > refers to the same data dir of W (defined in solrconfig via ). > > To make sure the R index sees the indexed documents of W , i am firing an > > empty commit on R. > > With this , I am getting performance improvement as compared to using the > > same index for reading and writing . > > Can anyone help me in knowing why this performance improvement is taking > > place even though both the indexeses are pointing to the same data > > directory. > > > > -- > > Thanks & Regards, > > Isan Fulia. > > > -- Thanks & Regards, Isan Fulia.
Turn off caching
Hi, My solrConfig file looks like *:* EveryTime I fire the same query so as to compare the results for different configurations , the query result time is getting reduced because of caching. So I want to turn off the cahing or clear the ache before i fire the same query . Does anyone know how to do it. -- Thanks & Regards, Isan Fulia.
StreamingUpdateSolrServer
Hi all, I am using StreamingUpdateSolrServer with queuesize = 5 and threadcount=4 The no. of connections created are same as threadcount. Is it that it creates a new connection for every thread. -- Thanks & Regards, Isan Fulia.
LucidGaze Monitoring tool
Hi all, Does anyone know what does m on the y -axis stands for in req/sec graph for update handler . -- Thanks & Regards, Isan Fulia.
RamBufferSize and AutoCommit
Hi all , I would like to know is there any relation between autocommit and rambuffersize. My solr config does not contain rambuffersize which mean its deault(32mb).Autocommit setting are after 500 docs or 80 sec whichever is first. Solr starts with Xmx 2700M .Total Ram is 4 GB. Does the rambufferSize is alloted outside the heap memory(2700M)? How does rambuffersize is related to out of memory errors. What is the optimal value for rambuffersize. -- Thanks & Regards, Isan Fulia.
Re: RamBufferSize and AutoCommit
Hi Eric , I m actually getting out of memory error. As I told earlier my rambuffersize is default(32mb).What could be the reasons for getting this error. Can u please share ur views. On 28 March 2011 17:55, Erick Erickson wrote: > Also note that making RAMBufferSize too big isn't useful. Lucid > recommends 128M as the point over which you hit diminishing > returns. But unless you're having problems speed-wise with the > default, why change it? > > And are you actually getting OOMs or is this a background question? > > Best > Erick > > On Mon, Mar 28, 2011 at 6:23 AM, Li Li wrote: > > there are 3 conditions that will trigger an auto flushing in lucene > > 1. size of index in ram is larger than ram buffer size > > 2. documents in mamory is larger than the number set by > setMaxBufferedDocs. > > 3. deleted term number is larger than the ratio set by > > setMaxBufferedDeleteTerms. > > > > auto flushing by time interval is added by solr > > > > rambufferSize will use estimated size and the real used memory may be > > larger than this value. So if your Xmx is 2700m, setRAMBufferSizeMB. > > should set value less than it. if you setRAMBufferSizeMB to 2700m and > > the other 3 conditions are not > > triggered, I think it will hit OOM exception. > > > > 2011/3/28 Isan Fulia : > >> Hi all , > >> > >> I would like to know is there any relation between autocommit and > >> rambuffersize. > >> My solr config does not contain rambuffersize which mean its > >> deault(32mb).Autocommit setting are after 500 docs or 80 sec > >> whichever is first. > >> Solr starts with Xmx 2700M .Total Ram is 4 GB. > >> Does the rambufferSize is alloted outside the heap memory(2700M)? > >> How does rambuffersize is related to out of memory errors. > >> What is the optimal value for rambuffersize. > >> > >> -- > >> Thanks & Regards, > >> Isan Fulia. > >> > > > -- Thanks & Regards, Isan Fulia.
Re: FW: Very very large scale Solr Deployment = how to do (Expert Question)?
Hi Ephraim/Jen, Can u share that diagram with all.It may really help all of us. Thanks, Isan Fulia. On 6 April 2011 10:15, Tirthankar Chatterjee wrote: > Hi Jen, > Can you please forward the diagram attachment too that Ephraim sent. :-) > Thanks, > Tirthankar > > -Original Message- > From: Jens Mueller [mailto:supidupi...@googlemail.com] > Sent: Tuesday, April 05, 2011 10:30 PM > To: solr-user@lucene.apache.org > Subject: Re: FW: Very very large scale Solr Deployment = how to do (Expert > Question)? > > Hello Ephraim, > > thank you so much for the great Document/Scaling-Concept!! > > First I think you really should publish this on the solr wiki. This > approach is nowhere documented there and not really obvious for newbies and > your document is great and explains this very well! > > Please allow me to further questions regarding your document: > 1.) Is it correct, that you mean by "DB" the Origin-Data-Source of the data > that is fed into the Solr "Cloud" for searching? > > 2.) Solr Aggregator: This term did not yeald any google results, but is a > very important aspect of your design (and this was the missing piece for me > when thinking about solr architectures): Is it cocrrec that the > "aggregators" are simply tomcat instances, with the solr webapp deployed? > These Aggregators do not have their own index but only run the solr webapp > and I access them via the ?shard= parameter giving the shards I want to > query? (So in the end they aggreate the data of the shards but do not have > their own data). This is really an important aspect that is not documented > well enough in the solr documentation. > > Thank you very much! > Jens > > > 2011/4/5 Ephraim Ofir > > > of course the attachment didn't get to the list, so here it is if you > > want it... > > > > Ephraim Ofir > > > > > > -Original Message- > > From: Ephraim Ofir > > Sent: Tuesday, April 05, 2011 10:20 AM > > To: 'solr-user@lucene.apache.org' > > Subject: RE: Very very large scale Solr Deployment = how to do (Expert > > Question)? > > > > I'm not sure about the scale you're aiming for, but you probably want > > to do both sharding and replication. There's no central server which > > would be the bottleneck. The guidelines should probably be something > like: > > 1. Split your index to enough shards so it can keep up with the update > > rate. > > 2. Have enough replicates of each shard master to keep up with the > > rate of queries. > > 3. Have enough aggregators in front of the shard replicates so the > > aggregation doesn't become a bottleneck. > > 4. Make sure you have good load balancing across your system. > > > > Attached is a diagram of the setup we have. You might want to look > > into SolrCloud as well. > > > > Ephraim Ofir > > > > > > -Original Message- > > From: Jens Mueller [mailto:supidupi...@googlemail.com] > > Sent: Tuesday, April 05, 2011 4:25 AM > > To: solr-user@lucene.apache.org > > Subject: Very very large scale Solr Deployment = how to do (Expert > > Question)? > > > > Hello Experts, > > > > > > > > I am a Solr newbie but read quite a lot of docs. I still do not > > understand what would be the best way to setup very large scale > > deployments: > > > > > > > > Goal (threoretical): > > > > A.) Index-Size: 1 Petabyte (1 Document is about 5 KB in Size) > > > > B) Queries: 10 Queries/ per Second > > > > C) Updates: 10 Updates / per Second > > > > > > > > > > Solr offers: > > > > 1.)Replication => Scales Well for B) BUT A) and C) are not > > satisfied > > > > > > 2.)Sharding => Scales well for A) BUT B) and C) are not satisfied > > (=> As > > I understand the Sharding approach all goes through a central server, > > that dispatches the updates and assembles the quries retrieved from > > the different shards. But this central server has also some capacity > > limits...) > > > > > > > > > > What is the right approach to handle such large deployments? I would > > be thankfull for just a rough sketch of the concepts so I can > > experiment/search further... > > > > > > Maybe I am missing something very trivial as I think some of the "Solr > > Users/Use Cases" on the homepage are that kind of large deployments. > > How are they implemented? > > > > > > > > Thanky very much!!! > > > > Jens > > > **Legal Disclaimer*** > "This communication may contain confidential and privileged > material for the sole use of the intended recipient. Any > unauthorized review, use or distribution by others is strictly > prohibited. If you have received the message in error, please > advise the sender by reply email and delete the message. Thank > you." > * > -- Thanks & Regards, Isan Fulia.
Migrating from solr 1.4.1 to 3.1.0
Hi all, Solr 3.1.0 uses different javabin format from 1.4.1 So if I use Solrj 1.4.1 jar , then i get javabin error while saving to 3.1.0 and if I use Solrj 3.1.0 jar , then I get javabin error while reading the document from solr 1.4.1. How to go for reindexing in this situation. -- Thanks & Regards, Isan Fulia.
Using lowercase as field type
Hi , My schema consists of a field of type lowercase(for applying the lowercase filter factory) and is the unique key . But its no longer behaving as unique key. Multiple documents with same value for the unique key are getting indexed. Does anyone know why this is happening or is it that the field of type lowercase cannot be unique key. -- Thanks & Regards, Isan Fulia.
Re: Using lowercase as field type
I want multiple documents with same unique key to overwrite each other but they are not overwriting because of lowercase field type as unique key On 4 May 2011 11:45, Markus Jelsma wrote: > So those multiple documents overwrite eachother? In that case, your data is > not suited for a lowercased docID. I'd recommend not doing any analysis on > the > docID to prevent such headaches. > > > Hi , > > > > My schema consists of a field of type lowercase(for applying the > lowercase > > filter factory) and is the unique key . But its no longer behaving as > > unique key. Multiple documents with same value for the unique key are > > getting indexed. > > Does anyone know why this is happening or is it that the field of type > > lowercase cannot be unique key. > -- Thanks & Regards, Isan Fulia.
