Solr Stats or Analytic Query Help

[Gopal Patwa]

I am trying write query to get below stats for orders data

Find Total Order Count, sum(Quantity) and sum(Cost) for specified date
range in a gap of 1 day. example if date range is for 10 days then get
these result for every day for 10 days.

Solr Version : 5.2.1

Example Order Solr Doc



2014-09-30T18:56:17Z

2

233.00



Query example using range facet to get count but not sure how to get
sum(quantity) and sum(cost) in same query. is there other way to get these
data from single or multiple query. single query would be better

solr/orders/select?q=*%3A*&wt=xml&indent=true&facet=true&facet.range=ORDER_DATE&f.ORDER_DATE.facet.range.start=NOW/DAY-1DAYS&f.ORDER_DATE.facet.range.end=NOW/DAY%2B10DAYS&f.ORDER_DATE.facet.range.gap=%2B1DAY


Response:















1289

295

0

0

0

0

0

0

0

0

0



+1DAY

2016-03-09T00:00:00Z

2016-03-20T00:00:00Z

Re: Solr Join between two indexes taking too long.

[Gopal Patwa]

may be this patch could help if ported to 5.x

https://issues.apache.org/jira/browse/SOLR-4787
*BitSetJoinQParserPlugin aka bjoin -> "*can provide sub-second response
times on result sets of tens of millions of records from the fromIndex and
hundreds of millions of records from the main query"

But this patch is not ported to 5.x, not sure why this patch was not added
as contrib



On Wed, Sep 9, 2015 at 2:12 PM, Mikhail Khludnev  wrote:

> On Wed, Sep 9, 2015 at 5:24 PM, Russell Taylor <
> russell.tay...@interactivedata.com> wrote:
>
> > er.
> >
> > Can you see a reason why where indexB has only 6 id's in its list it
> still
> > takes 46 seconds?
> >
> > Ids=6
> > {
> >
> >   },
> >   "debug": {
> > "join": {
> >   "{!join from=sedolKey to=sedolKey
> fromIndex=indexB}universe:55AL86":
> > {
> > "time": 46848,
> > "fromSetSize": 6,
> >
>
> In this particular case {!join .. score=none} should be faster. It's
> available since Solr 5.3.0. You don't need to reindex. Just launch new
> version of Solr with old index, and try this particular edge case with
> small "from". Let's discuss next steps afterwards.
>
>
> > "toSetSize": 0,
> > "fromTermCount": 11837043,
> > "fromTermTotalDf": 11837043,
> > "fromTermDirectCount": 11837043,
> > "fromTermHits": 6,
> > "fromTermHitsTotalDf": 6,
> > "toTermHits": 0,
> > "toTermHitsTotalDf": 0,
> > "toTermDirectCount": 0,
> > "smallSetsDeferred": 0,
> > "toSetDocsAdded": 0
> >   }
> > },
> > "rawquerystring": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:55AL86",
> > "querystring": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:55AL86",
> > "parsedquery": "JoinQuery({!join from=longValue to=longValue
> > fromIndex=indexB}universe:55AL86)",
> > "parsedquery_toString": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:55AL86",
> > "explain": {},
> > "QParser": "",
> > "timing": {
> >   "time": 46849,
> >   "prepare": {
> > "time": 0,
> > "query": {
> >   "time": 0
> > },
> > "facet": {
> >   "time": 0
> > },
> > "mlt": {
> >   "time": 0
> > },
> > "highlight": {
> >   "time": 0
> > },
> > "stats": {
> >   "time": 0
> > },
> > "expand": {
> >   "time": 0
> > },
> > "debug": {
> >   "time": 0
> > }
> >   },
> >   "process": {
> > "time": 46848,
> > "query": {
> >   "time": 46848
> > },
> > "facet": {
> >   "time": 0
> > },
> > "mlt": {
> >   "time": 0
> > },
> > "highlight": {
> >   "time": 0
> > },
> > "stats": {
> >   "time": 0
> > },
> > "expand": {
> >   "time": 0
> > },
> > "debug": {
> >   "time": 0
> > }
> >   }
> > }
> >   }
> > }
> >
> > ###
> > Ids=298
> > {
> >   "responseHeader": {
> > "status": 0,
> > "QTime": 51570,
> > "params": {
> >   "debugQuery": "true",
> >   "indent": "true",
> >   "q": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:16XO52",
> >   "_": "1441810442921",
> >   "wt": "json"
> > }
> >   },
> >   "response": {
> > "numFound": 0,
> > "start": 0,
> > "docs": []
> >   },
> >   "debug": {
> > "join": {
> >   "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:16XO52": {
> > "time": 51570,
> > "fromSetSize": 298,
> > "toSetSize": 0,
> > "fromTermCount": 11837043,
> > "fromTermTotalDf": 11837043,
> > "fromTermDirectCount": 11837043,
> > "fromTermHits": 298,
> > "fromTermHitsTotalDf": 298,
> > "toTermHits": 0,
> > "toTermHitsTotalDf": 0,
> > "toTermDirectCount": 0,
> > "smallSetsDeferred": 0,
> > "toSetDocsAdded": 0
> >   }
> > },
> > "rawquerystring": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:16XO52",
> > "querystring": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:16XO52",
> > "parsedquery": "JoinQuery({!join from=longValue to=longValue
> > fromIndex=indexB}universe:16XO52)",
> > "parsedquery_toString": "{!join from=longValue to=longValue
> > fromIndex=indexB}universe:16XO52",
> > "explain": {},
> > "QParser": "",
> > "timing": {
> >   "time": 51570,
> >   "prepare": {
> > "time": 0,
> > "query": {
> >   "time": 0
> > },
> > "facet": {
> >   "time": 0
> > },
> > "mlt": {
> >   "time": 0
> > },
> > "highlight": {
> >   "time": 0
> > },
> > "stats": {
> >   

Re: missing in json facet does not work for stream?

[Gopal Patwa]

Docs are available in Ref guide for Json facet and Json request api

https://cwiki.apache.org/confluence/display/solr/Faceted+Search
https://cwiki.apache.org/confluence/display/solr/JSON+Request+API


On Sun, Oct 25, 2015 at 7:08 PM, hao jin  wrote:

> Thanks, Yonik.
>
> When the shards parameter is specified in a json facet query with the
> stream method, it is still ordered by the count by default.
> From our perf. test with totally 100,000,000 docs, the stream method is
> the best and the enum method does not work for the field faceting.
> I saw JSON Facet API is in your blog. When will be it officially published
> and documented? Any schedule about it?
>
> Thanks
>
>
> 在 2015/10/23 22:38, Yonik Seeley 写道:
>
>> On Fri, Oct 23, 2015 at 10:24 AM, Shalin Shekhar Mangar
>>  wrote:
>>
>>> Now I am curious, what does it do!
>>>
>> It's basically like facet.method=enum, but it truly streams
>> (calculates each facet bucket on-the-fly and writes it to the
>> response).
>> Since it is streaming, it only supports sorting by term index order.
>>
>> Although if there is need/demand, we could also do a lightweight
>> ordering over the buckets first (ordering by count or other facet
>> function) and then still stream, creating the buckets and any
>> sub-facets on the fly.
>>
>> -Yonik
>>
>>
>>
>> On Fri, Oct 23, 2015 at 7:40 PM, Yonik Seeley  wrote:
>>>
 On Fri, Oct 23, 2015 at 5:55 AM, hao jin  wrote:

> Hi
> I found when the method of json facet is set to stream, the "missing"
> is not
> added to the result.
> Is it designed or a known issue?
>
 You found an undocumented feature (method=stream) ;-)
 That facet method doesn't have adequate testing yet, so I haven't
 publicized / documented it.
 Support for things like "missing" may be some of the stuff still TBD.

 -Yonik

>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>

Re: Search query speed

[Gopal Patwa]

You could start adding debug=true in your request, it will show complete
query execution time

On Tue, Nov 10, 2015 at 9:39 AM John Stric  wrote:

> The speed of particular query has gone from about 42 msec to 66 msec
> without any changes.
>
> How do I go about troubleshooting what may have happened? And how do I
> improve that speed?
>
> Thanks
>

Re: Slow cross-core joins

[Gopal Patwa]

You could give a try for this join contrib patch

https://issues.apache.org/jira/browse/SOLR-4787



On Mon, Mar 2, 2015 at 12:04 PM, Matt B  wrote:

> I've recently inherited a Solr instance that is required to perform
> numerous joins between two cores, usually as filter queries, similar to the
> one below:
>
> q=firstName=Matt&fq=-({!to=emailAddress toIndex=accounts type=join
> fromIndex=lists from=listValue}list_id:38f2-351b-11e4-9579-001e67654bce
> OR {!to=emailDomain toIndex=accounts type=join fromIndex=lists
> from=listValue}list_id:38f2-351b-11e4-9579-001e67654bce OR
> {!to=emailDomainReversed toIndex=accounts type=join fromIndex=lists
> from=listValue}list_id:38f2-351b-11e4-9579-001e67654bce)
>
> The accounts core is about 35GB with ~40,000,000 documents and the lists
> core is about 9 GB with 90,,000 documents.  There may be anywhere from
> one to one million documents in the lists core matching any particular
> list_id.  The idea is to filter a search query on the accounts core to
> include or exclude any documents with an email address, email domain, or
> reverse email domain that is found within the lists core for a particular
> list id.  The lists core is frequently updated on a daily basis with both
> additions and deletions.
>
> Not surprisingly, such queries are very slow, usually taking minutes to
> return any results.
>
> Are there any possible strategies to significantly increase the
> performance of such queries?  The JVM max heap size is set to 16 GB and the
> server has 64 GB RAM.

Re: A Synonym Searching for Phrase?

[Gopal Patwa]

you might have this filter in query analyzer, which can spit token "s-pass"

https://cwiki.apache.org/confluence/display/solr/Filter+Descriptions#FilterDescriptions-WordDelimiterFilter


On Sun, May 24, 2015 at 5:36 AM, Ryan Yacyshyn 
wrote:

> Thanks all for your suggestions.
>
> What we've done in the end - and I'm not so sure why it works - is adding
> "s-pass, spass, s pass" to the synonyms.txt file rather than s-pass, spass
> => s pass.
>
>
>
>
>
>
>
> On Fri, 15 May 2015 at 16:02 Rajani Maski  wrote:
>
> > Hi Ryan,
> >
> > I am not really sure whether this[1] solution mentioned in the link below
> > can work for your case considering its cons. However, I recommend having
> a
> > quick look at it.
> >
> > @Chris, Would eagerly wait for your contribution.
> >
> >
> > [1] https://support.lucidworks.com/hc/en-us/articles/205359448
> >
> >
> >
> > On Thu, May 14, 2015 at 11:30 PM, Chris Morley 
> > wrote:
> >
> > > I have implemented that but it's not open sourced yet.  It will be
> soon.
> > >
> > >  -Chris.
> > >
> > >
> > >
> > >
> > > 
> > >  From: "Ryan Yacyshyn" 
> > > Sent: Thursday, May 14, 2015 12:07 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: A Synonym Searching for Phrase?
> > > Hi All,
> > >
> > > I'm running into an issue where I have some tokens that really mean the
> > > same thing as two. For example, there are a couple ways users might
> want
> > > to
> > > search for certain type of visa called the "s pass", but they might
> query
> > > for spass or s-pass.
> > >
> > > I thought I could add a line in my synonym file to solve this, such as:
> > >
> > > s-pass, spass => s pass
> > >
> > > This doesn't seem to work. I found an Auto Phrase TokenFilter (
> > > https://github.com/LucidWorks/auto-phrase-tokenfilter) that looks like
> > it
> > > might help, but it sounds like it needs to use a specific query parser
> as
> > > well (we're using edismax).
> > >
> > > Has anyone came across this specific problem before? Would really
> > > appreciate your suggestions / help.
> > >
> > > We're using Solr 4.8.x (and lucidWorks 2.9).
> > >
> > > Thanks!
> > > Ryan
> > >
> > >
> > >
> >
>

Re: Support for Numeric DocValues Updates in Solr?

[Gopal Patwa]

I tried looking at Solr code but it seems it require more deeper
understanding of Solr code to add this support, may be other solr expert
can provide some pointers.

Did any one else tried Updating Numeric DocValues in Solr but not committed
the patch yet?

It would be really nice if this feature added, since our usecase required
to update few numeric fields at much higher rate than normal document
update.




On Wed, Nov 20, 2013 at 7:48 AM, Gopal Patwa  wrote:

> +1 to add this support in Solr
>
>
> On Wed, Nov 20, 2013 at 7:16 AM, Otis Gospodnetic <
> otis.gospodne...@gmail.com> wrote:
>
>> Hi,
>>
>> "Numeric DocValues Updates" functionality that came via
>> https://issues.apache.org/jira/browse/LUCENE-5189 sounds very
>> valuable, while we wait for full/arbitrary field updates
>> (https://issues.apache.org/jira/browse/LUCENE-4258).
>>
>> Would it make sense to add support for Numeric DocValues Updates to Solr?
>>
>> Thanks,
>> Otis
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>
>

Re: Zookeeper exceptions - SEVERE

[Gopal Patwa]

Shalin, "correlated with how frequently you call commit" is it soft commit
or hard commit? , I guess it should be later one.

just curious what data it update to zookeeper during commit


On Tue, Mar 18, 2014 at 9:12 PM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> SolrCloud will update Zookeeper on state changes (node goes to
> recovery, comes back up etc) or for leader election and during
> collection API commands. It doesn't correlate directly with indexing
> but is correlated with how frequently you call commit.
>
> On Wed, Mar 19, 2014 at 5:46 AM, Shawn Heisey  wrote:
> > On 3/18/2014 5:46 PM, Chris W wrote:
> >>
> >> I am running a 3 node zookeeper 3.4.5  Quorum. I am running into issues
> >> with Zookeeper transaction logs
> >>
> >>   [myid:2] - ERROR [main:QuorumPeer@453] - Unable to load database on
> disk
> >> java.io.IOException: Unreasonable length = 1048587
> >> at
> >>
> org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:100)
> >> at
> >> org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
> >> at
> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:602)
> >> at
> >>
> >>
> org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:157)
> >> at
> >> org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
> >>
> >> To unblock temporarily, i deleted the most recent txn log. How do i tell
> >> zookeeper to not grow the transaction log beyond x MegaBytes?
> >>
> >> How often does the transaction log get updated? Does zk transactions log
> >> grow everytime we index data into a new collection?
> >
> >
> > Zookeeper is a separate project at Apache.  ZK file management is
> discussed
> > in the ZK documentation.
> >
> >
> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> >
> > There is a bug filed on Zookeeper for the issue you are seeing, with a
> > fairly simple workaround.  It is fixed in the 3.4.6 version, which was
> > released last week.  I will see whether we can get ZK upgraded to 3.4.6
> in
> > the Solr 4.8.0 release.  I don't think we want to risk doing that
> upgrade in
> > 4.7.1, but I could be wrong.
> >
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1513
> > http://zookeeper.apache.org/releases.html
> >
> > I am actually not sure how often SolrCloud updates Zookeeper.  It happens
> > whenever the collections API is called for sure, and it may happen
> anytime
> > you index data as well.
> >
> > Thanks,
> > Shawn
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Identify specific document insert error inside a solrj batch request

[Gopal Patwa]

for reference implementation to Sanitize Unknown SolrFields, you can see
below link

https://github.com/cloudera/cdk/blob/master/cdk-morphlines/cdk-morphlines-solr-core/src/main/java/com/cloudera/cdk/morphline/solr/SanitizeUnknownSolrFieldsBuilder.java



On Sat, Aug 2, 2014 at 8:24 PM, Umesh Prasad  wrote:

> Solr  schema over REST https://wiki.apache.org/solr/SchemaRESTAPI
>
> https://cwiki.apache.org/confluence/display/solr/Schema+API
>
> You can use that for getting required fields and validate at client side ..
>
>
>
>
>
> On 31 July 2014 14:32, Liram Vardi  wrote:
>
> > Hi Jack,
> > Thank you for your reply.
> > This is the Solr stack trace. As you can see, the missing field is
> > "hourOfDay".
> >
> > Thanks,
> > Liram
> >
> > 2014-07-30 14:27:54,934 ERROR [qtp-608368492-19] (SolrException.java:108)
> > - org.apache.solr.common.SolrException:
> > [doc=53b16126--0002-2b03-17ac4d4a07b6] missing required field:
> hourOfDay
> > at
> >
> org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:189)
> > at
> >
> org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:73)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:210)
> > at
> >
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:556)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:692)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:435)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> > at
> >
> org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:94)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> > at
> >
> com.checkpoint.solr_plugins.MulticoreUpdateRequestProcessor.processAdd(MulticoreUpdateRequestProcessorFactory.java:152)
> > at
> >
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
> > at
> > org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
> > at
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > at
> >
> com.checkpoint.solr_plugins.MulticoreUpdateRequestProcessor.processAdd(MulticoreUpdateRequestProcessorFactory.java:248)
> > at
> >
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:86)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:143)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:123)
> > at
> > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:220)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:108)
> > at
> > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:185)
> > at
> > org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:111)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:150)
> > at
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:96)
> > at
> > org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> > at
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158

Re: Anybody uses Solr JMX?

[Gopal Patwa]

Another option to get JMX data from Solr to Graphite
 or Ganglia
 using
jmxtrans

https://github.com/jmxtrans/jmxtrans/wiki



On Wed, Aug 6, 2014 at 3:09 AM, rulinma  wrote:

> good job .
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Anybody-uses-Solr-JMX-tp4134598p4151408.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Autosuggest with spelling correction

[Gopal Patwa]

This jira has some documentation, may be this will help you..

https://issues.apache.org/jira/browse/SOLR-5683



On Wed, Aug 13, 2014 at 1:28 AM, Harun Reşit Zafer <
harun.za...@tubitak.gov.tr> wrote:

> Hi everyone,
>
> Currently I'm using AnalyzingInfixLookupFactory with a suggestions file
> containing up to 3 word phrases. However this component can't keep
> suggesting in case of spelling errors. I heard about FuzzySuggester and
> found some sample configurations here  asf/lucene/dev/trunk/solr/core/src/test-files/solr/
> collection1/conf/solrconfig-phrasesuggest.xml>. But I  couldn't make any
> of them work. I got the same error: ...solr-4.9.0\example\solr\
> collection1\data\fuzzy_suggest_analyzing\fwfsta.bin (The system cannot
> find the file specified).
>
> In short, is there a Suggester component that supports both infix lookup
> and fuzzy suggest, and where can I find a proper sample configuration.
>
> Thanks
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W  http://www.hrzafer.com
>
>

Re: Why do people want to deploy to Tomcat?

[Gopal Patwa]

My case is also similar to "Sujit Pal" but we have jboss6.


On Tue, Nov 12, 2013 at 9:47 AM, Sujit Pal  wrote:

> In our case, it is because all our other applications are deployed on
> Tomcat and ops is familiar with the deployment process. We also had
> customizations that needed to go in, so we inserted our custom JAR into the
> solr.war's WEB-INF/lib directory, so to ops the process of deploying Solr
> was (almost, except for schema.xml or solrconfig.xml changes) identical to
> any of the other apps. But I think if Solr becomes a server with clearly
> defined extension points (such as dropping your custom JARs into lib/ and
> custom configuration in conf/solrconfig.xml or similar like it already is)
> then it will be treated as something other than a webapp and the
> expectation that it runs on Tomcat will not apply.
>
> Just my $0.02...
>
> Sujit
>
>
>
> On Tue, Nov 12, 2013 at 9:13 AM, Siegfried Goeschl 
> wrote:
>
> > Hi ALex,
> >
> > in my case
> >
> > * ignorance that Tomcat is not fully supported
> > * Tomcat configuration and operations know-how inhouse
> > * could migrate to Jetty but need approved change request to do so
> >
> > Cheers,
> >
> > Siegfried Goeschl
> >
> > On 12.11.13 04:54, Alexandre Rafalovitch wrote:
> >
> >> Hello,
> >>
> >> I keep seeing here and on Stack Overflow people trying to deploy Solr to
> >> Tomcat. We don't usually ask why, just help when where we can.
> >>
> >> But the question happens often enough that I am curious. What is the
> >> actual
> >> business case. Is that because Tomcat is well known? Is it because other
> >> apps are running under Tomcat and it is ops' requirement? Is it because
> >> Tomcat gives something - to Solr - that Jetty does not?
> >>
> >> It might be useful to know. Especially, since Solr team is considering
> >> making the server part into a black box component. What use cases will
> >> that
> >> break?
> >>
> >> So, if somebody runs Solr under Tomcat (or needed to and gave up), let's
> >> use this thread to collect this knowledge.
> >>
> >> Regards,
> >> Alex.
> >> Personal website: http://www.outerthoughts.com/
> >> LinkedIn: http://www.linkedin.com/in/alexandrerafalovitch
> >> - Time is the quality of nature that keeps events from happening all at
> >> once. Lately, it doesn't seem to be working.  (Anonymous  - via GTD
> book)
> >>
> >>
>

Re: Adding a server to an existing SOLR cloud cluster

[Gopal Patwa]

Did you try adding core.properties file in your core folder with below
content and change the value for name and collection property

ex: core1/core.properties content:

numShards=1
name=core1
shard=shard1
collection=collection1


On Mon, Nov 11, 2013 at 8:14 AM, michael.boom  wrote:

> From my understanding, if your already existing cluster satisfies your
> collection (already live nodes >= nr shards * replication factor) there
> wouldn't be any need for creating additional replicas on the new server,
> unless you directly ask for them, after startup.
> I usually just add the machine to the cluster and the manually create the
> replicas I need.
>
>
>
> -
> Thanks,
> Michael
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Adding-a-server-to-an-existing-SOLR-cloud-cluster-tp4100275p4100313.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Support for Numeric DocValues Updates in Solr?

[Gopal Patwa]

+1 to add this support in Solr


On Wed, Nov 20, 2013 at 7:16 AM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:

> Hi,
>
> "Numeric DocValues Updates" functionality that came via
> https://issues.apache.org/jira/browse/LUCENE-5189 sounds very
> valuable, while we wait for full/arbitrary field updates
> (https://issues.apache.org/jira/browse/LUCENE-4258).
>
> Would it make sense to add support for Numeric DocValues Updates to Solr?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>

Re: csv does not return custom fields (distance)

[Gopal Patwa]

if you are using Solr 4.0 there was some issue related to field alias which
was fixed in Solr 4.3

https://issues.apache.org/jira/browse/SOLR-4671

you should try to reproduce this issue using latest Solr version 4.5.1



On Fri, Nov 22, 2013 at 11:28 AM, GaneshSe  wrote:

> Any help on this is greatly appreciated.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/csv-does-not-return-custom-fields-distance-tp4102313p4102656.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Solr use with Cloudera HDFS failed creating directory

[Gopal Patwa]

I am trying to setup Solr with HDFS following this wiki

https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS

My Setup:

***

VMWare: Cloudera Quick Start VM 4.4.0-1 default setup (only hdfs1,
hive1,hue1,mapreduce1 and zookeeper1 is running)

http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=F6mO278Rvo

SolrCloud:

Mac OS  10.7.5 ->  -Running Solr 4.6 with maven jetty plugin in eclipse
outside from HDFS (Cloudera VM) , so it is accessing HDFS as remote service

External zookeeper 3 nodes

Java 1.6, Jett Container 8.1

Collection with 1 shard and 1 replica



But I am getting below error "Problem creating directory:" I have created
this directory manually in hdfs. Do I need to setup some special user
permission in Solr?. or do I need to always run solr instance in HDFS (Data
Node)?

[cloudera@localhost ~]$ sudo -u hdfs hadoop fs -mkdir /solr-hdfs

Directory permisson in HDFS:

solr-hdfs rwxr-xr-x hdfs supergroup

Startup Log:

2014-01-01 20:21:57.433:INFO:oejs.Server:jetty-8.1.7.v20120910

2014-01-01 20:21:59.710:INFO:omjp.MavenWebInfConfiguration:Adding overlay:
file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/target/tmp/solr-4_6_0_war/

2014-01-01 20:22:02.249:INFO:oejpw.PlusConfiguration:No Transaction manager
found - if your webapp requires one, please configure one.

2014-01-01 20:22:03.368:INFO:oejsh.ContextHandler:started
o.m.j.p.JettyWebAppContext{/solr,[file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/,
file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/target/tmp/solr-4_6_0_war/]},file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/

2014-01-01 20:22:03.369:INFO:oejsh.ContextHandler:started
o.m.j.p.JettyWebAppContext{/solr,[file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/,
file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/target/tmp/solr-4_6_0_war/]},file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/

2014-01-01 20:22:03.369:INFO:oejsh.ContextHandler:started
o.m.j.p.JettyWebAppContext{/solr,[file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/,
file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/target/tmp/solr-4_6_0_war/]},file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/

0[main] INFO  org.apache.solr.servlet.SolrDispatchFilter  –
SolrDispatchFilter.init()

29   [main] INFO  org.apache.solr.core.SolrResourceLoader  – No /solr/home
in JNDI

30   [main] INFO  org.apache.solr.core.SolrResourceLoader  – using system
property solr.solr.home: /Users/gpatwa/opensource/solr-hdfs-home

32   [main] INFO  org.apache.solr.core.SolrResourceLoader  – new
SolrResourceLoader for directory: '/Users/gpatwa/opensource/solr-hdfs-home/'

220  [main] INFO  org.apache.solr.core.ConfigSolr  – Loading container
configuration from /Users/gpatwa/opensource/solr-hdfs-home/solr.xml

348  [main] INFO  org.apache.solr.core.ConfigSolrXml  – Config-defined core
root directory:

358  [main] INFO  org.apache.solr.core.CoreContainer  – New CoreContainer
445620464

359  [main] INFO  org.apache.solr.core.CoreContainer  – Loading cores into
CoreContainer [instanceDir=/Users/gpatwa/opensource/solr-hdfs-home/]

374  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
socketTimeout to: 12

375  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
urlScheme to: http://

375  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
connTimeout to: 15000

375  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
maxConnectionsPerHost to: 20

375  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
corePoolSize to: 0

376  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
maximumPoolSize to: 2147483647

376  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
maxThreadIdleTime to: 5

376  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
sizeOfQueue to: -1

378  [main] INFO
org.apache.solr.handler.component.HttpShardHandlerFactory  – Setting
fairnessPolicy to: false

645  [main] INFO  org.apache.solr.logging.LogWatcher  – SLF4J impl is
org.slf4j.impl.Log4jLoggerFactory

646  [main] INFO  org.apache.solr.logging.LogWatcher  – Registering Log
Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]

647  [main] INFO  org.apache.solr.core.ZkContainer  – Zookeeper
client=localhost:2181/search/catalog

653  [main] INFO  org.apache.solr.cloud.ZkController  – zkHost includes
chroot

762  [main] INFO  org.apache.solr.common.cloud.ConnectionManager  – Waiting
for client to connect to ZooKeeper

5781 [main-EventThread] IN

Re: Solr use with Cloudera HDFS failed creating directory

[Gopal Patwa]

I gave another try with Solr 4.4 which ship with Cloudera VM as Cloudera
Search but same result. It seems there is compitability issue with protobuf
library dependecy in haddop java client and HDFS server it self.

Solr 4.4 depend on protobuf-java-2.4.0a.jar
Solr 4.6  depend on protobuf-java-2.5.0.jar

Finally I tried with Horton work HDFS distribution
http://hortonworks.com/products/hortonworks-sandbox/#install

Wow!!! it worked without any issue.

Log Snippet:

4933 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.JmxMonitoredMap  – No JMX servers found, not exposing
Solr information with JMX.

4942 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.HdfsDirectoryFactory  – creating directory factory for
path hdfs://10.249.132.15:8020/solr-hdfs

4953 [coreLoadExecutor-4-thread-1] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics  – Initializing JVM Metrics with
processName=blockcache, sessionId=1388960072262

2014-01-05 14:14:32.403 java[46758:10b03] Unable to load realm info from
SCDynamicStore

5115 [coreLoadExecutor-4-thread-1] WARN
org.apache.hadoop.util.NativeCodeLoader  – Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable

5962 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.CachingDirectoryFactory  – return new directory for
hdfs://10.249.132.15:8020/solr-hdfs

5999 [coreLoadExecutor-4-thread-1] INFO  org.apache.solr.core.SolrCore  –
New index directory detected: old=null new=hdfs://
10.249.132.15:8020/solr-hdfs/index/

6075 [coreLoadExecutor-4-thread-1] WARN  org.apache.solr.core.SolrCore  –
[event_shard1_replica1] Solr index directory 'hdfs:/
10.249.132.15:8020/solr-hdfs/index' doesn't exist. Creating new index...

6085 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.HdfsDirectoryFactory  – creating directory factory for
path hdfs://10.249.132.15:8020/solr-hdfs/index

6086 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.HdfsDirectoryFactory  – Number of slabs of block cache
[1] with direct memory allocation set to [true]

6086 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.HdfsDirectoryFactory  – Block cache target memory
usage, slab size of [134217728] will allocate [1] slabs and use
~[134217728] bytes

6087 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.store.blockcache.BufferStore  – Initializing the 1024
buffers with [8192] buffers.

6114 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.store.blockcache.BufferStore  – Initializing the 8192
buffers with [8192] buffers.

6408 [coreLoadExecutor-4-thread-1] INFO
org.apache.solr.core.CachingDirectoryFactory  – return new directory for
hdfs://10.249.132.15:8020/solr-hdfs/index

7907 [coreLoadExecutor-4-thread-1] INFO  org.apache.solr.core.SolrCore  –
SolrDeletionPolicy.onCommit: commits: num=1

commit{dir=NRTCachingDirectory(org.apache.solr.store.hdfs.HdfsDirectory@6cab6dcblockFactory=org.apache.solr.store.hdfs.HdfsLockFactory@4a6d0362;
maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}




On Thu, Jan 2, 2014 at 8:20 AM, Gopal Patwa  wrote:

> I am trying to setup Solr with HDFS following this wiki
>
> https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS
>
> My Setup:
>
> ***
>
> VMWare: Cloudera Quick Start VM 4.4.0-1 default setup (only hdfs1,
> hive1,hue1,mapreduce1 and zookeeper1 is running)
>
>
> http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html?productID=F6mO278Rvo
>
> SolrCloud:
>
> Mac OS  10.7.5 ->  -Running Solr 4.6 with maven jetty plugin in eclipse
> outside from HDFS (Cloudera VM) , so it is accessing HDFS as remote service
>
> External zookeeper 3 nodes
>
> Java 1.6, Jett Container 8.1
>
> Collection with 1 shard and 1 replica
>
> 
>
> But I am getting below error "Problem creating directory:" I have created
> this directory manually in hdfs. Do I need to setup some special user
> permission in Solr?. or do I need to always run solr instance in HDFS (Data
> Node)?
>
> [cloudera@localhost ~]$ sudo -u hdfs hadoop fs -mkdir /solr-hdfs
>
> Directory permisson in HDFS:
>
> solr-hdfs rwxr-xr-x hdfs supergroup
>
> Startup Log:
>
> 2014-01-01 20:21:57.433:INFO:oejs.Server:jetty-8.1.7.v20120910
>
> 2014-01-01 20:21:59.710:INFO:omjp.MavenWebInfConfiguration:Adding overlay:
> file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/target/tmp/solr-4_6_0_war/
>
> 2014-01-01 20:22:02.249:INFO:oejpw.PlusConfiguration:No Transaction
> manager found - if your webapp requires one, please configure one.
>
> 2014-01-01 20:22:03.368:INFO:oejsh.ContextHandler:started
> o.m.j.p.JettyWebAppContext{/solr,[file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdfs/src/main/webapp/,
> file:/Users/gpatwa/workspaces/workspace_pb_search_platform_fr/solr-hdf

Replica node down but zookeeper clusterstate not updated

[Gopal Patwa]

Solr = 4.6.1, attached solrcloud admin console view
Zookeeper 3.4.5  = 3 node ensemble

In my test setup, I have 3 Node SolrCloud setup with 2 shard. Today we had
power failure and all node went down.

I started 3 node zookeeper ensemble first then followed with 3 node
solrcloud, and one of replica ip address was change due to dynamic ip
allocation but zookeeper
clusterstate is not updated with new ip address and it was still holding
old ip address for that bad node.

Do I need to manually update clusterstate in zookeeper? what are my options
if this could happen in production.

Bad node:
old IP:10.249.132.35 (still exist in zookeeper)
new IP: 10.249.133.10

Log from Node1:

11:26:25,242 INFO  [STDOUT] 49170786 [Thread-2-EventThread] INFO
 org.apache.solr.common.cloud.ZkStateReader  â A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
11:26:41,072 INFO  [STDOUT] 49186615 [RecoveryThread] INFO
 org.apache.solr.cloud.ZkController  â publishing
core=genre_shard1_replica1 state=recovering
11:26:41,079 INFO  [STDOUT] 49186622 [RecoveryThread] ERROR
org.apache.solr.cloud.RecoveryStrategy  â Error while trying to recover.
core=genre_shard1_replica1:org.apache.solr.client.solrj.SolrServerException:
Server refused connection at: http://10.249.132.35:8080/solr
11:26:41,079 INFO  [STDOUT] at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:496)
11:26:41,079 INFO  [STDOUT] at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
11:26:41,079 INFO  [STDOUT] at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:221)
11:26:41,079 INFO  [STDOUT] at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:367)
11:26:41,079 INFO  [STDOUT] at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)
11:26:41,079 INFO  [STDOUT] Caused by:
org.apache.http.conn.HttpHostConnectException: Connection to
http://10.249.132.35:8080 refused


11:27:14,036 INFO  [STDOUT] 49219580 [RecoveryThread] ERROR
org.apache.solr.cloud.RecoveryStrategy  â Recovery failed - trying again...
(9) core=geo_shard1_replica1
11:27:14,037 INFO  [STDOUT] 49219581 [RecoveryThread] INFO
 org.apache.solr.cloud.RecoveryStrategy  â Wait 600.0 seconds before trying
to recover again (10)
11:27:14,958 INFO  [STDOUT] 49220498 [Thread-40] INFO
 org.apache.solr.common.cloud.ZkStateReader  â Updating cloud state from
ZooKeeper...



Log from bad node with new ip address:

11:06:29,551 INFO  [STDOUT] 6234 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.cloud.ShardLeaderElectionContext  â Enough replicas found
to continue.
11:06:29,552 INFO  [STDOUT] 6236 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.cloud.ShardLeaderElectionContext  â I may be the new
leader - try and sync
11:06:29,554 INFO  [STDOUT] 6237 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.cloud.SyncStrategy  â Sync replicas to
http://10.249.132.35:8080/solr/venue_shard2_replica2/
11:06:29,555 INFO  [STDOUT] 6239 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.update.PeerSync  â PeerSync: core=venue_shard2_replica2
url=http://10.249.132.35:8080/solr START replicas=[
http://10.249.132.56:8080/solr/venue_shard2_replica1/] nUpdates=100
11:06:29,556 INFO  [STDOUT] 6240 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.update.PeerSync  â PeerSync: core=venue_shard2_replica2
url=http://10.249.132.35:8080/solr DONE.  We have no versions.  sync failed.
11:06:29,556 INFO  [STDOUT] 6241 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.cloud.SyncStrategy  â Leader's attempt to sync with shard
failed, moving to the next candidate
11:06:29,558 INFO  [STDOUT] 6241 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.cloud.ShardLeaderElectionContext  â We failed sync, but we
have no versions - we can't sync in that case - we were active before, so
become leader anyway
11:06:29,559 INFO  [STDOUT] 6243 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.cloud.ShardLeaderElectionContext  â I am the new leader:
http://10.249.132.35:8080/solr/venue_shard2_replica2/ shard2
11:06:29,561 INFO  [STDOUT] 6245 [coreLoadExecutor-4-thread-10] INFO
 org.apache.solr.common.cloud.SolrZkClient  â makePath:
/collections/venue/leaders/shard2
11:06:29,577 INFO  [STDOUT] 6261 [Thread-2-EventThread] INFO
 org.apache.solr.update.PeerSync  â PeerSync: core=event_shard2_replica2
url=http://10.249.132.35:8080/solr  Received 18 versions from
10.249.132.56:8080/solr/event_shard2_replica1/
11:06:29,578 INFO  [STDOUT] 6263 [Thread-2-EventThread] INFO
 org.apache.solr.update.PeerSync  â PeerSync: core=event_shard2_replica2
url=http://10.249.132.35:8080/solr Requesting updates from
10.249.132.56:8080/solr/event_shard2_replica1/n=10versions=[1457764666067386368,
1456709993140060160, 1456709989863260160,
1456709986075803648, 1456709971758546944, 1456709179685208064,
1456709137524064256, 1456709130

Re: Autocommit, opensearchers and ingestion

[Gopal Patwa]

This blog by Eric will help you to understand different commit option and
transaction logs and it does provide some recommendation for ingestion
process.

http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/


On Tue, Feb 25, 2014 at 11:40 AM, Furkan KAMACI wrote:

> Hi;
>
> You should read here:
>
> http://wiki.apache.org/solr/FAQ#What_does_.22exceeded_limit_of_maxWarmingSearchers.3DX.22_mean.3F
>
> On the other hand do you have 4 Zookeeper instances as a quorum?
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-02-25 20:31 GMT+02:00 Joel Cohen :
>
> > Hi all,
> >
> > I'm working with Solr 4.6.1 and I'm trying to tune my ingestion process.
> > The ingestion runs a big DB query and then does some ETL on it and
> inserts
> > via SolrJ.
> >
> > I have a 4 node cluster with 1 shard per node running in Tomcat with
> > -Xmx=4096M. Each node has a separate instance of Zookeeper on it, plus
> the
> > ingestion server has one as well. The Solr servers have 8 cores and 64 Gb
> > of total RAM. The ingestion server is a VM with 8 Gb and 2 cores.
> >
> > My ingestion code uses a few settings to control concurrency and batch
> > size.
> >
> > solr.update.batchSize=500
> > solr.threadCount=4
> >
> > With this setup, I'm getting a lot of errors and the ingestion is taking
> > much longer than it should.
> >
> > Every so often during the ingestion I get these errors on the Solr
> servers:
> >
> > WARN  shard1 - 2014-02-25 11:18:34.341;
> > org.apache.solr.update.UpdateLog$LogReplayer; Starting log replay
> >
> >
> tlog{file=/usr/local/solr_shard1/productCatalog/data/tlog/tlog.0014074
> > refcount=2} active=true starting pos=776774
> > WARN  shard1 - 2014-02-25 11:18:37.275;
> > org.apache.solr.update.UpdateLog$LogReplayer; Log replay finished.
> > recoveryInfo=RecoveryInfo{adds=4065 deletes=0 deleteByQuery=0 errors=0
> > positionOfStart=776774}
> > WARN  shard1 - 2014-02-25 11:18:37.960; org.apache.solr.core.SolrCore;
> > [productCatalog] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
> > WARN  shard1 - 2014-02-25 11:18:37.961; org.apache.solr.core.SolrCore;
> > [productCatalog] Error opening new searcher. exceeded limit of
> > maxWarmingSearchers=2, try again later.
> > WARN  shard1 - 2014-02-25 11:18:37.961; org.apache.solr.core.SolrCore;
> > [productCatalog] Error opening new searcher. exceeded limit of
> > maxWarmingSearchers=2, try again later.
> > ERROR shard1 - 2014-02-25 11:18:37.961;
> > org.apache.solr.common.SolrException;
> org.apache.solr.common.SolrException:
> > Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try
> > again later.
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1575)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1346)
> > at
> >
> >
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:592)
> >
> > I cut threads down to 1 and batchSize down to 100 and the errors go away,
> > but the upload time jumps up by a factor of 25.
> >
> > My solrconfig.xml has:
> >
> >  
> >${solr.autoCommit.maxDocs:1}
> >${solr.autoCommit.maxTime:15000}
> >false
> >  
> >
> >  
> >${solr.autoSoftCommit.maxTime:1000}
> >  
> >
> > I turned autowarmCount down to 0 for all the caches. What else can I tune
> > to allow me to run bigger batch sizes and more threads in my upload
> script?
> >
> > --
> >
> > joel cohen, senior system engineer
> >
> > e joel.co...@bluefly.com p 212.944.8000 x276
> > bluefly, inc. 42 w. 39th st. new york, ny 10018
> > www.bluefly.com  | *fly since
> > 2013...*
> >
>

Re: Know indexing time of a document

[Gopal Patwa]

you could just add a field with default value NOW in schema.xml, for example

  


On Wed, Feb 26, 2014 at 10:44 PM, pratpor  wrote:

> Is it possible to know the indexing time of a document in solr. Like there
> is
> a implicit field for "score" which automatically gets added to a document,
> is there a field that stores value of indexing time?
>
> Thanks
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Know-indexing-time-of-a-document-tp4120051.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Solr is NoSQL database or not?

[Gopal Patwa]

Well said Jack, we are using Solr as NoSQL solution as Jack describe from
Solr version 3.x and still using it in Production with 4.x and on our
Stubhub site most visited page.

https://m.stubhub.com/los-angeles-kings-tickets/los-angeles-kings-los-angeles-staples-center-1-3-2014-4323511/



On Sat, Mar 1, 2014 at 6:14 AM, Jack Krupansky wrote:

> A database is a place you store information for relatively permanent
> reference, called a "system of record". Most commonly data is accessed by a
> primary key. Update of existing data by individual row is a common
> operation.
>
> Solr is a "search server" or "search platform". The focus of a search
> server is to support rapid and relevant rich search, especially keyword
> text search. Data itself usually lives elsewhere, but is loaded into the
> search server whenever it changes, a process known as "indexing". It is not
> uncommon with a search server to "reindex" data, which means to throw all
> the data away and start over, rereading the data from its source(s)
> (system(s) of record). Update of existing data is usually in "batches", not
> individual rows. Data tends to be added rather than updated.
>
> Commonly a search server is used in conjunction with some number of
> databases.
>
> Can one use Solr as a database as well? Sure, its possible, but that's not
> its primary and most popular use at this point.
>
> I mean, what's one of the most commonly used verbs on the Solr email list?
> We're always telling people to "reindex". Can you imagine database
> developers being told that they must delete all their existing data and
> "start over"?
>
> -- Jack Krupansky
>
> -Original Message- From: nutchsolruser
> Sent: Friday, February 28, 2014 11:09 PM
> To: solr-user@lucene.apache.org
> Subject: Solr is NoSQL database or not?
>
>
> You may think this is silly question but let me ask this because i am
> confused ,
> http://www.lucidworks.com/webinar-solr-4-the-nosql-search-server/  this
> says
> Solr is NoSQL but many other links dont have solr in their list as NoSQL
> database.
>
> http://en.wikipedia.org/wiki/NoSQL
> http://en.wikipedia.org/wiki/Document-oriented_database
>
> it's really confusing what is real meaning of NoSQL database?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Solr-is-NoSQL-database-or-not-tp4120554.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

java.lang.Exception: Conflict with StreamingUpdateSolrServer

[Gopal Patwa]

We have non SolrCloud cluster with Solr 4.1 version and often we get this
error during indexing, Did anyone else had similar experience with Indexing
or have seen this error.


2014-03-01 16:52:16,857 [1a44#fb2a/ActiveListingPump] priority=INFO
app_name=listing-search-index thread=pool-15-thread-336
location=StreamingUpdateSolrServer line=162 Status for: null is 409

2014-03-01 16:52:16,858 [1a44#fb2a/ActiveListingPump] priority=ERROR
app_name=listing-search-index thread=pool-15-thread-336
location=StreamingUpdateSolrServer line=304 error
java.lang.Exception: Conflict
Conflict
request: http://localhost:8080/solr/activeInventory/update/javabin
at 
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner.run(StreamingUpdateSolrServer.java:170)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

2014-03-01 16:52:16,858 [1a44#fb2a/ActiveListingPump] priority=INFO
app_name=listing-search-index thread=pool-15-thread-336
location=StreamingUpdateSolrServer line=200 finished:
org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer$Runner@331c403a

Re: java.lang.Exception: Conflict with StreamingUpdateSolrServer

[Gopal Patwa]

Thanks Chirs,  I found in our application code it was related to optimistic
concurrency failure.


On Mon, Mar 3, 2014 at 6:13 PM, Chris Hostetter wrote:

>
> : Subject: java.lang.Exception: Conflict with StreamingUpdateSolrServer
>
> the fact that you are using StreamingUpdateSolrServer isn't really a
> factor here -- what matters is the data you are sending to solr in the
> updates...
>
> : location=StreamingUpdateSolrServer line=162 Status for: null is 409
> ...
> : Conflict
>
> A "409" HTTP Status is a "Conflict".
>
> It means that Optimistic concurrency failed.  Your update indicated a
> document version but the version of hte document on the server didn't meet
> the version requirements...
>
>
> https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents#UpdatingPartsofDocuments-OptimisticConcurrency
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Apache solr sink issue

[Gopal Patwa]

Do you have this tag "id" define in your schema , it
is not mandatory to have unique field but if you need it then u have to
provide it else you can remove it, see below wiki page for more details

http://wiki.apache.org/solr/SchemaXml#The_Unique_Key_Field

Some options to generate this field if your document cannot derive one

https://wiki.apache.org/solr/UniqueKey




On Mon, Aug 18, 2014 at 10:48 PM, Jeniba Johnson <
jeniba.john...@lntinfotech.com> wrote:

> Hi,
>
> I want to index a log file in Solr using Flume + Apache Solr sink
> Iam referring this below mentioned URL
>
> https://cwiki.apache.org/confluence/display/FLUME/How+to+Setup+Solr+Sink+for+Flume
>
>
> Error  from flume console
> 2014-08-19 15:38:56,451 (concurrentUpdateScheduler-2-thread-1) [ERROR -
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer.handleError(ConcurrentUpdateSolrServer.java:354)]
> error
> java.lang.Exception: Bad Request
> request: http://xxx.xx.xx:8983/solr/update?wt=javabin&version=2
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:208)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
>
> Error  from solr console
> 473844 [qtp176433427-19] ERROR org.apache.solr.core.SolrCore  â
> org.apache.solr.common.SolrException: Document is missing mandatory
> uniqueKey field: id
>
>
> Csn anyone help me with this issue and help me with the steps for
> integrating flume with solr sink
>
>
>
> Regards,
> Jeniba Johnson
>
>
> 
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>

Re: New tiny high-performance HTTP/Servlet server for Solr

[Gopal Patwa]

Thanks for sharing, since in future Solr may move towards standalone server
this (undertow) could be one option.

On Sat, Sep 13, 2014 at 9:36 PM, William Bell  wrote:

> Can we get some stats? Do you have any numbers on performance?
>
> On Sat, Sep 13, 2014 at 3:03 PM, Jayson Minard  >
> wrote:
>
> > Instead of within an Application Server such as Jetty, Tomcat or Wildly
> ...
> > Solr can also now be run standalone on Undertow without the overhead or
> > complexity of a full application server. Open-sourced on
> > https://github.com/bremeld/solr-undertow
> >
> > solr-undertow
> >
> > Solr running in standalone server - High Performance, tiny, fast, easy,
> > standalone deployment. Requires JDK 1.7 or newer. Less than 4MB download,
> > faster than Jetty, Tomcat and all the others. Written in the Kotlin
> > language
> >  for the JVM.
> >
> > Releases are available here
> >  on GitHub.
> >
> > This application launches a Solr WAR file as a standalone server running
> a
> > high performance HTTP front-end based on undertow.io (the engine behind
> > WildFly, the new JBoss). It has no features of an application server,
> does
> > nothing more than load Solr servlets and also service the Admin UI. It is
> > production-quality for a stand-alone Solr server.
> >
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>

Re: [ANN] Lucidworks Fusion 1.0.0

[Gopal Patwa]

you can register for webinar also to know more about Fusion on Oct 2nd.

http://lucidworks.com/blog/say-hello-to-lucidworks-fusion/



On Tue, Sep 23, 2014 at 7:39 AM, Jack Krupansky 
wrote:

> You simply download it yourself and give yourself a demo!!
>
> http://lucidworks.com/product/fusion/
>
> -- Jack Krupansky
>
> -Original Message- From: Thomas Egense
> Sent: Tuesday, September 23, 2014 2:00 AM
> To: solr-user@lucene.apache.org
> Subject: Re: [ANN] Lucidworks Fusion 1.0.0
>
>
> Hi Grant.
> Will there be a Fusion demostration/presentation  at Lucene/Solr Revolution
> DC? (Not listed in the program yet).
>
>
> Thomas Egense
>
> On Mon, Sep 22, 2014 at 3:45 PM, Grant Ingersoll 
> wrote:
>
>  Hi All,
>>
>> We at Lucidworks are pleased to announce the release of Lucidworks Fusion
>> 1.0.   Fusion is built to overlay on top of Solr (in fact, you can manage
>> multiple Solr clusters -- think QA, staging and production -- all from our
>> Admin).In other words, if you already have Solr, simply point Fusion
>> at
>> your instance and get all kinds of goodies like Banana (
>> https://github.com/LucidWorks/Banana -- our port of Kibana to Solr + a
>> number of extensions that Kibana doesn't have), collaborative filtering
>> style recommendations (without the need for Hadoop or Mahout!), a modern
>> signal capture framework, analytics, NLP integration, Boosting/Blocking
>> and
>> other relevance tools, flexible index and query time pipelines as well as
>> a
>> myriad of connectors ranging from Twitter to web crawling to Sharepoint.
>> The best part of all this?  It all leverages the infrastructure that you
>> know and love: Solr.  Want recommendations?  Deploy more Solr.  Want log
>> analytics?  Deploy more Solr.  Want to track important system metrics?
>> Deploy more Solr.
>>
>> Fusion represents our commitment as a company to continue to contribute a
>> large quantity of enhancements to the core of Solr while complementing and
>> extending those capabilities with value adds that integrate a number of
>> 3rd
>> party (e.g connectors) and home grown capabilities like an all new,
>> responsive UI built in AngularJS.  Fusion is not a fork of Solr.  We do
>> not
>> hide Solr in any way.  In fact, our goal is that your existing
>> applications
>> will work out of the box with Fusion, allowing you to take advantage of
>> new
>> capabilities w/o overhauling your existing application.
>>
>> If you want to learn more, please feel free to join our technical webinar
>> on October 2: http://lucidworks.com/blog/say-hello-to-lucidworks-fusion/.
>> If you'd like to download: http://lucidworks.com/product/fusion/.
>>
>> Cheers,
>> Grant Ingersoll
>>
>> 
>> Grant Ingersoll | CTO
>> gr...@lucidworks.com | @gsingers
>> http://www.lucidworks.com
>>
>>
>>
>

Re: Keeping capitalization in suggestions?

[Gopal Patwa]

More detail can be found in Solr Docs

https://cwiki.apache.org/confluence/display/solr/Suggester


On Thu, Dec 4, 2014 at 6:33 AM, Clemens Wyss DEV 
wrote:

> Enter the "factory"! ;)
>  name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingInfixLookupFactory
>
> -Ursprüngliche Nachricht-
> Von: Clemens Wyss DEV [mailto:clemens...@mysign.ch]
> Gesendet: Donnerstag, 4. Dezember 2014 14:46
> An: solr-user@lucene.apache.org
> Betreff: AW: Keeping capitalization in suggestions?
>
> Thx.
> Where (in which jar) do I find
> org.apache.solr.spelling.suggest.AnalyzingInfixSuggester ?
> Or:
> How do I declare the suggest-searchComponent in solrconfig.xml to make use
> of (Lucene's?) AnalyzingInfixSuggester
>
> -Ursprüngliche Nachricht-
> Von: Michael Sokolov [mailto:msoko...@safaribooksonline.com]
> Gesendet: Donnerstag, 4. Dezember 2014 14:05
> An: solr-user@lucene.apache.org
> Betreff: Re: Keeping capitalization in suggestions?
>
> Have a look at AnalyzingInfixSuggester - it does what you want.
>
> -Mike
>
> On 12/4/14 3:05 AM, Clemens Wyss DEV wrote:
> > When I index a text such as "Chamäleon" and look for suggestions for
> "chamä" and/or "Chamä", I'd expect to get "Chamäleon" (uppercased).
> > But what happens is
> >
> > If lowecasefilter (see below (1)) set
> > "chamä" returns "chamäleon"
> > "Chamä" does not match
> >
> > If lowecasefilter (1) not set
> > "Chamä" returns "Chamäleon"
> > "chamä" does not match
> >
> > I guess lowecasefilter should not be set/active, but then how do I get
> matches even if the search term is lowercased?
> >
> > Context:
> > schema.xml
> > ...
> >   positionIncrementGap="100">
> >
> >  
> >  
> >   words="lang/stopwords_de.txt"/>
> >  
> >
> >
> >  
> >   ignoreCase="true" synonyms="synonyms.txt"/>
> >  
> >   words="lang/stopwords_de.txt"/>
> >  
> >
> >  
> > ...
> >   positionIncrementGap="100">
> >
> >  
> >   words="stopwords.txt"/>
> >   
> >
> >  
> >
> > solrconfig.xml
> > -
> > ...
> >   class="org.apache.solr.handler.component.SearchHandler" name="/suggest">
> >  
> >  none
> >  json
> >  false
> >  true
> >  suggestDictionary
> >  true
> >  5
> >  false
> >  
> >  
> >  suggest
> >  
> >  
> > ...
> >  
> >  
> >  suggestDictionary
> >   name="classname">org.apache.solr.spelling.suggest.Suggester
> >   name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookupFactory
> >  suggest
> >  0.
> >  true
> >  
> >  
> > ...
> >
>
>

Re: Solr on Tomcat

[Gopal Patwa]

I think until Solr become completely standalone, it could be major task for
all folks who run Solr as war or repackage Solr war maven release to adopt
5.0 release, since they need to remove tomcat or any other  container they
have in production for running Solr.

Not to mention there will tools build for those container to support
logging and monitoring.

And possibly people will still use unsupported war release to fulfill their
needs until Solr standalone release become stable.

For our use case we repackage solr war (maven release) to include some
custom code which help us to test/debug/deploy our code easily.

Most likely for us we have to create maven war artifact from solr.war
locally and publish to our internal maven repository, so our current build
and deploy process does not break.

On Tue, Feb 10, 2015 at 10:12 AM, Matt Kuiper 
wrote:

> Thanks for all the responses.  I am planning a new project, and
> considering deployment options at this time.  It's helpful to see where
> Solr is headed.
>
> Thanks,
>
> Matt Kuiper
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Tuesday, February 10, 2015 10:05 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr on Tomcat
>
> On 2/10/2015 9:48 AM, Matt Kuiper wrote:
> > I am starting to look in to Solr 5.0.  I have been running Solr 4.* on
> Tomcat.   I was surprised to find the following notice on
> https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+Tomcat
>  (Marked as Unreleased)
> >
> >  Beginning with Solr 5.0, Support for deploying Solr as a WAR in
> servlet containers like Tomcat is no longer supported.
> >
> > I want to verify that it is true that Solr 5.0 will not be able to run
> on Tomcat, and confirm that the recommended way to deploy Solr 5.0 is as a
> Linux service.
>
> Solr will eventually (hopefully soon) be entirely its own application.
> The documentation you have seen in the reference guide is there to prepare
> users for this eventuality.
>
> Right now we are in a transition period.  We have built scripts for
> controlling the start and stop of the example server installation.
> Under the covers, Solr is still a web application contained in a war and
> the example server still runs an unmodified copy of jetty.  Down the road,
> when Solr will becomes a completely standalone application, we will merely
> have to modify the script wrapper to use it, and the user may not even
> notice the change.
>
> With 5.0, if you want to run in tomcat, you will be able to find the war
> in the download's server/webapps directory and use it just like you do now
> ... but we will be encouraging people to NOT do this, because eventually it
> will be completely unsupported.
>
> Thanks,
> Shawn
>
>

Re: SOLR 4.2 SolrQuery exception

[Gopal Patwa]

manually delete lock file
"/data/solr1/example/solr/collection1/./data/index/write.lock",
And restart solr


On Sun, Mar 24, 2013 at 9:32 PM, Sandeep Kumar Anumalla <
sanuma...@etisalat.ae> wrote:

> Hi,
>
> I managed to resolve this issue and I am getting the results also. But
> this time I am getting a different exception while loading Solr Container
>
> Here is the Code.
>
> String SOLR_HOME = "/data/solr1/example/solr/collection1";
> CoreContainer coreContainer = new CoreContainer(SOLR_HOME);
> CoreDescriptor discriptor = new CoreDescriptor(coreContainer,
> "collection1", new File(SOLR_HOME).getAbsolutePath());
> SolrCore solrCore = coreContainer.create(discriptor);
> coreContainer.register(solrCore, false);
> File home = new File( SOLR_HOME );
> File f = new File( home, "solr.xml" );
> coreContainer.load( SOLR_HOME, f );
> server = new EmbeddedSolrServer( coreContainer, "collection1" );
> SolrQuery q = new SolrQuery();
>
>
> Parameters inside Solrconfig.xml
> 
> simple
> true
>
>
> WARNING: Unable to get IndexCommit on startup
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
> SimpleFSLock@/data/solr1/example/solr/collection1/./data/index/write.lock
>at org.apache.lucene.store.Lock.obtain(Lock.java:84)
>at org.apache.lucene.index.IndexWriter.(IndexWriter.java:636)
>at
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:77)
>at
> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
>at
> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:192)
>at
> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:106)
>at
> org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:904)
>at
> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:592)
>at org.apache.solr.core.SolrCore.(SolrCore.java:801)
>at org.apache.solr.core.SolrCore.(SolrCore.java:619)
>at
> org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1021)
>at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051)
>at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634)
>at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>at java.lang.Thread.run(Thread.java:679)
>
>
>
> From: Sandeep Kumar Anumalla
> Sent: 24 March, 2013 03:44 PM
> To: solr-user@lucene.apache.org
> Subject: SOLR 4.2 SolrQuery exception
>
> I am using the below code and getting the exception while using SolrQuery
>
>
>
> Mar 24, 2013 3:08:07 PM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener sending requests to 
> Searcher@795e0c2bmain{StandardDirectoryReader(segments_49:524 _4v(4.2):C299313
> _4x(4.2):C2953/1396 _4y(4.2):C2866/1470 _4z(4.2):C4263/2793
> _50(4.2):C3554/761 _51(4.2):C1126/365 _52(4.2):C650/285 _53(4.2):C500/215
> _54(4.2):C1808/1593 _55(4.2):C1593)}
> Mar 24, 2013 3:08:07 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.NullPointerException
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1797)
> at
> org.apache.solr.core.QuerySenderListener.newSearcher(QuerySenderListener.java:64)
> at org.apache.solr.core.SolrCore$5.call(SolrCore.java:1586)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
>
> Mar 24, 2013 3:08:07 PM org.apache.solr.core.SolrCore execute
> INFO: [collection1] webapp=null path=null
> params={event=firstSearcher&q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false}
> status=500 QTime=4
> Mar 24, 2013 3:08:07 PM org.apache.solr.core.QuerySenderListener
> newSearcher
> INFO: QuerySenderListener done.
> Mar 24, 2013 3:08:07 PM
> org.apache.solr.handler.component.SpellCheckComponent$SpellCheckerListener
> newSearcher
> INFO: Lo

Re: Multi-core and replicated Solr cloud testing. Data-directory mis-configures

[Gopal Patwa]

if you use default directory then it will use solr.home directory, I have
tested solr cloud example on local machine with 5-6 nodes.And data
directory was created under core name, like

"example2/solr/collection1/data". you could see example startup script from
source code "solr/cloud-dev/solrcloud-multi-start.sh"

example solrconfig.xml

  ${solr.data.dir:}

On Sun, Mar 24, 2013 at 10:44 PM, Trevor Campbell
wrote:

> I have three indexes which I have set up as three separate cores, using
> this solr.xml config.
>
>hostPort="${jetty.port:}">
> 
>
> 
> 
>
> 
> 
>
> 
>   
>
> This works just fine a standalone solr.
>
> I duplicated this setup on the same machine under a completely separate
> solr installation (solr-nodeb) and modified all the data directroies to
> point to the direstories in nodeb.  This all worked fine.
>
> I then connected the 2 instances together with zoo-keeper using settings
> "-Dbootstrap_conf=true -Dcollection.configName=**jiraCluster -DzkRun
> -DnumShards=1" for the first intsance and "-DzkHost=localhost:9080" for
>  the second. (I'm using tomcat and ports 8080 and 8081 for the 2 Solr
> instances)
>
> Now the data directories of the second node point to the data directories
> in the first node.
>
> I have tried many settings in the solrconfig.xml for each core but am now
> using absolute paths, e.g.
> /home//solr-**4.2.0-nodeb/example/multicore/**
> jira-comment/data
>
> previously I used
> ${solr.jira-comment.data.dir:/**home/tcampbell/solr-4.2.0-**
> nodeb/example/multicore/jira-**comment/data}
> but that had the same result.
>
> It seems zookeeper is forcing data directory config from the uploaded
> configuration on all the nodes in the cluster?
>
> How can I do testing on a single machine? Do I really need identical
> directory layouts on all machines?
>
>
>

Re: Basic auth on SolrCloud /admin/* calls

[Gopal Patwa]

I was looking too for this feature and it seems SOLR-4470 can work but I
haven't tried yet.

+1


On Sun, Apr 14, 2013 at 12:14 PM, Tim Vaillancourt wrote:

> I've thought about this too, and have heard of some people running a
> lightweight http proxy upstream of Solr.
>
> With the right network restrictions (only way for a client to reach solr
> is via a proxy + the nodes can still talk to each other), you could achieve
> the same thing SOLR-4470 is doing, with the drawback of additional proxy
> and firewall components to maintain, plus added overhead on HTTP calls.
>
> A benefit though is a lightweight proxy ahead of Solr could implement HTTP
> caching, taking some load off of Solr.
>
> In a perfect world, I'd say rolling out SOLR-4470 is the best solution,
> but again, it seems to be losing momentum (please Vote/support the
> discussion!). While proxies can achieve this, I think enough people have
> pondered about this to implement this as a feature in Solr.
>
> Tim
>
>
> On 14/04/13 12:32 AM, adfel70 wrote:
>
>> Did anyone try blocking access to the ports in the firewall level, and
>> allowing all the solr servers in the cluster+given control-machines?
>> Assuming that search request to solr run though a proxy..
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.**
>> nabble.com/Basic-auth-on-**SolrCloud-admin-calls-**tp4052266p4055868.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>

Re: Doubts about solr stats component

[Gopal Patwa]

please post field defination from solr schema.xml for
stats.field=login_attempts
,
it depends how you have defined stats field

Re: commit in solr4 takes a longer time

[Gopal Patwa]

you might want to added openSearcher=false for hard commit, so hard commit
also act like soft commit

   
5
30
   false




On Thu, May 2, 2013 at 12:16 AM, vicky desai wrote:

> Hi,
>
> I am using 1 shard and two replicas. Document size is around 6 lakhs
>
>
> My solrconfig.xml is as follows
> 
> 
> LUCENE_40
> 
>
>
> 2147483647
> simple
> true
> 
> 
> 
> 500
> 1000
> 
> 
> 5
> 30
> 
> 
>
> 
>  multipartUploadLimitInKB="204800" />
> 
>
>  default="true" />
> 
>  class="org.apache.solr.handler.admin.AdminHandlers" />
>  class="solr.ReplicationHandler" />
>  class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}" />
> true
> 
> *:*
> 
> 
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060402.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: commit in solr4 takes a longer time

[Gopal Patwa]

Since you have define commit option as "Auto Commit" for hard and soft
commit then you don't have to explicitly call commit from SolrJ client. And
openSearcher=false for hard commit will make hard commit faster since it is
only makes sure that recent changes are flushed to disk (for durability)
 and not opening any searcher.

can you post you log when soft commit and hard commit happens?

You can read about waitFlush=false and waitSearcher=false which are default
to true, see below from  java doc

JavaDoc:
*waitFlush* block until index changes are flushed to disk
*waitSearcher* block until a new searcher is opened and registered as the
main query searcher, making the changes visible*T*


On Fri, May 3, 2013 at 7:19 AM, vicky desai wrote:

> Hi All,
>
> setting opensearcher flag to true solution worked and it give me visible
> improvement in commit time. One thing to make note of is that while using
> solrj client we have to call server.commit(false,false) which i was doing
> incorrectly and hence was not able to see the improvement earliear.
>
> Thanks everyone
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/commit-in-solr4-takes-a-longer-time-tp4060396p4060688.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: disaster recovery scenarios for solr cloud and zookeeper

[Gopal Patwa]

agree with Anshum and Netflix has very nice supervisor system for ZooKeeper
if they goes down it will restart them automatically

http://techblog.netflix.com/2012/04/introducing-exhibitor-supervisor-system.html
https://github.com/Netflix/exhibitor




On Fri, May 3, 2013 at 6:53 PM, Anshum Gupta  wrote:

> In case all your Zk nodes go down, the querying would continue to work
> fine (as far as no nodes fail) but you'd not be able to add docs.
>
> Sent from my iPhone
>
> On 03-May-2013, at 17:52, Shawn Heisey  wrote:
>
> > On 5/3/2013 6:07 PM, Walter Underwood wrote:
> >> Ideally, the Solr nodes should be able to continue as long as no node
> fails. Failure of a leader would be bad, failure of non-leader replicas
> might cause some timeouts, but could be survivable.
> >>
> >> Of course, nodes could not be added.
> >
> > I have read a few things that say things go read only when the zookeeper
> > ensemble loses quorum.  I'm not sure whether that means that Solr goes
> > read only or zookeeper goes read only.  I would be interested in knowing
> > exactly what happens when zookeeper loses quorum as well as what happens
> > if all three (or more) zookeeper nodes in the ensemble go away entirely.
> >
> > I have a SolrCloud I can experiment with, but I need to find a
> > maintenance window for testing, so I can't check right now.
> >
> > Thanks,
> > Shawn
> >
>

Re: SolrCloud and exernal file fields

[Gopal Patwa]

Hi, I am also very much interested in this, since we use Solr 4 with NRT
where we update index every second but most of time it update only stored
filed.
 if Solr/Lucene could provide external datastore without re-indexing even
for stored field only, it would be very beneficial for frequent update use
case, where cache invalidation will not happen for stored fields update and
it will improve indexing performance due to smaller index size.

Here is below link for similar work.

http://www.flax.co.uk/blog/2012/06/22/updating-individual-fields-in-lucene-with-a-redis-backed-codec/


On Fri, Nov 23, 2012 at 11:42 AM, Simone Gianni  wrote:

> Posted,
> see it here
>
> http://lucene.472066.n3.nabble.com/Possible-sharded-and-replicated-replacement-for-ExternalFileFields-in-SolrCloud-td4022108.html
>
> Simone
>
>
> 2012/11/23 Simone Gianni 
>
> > 2012/11/22 Martin Koch 
> >
> >> IMO it would be ideal if the lucene/solr community could come up with a
> >> good way of updating fields in a document without reindexing. This could
> >> be
> >> by linking to some external data store, or in the lucene/solr internals.
> >> If
> >> it would make things easier, a good first step would be to have
> >> dynamically
> >> updateable numerical fields only.
> >>
> >
> > Hi Martin,
> > I'm working on implementing exactly this, and I have a working prototype
> > right now. I'm going to write on lucene dev about the details and asking
> > advice there. I'll contribute the code, so anyone interested followup on
> > dev.
> >
> > Simone
> >
> >
>

Re: Problem querying collection in Solr 4.1

[Gopal Patwa]

one thing I noticed in solrconfig xml that it set to use Lucene version 4.0
index format but you  mention you are using it 4.1

  LUCENE_40



On Mon, Jan 21, 2013 at 4:26 PM, Brett Hoerner wrote:

> I have a collection in Solr 4.1 RC1 and doing a simple query like
> text:"puppy dog" is causing an exception. Oddly enough, I CAN query for
> text:puppy or text:"puppy", but adding the space breaks everything.
>
> Schema and config: https://gist.github.com/f49da15e39e5609b75b1
>
> This happens whether I query the whole collection or a single direct core.
> I haven't tested whether this would happen outside of SolrCloud.
>
> http://localhost:8984/solr/timeline/select?q=text%3A%22puppy+dog%22&wt=xml
>
>
> http://localhost:8984/solr/timeline_shard4_replica1/select?q=text%3A%22puppy+dog%22&wt=xml
>
> Jan 22, 2013 12:07:24 AM org.apache.solr.common.SolrException log
> SEVERE: null:org.apache.solr.common.SolrException:
> org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> available to handle this request:[
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard2_replica1,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard1_replica2,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard3_replica2,
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard4_replica1,
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard1_replica1,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard2_replica2,
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard3_replica1,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard4_replica2]
>  at
>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302)
> at
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
> at
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
>  at
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
>  at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>  at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
> at
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>  at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
>  at
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
>  at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>  at
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>  at org.eclipse.jetty.server.Server.handle(Server.java:365)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
>  at
>
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
>  at
>
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
>  at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
> at
>
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
>  at
>
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
> at
>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>  at
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers available to handle this request:[
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard2_replica1,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard1_replica2,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard3_replica2,
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard4_replica1,
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard1_replica1,
> http://timelinesearch-2d.i.massrel.com:8983/solr/timeline_shard2_replica2,
> http://timelinesearch-1d.i.massrel.com:8983/solr/timeline_shard3_replica1,
> http://

Re: Upgrade to 4.1.0 and possible deprecation of StreamingUpdateSolrServer & CommonsHttpSolrServer

[Gopal Patwa]

These classes were deprecated from 4.0 and it is replaced with
ConcurrentUpdateSolrServer and HttpSolrServer

On Tue, Jan 22, 2013 at 5:28 PM, Lewis John Mcgibbney <
lewis.mcgibb...@gmail.com> wrote:

> Hi All,
>
> As above, I am upgrading our application and I wish to give it a facelift
> to use the new shiny 4.1.0 deps.
> I notice that not both the packages above are no longer included in the
> 4.1.0 solr-solrj artifact.
> Can someone please explain what I should replace this with?
> I also notice that the wiki is out [0], if someone can please explain/help
> then I will update.
>
> Thank you very much in advance.
>
> Lewis
>
> [0]
> http://wiki.apache.org/solr/SolrPerformanceFactors#Indexing_Performance
>
> --
> *Lewis*
>

Large Index and OutOfMemoryError: Map failed

[Gopal Patwa]

 2147483647
1
4096
10
1000
1
single


  0.0
  10.0



  false
  0






1000
 
   90
   false
 
 
   ${inventory.solr.softcommit.duration:1000}
 





Thanks
Gopal Patwa
*

Open deleted index file failing jboss shutdown with Too many open files Error

[Gopal Patwa]

I am using Solr 4.0 nightly build with NRT and I often get this
error during auto commit "Too many open files". I have search this forum
and what I found it is related to OS ulimit setting, please see below my
ulimit settings. I am not sure what ulimit setting I should have for open
file? ulimit -n unlimited?.

Even if I set to higher number, it will just delay the issue until it reach
new open file limit. What I have seen that Solr has kept deleted index file
open by java process, which causing issue for our application server jboss
to shutdown gracefully due this open files by java process.

I have seen recently this issue was resolved in lucene, is it TRUE?

https://issues.apache.org/jira/browse/LUCENE-3855


I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3
- 15GB, with Single shard

We update the index every 5 seconds, soft commit every 1 second and hard
commit every 15 minutes

Environment: Jboss 4.2, JDK 1.6 64 bit, CentOS , JVM Heap Size = 24GB*


ulimit:

core file size  (blocks, -c) 0

data seg size   (kbytes, -d) unlimited

scheduling priority (-e) 0

file size   (blocks, -f) unlimited

pending signals (-i) 401408

max locked memory   (kbytes, -l) 1024

max memory size (kbytes, -m) unlimited

open files  (-n) 4096

pipe size(512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority  (-r) 0

stack size  (kbytes, -s) 10240

cpu time   (seconds, -t) unlimited

max user processes  (-u) 401408

virtual memory  (kbytes, -v) unlimited

file locks  (-x) unlimited


ERROR:*

*2012-04-01* *20:08:35*,*323* [] *priority=ERROR* *app_name=*
*thread=pool-10-thread-1* *location=CommitTracker* *line=93* *auto*
*commit* *error...:org.apache.solr.common.SolrException:* *Error*
*opening* *new* *searcher*
*at* 
*org.apache.solr.core.SolrCore.openNewSearcher*(*SolrCore.java:1138*)
*at* *org.apache.solr.core.SolrCore.getSearcher*(*SolrCore.java:1251*)
*at* 
*org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:409*)
*at* 
*org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*)
*at* 
*java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*)
*at* 
*java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*)
*at* *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*)
*at* 
*java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301*(*ScheduledThreadPoolExecutor.java:98*)
*at* 
*java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run*(*ScheduledThreadPoolExecutor.java:206*)
*at* 
*java.util.concurrent.ThreadPoolExecutor$Worker.runTask*(*ThreadPoolExecutor.java:886*)
*at* 
*java.util.concurrent.ThreadPoolExecutor$Worker.run*(*ThreadPoolExecutor.java:908*)
*at* *java.lang.Thread.run*(*Thread.java:662*)*Caused* *by:*
*java.io.FileNotFoundException:*
*/opt/mci/data/srwp01mci001/inventory/index/_4q1y_0.tip* (*Too many
open files*)
*at* *java.io.RandomAccessFile.open*(*Native* *Method*)
*at* *java.io.RandomAccessFile.*<*init*>(*RandomAccessFile.java:212*)
*at* 
*org.apache.lucene.store.FSDirectory$FSIndexOutput.*<*init*>(*FSDirectory.java:449*)
*at* 
*org.apache.lucene.store.FSDirectory.createOutput*(*FSDirectory.java:288*)
*at* 
*org.apache.lucene.codecs.BlockTreeTermsWriter.*<*init*>(*BlockTreeTermsWriter.java:161*)
*at* 
*org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsConsumer*(*Lucene40PostingsFormat.java:66*)
*at* 
*org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField*(*PerFieldPostingsFormat.java:118*)
*at* 
*org.apache.lucene.index.FreqProxTermsWriterPerField.flush*(*FreqProxTermsWriterPerField.java:322*)
*at* 
*org.apache.lucene.index.FreqProxTermsWriter.flush*(*FreqProxTermsWriter.java:92*)
*at* *org.apache.lucene.index.TermsHash.flush*(*TermsHash.java:117*)
*at* *org.apache.lucene.index.DocInverter.flush*(*DocInverter.java:53*)
*at* 
*org.apache.lucene.index.DocFieldProcessor.flush*(*DocFieldProcessor.java:81*)
*at* 
*org.apache.lucene.index.DocumentsWriterPerThread.flush*(*DocumentsWriterPerThread.java:475*)
*at* 
*org.apache.lucene.index.DocumentsWriter.doFlush*(*DocumentsWriter.java:422*)
*at* 
*org.apache.lucene.index.DocumentsWriter.flushAllThreads*(*DocumentsWriter.java:553*)
*at* 
*org.apache.lucene.index.IndexWriter.getReader*(*IndexWriter.java:354*)
*at* 
*org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter*(*StandardDirectoryReader.java:258*)
*at* 
*org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged*(*StandardDirectoryReader.java:243*)
*at* 
*org.apache.lucene.index.Director

Re: Open deleted index file failing jboss shutdown with Too many open files Error

[Gopal Patwa]

Here is SolrConfig.xml, and I am using Lucene NRT with soft commit and
 update the index every 5 seconds, soft commit every 1 second and hard
commit every 15 minutes

> SolrConfig.xml:
>
>
>
>false
>10
>2147483647
>1
>4096
>10
>1000
>1
>single
>
>
>  0.0
>  10.0
>
>
>
>  false
>  0
>
>
>
>
>
>
>1000
> 
>   90
>   false
> 
> 
>   ${inventory.solr.softcommit.duration:1000}
> 
>
>

On Sun, Apr 1, 2012 at 7:47 PM, Gopal Patwa  wrote:

> I am using Solr 4.0 nightly build with NRT and I often get this
> error during auto commit "Too many open files". I have search this forum
> and what I found it is related to OS ulimit setting, please see below my
> ulimit settings. I am not sure what ulimit setting I should have for open
> file? ulimit -n unlimited?.
>
> Even if I set to higher number, it will just delay the issue until it
> reach new open file limit. What I have seen that Solr has kept deleted
> index file open by java process, which causing issue for our application
> server jboss to shutdown gracefully due this open files by java process.
>
> I have seen recently this issue was resolved in lucene, is it TRUE?
>
> https://issues.apache.org/jira/browse/LUCENE-3855
>
>
> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3
> - 15GB, with Single shard
>
> We update the index every 5 seconds, soft commit every 1 second and hard
> commit every 15 minutes
>
> Environment: Jboss 4.2, JDK 1.6 64 bit, CentOS , JVM Heap Size = 24GB*
>
>
> ulimit:
>
> core file size  (blocks, -c) 0
>
> data seg size   (kbytes, -d) unlimited
>
> scheduling priority (-e) 0
>
> file size   (blocks, -f) unlimited
>
> pending signals (-i) 401408
>
> max locked memory   (kbytes, -l) 1024
>
> max memory size (kbytes, -m) unlimited
>
> open files  (-n) 4096
>
> pipe size(512 bytes, -p) 8
>
> POSIX message queues (bytes, -q) 819200
>
> real-time priority  (-r) 0
>
> stack size  (kbytes, -s) 10240
>
> cpu time   (seconds, -t) unlimited
>
> max user processes  (-u) 401408
>
> virtual memory  (kbytes, -v) unlimited
>
> file locks  (-x) unlimited
>
>
> ERROR:*
>
> *2012-04-01* *20:08:35*,*323* [] *priority=ERROR* *app_name=* 
> *thread=pool-10-thread-1* *location=CommitTracker* *line=93* *auto* *commit* 
> *error...:org.apache.solr.common.SolrException:* *Error* *opening* *new* 
> *searcher*
>   *at* 
> *org.apache.solr.core.SolrCore.openNewSearcher*(*SolrCore.java:1138*)
>   *at* *org.apache.solr.core.SolrCore.getSearcher*(*SolrCore.java:1251*)
>   *at* 
> *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:409*)
>   *at* 
> *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*)
>   *at* 
> *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*)
>   *at* 
> *java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*)
>   *at* *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*)
>   *at* 
> *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301*(*ScheduledThreadPoolExecutor.java:98*)
>   *at* 
> *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run*(*ScheduledThreadPoolExecutor.java:206*)
>   *at* 
> *java.util.concurrent.ThreadPoolExecutor$Worker.runTask*(*ThreadPoolExecutor.java:886*)
>   *at* 
> *java.util.concurrent.ThreadPoolExecutor$Worker.run*(*ThreadPoolExecutor.java:908*)
>   *at* *java.lang.Thread.run*(*Thread.java:662*)*Caused* *by:* 
> *java.io.FileNotFoundException:* 
> */opt/mci/data/srwp01mci001/inventory/index/_4q1y_0.tip* (*Too many open 
> files*)
>   *at* *java.io.RandomAccessFile.open*(*Native* *Method*)
>   *at* *java.io.RandomAccessFile.*<*init*>(*RandomAccessFile.java:212*)
>   *at* 
> *org.apache.lucene.store.FSDirectory$FSIndexOutput.*<*init*>(*FSDirectory.java:449*)
>   *at* 
> *org.apache.lucene.store.FSDirectory.createOutput*(*FSDirectory.java:288*)
>   *at* 
> *org.apache.lucene.codecs.BlockTreeTermsWri

Re: Large Index and OutOfMemoryError: Map failed

[Gopal Patwa]

Michael, Thanks for response

it was 65K as you mention the default value for "cat
/proc/sys/vm/max_map_count" . How we determine what value this should be?
 is it number of document during hard commit in my case it is 15 minutes?
or it is number of  index file or number of documents we have in all cores.

I have raised the number to 140K but I still get when it reaches to 140K,
we have to restart jboss server to free up the map count, sometime OOM
error happen during "*Error opening new searcher"*

is making this number to unlimited is only solution?''


Error log:

*location=CommitTracker line=93 auto commit
error...:org.apache.solr.common.SolrException: Error opening new
searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)Caused by:
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at 
org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:293)
at 
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.(PerFieldPostingsFormat.java:262)
at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.(PerFieldPostingsFormat.java:316)
at 
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316)
at org.apache.lucene.codecs.Codec.files(Codec.java:56)
at org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423)
at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215)
at 
org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220)
at 
org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497)
at 
org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477)
at 
org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201)
at 
org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119)
at 
org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148)
at 
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:438)
at 
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:553)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:354)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:258)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:243)
at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:250)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1091)
... 11 moreCaused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)*



And one more issue we came across i.e

On Sat, Mar 31, 2012 at 3:15 AM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> It's the virtual memory limit that matters; yours says unlimited below
> (good!), but, are you certain that's really the limit your Solr
> process runs with?
>
> On Linux, there is also a per-process map count:
>
>cat /proc/sys/vm/max_map_count
>
> I think it typically defaults to 65,536 but you should check on your
> env.  If a process tries to map more than this many regions, you'll
> hit that exception.
>
> I think you can:
>
>  cat /proc//maps | wc
>
> to see how many maps your Solr process currently has... if that is
> anywhere near the limit then it could be the cause.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Mar 31, 2012 at 1:26 AM, Gopal Patwa  wrote:
> &g

Re: Large Index and OutOfMemoryError: Map failed

[Gopal Patwa]

I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
system data. Is their any existing test case to reproduce this issue? I am
trying understand how I can reproduce this issue with unit/integration test

I will try recent solr trunk build too,  if it is some bug in solr or
lucene keeping old searcher open then how to reproduce it?

SYSTEM DATA
===
PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
SYSTEM ID: x86_64
CURRENT CPU SPEED: 1600.000 MHz
CPUS: 8 processor(s)
MEMORY: 49449296 kB
DISTRIBUTION: CentOS release 5.3 (Final)
KERNEL NAME: 2.6.18-128.el5
UPTIME: up 71 days
LOAD AVERAGE: 1.42, 1.45, 1.53
JBOSS Version: Implementation-Version: 4.2.2.GA (build:
SVNTag=JBoss_4_2_2_GA date=20
JAVA Version: java version "1.6.0_24"


On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> Your largest index has 66 segments (690 files) ... biggish but not
> insane.  With 64K maps you should be able to have ~47 searchers open
> on each core.
>
> Enabling compound file format (not the opposite!) will mean fewer maps
> ... ie should improve this situation.
>
> I don't understand why Solr defaults to compound file off... that
> seems dangerous.
>
> Really we need a Solr dev here... to answer "how long is a stale
> searcher kept open".  Is it somehow possible 46 old searchers are
> being left open...?
>
> I don't see any other reason why you'd run out of maps.  Hmm, unless
> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> Which exact JVM are you using?  If you can print the
> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>
> Yes, switching away from MMapDir will sidestep the "too many maps"
> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> there really is a leak here (Solr not closing the old searchers or a
> Lucene bug or something...) then you'll eventually run out of file
> descriptors (ie, same  problem, different manifestation).
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> 2012/4/11 Gopal Patwa :
> >
> > I have not change the mergefactor, it was 10. Compound index file is
> disable
> > in my config but I read from below post, that some one had similar issue
> and
> > it was resolved by switching from compound index file format to
> non-compound
> > index file.
> >
> > and some folks resolved by "changing lucene code to disable
> MMapDirectory."
> > Is this best practice to do, if so is this can be done in configuration?
> >
> >
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
> >
> > I have index document of core1 = 5 million, core2=8million and
> > core3=3million and all index are hosted in single Solr instance
> >
> > I am going to use Solr for our site StubHub.com, see attached "ls -l"
> list
> > of index files for all core
> >
> > SolrConfig.xml:
> >
> >
> >   
> >   false
> >   10
> >   2147483647
> >   1
> >   4096
> >   10
> >   1000
> >   1
> >   single
> >
> >   
> > 0.0
> > 10.0
> >   
> >
> >   
> > false
> > 0
> >   
> >
> >   
> >
> >
> >   
> >   1000
> >
> >  90
> >  false
> >
> >
> >
>  ${inventory.solr.softcommit.duration:1000}
> >
> >
> >   
> >
> >
> > Forwarded conversation
> > Subject: Large Index and OutOfMemoryError: Map failed
> > 
> >
> > From: Gopal Patwa 
> > Date: Fri, Mar 30, 2012 at 10:26 PM
> > To: solr-user@lucene.apache.org
> >
> >
> > I need help!!
> >
> >
> >
> >
> >
> > I am using Solr 4.0 nightly build with NRT and I often get this error
> during
> > auto commit "java.lang.OutOfMemoryError: Map failed". I have search this
> > forum and what I found it is related to OS ulimit setting, please se
> below
> > my ulimit settings. I am not sure what ulimit setting I should have? and
> we
> > also get "java.net.SocketException: Too many open files" NOT sure how
> many
> > open file we need to set?
> >
> >
> > I have 3 core with index size : core1 - 70GB, Core2 - 50G

Re: Large Index and OutOfMemoryError: Map failed

[Gopal Patwa]

We cannot avoid auto soft commit, since we need Lucene NRT feature. And I
use StreamingUpdateSolrServer for adding/updating index.

On Thu, Apr 19, 2012 at 7:42 AM, Boon Low  wrote:

> Hi,
>
> Also came across this error recently, while indexing with > 10 DIH
> processes in parallel + default index setting. The JVM grinds to a halt and
> throws this error. Checking the index of a core reveals thousands of files!
> Tuning the default autocommit from 15000ms to 90ms solved the problem
> for us. (no 'autosoftcommit').
>
> Boon
>
> -
> Boon Low
> Search UX and Engine Developer
> brightsolid Online Publishing
>
> On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
>
> > I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
> > system data. Is their any existing test case to reproduce this issue? I
> am
> > trying understand how I can reproduce this issue with unit/integration
> test
> >
> > I will try recent solr trunk build too,  if it is some bug in solr or
> > lucene keeping old searcher open then how to reproduce it?
> >
> > SYSTEM DATA
> > ===
> > PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
> > SYSTEM ID: x86_64
> > CURRENT CPU SPEED: 1600.000 MHz
> > CPUS: 8 processor(s)
> > MEMORY: 49449296 kB
> > DISTRIBUTION: CentOS release 5.3 (Final)
> > KERNEL NAME: 2.6.18-128.el5
> > UPTIME: up 71 days
> > LOAD AVERAGE: 1.42, 1.45, 1.53
> > JBOSS Version: Implementation-Version: 4.2.2.GA (build:
> > SVNTag=JBoss_4_2_2_GA date=20
> > JAVA Version: java version "1.6.0_24"
> >
> >
> > On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
> > luc...@mikemccandless.com> wrote:
> >
> >> Your largest index has 66 segments (690 files) ... biggish but not
> >> insane.  With 64K maps you should be able to have ~47 searchers open
> >> on each core.
> >>
> >> Enabling compound file format (not the opposite!) will mean fewer maps
> >> ... ie should improve this situation.
> >>
> >> I don't understand why Solr defaults to compound file off... that
> >> seems dangerous.
> >>
> >> Really we need a Solr dev here... to answer "how long is a stale
> >> searcher kept open".  Is it somehow possible 46 old searchers are
> >> being left open...?
> >>
> >> I don't see any other reason why you'd run out of maps.  Hmm, unless
> >> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> >> Which exact JVM are you using?  If you can print the
> >> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
> >>
> >> Yes, switching away from MMapDir will sidestep the "too many maps"
> >> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> >> there really is a leak here (Solr not closing the old searchers or a
> >> Lucene bug or something...) then you'll eventually run out of file
> >> descriptors (ie, same  problem, different manifestation).
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> 2012/4/11 Gopal Patwa :
> >>>
> >>> I have not change the mergefactor, it was 10. Compound index file is
> >> disable
> >>> in my config but I read from below post, that some one had similar
> issue
> >> and
> >>> it was resolved by switching from compound index file format to
> >> non-compound
> >>> index file.
> >>>
> >>> and some folks resolved by "changing lucene code to disable
> >> MMapDirectory."
> >>> Is this best practice to do, if so is this can be done in
> configuration?
> >>>
> >>>
> >>
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
> >>>
> >>> I have index document of core1 = 5 million, core2=8million and
> >>> core3=3million and all index are hosted in single Solr instance
> >>>
> >>> I am going to use Solr for our site StubHub.com, see attached "ls -l"
> >> list
> >>> of index files for all core
> >>>
> >>> SolrConfig.xml:
> >>>
> >>>
> >>>  
> >>>  false
> >>>  10
> >>>  2147483647
> >>>  1
> >>>  4096
> >>>  10
> >>>  1

Re: # open files with SolrCloud

[Gopal Patwa]

Yonik, This same issue we have on our production with Solr 4 Trunk build
running on Cent OS, JDK 6 64-bit

I have reported "java.io.IOException: Map failed" and "Too many open files"
issue, i seems their is a search leak in Solr which is not closing them and
file being kept open.

It would be great help if we can resolve this issue, I was going to try
latest build but it seems this issue is not resolved yet

http://lucene.472066.n3.nabble.com/Large-Index-and-OutOfMemoryError-Map-failed-td3872891.html


On Sat, Apr 21, 2012 at 11:57 AM, Yonik Seeley
wrote:

> I can reproduce some kind of searcher leak issue here, even w/o
> SolrCloud, and I've opened
> https://issues.apache.org/jira/browse/SOLR-3392
>
> -Yonik
> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
> Boston May 7-10
>

Re: # open files with SolrCloud

[Gopal Patwa]

forgot to mention we are not using Solr Cloud yet but we use Lucene NRT
feature,  This issue is happening WITHOUT Solr Cloud

On Sat, Apr 21, 2012 at 8:14 PM, Gopal Patwa  wrote:

> Yonik, This same issue we have on our production with Solr 4 Trunk build
> running on Cent OS, JDK 6 64-bit
>
> I have reported "java.io.IOException: Map failed" and "Too many open
> files" issue, i seems their is a search leak in Solr which is not closing
> them and file being kept open.
>
> It would be great help if we can resolve this issue, I was going to try
> latest build but it seems this issue is not resolved yet
>
>
> http://lucene.472066.n3.nabble.com/Large-Index-and-OutOfMemoryError-Map-failed-td3872891.html
>
>
> On Sat, Apr 21, 2012 at 11:57 AM, Yonik Seeley  > wrote:
>
>> I can reproduce some kind of searcher leak issue here, even w/o
>> SolrCloud, and I've opened
>> https://issues.apache.org/jira/browse/SOLR-3392
>>
>> -Yonik
>> lucenerevolution.com - Lucene/Solr Open Source Search Conference.
>> Boston May 7-10
>>
>
>

Re: # open files with SolrCloud

[Gopal Patwa]

Great! I am going to try new Solr 4 build from April 23rd

On Sun, Apr 22, 2012 at 11:35 PM, Sami Siren  wrote:

> On Sat, Apr 21, 2012 at 9:57 PM, Yonik Seeley
>  wrote:
> > I can reproduce some kind of searcher leak issue here, even w/o
> > SolrCloud, and I've opened
> > https://issues.apache.org/jira/browse/SOLR-3392
>
> With the fix integrated. I do not see the leaking problem anymore with
> my setup so it seems to be working now.
>
> --
>  Sami Siren
>

ConcurrentUpdateSolrServer and unable to override default http settings

[Gopal Patwa]

In Solr4j client trunk build for 4.0, ConcurrentUpdateSolrServer class does
not allow to override default http settings
like HttpConnectionParams.setConnectionTimeout,
HttpConnectionParams.setSoTimeout, DefaultMaxConnectionsPerHost


Due to HttpSolrServer is not accessible from ConcurrentUpdateSolrServer
class,  since most of time you just need to override default http settings,
I know we can pass HttpClient but it would be nice
if ConcurrentUpdateSolrServer can allow to get access to HttpSolrServer
from some getter method.

Otherwise anyone who need to override default http settings need to pass
HttpClient.


-Gopal Patwa

Re: Latest solr4 snapshot seems to be giving me a lot of unhappy logging about 'Log4j', should I be concerned?

[Gopal Patwa]

I have similar issue using log4j for logging with trunk build, the
CoreConatainer class print big stack trace on our jboss 4.2.2 startup, I am
using sjfj 1.5.2

10:07:45,918 WARN  [CoreContainer] Unable to read SLF4J version
java.lang.NoSuchMethodError:
org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder;
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:395)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:355)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:304)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:101)



On Tue, May 1, 2012 at 9:25 AM, Benson Margulies wrote:

> On Tue, May 1, 2012 at 12:16 PM, Mark Miller 
> wrote:
> > There is a recent JIRA issue about keeping the last n logs to display in
> the admin UI.
> >
> > That introduced a problem - and then the fix introduced a problem - and
> then the fix mitigated the problem but left that ugly logging as a by
> product.
> >
> > Don't remember the issue # offhand. I think there was a dispute about
> what should be done with it.
> >
> > On May 1, 2012, at 11:14 AM, Benson Margulies wrote:
> >
> >> CoreContainer.java, in the method 'load', finds itself calling
> >> loader.NewInstance with an 'fname' of Log4j of the slf4j backend is
> >> 'Log4j'.
>
> Couldn't someone just fix the if statement to say, 'OK, if we're doing
> log4j, we have no log watcher' and skip all the loud failing on the
> way?
>
>
>
> >>
> >> e.g.:
> >>
> >> 2012-05-01 10:40:32,367 org.apache.solr.core.CoreContainer  - Unable
> >> to load LogWatcher
> >> org.apache.solr.common.SolrException: Error loading class 'Log4j'
> >>
> >> What is it actually looking for? Have I misplaced something?
> >
> > - Mark Miller
> > lucidimagination.com
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: ConcurrentUpdateSolrServer and unable to override default http settings

[Gopal Patwa]

Is this possible to make this improvement, so it can save lot of time and
code for using ConcurrentUpdateSolrServer with allowing to override default
http settings

On Sun, Apr 29, 2012 at 8:56 PM, Gopal Patwa  wrote:

> In Solr4j client trunk build for 4.0, ConcurrentUpdateSolrServer class
> does not allow to override default http settings
> like HttpConnectionParams.setConnectionTimeout, 
> HttpConnectionParams.setSoTimeout, DefaultMaxConnectionsPerHost
>
>
> Due to HttpSolrServer is not accessible from ConcurrentUpdateSolrServer
> class,  since most of time you just need to override default http settings,
> I know we can pass HttpClient but it would be nice
> if ConcurrentUpdateSolrServer can allow to get access to HttpSolrServer
> from some getter method.
>
> Otherwise anyone who need to override default http settings need to pass
> HttpClient.
>
>
> -Gopal Patwa
>
>
>
>
>

Re: PointType doc reindex issue

[Gopal Patwa]

You need remove field after read solr doc,  when u add new field it will
add to list,  so when u try to commit the update field,  it will be multi
value and in your schema it is single value
On Oct 10, 2012 9:26 AM, "Ravi Solr"  wrote:

> Hello,
> I have a weird problem, Whenever I read the doc from solr and
> then index the same doc that already exists in the index (aka
> reindexing) I get the following error. Can somebody tell me what I am
> doing wrong. I use solr 3.6 and the definition of the field is given
> below
>
>  subFieldSuffix="_coordinate"/>
>  stored="true"/>
>
> Exception in thread "main"
> org.apache.solr.client.solrj.SolrServerException: Server at
> http://testsolr:8080/solr/mycore returned non ok status:400,
> message:ERROR: [doc=1182684] multiple values encountered for non
> multiValued field geolocation_0_coordinate: [39.017608, 39.017608]
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:328)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
> at com.wpost.search.indexing.MyTest.main(MyTest.java:31)
>
>
> The data in the index looks as follows
>
> 39.017608,-77.375239
> 
>  39.017608
>  39.017608
> 
> 
> -77.375239
> -77.375239
> 
>
> Thanks
>
> Ravi Kiran Bhaskar
>

Re: PointType doc reindex issue

[Gopal Patwa]

Instead addfield method use setfield
On Oct 10, 2012 9:54 AM, "Ravi Solr"  wrote:

> Gopal I did in fact test the same and it worked when I delete ted the
> geolocation_0_coordinate and geolocation_1_coordinate. But that seems
> weird, so I was thinking if there is something else I need to do to
> avoid doing this awkward workaround.
>
> Ravi Kiran Bhaskar
>
> On Wed, Oct 10, 2012 at 12:36 PM, Gopal Patwa 
> wrote:
> > You need remove field after read solr doc,  when u add new field it will
> > add to list,  so when u try to commit the update field,  it will be multi
> > value and in your schema it is single value
> > On Oct 10, 2012 9:26 AM, "Ravi Solr"  wrote:
> >
> >> Hello,
> >> I have a weird problem, Whenever I read the doc from solr and
> >> then index the same doc that already exists in the index (aka
> >> reindexing) I get the following error. Can somebody tell me what I am
> >> doing wrong. I use solr 3.6 and the definition of the field is given
> >> below
> >>
> >>  >> subFieldSuffix="_coordinate"/>
> >>  >> stored="true"/>
> >>
> >> Exception in thread "main"
> >> org.apache.solr.client.solrj.SolrServerException: Server at
> >> http://testsolr:8080/solr/mycore returned non ok status:400,
> >> message:ERROR: [doc=1182684] multiple values encountered for non
> >> multiValued field geolocation_0_coordinate: [39.017608, 39.017608]
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:328)
> >> at
> >>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
> >> at com.wpost.search.indexing.MyTest.main(MyTest.java:31)
> >>
> >>
> >> The data in the index looks as follows
> >>
> >> 39.017608,-77.375239
> >> 
> >>  39.017608
> >>  39.017608
> >> 
> >> 
> >> -77.375239
> >> -77.375239
> >> 
> >>
> >> Thanks
> >>
> >> Ravi Kiran Bhaskar
> >>
>

Re: SolrJ, optimize, maxSegments

[Gopal Patwa]

Did you tried below options


  0.0
  10.0


This is from java doc

/** When forceMergeDeletes is called, we only merge away a

   *  segment if its delete percentage is over this

   *  threshold.  Default is 10%. */

  public TieredMergePolicy setForceMergeDeletesPctAllowed(double v) {

if (v < 0.0 || v > 100.0) {

  throw new IllegalArgumentException("forceMergeDeletesPctAllowed must
be between 0.0 and 100.0 inclusive (got " + v + ")");

}

forceMergeDeletesPctAllowed = v;

return this;

  }

On Fri, Oct 12, 2012 at 3:16 PM, Erick Erickson wrote:

> Sounds reasonable although I admit I haven't looked deeply.
>
> Erick
>
> On Fri, Oct 12, 2012 at 3:41 PM, Shawn Heisey  wrote:
> > On 10/12/2012 6:04 AM, Erick Erickson wrote:
> >>
> >> Hmmm, I dug around in the code and found this bit:
> >> *  Forces merging of all segments that have deleted
> >> *  documents.  The actual merges to be executed are
> >> *  determined by the {@link MergePolicy}.  For example,
> >> *  the default {@link TieredMergePolicy} will only
> >> *  pick a segment if the percentage of
> >> *  deleted docs is over 10%.
> >>
> >> see IndexWriter.forceMergeDeletes. So perhaps this limit
> >> was never hit?
> >
> >
> > My own digging based on yours turned up this:
> >
> > https://issues.apache.org/jira/browse/SOLR-2725
> >
> > This sounds like there is currently no way to change this in the Solr
> > config, so it looks like my choices right now are to make a source code
> > change and respin the Solr 3.5.0 that I'm using or just continue to use
> > optimize with no options until SOLR-2725 gets resolved and I can upgrade.
> > Is that right?
> >
> > Thanks,
> > Shawn
> >
>