Re: Can't get q.op working

[Erik Hatcher]

Kevin,

I've just tried this locally using the tutorial example data, using  
both a default (in schema.xml) of AND and OR.  (I use the Ruby  
response writer because it's easier to read than XML ;)


Use the default operator from schema.xml:


Override with AND:
	


Override with OR:
	


All worked as expected in all cases.  There is one result with AND  
and three results with OR.


I recommend you try this same scenario out with the tutorial example  
data and ensure things work as I've stated here.  Let us know more  
details if the problem persists.


Erik


On Sep 26, 2006, at 11:02 PM, Kevin Lewandowski wrote:


I'm running the latest nightly build (2006-09-27) and cannot seem to
get the q.op parameter working. I have the default operator set to AND
and am testing with a two word query that returns no results. If I add
"OR" to the query I get results. But if I remove the OR and add
"q.op=OR" to the Solr query I still get no results.

Is there anything I could be doing wrong?

thanks
Kevin

Re: Extending Solr's Admin functionality

[Simon Willnauer]

@Otis: I suggest we go a bit more in detail about the features solr
should expose via JMX and talk about the contribution. I'd love to
extend solr with more JMX support.



On 9/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:

On 9/26/06, Otis Gospodnetic <[EMAIL PROTECTED]> wrote:
> On the other hand, some people I talked to also expressed interest in JMX, so 
I'd encourage Simon to make that contribution.

I'm also interested in JMX.
It has different adapters, including an HTTP one AFAIK, but I don't
know how easy it is to use.


The application should only provide mbeans as an interface for the JMX
kernel to expose these interfaces to the adapter. Which adapter you
use depends on you personal preferences. There are lots of JMX Monitor
apps around with http adaptors like mx4j (http://mx4j.sourceforge.net)
if deployed in the same container all mbeans are exposed to the
connector via the mbean registry / server.

@Yonik: What are you interests in JMX?

best regards Simon


-Yonik

Re: Extending Solr's Admin functionality

[Yonik Seeley]

On 9/27/06, Simon Willnauer <[EMAIL PROTECTED]> wrote:

@Yonik: What are you interests in JMX?


More standard & easier integration with many different tools
(scripting languages, 3rd party consoles, etc).  The hard part is
figuring out exactly what should be exported as an MBean.

-Yonik

Re: Is solr scalable with respect to number of documents?

[Vish D.]

Are there any plans on implementing a MultiSearcher into solr?

I have been following the list for a while, and read quite a few topic on
multiple instances of solr, in order to accomodate multiple schemas as well
as break down index sizes for performance reasons. I have a use case that
sits right in the middle of these. I need to be able to break down the data
into collections, which might be defined by business factors, but also be
able search across these collections. When I say search across, I would
expect to see a results list ordered in relevance, just as you would expect
when searching a single collection.

Does solr support such a use case? Does anyone have a working implementation
of it that you would like to share? If there aren't any plans on
implementing the above, any suggestions on tackling the above requirement?



On 7/11/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:


On 7/11/06, Wang, Ningjun (LNG-NPV) <[EMAIL PROTECTED]> wrote:
> Can I solve this problem by using lucene directly without solr? In
> another word, does lucene offer and API to merge search results from
> different machine?

A MultiSearcher over multiple RemoteSearchers can work in Lucene.

-Yonik

Re: Extending Solr's Admin functionality

[Erik Hatcher]

Ah, so I'm beginning to get it.  If we build Solr with JMX support,  
the admin HTTP/XML(err, Ruby) interface could be written into the JMX  
HTTP adapter as a separate web application, and allowing users to  
plug it in or not.  If I'm understanding that correctly then I'm  
quite +1 on JMX!  And I suppose some of these adapters already have  
built in web service interfaces.


Erik


On Sep 27, 2006, at 6:20 AM, Simon Willnauer wrote:


@Otis: I suggest we go a bit more in detail about the features solr
should expose via JMX and talk about the contribution. I'd love to
extend solr with more JMX support.



On 9/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:

On 9/26/06, Otis Gospodnetic <[EMAIL PROTECTED]> wrote:
> On the other hand, some people I talked to also expressed  
interest in JMX, so I'd encourage Simon to make that contribution.


I'm also interested in JMX.
It has different adapters, including an HTTP one AFAIK, but I don't
know how easy it is to use.


The application should only provide mbeans as an interface for the JMX
kernel to expose these interfaces to the adapter. Which adapter you
use depends on you personal preferences. There are lots of JMX Monitor
apps around with http adaptors like mx4j (http://mx4j.sourceforge.net)
if deployed in the same container all mbeans are exposed to the
connector via the mbean registry / server.

@Yonik: What are you interests in JMX?

best regards Simon


-Yonik

Re: Is solr scalable with respect to number of documents?

[Vish D.]

I just noticed that link on the first reply from Yonik about
FederatedSearch. I see that a lot of thought went in to it. I guess the
question to ask would be, any progress on it, Yonik? :)

On 9/27/06, Vish D. <[EMAIL PROTECTED]> wrote:


Are there any plans on implementing a MultiSearcher into solr?

I have been following the list for a while, and read quite a few topic on
multiple instances of solr, in order to accomodate multiple schemas as well
as break down index sizes for performance reasons. I have a use case that
sits right in the middle of these. I need to be able to break down the data
into collections, which might be defined by business factors, but also be
able search across these collections. When I say search across, I would
expect to see a results list ordered in relevance, just as you would expect
when searching a single collection.

Does solr support such a use case? Does anyone have a working
implementation of it that you would like to share? If there aren't any plans
on implementing the above, any suggestions on tackling the above
requirement?



On 7/11/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>
> On 7/11/06, Wang, Ningjun (LNG-NPV) <[EMAIL PROTECTED]> wrote:
> > Can I solve this problem by using lucene directly without solr? In
> > another word, does lucene offer and API to merge search results from
> > different machine?
>
> A MultiSearcher over multiple RemoteSearchers can work in Lucene.
>
> -Yonik
>

Re: Can't get q.op working

[Kevin Lewandowski]

Now I feel dumb! I hadn't deployed the latest build properly. The new
.war file was there but for some reason restarting tomcat didn't
reload it. Anyway, q.op is working fine now.

On 9/27/06, Erik Hatcher <[EMAIL PROTECTED]> wrote:

Kevin,

I've just tried this locally using the tutorial example data, using
both a default (in schema.xml) of AND and OR.  (I use the Ruby
response writer because it's easier to read than XML ;)

Use the default operator from schema.xml:


Override with AND:


Override with OR:


All worked as expected in all cases.  There is one result with AND
and three results with OR.

I recommend you try this same scenario out with the tutorial example
data and ensure things work as I've stated here.  Let us know more
details if the problem persists.

Erik


On Sep 26, 2006, at 11:02 PM, Kevin Lewandowski wrote:

> I'm running the latest nightly build (2006-09-27) and cannot seem to
> get the q.op parameter working. I have the default operator set to AND
> and am testing with a two word query that returns no results. If I add
> "OR" to the query I get results. But if I remove the OR and add
> "q.op=OR" to the Solr query I still get no results.
>
> Is there anything I could be doing wrong?
>
> thanks
> Kevin

Re: Extending Solr's Admin functionality

[Simon Willnauer]

First I agree with yonik, the main point is to define which classes /
parts / mbeans should be exposed to JMX is the hard part and should be
planned carefully. I could imagine a very flexible layer between jmx
and solr using 1.5 annotations and an integration of commons-modeler.

Erik, am I get you right that you want to connect to jmx via an extern
webapplication which acts as an admin interface (written in ruby or
whatever) or are you pointing to a http/XML connector for jmx?
JSR-160 permits extensions to the way in which communication is done
between the client and the server. Basic implementations are using the
mandatory RMI-based implementation required by the JSR-160
specification (IIOP and JRMP) and the (optional) JMXMP. By using other
providers or JMX implementations (such as MX4J) you can take advantage
of protocols like SOAP, Hessian, Burlap over simple HTTP or SSL and
others. (http://mx4j.sourceforge.net)

best regards simon



On 9/27/06, Erik Hatcher <[EMAIL PROTECTED]> wrote:

Ah, so I'm beginning to get it.  If we build Solr with JMX support,
the admin HTTP/XML(err, Ruby) interface could be written into the JMX
HTTP adapter as a separate web application, and allowing users to
plug it in or not.  If I'm understanding that correctly then I'm
quite +1 on JMX!  And I suppose some of these adapters already have
built in web service interfaces.

Erik


On Sep 27, 2006, at 6:20 AM, Simon Willnauer wrote:

> @Otis: I suggest we go a bit more in detail about the features solr
> should expose via JMX and talk about the contribution. I'd love to
> extend solr with more JMX support.
>
>
>
> On 9/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>> On 9/26/06, Otis Gospodnetic <[EMAIL PROTECTED]> wrote:
>> > On the other hand, some people I talked to also expressed
>> interest in JMX, so I'd encourage Simon to make that contribution.
>>
>> I'm also interested in JMX.
>> It has different adapters, including an HTTP one AFAIK, but I don't
>> know how easy it is to use.
>
> The application should only provide mbeans as an interface for the JMX
> kernel to expose these interfaces to the adapter. Which adapter you
> use depends on you personal preferences. There are lots of JMX Monitor
> apps around with http adaptors like mx4j (http://mx4j.sourceforge.net)
> if deployed in the same container all mbeans are exposed to the
> connector via the mbean registry / server.
>
> @Yonik: What are you interests in JMX?
>
> best regards Simon
>>
>> -Yonik
>>

Re: Can't get q.op working

[Erik Hatcher]

On Sep 27, 2006, at 12:06 PM, Kevin Lewandowski wrote:


Now I feel dumb! I hadn't deployed the latest build properly. The new
.war file was there but for some reason restarting tomcat didn't
reload it. Anyway, q.op is working fine now.


hey, no worries... it happens to all of us.  i'm just glad that was  
the solution and not a solr bug ;)





On 9/27/06, Erik Hatcher <[EMAIL PROTECTED]> wrote:

Kevin,

I've just tried this locally using the tutorial example data, using
both a default (in schema.xml) of AND and OR.  (I use the Ruby
response writer because it's easier to read than XML ;)

Use the default operator from schema.xml:



Override with AND:


Override with OR:


All worked as expected in all cases.  There is one result with AND
and three results with OR.

I recommend you try this same scenario out with the tutorial example
data and ensure things work as I've stated here.  Let us know more
details if the problem persists.

Erik


On Sep 26, 2006, at 11:02 PM, Kevin Lewandowski wrote:

> I'm running the latest nightly build (2006-09-27) and cannot  
seem to
> get the q.op parameter working. I have the default operator set  
to AND
> and am testing with a two word query that returns no results. If  
I add

> "OR" to the query I get results. But if I remove the OR and add
> "q.op=OR" to the Solr query I still get no results.
>
> Is there anything I could be doing wrong?
>
> thanks
> Kevin

Re: Extending Solr's Admin functionality

[Erik Hatcher]

On Sep 27, 2006, at 12:07 PM, Simon Willnauer wrote:

Erik, am I get you right that you want to connect to jmx via an extern
webapplication which acts as an admin interface (written in ruby or
whatever) or are you pointing to a http/XML connector for jmx?


Right.  I'd love to provide Solr applications using Ruby on Rails  
(Solr on Rails, heh) with all the information it can get from Solr.


Erik



JSR-160 permits extensions to the way in which communication is done
between the client and the server. Basic implementations are using the
mandatory RMI-based implementation required by the JSR-160
specification (IIOP and JRMP) and the (optional) JMXMP. By using other
providers or JMX implementations (such as MX4J) you can take advantage
of protocols like SOAP, Hessian, Burlap over simple HTTP or SSL and
others. (http://mx4j.sourceforge.net)

best regards simon



On 9/27/06, Erik Hatcher <[EMAIL PROTECTED]> wrote:

Ah, so I'm beginning to get it.  If we build Solr with JMX support,
the admin HTTP/XML(err, Ruby) interface could be written into the JMX
HTTP adapter as a separate web application, and allowing users to
plug it in or not.  If I'm understanding that correctly then I'm
quite +1 on JMX!  And I suppose some of these adapters already have
built in web service interfaces.

Erik


On Sep 27, 2006, at 6:20 AM, Simon Willnauer wrote:

> @Otis: I suggest we go a bit more in detail about the features solr
> should expose via JMX and talk about the contribution. I'd love to
> extend solr with more JMX support.
>
>
>
> On 9/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>> On 9/26/06, Otis Gospodnetic <[EMAIL PROTECTED]> wrote:
>> > On the other hand, some people I talked to also expressed
>> interest in JMX, so I'd encourage Simon to make that contribution.
>>
>> I'm also interested in JMX.
>> It has different adapters, including an HTTP one AFAIK, but I  
don't

>> know how easy it is to use.
>
> The application should only provide mbeans as an interface for  
the JMX

> kernel to expose these interfaces to the adapter. Which adapter you
> use depends on you personal preferences. There are lots of JMX  
Monitor
> apps around with http adaptors like mx4j (http:// 
mx4j.sourceforge.net)

> if deployed in the same container all mbeans are exposed to the
> connector via the mbean registry / server.
>
> @Yonik: What are you interests in JMX?
>
> best regards Simon
>>
>> -Yonik
>>

Re: Extending Solr's Admin functionality

[Walter Underwood]

On 9/27/06 9:07 AM, "Simon Willnauer" <[EMAIL PROTECTED]>
wrote:
>
> First I agree with yonik, the main point is to define which classes /
> parts / mbeans should be exposed to JMX is the hard part and should be
> planned carefully.

That is the hard part regardless of whether we use JMX or bare-metal
HTTP+XML. What operations are exposed for admin?

wunder
--
Search Guru, Netflix

Re: Extending Solr's Admin functionality

[Erik Hatcher]

On Sep 27, 2006, at 12:42 PM, Walter Underwood wrote:


On 9/27/06 9:07 AM, "Simon Willnauer" <[EMAIL PROTECTED]>
wrote:


First I agree with yonik, the main point is to define which classes /
parts / mbeans should be exposed to JMX is the hard part and  
should be

planned carefully.


That is the hard part regardless of whether we use JMX or bare-metal
HTTP+XML. What operations are exposed for admin?


Well, for long-hanging fruit (autumn Virginia apples):

  * In a read-only fashion, everything from stats.jsp and registry.jsp

This could already be done by making a request handler do what those  
JSP pages do - piece of cake.  But I can see Solr also providing more  
interactive type of administration in the future, such as providing  
analysis capabilities, spell checking, and (*gasp*) dynamic  
configuration of the schema and configuration even.


Erik

Re: Extending Solr's Admin functionality

[Erik Hatcher]

On Sep 27, 2006, at 1:49 PM, Erik Hatcher wrote:

Well, for long-hanging fruit (autumn Virginia apples):


uh, it's been a "long" day: 


but i meant low-hanging fruit.


  * In a read-only fashion, everything from stats.jsp and registry.jsp

This could already be done by making a request handler do what  
those JSP pages do - piece of cake.  But I can see Solr also  
providing more interactive type of administration in the future,  
such as providing analysis capabilities, spell checking, and  
(*gasp*) dynamic configuration of the schema and configuration even.

Re: Is solr scalable with respect to number of documents?

[Yonik Seeley]

On 9/27/06, Vish D. <[EMAIL PROTECTED]> wrote:

I just noticed that link on the first reply from Yonik about
FederatedSearch. I see that a lot of thought went in to it. I guess the
question to ask would be, any progress on it, Yonik? :)


No code, but great progress at shooting holes in various strategies ;-)

I'm currently thinking about doing federated search at a higher level,
with slightly modified standard request handlers, and another
top-level request handler that can combine requests.  The biggest
downside: no custom query handlers.

The other option: do federated search like a lucene MultiSearcher...
(a federated version of the SolrIndexSearcher).  The downside is that
existing interfaces would not be usable... we can't be shipping tons
of BitDocSets across the network.  Things like highlighting, federated
search, etc, would need to be pushed down into this interface.  New
interfaces means lots of changes to request handler code.  Upside
would be that custom request handlers would still work and be
automatically parallelized.

Anyone have any thoughts on this stuff?
http://wiki.apache.org/solr/FederatedSearch

-Yonik

Price facet counts based on price[] query or text field

[Mike Austin]

I'm trying to figure out the best way to do filters based on price.  I've
been thinking that I would do it based on the query such as price[10-20],
however I wanted opinions on whether it would be better to do it based on
pre-filtering the item into a price range and searching for the string that
represents the price range in a multivalued text field in the schema (beside
other filtering values)? I was thinking that pre-filtering or grouping items
into the respected price range would be more work operationally but might
pay off with better performance. It would limit my searches to only the
predefined price ranges, however I don't know if that is a big problem.

Maybe some info on how other people do this and why would be helpful?

Thanks,
Mike

Re: Price facet counts based on price[] query or text field

[Chris Hostetter]

if you are looking at totally cold searches, with no caching, then you
might see a small performance using pre-defined constraints flagged with
special terms then with range filters -- just becuase looking up the
documents matching one term is allways going to be a little faster then
looking up the documents that match a range of terms -- but that
differnece is probably going to be so small you won't even notice it.

Even if you do find that it's measurable given your index size, the
headaches of needing to reindex all of your data everytime you decide to
tweak your price ranges may overshadow the slight performance gains.

That said: as long as you've got some memory to spare, you should never
really need to worry about the speed of totally cold searches with no
caching.  Once the filterCache is populated with sets for each price facet
constraint, it won't matter if that constraint was specified with a range
or with a term.



: Date: Wed, 27 Sep 2006 17:44:11 -0500
: From: Mike Austin <[EMAIL PROTECTED]>
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Price facet counts based on price[] query or text field
:
: I'm trying to figure out the best way to do filters based on price.  I've
: been thinking that I would do it based on the query such as price[10-20],
: however I wanted opinions on whether it would be better to do it based on
: pre-filtering the item into a price range and searching for the string that
: represents the price range in a multivalued text field in the schema (beside
: other filtering values)? I was thinking that pre-filtering or grouping items
: into the respected price range would be more work operationally but might
: pay off with better performance. It would limit my searches to only the
: predefined price ranges, however I don't know if that is a big problem.
:
: Maybe some info on how other people do this and why would be helpful?
:
: Thanks,
: Mike
:



-Hoss

How much ram can Solr use?

[Kevin Lewandowski]

On the performace wiki page it mentions a test box with 16GB ram. Did
anything special need to be done to use that much ram (with the OS or
java)? Would Solr on a system with Linux x86_64 and Tomcat be able to
use that much ram? (sorry, I don't know Java so I don't know if there
are any limitations there).

thanks,
Kevin

Re: How much ram can Solr use?

[Otis Gospodnetic]

Yes, you'll need a 64-bit machine in order to use over 2GB of JVM heap.  You 
can set the maximum with -Xmx16g for example.  Try that on a 32-bit machines, 
and you'll immediately get an error from JVM telling you to go get yourself a 
64-bit box.

Otis

- Original Message 
From: Kevin Lewandowski <[EMAIL PROTECTED]>
To: solr-user@lucene.apache.org
Sent: Wednesday, September 27, 2006 7:16:53 PM
Subject: How much ram can Solr use?

On the performace wiki page it mentions a test box with 16GB ram. Did
anything special need to be done to use that much ram (with the OS or
java)? Would Solr on a system with Linux x86_64 and Tomcat be able to
use that much ram? (sorry, I don't know Java so I don't know if there
are any limitations there).

thanks,
Kevin

Re: How much ram can Solr use?

[Yonik Seeley]

On 9/27/06, Kevin Lewandowski <[EMAIL PROTECTED]> wrote:

On the performace wiki page it mentions a test box with 16GB ram. Did
anything special need to be done to use that much ram (with the OS or
java)? Would Solr on a system with Linux x86_64 and Tomcat be able to
use that much ram? (sorry, I don't know Java so I don't know if there
are any limitations there).


Yep, works fine  We don't use the full 16GB for the java heap
though because:
- you want to leave RAM for the OS cache of the index files
- bigger heaps = longer gc pauses (at least with Java5).

-Yonik

error on optimize

[steve.whitley]

We're having something strange with our SOLR instance.  

 

When we post a commit, we get an empty reply from the server:

 

$ curl http://search-campaign.unitedeway.org/solr/update --data-binary
''

curl: (52) Empty reply from server

 

When we post the optimize xml the following stack trace is returned:

 

$ curl http://search-campaign.unitedeway.org/solr/update --data-binary
''

java.io.IOException: Lock obtain timed out:
Lock@/web/search/campaign/jetty.tmp/lucene-d1bba62e1f2e75d919a17dcaa15a9
1a7-write.lock

at org.apache.lucene.store.Lock.obtain(Lock.java:56)

at
org.apache.lucene.index.IndexWriter.(IndexWriter.java:256)

at
org.apache.lucene.index.IndexWriter.(IndexWriter.java:206)

at
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:65
)

at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler
.java:118)

at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandl
er2.java:153)

at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.
java:458)

at org.apache.solr.core.SolrCore.update(SolrCore.java:755)

at
org.apache.solr.servlet.SolrUpdateServlet.doPost(SolrUpdateServlet.java:
52)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)

at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)

at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)

at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)

at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:1
64)

at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)

at org.mortbay.jetty.Server.handle(Server.java:309)

at org.mortbay.jetty.Server.handle(Server.java:285)

at
org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)

at
org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)

at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.j
ava:625)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)

at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)

at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)

at
org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChan
nelConnector.java:680)

at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.ja
va:412)



 

Really at a loss as what to do.  Our cache is huge and we'd like to
optimize things up a bit.

 

Thoughts?

 

--Steve

Re: error on optimize

[Yonik Seeley]

Hi Steve,
It looks like the commit is taking a long time and jetty is timing it out.

See this thread:
http://www.nabble.com/Synchronizing-commit-and-optimize-tf1498513.html#a4067023

-Yonik

On 9/27/06, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:

We're having something strange with our SOLR instance.



When we post a commit, we get an empty reply from the server:



$ curl http://search-campaign.unitedeway.org/solr/update --data-binary
''

curl: (52) Empty reply from server



When we post the optimize xml the following stack trace is returned:



$ curl http://search-campaign.unitedeway.org/solr/update --data-binary
''

java.io.IOException: Lock obtain timed out:
Lock@/web/search/campaign/jetty.tmp/lucene-d1bba62e1f2e75d919a17dcaa15a9
1a7-write.lock

at org.apache.lucene.store.Lock.obtain(Lock.java:56)

at
org.apache.lucene.index.IndexWriter.(IndexWriter.java:256)

at
org.apache.lucene.index.IndexWriter.(IndexWriter.java:206)

at
org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:65
)

at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler
.java:118)

at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandl
er2.java:153)

at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.
java:458)

at org.apache.solr.core.SolrCore.update(SolrCore.java:755)

at
org.apache.solr.servlet.SolrUpdateServlet.doPost(SolrUpdateServlet.java:
52)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)

at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)

at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)

at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)

at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:1
64)

at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)

at org.mortbay.jetty.Server.handle(Server.java:309)

at org.mortbay.jetty.Server.handle(Server.java:285)

at
org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)

at
org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)

at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.j
ava:625)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)

at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)

at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)

at
org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChan
nelConnector.java:680)

at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.ja
va:412)





Really at a loss as what to do.  Our cache is huge and we'd like to
optimize things up a bit.



Thoughts?



--Steve

Re: error on optimize

[Yonik Seeley]

Steve, something else I notice:

http://search-campaign.unitedeway.org/solr/admin/stats.jsp
shows that there are *many* SolrIndexSearchers open (and that's going
to take up a lot of menory).

There should normally be 1 and occasionally 2 when autowarming.  There
is also sometimes an additional non-caching SolrIndexSearcher opened
by the UpdateHandler for use in deleting documents.

I've never seen this happen.  Is this stock Solr, or have any
modifications been made?
Do you have logs to check to see what error messages or exceptions may
appear?  Search for "Memory" to see if you hit an out-of-memory error
too.

Oh, and there's no security in Solr, so I wouldn't expose it directly
to the internet indefinitely w/o something protecting it from vandals
:-)

-Yonik

On 9/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:

Hi Steve,
It looks like the commit is taking a long time and jetty is timing it out.

See this thread:
http://www.nabble.com/Synchronizing-commit-and-optimize-tf1498513.html#a4067023

-Yonik

On 9/27/06, [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:
> We're having something strange with our SOLR instance.
>
>
>
> When we post a commit, we get an empty reply from the server:
>
>
>
> $ curl http://search-campaign.unitedeway.org/solr/update --data-binary
> ''
>
> curl: (52) Empty reply from server
>
>
>
> When we post the optimize xml the following stack trace is returned:
>
>
>
> $ curl http://search-campaign.unitedeway.org/solr/update --data-binary
> ''
>
> java.io.IOException: Lock obtain timed out:
> Lock@/web/search/campaign/jetty.tmp/lucene-d1bba62e1f2e75d919a17dcaa15a9
> 1a7-write.lock
>
> at org.apache.lucene.store.Lock.obtain(Lock.java:56)
>
> at
> org.apache.lucene.index.IndexWriter.(IndexWriter.java:256)
>
> at
> org.apache.lucene.index.IndexWriter.(IndexWriter.java:206)
>
> at
> org.apache.solr.update.SolrIndexWriter.(SolrIndexWriter.java:65
> )
>
> at
> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler
> .java:118)
>
> at
> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandl
> er2.java:153)
>
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.
> java:458)
>
> at org.apache.solr.core.SolrCore.update(SolrCore.java:755)
>
> at
> org.apache.solr.servlet.SolrUpdateServlet.doPost(SolrUpdateServlet.java:
> 52)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:767)
>
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:860)
>
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:408)
>
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:350)
>
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:195)
>
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:1
> 64)
>
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:536)
>
> at org.mortbay.jetty.Server.handle(Server.java:309)
>
> at org.mortbay.jetty.Server.handle(Server.java:285)
>
> at
> org.mortbay.jetty.HttpConnection.doHandler(HttpConnection.java:363)
>
> at
> org.mortbay.jetty.HttpConnection.access$1600(HttpConnection.java:45)
>
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.j
> ava:625)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:613)
>
> at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:195)
>
> at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:297)
>
> at
> org.mortbay.jetty.nio.SelectChannelConnector$HttpEndPoint.run(SelectChan
> nelConnector.java:680)
>
> at
> org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.ja
> va:412)
>
> 
>
>
>
> Really at a loss as what to do.  Our cache is huge and we'd like to
> optimize things up a bit.
>
>
>
> Thoughts?
>
>
>
> --Steve

Re: How much ram can Solr use?

[Ken Krugler]

On 9/27/06, Kevin Lewandowski <[EMAIL PROTECTED]> wrote:

On the performace wiki page it mentions a test box with 16GB ram. Did
anything special need to be done to use that much ram (with the OS or
java)? Would Solr on a system with Linux x86_64 and Tomcat be able to
use that much ram? (sorry, I don't know Java so I don't know if there
are any limitations there).


Yep, works fine  We don't use the full 16GB for the java heap
though because:
- you want to leave RAM for the OS cache of the index files
- bigger heaps = longer gc pauses (at least with Java5).


Yes, we came to exactly the same conclusion.

You could try to use all of that RAM in the JVM, and load all of the 
index with a RAMDirectory (or MMapDirectory), but both of those wound 
up being slower than the configuration where we had a smaller JVM and 
relied more on the OS-level cache to make index accesses really fast.


One trick I've heard is to do cat directory>/_*.* > /dev/null to force all of the index data into the 
OS cache.


-- Ken
--
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"Find Code, Find Answers"

Re: How much ram can Solr use?

[WHIRLYCOTT]

Does that actually work?

phil.

On Sep 27, 2006, at 11:27 PM, Ken Krugler wrote:

One trick I've heard is to do cat / 
_*.* > /dev/null to force all of the index data into the OS cache.



--
   Whirlycott
   Philip Jacob
   [EMAIL PROTECTED]
   http://www.whirlycott.com/phil/