Fixed my problem, the implementation of solPHP was faulty. It was
sending one doc at a time (one curl per doc) and the system quickly ran
out of resources. Now I modified it to send by batch (1000 at a time)
and everything is #1!
Michael Imbeault wrote:
Old issue (see
http://www.mail-archive.
I sent out an email about this a while back, but basically this limit
appears only on Tomcat and only when Solr attempts to write to the response.
You can work around it by splitting up your posts so that you're posting
less than 5000 (or whatever your limit seems to be) at a time. You DO NOT
ha
: Old issue (see
: http://www.mail-archive.com/solr-user@lucene.apache.org/msg00651.html),
: but I'm experiencing the same exact thing on windows xp, latest tomcat.
did you notice the followup thread from sangraal where he mentioned that
he'd narrowed the problem down to both using Tomcat and sub
:
: The only output I get from 'lsof -p' that pertains to TCP connections are
: the following...I'm not too sure how to interpret it though:
i'm not much of an lsof expert either, but maybe someone else has some
suggestions -- one thing though, it's not just the TCP connections i'm
interested in,
Just an update, I changed my doUpdate method to use the HTTPClient API and
have the exact same problem... the update hangs at exactly the same point...
6,144.
private String doUpdate(String sw) {
StringBuffer updateResult = new StringBuffer();
try {
// open connection
log.info("C
Very interesting... thanks Thom. I haven't given HttpClient a shot yet, but
will be soon.
-S
On 7/31/06, Thom Nelson <[EMAIL PROTECTED]> wrote:
I had a similar problem and was able to fix it in Solr by manually
buffering the responses to a StringWriter before sending it to Tomcat.
Essentially,
Chris, my response is below each of your paragraphs...
I don't have the means to try out this code right now ... but i can't see
any obvious problems with it (there may be somewhere that you are opening
a stream or reader and not closing it, but i didn't see one) ... i notice
you are running th
I had a similar problem and was able to fix it in Solr by manually
buffering the responses to a StringWriter before sending it to Tomcat.
Essentially, Tomcat's buffer will only hold so much and at that point
it blocks (thus it always hangs at a constant number of documents).
However, a better solu
Those are some great ideas Chris... I'm going to try some of them out. I'll
post the results when I get a chance to do more testing. Thanks.
At this point I can work around the problem by ignoring Solr's response but
this is obviously not ideal. I would feel better knowing what is causing the
is
: Sure, the method that does all the work updating Solr is the doUpdate(String
: s) method in the GanjaUpdate class I'm pasting below. It's hanging when I
: try to read the response... the last output I receive in my log is Got
: Reader...
I don't have the means to try out this code right now ...
Does anyone know if the following table is still valid for HttpUrlConnection:
http://www.innovation.ch/java/HTTPClient/urlcon_vs_httpclient.html
If so, there are a couple of advantages to using HTTPClient with Solr:
- direct streaming to/from socket (could be important for very large
requests/res
I'm using HttpClient for indexing and searching and it seems to work well. You can either
POST files directly (only works in 3.1 alpha, use InputStreamRequestEntity in 3.0):
PostMethod post = new PostMethod(solrUrl);
post.setRequestEntity(new FileRequestEntity(file, "application/xml"
On 7/28/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
...Getting all the little details of connection handling correct can be
tough... it's probably a good idea if we work toward common client
libraries so everyone doesn't have to reinvent them
Jakarta's HttpClient [1] is IMHO a good base fo
Yeah that code is pretty bare bones... I'm still in the initial testing
stage. You're right it definitely needs some more thourough work.
I did try removing all the conn.disconnect(); statements and there was no
change.
I'm going to give the Java Client code you sent me yesterday a shot and see
It may be some sort of weird interaction with persistent connections
and timeouts (both client and server have connection timeouts I
assume).
Does anything change if you remove your .disconnect() call (it
shouldn't be needed).
Do you ever see any exceptions in the client side?
The code you show
Sure, the method that does all the work updating Solr is the doUpdate(String
s) method in the GanjaUpdate class I'm pasting below. It's hanging when I
try to read the response... the last output I receive in my log is Got
Reader...
--
package com.iceninetech.solr.update;
import com.icen
: I'm sure... it seems like solr is having trouble writing to a tomcat
: response that's been inactive for a bit. It's only 30 seconds though, so I'm
: not entirely sure why that would happen.
but didn't you say you don't have this problem when you use curl -- just
your java client code?
Did you
I'm sure... it seems like solr is having trouble writing to a tomcat
response that's been inactive for a bit. It's only 30 seconds though, so I'm
not entirely sure why that would happen.
I use the same client code for DL'ing XSL sheets from external servers and
it works fine, but in those instanc
I'll give that a shot...
Thanks again for all your help.
-S
On 7/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
You might also try the Java update client here:
http://issues.apache.org/jira/browse/SOLR-20
-Yonik
On 7/27/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
Commenting out the following line in SolrCore fixes my problem... but of
course I don't get the result status info... but this isn't a problem for me
really.
-Sangraal
writer.write("");
While it's possible you hit a Tomcat bug, I think it'
thing
> Jerry or Jerrymouse.
>
> - Original Message
> From: Mike Klaas < [EMAIL PROTECTED]>
> To: solr-user@lucene.apache.org
> Sent: Thursday, July 27, 2006 6:33:16 PM
> Subject: Re: Doc add limit
>
> Hi Sangraal:
>
> Sorry--I tried not to imply that th
You might also try the Java update client here:
http://issues.apache.org/jira/browse/SOLR-20
-Yonik
MAIL PROTECTED]>
To: solr-user@lucene.apache.org
Sent: Thursday, July 27, 2006 6:33:16 PM
Subject: Re: Doc add limit
Hi Sangraal:
Sorry--I tried not to imply that this might affect your issue. You
may have to crank up the solr logging to determine where it is
freezing (and what might be happen
eir thing Jerry
or Jerrymouse.
- Original Message
From: Mike Klaas <[EMAIL PROTECTED]>
To: solr-user@lucene.apache.org
Sent: Thursday, July 27, 2006 6:33:16 PM
Subject: Re: Doc add limit
Hi Sangraal:
Sorry--I tried not to imply that this might affect your issue. You
may have to cr
Yeah, I'm closing them. Here's the method:
-
private String doUpdate(String sw) {
StringBuffer updateResult = new StringBuffer();
try {
// open connection
log.info("Connecting to and preparing to post to SolrUpdate
servlet.");
URL url = new URL("http://localhost:808
Are you reading the response and closing the connection? If not, you
are probably running out of socket connections.
-Yonik
On 7/27/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
Yonik,
It looks like the problem is with the way I'm posting to the SolrUpdate
servlet. I am able to use curl to pos
I think you're right... I will probably work on splitting the batches up
into smaller pieces at some point in the future. I think I will need the
capability to do large batches at some point though, so I want to make sure
the system can handle it. I also want to make sure this problem doesn't pop
Hi Sangraal:
Sorry--I tried not to imply that this might affect your issue. You
may have to crank up the solr logging to determine where it is
freezing (and what might be happening).
It is certainly worth investigating why this occurs, but I wonder
about the advantages of using such huge batche
Mike,
I've been posting with the content type set like this:
conn.setRequestProperty("Content-Type", "application/octet-stream");
I tried your suggestion though, and unfortunately there was no change.
conn.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
-Sangraal
On 7/
On 7/27/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
class SolrConnection:
def __init__(self, host='localhost:8983', solrBase='/solr'):
self.host = host
self.solrBase = solrBase
#a connection to the server is not opened at this point.
self.conn = httplib.HTTPConnection(self.host
Yonik,
It looks like the problem is with the way I'm posting to the SolrUpdate
servlet. I am able to use curl to post the data to my tomcat instance
without a problem. It only fails when I try to handle the http post from
java... my code is below:
URL url = new URL("http://localhost:8983/sol
On 7/26/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
I removed everything from the Add xml so the docs looked like this:
187880
187852
and it still hung at 6,144...
Maybe you can try the following simple Python client to try and rule
out some kind of different client interactions... the
I removed everything from the Add xml so the docs looked like this:
187880
187852
and it still hung at 6,144...
-S
On 7/26/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
If you narrow the docs down to just the "id" field, does it still
happen at the same place?
-Yonik
On 7/26/06, sangraa
If you narrow the docs down to just the "id" field, does it still
happen at the same place?
-Yonik
On 7/26/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
I see the problem on Mac OS X/JDK: 1.5.0_06 and Debian/JDK: 1.5.0_07.
I don't think it's a socket problem, because I can initiate additional
I see the problem on Mac OS X/JDK: 1.5.0_06 and Debian/JDK: 1.5.0_07.
I don't think it's a socket problem, because I can initiate additional
updates while the server is hung... weird I know.
Thanks for all your help, I'll send a post if/when I find a solution.
-S
On 7/26/06, Yonik Seeley <[EMA
Tomcat problem, or a Solr problem that is only manifesting on your
platform, or a JVM or libc problem, or even a client update problem...
(possibly you might be exhausting the number of sockets in the server
by using persistent connections with a long timeout and never reusing
them?)
What is your
Right now the heap is set to 512M but I've increased it up to 2GB and yet it
still hangs at the same number 6,144...
Here's something interesting... I pushed this code over to a different
server and tried an update. On that server it's hanging on #5,267. Then
tomcat seems to try to reload the web
So it looks like your client is hanging trying to send somethig over
the socket to the server and blocking... probably because Tomcat isn't
reading anything from the socket because it's busy trying to restart
the webapp.
What is the heap size of the server? try increasing it... maybe tomcat
could
Thanks for you help Yonik, I've responded to your questions below:
On 7/26/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
It's possible it's not hanging, but just takes a long time on a
specific add. This is because Lucene will occasionally merge
segments. When very large segments are merged, it
It's possible it's not hanging, but just takes a long time on a
specific add. This is because Lucene will occasionally merge
segments. When very large segments are merged, it can take a long
time.
In the log file, add commands are followed by the number of
milliseconds the operation took. Next
40 matches
Mail list logo
