So it looks like your client is hanging trying to send somethig over
the socket to the server and blocking... probably because Tomcat isn't
reading anything from the socket because it's busy trying to restart
the webapp.
What is the heap size of the server? try increasing it... maybe tomcat
could have detected low memory and tried to reload the webapp.
-Yonik
On 7/26/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
Thanks for you help Yonik, I've responded to your questions below:
On 7/26/06, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>
> It's possible it's not hanging, but just takes a long time on a
> specific add. This is because Lucene will occasionally merge
> segments. When very large segments are merged, it can take a long
> time.
I've left it running (hung) for up to a half hour at a time and I've
verified that my cpu idles during the hang. I have witnessed much shorter
hangs on the ramp up to my 6,144 limit but they have been more like 2 - 10
seconds in length. Perhaps this is the Lucene merging you mentioned.
In the log file, add commands are followed by the number of
> milliseconds the operation took. Next time Solr hangs, wait for a
> number of minutes until you see the operation logged and note how long
> it took.
Here are the last 5 log entries before the hang the last one is doc #6,144.
Also it looks like Tomcat is trying to redeploy the webapp those last tomcat
entries repeat indefinitely every 10 seconds or so. Perhaps this is a Tomcat
problem?
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110705) 0 36596
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110700) 0 36600
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110688) 0 36603
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110690) 0 36608
Jul 26, 2006 1:25:28 PM org.apache.solr.core.SolrCore update
INFO: add (id=110686) 0 36611
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] redeploy resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] redeploy resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT/META-INF/context.xml
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT/WEB-INF/web.xml
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
/webapps/ROOT/META-INF/context.xml
Jul 26, 2006 1:25:36 PM org.apache.catalina.startup.HostConfigcheckResources
FINE: Checking context[] reload resource /source/solr/apache-tomcat-5.5.17
/conf/context.xml
How many documents are in the index before you do a batch that causes
> a hang? Does it happen on the first batch? If so, you might be
> seeing some other bug. What appserver are you using? Do the admin
> pages respond when you see this hang? If so, what does a stack trace
> look like?
I actually don't think I had the problem on the first batch, in fact my
first batch contained very close to 6,144 documents so perhaps there is a
relation there. Right now, I'm adding to an index with close to 90,000
documents in it.
I'm running Tomcat 5.5.17 and the admin pages respond just fine when it's
hung... I did a thread dump and this is the trace of my update:
"http-8080-Processor25" Id=33 in RUNNABLE (running in native) total cpu
time=6330.7360ms user time=5769.5920ms
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.PrintStream.write(PrintStream.java:412)
at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java
:112)
at sun.net.www.http.HttpClient.writeRequests(HttpClient.java:533)
at sun.net.www.protocol.http.HttpURLConnection.writeRequests(
HttpURLConnection.java:410)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(
HttpURLConnection.java:934)
at com.gawker.solr.update.GanjaUpdate.doUpdate(GanjaUpdate.java:169)
at com.gawker.solr.update.GanjaUpdate.update(GanjaUpdate.java:62)
at org.apache.jsp.update_jsp._jspService(update_jsp.java:57)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.jasper.servlet.JspServletWrapper.service(
JspServletWrapper.java:332)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java
:314)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(
Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:613)
-Yonik
>
>
> On 7/26/06, sangraal aiken <[EMAIL PROTECTED]> wrote:
> > Hey there... I'm having an issue with large doc updates on my solr
> > installation. I'm adding in batches between 2-20,000 docs at a time and
> I've
> > noticed Solr seems to hang at 6,144 docs every time. Breaking the adds
> into
> > smaller batches works just fine, but I was wondering if anyone knew why
> this
> > would happen. I've tried doubling memory as well as tweaking various
> config
> > options but nothing seems to let me break the 6,144 barrier.
> >
> > This is the output from Solr admin. Any help would be greatly
> appreciated.
> >
> >
> > *name: * updateHandler *class: *
> > org.apache.solr.update.DirectUpdateHandler2 *version: * 1.0
> *description:
> > * Update handler that efficiently directly updates the on-disk main
> lucene
> > index *stats: *commits : 0
> > optimizes : 0
> > docsPending : 6144
> > deletesPending : 6144
> > adds : 6144
> > deletesById : 0
> > deletesByQuery : 0
> > errors : 0
> > cumulative_adds : 6144
> > cumulative_deletesById : 0
> > cumulative_deletesByQuery : 0
> > cumulative_errors : 0
> > docsDeleted : 0
