I have implemented dataimporthandler scheduling based on
http://wiki.apache.org/solr/DataImportHandler#Scheduling. It
periodically triggers full and delta updates. I'm unpacking the
original solr.war, adding a few scheduling-related classes such as
ApplicationListener etc (I have modified the example a lot) and
repacking the web application.
The scheduling works fine, but when I undeploy solr web application,
Tomcat gives errors about ThreadLocals that were not cleared:
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [org.apache.solr.handler.dataimport.DataImporter$2] (value
[org.apache.solr.
handler.dataimport.DataImporter$2@b0e2096]) and a value of type
[java.util.concurrent.atomic.AtomicLong] (value [2]) but failed to
remove it when the web applic
ation was stopped. Threads are going to be renewed over time to try
and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [org.apache.solr.handler.dataimport.DataImporter$3] (value
[org.apache.solr.
handler.dataimport.DataImporter$3@4c7d5d85]) and a value of type
[java.text.SimpleDateFormat] (value
[java.text.SimpleDateFormat@4f76f1a0]) but failed to remove
it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@3a86edfe]) and a value
of type [org.apache.solr.handler.dataimport.ContextImpl] (value
[org.apache.solr.handler.dataimport.ContextImpl@7072dcb6]) but failed
to remove it when the web
application was stopped. Threads are going to be renewed over time to
try and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [org.apache.solr.schema.DateField.ThreadLocalDateFormat]
(value [org.apache.
solr.schema.DateField$ThreadLocalDateFormat@4f86a67]) and a value of
type [org.apache.solr.schema.DateField.ISO8601CanonicalDateFormat]
(value [org.apache.solr.
schema.DateField$ISO8601CanonicalDateFormat@6b2ed43a]) but failed to
remove it when the web application was stopped. Threads are going to
be renewed over time t
o try and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [org.apache.solr.handler.dataimport.DataImporter$2] (value
[org.apache.solr.
handler.dataimport.DataImporter$2@b0e2096]) and a value of type
[java.util.concurrent.atomic.AtomicLong] (value [2]) but failed to
remove it when the web applic
ation was stopped. Threads are going to be renewed over time to try
and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [java.lang.ThreadLocal] (value
[java.lang.ThreadLocal@3a86edfe]) and a value
of type [org.apache.solr.handler.dataimport.ContextImpl] (value
[org.apache.solr.handler.dataimport.ContextImpl@511192bd]) but failed
to remove it when the web
application was stopped. Threads are going to be renewed over time to
try and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [org.apache.solr.handler.dataimport.DataImporter$3] (value
[org.apache.solr.
handler.dataimport.DataImporter$3@4c7d5d85]) and a value of type
[java.text.SimpleDateFormat] (value
[java.text.SimpleDateFormat@4f76f1a0]) but failed to remove
it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
3.4.2012 18:36:49 org.apache.catalina.loader.WebappClassLoader
checkThreadLocalMapForLeaks
SEVERE: The web application [/my-solr] created a ThreadLocal with key
of type [org.apache.solr.schema.DateField.ThreadLocalDateFormat]
(value [org.apache.
solr.schema.DateField$ThreadLocalDateFormat@4f86a67]) and a value of
type [org.apache.solr.schema.DateField.ISO8601CanonicalDateFormat]
(value [org.apache.solr.
schema.DateField$ISO8601CanonicalDateFormat@6b2ed43a]) but failed to
remove it when the web application was stopped. Threads are going to
be renewed over time t
o try and avoid a probable memory leak.
I have rechecked my code to make sure it should not have any memory
leaks. I have identified the cause to method:
private void sendHttpPost(String completeUrl, String coreName) {
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(completeUrl);
try {
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
method.getResponseBody();
} catch (HttpException e) {
System.err.println("Fatal protocol violation: " + e.getMessage());
e.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}
}
If I comment out that code, the errors about memory leaks disappear.
This method simply starts solr indexing operation by calling localhost
url such as my-solr/mycore/dataimport?command=full-import
The method is rewritten to use HttpClient but I got the same mem leak
errors when I used the original HTTPPostScheduler class from
http://wiki.apache.org/solr/DataImportHandler#Scheduling
So it looks like the mem leak errors do not come from my added code
but from original Solr code (which receives the http requests).
What is even more confusing is that if I disable the sendHttpPost(),
the errors disappear as I mentioned - but if I manually request the
same urls to initiate the full / delta imports, the mem leak errors do
not show up when webapp is undeployed....??
