Re: Deleting Fields

Joseph Obernberger Mon, 01 Jun 2015 06:15:08 -0700

Hi - we are using 64bit OS and 64bit JVM.  The JVM settings are currently:
-----------------------------------------
-DSTOP.KEY=solrrocks
-DSTOP.PORT=8100
-Dhost=helios
-Djava.net.preferIPv4Stack=true
-Djetty.port=9100
-DnumShards=27
-Dsolr.clustering.enabled=true
-Dsolr.install.dir=/opt/solr
-Dsolr.lock.type=hdfs
-Dsolr.solr.home=/opt/solr/server/solr
-Duser.timezone=UTC-DzkClientTimeout=15000
-DzkHost=eris.querymasters.com:2181,daphnis.querymasters.com:2181,triton.querymasters.com:2181,oberon.querymasters.com:2181,portia.querymasters.com:2181,puck.querymasters.com:2181/solr5
-XX:+CMSParallelRemarkEnabled
-XX:+CMSScavengeBeforeRemark
-XX:+ParallelRefProcEnabled
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseConcMarkSweepGC
-XX:+UseLargePages
-XX:+UseParNewGC-XX:CMSFullGCsBeforeCompaction=1
-XX:CMSInitiatingOccupancyFraction=50
-XX:CMSMaxAbortablePrecleanTime=6000
-XX:CMSTriggerPermRatio=80
-XX:ConcGCThreads=8
-XX:MaxDirectMemorySize=26g
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 9100 /opt/solr/server/logs
-XX:ParallelGCThreads=8
-XX:PretenureSizeThreshold=64m
-XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-Xloggc:/opt/solr/server/logs/solr_gc.log
-Xms8g
-Xmx16g
-Xss256k
-verbose:gc
-----------------------------------------

At the time out of the OOM error, Xmx was set to 10g. OS limits, forthe most part, are 'factory' Scientific Linux 6.6. I didn't see anymessages in the log about too many open files. Thank you for the tips!


-Joe

On 5/31/2015 4:24 AM, Tomasz Borek wrote:

Joseph,

You are doing a memory intensive operation and perhaps an IO intensive
operation at once. That makes your C-heap run out of memory or hit a thread
limit (thus first problem, java.lang.OutOfMemoryError: unable to create new
native thread) and later you're also hitting the problem of Java heap being
full or - more precisely - GC being unable to free enough space there even
with collection, to allocate new object that you want allocated (thus
second throw: java.lang.OutOfMemoryError: Java heap space).

Important is:
- whether your OS is 32-bit or 64-bit
- whether your JVM is 32-bit or 64-bit
- what are your OS limits on thread creation and have you touched them or
changed them (1st problem)
- how do you start JVM (Xms, Xmx, Xss, dumps, direct memory, permgen size -
both problems)

What you can do to solve your problems differs depending on what exactly
causes them, but in general:

NATIVE:
1) Either your operation causes many threads to spawn and you hit your
thread limit (OS limits how many threads process can create) - have a
threaddump and see
2) Or your op causes many thread creation and the memory settings you start
JVM with plus 32/64 bits of OS and JVM make it impossible for the C-heap to
have this much memory thus you're hitting the OOM error - adjust settings,
move to 64-bit architectures, add RAM while on 64-bit (32-bit really chokes
you down, less than 4GB is available for you for EVERYTHING: Java heap,
PermGen space AND C-Heap)

Usually, with such thread-greedy operation, it's also nice to look at the
code and see if perhaps one can optimize thread creation/management.

OOM on Java heap:
Add crash on memory dump parameter to your JVM and walk dominator tree or
see histogram to actually tell what's eating your heap space. MAT is a good
tool for this, unless your heap is like 150GB, then Netbeans may help or
see Alexey Ragozin's work, I think he forked the code for NetBeans heap
analyzer and made some adjustments specially for such cases. Light Google
search and here it is: http://blog.ragozin.info/


pozdrawiam,
LAFK

2015-05-30 20:48 GMT+02:00 Erick Erickson <erickerick...@gmail.com>:

Faceting on very high cardinality fields can use up memory, no doubt
about that. I think the entire delete question was a red herring, but
you know that already ;)....

So I think you can forget about the delete stuff. Although do note
that if you do re-index your old documents, the new version won't have
the field, and as segments are merged the deleted documents will have
all their resources reclaimed, effectively deleting the field from the
old docs.... So you could gradually re-index your corpus and get this
stuff out of there.

Best,
Erick

On Sat, May 30, 2015 at 5:18 AM, Joseph Obernberger
<j...@lovehorsepower.com> wrote:

Thank you Erick.  I was thinking that it actually went through and

removed

the index data; that you for the clarification.  What happened was I had
some bad data that created a lot of fields (some 8000).  I was getting

some

errors adding new fields where solr could not talk to zookeeper, and I
thought it may be because there are so many fields.  The index size is

some

420million docs.
I'm hesitant to try to re-create as when the shards crash, they leave a
write.lock file in HDFS, and I need to manually delete that file (on 27
machines) before bringing them back up.
I believe this is the stack trace - but this looks to be related to

facets,

and I'm not 100% sure that this is the correct trace!  Sorry - I if it
happens again I will update.

ERROR - 2015-05-29 20:39:34.707; [UNCLASS shard9 core_node14 UNCLASS]
org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
java.lang.OutOfMemoryError: unable to create new native thread
         at

org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)

at

org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)

at

org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)

at

org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)

at

org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)

at

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)

at

org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)

at

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)

at

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)

         at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
         at

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

at

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)

at

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

at

org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)

at

org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)

at

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

         at org.eclipse.jetty.server.Server.handle(Server.java:368)
         at

org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)

at

org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)

at

org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)

at

org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)

at

org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)

         at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
         at

org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)

at

org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)

at

org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

at

org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

         at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
         at java.lang.Thread.start0(Native Method)
         at java.lang.Thread.start(Thread.java:714)
         at

java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)

at

java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)

at

org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)

at

org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)

at

org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)

at

org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)

at

org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)

         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
         at

org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)

at

org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)

         ... 26 more

Then later:

ERROR - 2015-05-29 21:57:22.370; [UNCLASS shard9 core_node14 UNCLASS]
org.apache.solr.common.SolrException; null:java.lang.RuntimeException:
java.lang.OutOfMemoryError: Java heap space
         at

org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:854)

at

org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:463)

at

org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)

at

org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)

at

org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)

at

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)

at

org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)

at

org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)

at

org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)

         at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
         at

org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)

at

org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)

at

org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)

at

org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)

at

org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)

at

org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)

         at org.eclipse.jetty.server.Server.handle(Server.java:368)
         at

org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)

at

org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)

at

org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)

at

org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)

at

org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)

         at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
         at

org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)

at

org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)

at

org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)

at

org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)

         at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space



-Joe


On 5/30/2015 12:32 AM, Erick Erickson wrote:

Yes, but deleting fields from the schema only means that _future_
documents will throw an "undefined field" error. All the documents
currently in the index will retain that field.

Why you're hitting an OOM is a mystery though. But delete field isn't
removing the contents if indexed documents. Showing us the full stack
when you hit an OOM would be helpful.

Best,
Erick

On Fri, May 29, 2015 at 4:58 PM, Joseph Obernberger
<j...@lovehorsepower.com> wrote:

Thank you Shawn - I'm referring to fields in the schema.  With Solr 5,
you
can delete fields from the schema.

https://cwiki.apache.org/confluence/display/solr/Schema+API#SchemaAPI-DeleteaField

-Joe


On 5/29/2015 7:30 PM, Shawn Heisey wrote:

On 5/29/2015 5:08 PM, Joseph Obernberger wrote:

Hi All - I have a lot of fields to delete, but noticed that once I
started deleting them, I quickly ran out of heap space.  Is
delete-field a memory intensive operation?  Should I delete one

field,

wait a while, then delete the next?

I'm not aware of a way to delete a field.  I may have a different
definition of what a field is than you do, though.

Solr lets you delete entire documents, but deleting a field from the
entire index would involve re-indexing every document in the index,
excluding that field.

Can you be more specific about exactly what you are doing, what you

are

seeing, and what you want to see instead?

Also, please be aware of this:

http://people.apache.org/~hossman/#threadhijack

Thanks,
Shawn

Re: Deleting Fields

Reply via email to