Re: Large Index and OutOfMemoryError: Map failed
It's the virtual memory limit that matters; yours says unlimited below (good!), but, are you certain that's really the limit your Solr process runs with? On Linux, there is also a per-process map count: cat /proc/sys/vm/max_map_count I think it typically defaults to 65,536 but you should check on your env. If a process tries to map more than this many regions, you'll hit that exception. I think you can: cat /proc//maps | wc to see how many maps your Solr process currently has... if that is anywhere near the limit then it could be the cause. Mike McCandless http://blog.mikemccandless.com On Sat, Mar 31, 2012 at 1:26 AM, Gopal Patwa wrote: > *I need help!!* > > * > * > > *I am using Solr 4.0 nightly build with NRT and I often get this error > during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I > have search this forum and what I found it is related to OS ulimit > setting, please se below my ulimit settings. I am not sure what ulimit > setting I should have? and we also get "**java.net.SocketException:* > *Too* *many* *open* *files" NOT sure how many open file we need to > set?* > > > I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 - > 15GB, with Single shard > > * > * > > *We update the index every 5 seconds, soft commit every 1 second and > hard commit every 15 minutes* > > * > * > > *Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB* > > * > * > > ulimit: > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 401408 > max locked memory (kbytes, -l) 1024 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 10240 > cpu time (seconds, -t) unlimited > max user processes (-u) 401408 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > * > * > > *ERROR:* > > * > * > > *2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=* > *thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto* > *commit* *error...:java.io.IOException:* *Map* *failed* > *at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*) > *at* > *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*) > *at* > *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*) > *at* > *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*) > *at* > *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*) > *at* > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*) > *at* > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*) > *at* > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*) > *at* > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*) > *at* > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*) > *at* > *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*) > *at* > *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*) > *at* > *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*) > *at* > *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*) > *at* > *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*) > *at* > *org.apache.lucene.index.IndexWriter.maybeApplyDeletes*(*IndexWriter.java:2930*) > *at* > *org.apache.lucene.index.IndexWriter.prepareCommit*(*IndexWriter.java:2681*) > *at* > *org.apache.lucene.index.IndexWriter.commitInternal*(*IndexWriter.java:2804*) > *at* > *org.apache.lucene.index.IndexWriter.commit*(*IndexWriter.java:2786*) > *at* > *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:391*) > *at* > *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*) > *at* > *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*) > *at* > *java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*) > *at* *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*) > *at* > *java.util.concurrent.ScheduledThreadPoolExecutor$Schedul
Solr caching memory consumption Problem
Hello friends, I am using DIH for solr indexing. I have 60 million records in SQL which need to upload on solr. i started caching its smoothly working and memory consumption is normal, But after some time incrementally memory consumption going high and process reach more then 6 gb. that the reason i am not able to caching my data. please advise me if anything need to be done in configuration or in tomcat configuration. this will be very help full for me. - Regards, Suneel Pandey Sr. Software Developer -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-caching-memory-consumption-Problem-tp3873158p3873158.html Sent from the Solr - User mailing list archive at Nabble.com.
Content privacy, search & index
Hi, I'm relatively new to Solr, new in the way that i already used Solr several times but always with a very simple approach, meaning simple fulltext search with faceting and filtering. Today, i've to go a bit further and before i do, i'd like to get your point of view ;-) I need to index users and user contents that are subject to privacy levels like for instance : * Anyone * Only me * Only my friends * Only people i choose ...really classic. So, when an user searches for contents on the website, in the results, we can't show him the content elements he is not allowed to see. My first thought was : "There might be a way to do that with complex solr queries" So i start reading the documentation, and i have to say that i understand half of the things i read :-) And then, a new idea came to my mind. I was thinking about this process : 1- The user submits the search form with his keywords 2- I prepare a classic fulltext search query 3- I compute some way the friend list of the current user 4- I add a filter to the Solr query with the result of that 5- I send the query While this seems reasonable since i can add some cache system in the way to avoid computing the friend list each time, i don't know why, it doesn't feel right ;-) The other way would be to index users and users friends and somehow letting solr doing all the job. What do you think ? Is the second solution even possible ? Thanks ! Br, Benjamin. -- View this message in context: http://lucene.472066.n3.nabble.com/Content-privacy-search-index-tp3873462p3873462.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: How do I use localparams/joins using SolrJ and/or the Admin GUI
Try escaping the '+' with %2B (as I remember).
Your stack trace didn't come through, at least by the time
it got to my mail so Im guessing a bit.
Best
Erick
On Fri, Mar 30, 2012 at 5:33 PM, vybe3142 wrote:
> Here's a JOIN query using local params that I can sucessfully execute on a
> browser window:
>
>
> When I paste the relevant part of the query into the SOLR admin UI query
> interface,
> {!join+from=join_id+to=id}attributes_AUTHORS.4:4, I fail to retrieve any
> documents
>
> The query transaltes to :
>
>
>
>
>
> Server stack trace
>
>
> There appears to be some sort of URL translation going on but I'd like to
> understand how to make this work.
>
> Next, I'd like to understand how to make this work with SOLRJ
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-do-I-use-localparams-joins-using-SolrJ-and-or-the-Admin-GUI-tp3872088p3872088.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: How do I use localparams/joins using SolrJ and/or the Admin GUI
On Sat, Mar 31, 2012 at 11:50 AM, Erick Erickson wrote: > Try escaping the '+' with %2B (as I remember). Shouldn't that be the other way? The admin UI should do any necessary escaping, so those "+" chars should instead be a spaces? -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
Re: Content privacy, search & index
The second option is actually possible, and actually easiest in terms of letting Solr do the most work. Presumably you have some web-app facing the user, you could pre-calculate the list of authorized viewers there on some kind of session basis. Be careful, however, that this list of IDs doesn't grow unbounded. Anything up to several hundred is probably OK, but thousands gets to be problematical. The other thing to consider here is whether the list of authorized users changes much. The downside is that you have to index the authorized users with the document. So if I have 10,000 docs and add someone to my friend list, you have to re-index all 10,000 docs. Complex Solr queries... don't quite know what that would look like. My rule of thumb is that when I try to make Solr behave like a DB through queries...it's probably wrong. There is another option, the "no cache" filter queries, see: https://issues.apache.org/jira/browse/SOLR-2429 This was actually designed for ACL checking. You say you're relatively new to Solr, so making sense of this may be a bit daunting. The basic idea is that you can implement a PostFilter where each matching document, after all other selection criteria have been met, is sent to your code which can then give a go-no go answer. This keeps all the facets etc. accurate. Best Erick On Sat, Mar 31, 2012 at 9:57 AM, dbenjamin wrote: > Hi, > > I'm relatively new to Solr, new in the way that i already used Solr several > times but always with a very simple approach, meaning simple fulltext search > with faceting and filtering. > > Today, i've to go a bit further and before i do, i'd like to get your point > of view ;-) > > I need to index users and user contents that are subject to privacy levels > like for instance : > > * Anyone > * Only me > * Only my friends > * Only people i choose > > ...really classic. > > So, when an user searches for contents on the website, in the results, we > can't show him the content elements he is not allowed to see. > > My first thought was : "There might be a way to do that with complex solr > queries" > > So i start reading the documentation, and i have to say that i understand > half of the things i read :-) > > And then, a new idea came to my mind. I was thinking about this process : > > 1- The user submits the search form with his keywords > 2- I prepare a classic fulltext search query > 3- I compute some way the friend list of the current user > 4- I add a filter to the Solr query with the result of that > 5- I send the query > > While this seems reasonable since i can add some cache system in the way to > avoid computing the friend list each time, i don't know why, it doesn't feel > right ;-) > > The other way would be to index users and users friends and somehow letting > solr doing all the job. > > What do you think ? Is the second solution even possible ? > > > Thanks ! > Br, > > Benjamin. > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Content-privacy-search-index-tp3873462p3873462.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: How do I use localparams/joins using SolrJ and/or the Admin GUI
Gh, of course it should be the other way. That'll teach me to not try to listen to "wait wait, don't tell me" and reply to the list at the same time... Thanks, Erick On Sat, Mar 31, 2012 at 12:01 PM, Yonik Seeley wrote: > On Sat, Mar 31, 2012 at 11:50 AM, Erick Erickson > wrote: >> Try escaping the '+' with %2B (as I remember). > > Shouldn't that be the other way? The admin UI should do any necessary > escaping, so those "+" chars should instead be a spaces? > > -Yonik > lucenerevolution.com - Lucene/Solr Open Source Search Conference. > Boston May 7-10
Re: Solr Tomcat Install
i also encountered this same problem during an installation. i found out that the config.xml file was a malformed XML document. you might want to check there. - Original Message - From: rdancy To: solr-user@lucene.apache.org Sent: Thursday, March 29, 2012 1:20 AM Subject: Solr Tomcat Install Hello, I have configured Solr inside Tomcat and I get the following error when I go to browser and click on the solr admin link: HTTP Status 500 - Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. If you want solr to continue after configuration errors, change: false in solr.xml - java.lang.NoClassDefFoundError: org/apache/velocity/context/Context at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:383) at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:425) at org.apache.solr.core.SolrCore.createInitInstance(SolrCore.java:447) at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:1556) at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:1550) at org.apache.solr.core.SolrCore.initPlugins(SolrCore.java:1583) at org.apache.solr.core.SolrCore.initWriters(SolrCore.java:1466) at org.apache.solr.core.SolrCore.(SolrCore.java:556) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3696) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4343) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:525) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:626) at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1138) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:719) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:516) at org.apache.catalina.core.StandardServer.start(StandardServer.java:710) at org.apache.catalina.startup.Catalina.start(Catalina.java:566) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413) Caused by: java.lang.ClassNotFoundException: org.apache.velocity.context.Context at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:627) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) ... 42 more type Status report message Severe errors in solr configuration. Check your log files for more detailed information on what may be wrong. If you want solr to continue after configuration errors, change: false in solr.xml - java.lang.NoClassDefFoundError: org/apache/velocity/context/Context at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:383) at org.apache.
Re: Why my highlights are wrong(one character offset)?
I have fixed the problem. The problem is that there are line change delimiter($) between the xml tag and my index content. It causes the problem. I reprocess my index content and eliminate the delimiter and the highlight offset is correct now. -- View this message in context: http://lucene.472066.n3.nabble.com/Why-my-highlights-are-wrong-one-character-offset-tp3860286p3873959.html Sent from the Solr - User mailing list archive at Nabble.com.
ExtractingRequestHandler
Hi, I want to index various filetypes in solr, this can easily done with ExtractingRequestHandler. But I also need the extracted content back. I know ext.extract.only but then nothing gets indexed, right? Can I index the document AND get the content back as with ext.extract.only? In a single request? Thank you
RE: Client-side failover with SolrJ
> Did you try > http://lucene.apache.org/solr/api/org/apache/solr/client/solrj > /impl/LBHttpSolrServer.html? > This might be what you're looking for. Cool! Thx!
Re: Content privacy, search & index
Hi Erick and thanks for the quick reply. Well, my intend would not to index all content elements with its own list of authorized users IDs. I was thinking more of something like I index the contents and the users + their friend list separatly, and then being able somehow to ask Solr to filter results of one index depending on the other. In SQL it would be performed with a sub-query, for instance : SELECT a.* FROM albums a LEFT JOIN albums_settings as ON as.album_id = a.id WHERE as.privacy_level = 'anyone' OR (as.privacy_level = 'friends' AND a.owner_id IN ( SELECT friend_id FROM friendships WHERE user_id = :currentUserId ) ) ; It would require some tests, but this request should return only what we want. I was assuming by "complex query" that Solr would be able to perform that kind of operation. But if it does, that raises some questions : - Is it the best way to do that ? - It's obvious that i need to index the registered users in Solr (because an user can search for others), but is it clever to index friend list for each user as well ? (if we take a look at the search box on Facebook, or other any sexy social network, they propose auto-complete for current user friends, so maybe it makes sense...) Br, Benjamin. -- View this message in context: http://lucene.472066.n3.nabble.com/Content-privacy-search-index-tp3873462p3874112.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Content privacy, search & index
Benjamin, I think implementing a QueryHandler that adds the necessary query is the right way to do that. It'd transform a query for "a b" into "+(a b) +(authorizedBit)" (to use the language of the default QueryParser but please not by substring, using the real query objects!). Recalculating the friends-list... well... that all depends on your authorization system, this should be cached somewhere in a session or so, ideally you'd even cache somewhere close to there the queries that you add. Then performance is likely to be ok. paul Le 31 mars 2012 à 15:57, dbenjamin a écrit : > Hi, > > I'm relatively new to Solr, new in the way that i already used Solr several > times but always with a very simple approach, meaning simple fulltext search > with faceting and filtering. > > Today, i've to go a bit further and before i do, i'd like to get your point > of view ;-) > > I need to index users and user contents that are subject to privacy levels > like for instance : > > * Anyone > * Only me > * Only my friends > * Only people i choose > > ...really classic. > > So, when an user searches for contents on the website, in the results, we > can't show him the content elements he is not allowed to see. > > My first thought was : "There might be a way to do that with complex solr > queries" > > So i start reading the documentation, and i have to say that i understand > half of the things i read :-) > > And then, a new idea came to my mind. I was thinking about this process : > > 1- The user submits the search form with his keywords > 2- I prepare a classic fulltext search query > 3- I compute some way the friend list of the current user > 4- I add a filter to the Solr query with the result of that > 5- I send the query > > While this seems reasonable since i can add some cache system in the way to > avoid computing the friend list each time, i don't know why, it doesn't feel > right ;-) > > The other way would be to index users and users friends and somehow letting > solr doing all the job. > > What do you think ? Is the second solution even possible ? > > > Thanks ! > Br, > > Benjamin. > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Content-privacy-search-index-tp3873462p3873462.html > Sent from the Solr - User mailing list archive at Nabble.com.
RE: Content privacy, search & index
> - Is it the best way to do that ? > - It's obvious that i need to index the registered users in > Solr (because an > user can search for others), but is it clever to index friend > list for each > user as well ? (if we take a look at the search box on > Facebook, or other > any sexy social network, they propose auto-complete for current user > friends, so maybe it makes sense...) This is a common question: How to merge the resultlist from solr (A) with a resultlist from elsewhere (B) (offen a RDBMS like in you case). 3 options: 1) do the merge in A: * fetch the ids from B and do the merge in A (e.g. filterQuery in Solr, be aware of maxBooleanClauses). 2) do the merge in B: * fetch the ids from A and do the merge in B (e.g. subselect, has limitation in big number of Ids too). 3) do the merge in the application (C): * fetch the ids from A and B and intersect them in C Depending on the size of the resultsets one of the 3 options is the best ;)
RE: 400 Error adding field 'tags'='[a,b,c]'
[Solved]
Turns out the text field I was copying all these to was not multivalued!
make sure that if you're copying multiple fields to a single field that the
single field has multivalued=true.
phew.
Thanks for the Help.
Date: Thu, 15 Mar 2012 03:19:15 -0700
From: ml-node+s472066n3828342...@n3.nabble.com
To: alpti...@hotmail.com
Subject: Re: 400 Error adding field 'tags'='[a,b,c]'
Hi Alp,
if you have not changed how SOLR logs in general, you should find the
log output in the regular server logfile. For Tomcat you can find this
in TOMCAT_HOME/catalina.out (or search for that name).
If there is a problem with your schema, SOLR should be complaining about
it during application/server start up. It would definitely print
something if there is a field declared in your schema but cannot be
initialized for some reason.
I don't think that the names of the fields themselves are the problem. I
never had an issue with the field name 'name'.
Cheers,
Chantal
On Wed, 2012-03-14 at 02:53 +0100, jlark wrote:
> Interestingly I'm getting this on other fields now.
>
> I have the field stored="true" />
>
> which is copied to text
>
> and my text field is simply indexed="true" stored="true" />
>
> I'm feedin my test document
>
> {"url" : "TestDoc2", "title" : "another test", "ptag":["a","b"],"name":"foo
> bar"},
>
> and when I try to feed I get.
>
> HTTP request sent, awaiting response... 400 ERROR: [doc=TestDoc2] Error
> adding field 'name'='foo bar'
>
> If I remove the field from the document though it works fine.
> I'm wondering if there is a set of reserved names that I'm using at this
> point.
>
> Jus twhish there was a way to get more helpfull error messages.
>
> Thanks for the help.
> Alp
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/400-Error-adding-field-tags-a-b-c-tp3823853p3824126.html
> Sent from the Solr - User mailing list archive at Nabble.com.
If you reply to this email, your message will be added to the
discussion below:
http://lucene.472066.n3.nabble.com/400-Error-adding-field-tags-a-b-c-tp3823853p3828342.html
To unsubscribe from 400 Error adding field 'tags'='[a,b,c]',
click here.
NAML
--
View this message in context:
http://lucene.472066.n3.nabble.com/400-Error-adding-field-tags-a-b-c-tp3823853p3874445.html
Sent from the Solr - User mailing list archive at Nabble.com.
Determining individual Field matches within a SolrRequestHandler or SearchComponent
Hi, Is it possible to easily and programmatically determine whether one or more solr fields were a successful match within a SolrRequestHandler or SearchComponent? The DebugComponent provides similar text output when debugging/explaining a query: 8.249506 = (MATCH) fieldWeight(attribution_en:leigh in 2313441) Class org.apache.lucene.search.Explanation has an isMatch() - is there anything similar, but on a per-field basis? I've dug through the solr and lucene source a bit, and it doesn't seem like this is available anywhere. If this is the case, what's my next best option? Comparing the query text to the field(s) in question? Thanks, Josh
