Re: Copying part of index directory
Mike Klaas wrote: On 15-Jun-07, at 4:25 AM, Roopesh P Raj wrote: Hi, I am new to solr, lucene. In my project I want to copy index directory based on some query (copy may not copy the whole index directory). I came across a backup script in solr/bin folder but it seems to be copying the whole index directory. My query is, what is the procedure for copying a part of the index. Is it like - do a query, get all the fields, make a new index with those queried results? Or is there any other way ? You can't easily copy parts of the physical file. I suggest performing the query, fetching some unique key (application-specific), and re-indexing those documents from your original source. It can be done through Solr too, but you have to be careful to store all the relevant fields to being with. Thanks for the reply. I have one more query. My doubt is where to re-index (location of the index directory) ? For this should I run another instance of solr? Is this the preferred approach ? Roopesh -- DigitalGlue, India
Filtering on a 'unique key' set
Merely an efficiency related question: is there any other way to filter on a uniqueKey set than using the 'fq' parameter & building a list of the uniqueKeys? In 'raw' Lucene, you could use filters directly in search; is this (close to) equivalent efficiency wise? Thanks -- View this message in context: http://www.nabble.com/Filtering-on-a-%27unique-key%27-set-tf3935694.html#a11162273 Sent from the Solr - User mailing list archive at Nabble.com.
Re: Filtering on a 'unique key' set
On 6/17/07, Henrib <[EMAIL PROTECTED]> wrote: Merely an efficiency related question: is there any other way to filter on a uniqueKey set than using the 'fq' parameter & building a list of the uniqueKeys? I don't thnik so... In 'raw' Lucene, you could use filters directly in search; is this (close to) equivalent efficiency wise? Yes, any fq params are turned into filters. -Yonik
can't post.sh/post.jar
hello solrs! I get the following error on windows when trying to index an ~60mb xml file with post.jar. Also couldn't get post.sh to work - anyone successfully ran it on windows? C:\solr\example\exampledocs>java -jar post.jar flix.xml SimplePostTool: version 1.2 SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, othe r encodings are not currently supported SimplePostTool: POSTing files to http://localhost:8983/solr/update.. SimplePostTool: POSTing file flix.xml Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at sun.net.www.http.PosterOutputStream.write(Unknown Source) at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) at sun.nio.cs.StreamEncoder.write(Unknown Source) at java.io.OutputStreamWriter.write(Unknown Source) at org.apache.solr.util.SimplePostTool.pipe(SimplePostTool.java:281) at org.apache.solr.util.SimplePostTool.postData(SimplePostTool.java:247) at org.apache.solr.util.SimplePostTool.postFile(SimplePostTool.java:213) at org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:152 ) at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:112) C:\solr\example\exampledocs> - Pinpoint customers who are looking for what you sell.
Re: can't post.sh/post.jar
If you are dealing with such large files, you need to make sure the JVM has a big enough heap. Try starting java with -mx100m (-mx2G if you have it) java -mx100m -jar post.jar flix.xml The solr server also needs to be started with enough memory... ryan michael ravits wrote: hello solrs! I get the following error on windows when trying to index an ~60mb xml file with post.jar. Also couldn't get post.sh to work - anyone successfully ran it on windows? C:\solr\example\exampledocs>java -jar post.jar flix.xml SimplePostTool: version 1.2 SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, othe r encodings are not currently supported SimplePostTool: POSTing files to http://localhost:8983/solr/update.. SimplePostTool: POSTing file flix.xml Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at sun.net.www.http.PosterOutputStream.write(Unknown Source) at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) at sun.nio.cs.StreamEncoder.write(Unknown Source) at java.io.OutputStreamWriter.write(Unknown Source) at org.apache.solr.util.SimplePostTool.pipe(SimplePostTool.java:281) at org.apache.solr.util.SimplePostTool.postData(SimplePostTool.java:247) at org.apache.solr.util.SimplePostTool.postFile(SimplePostTool.java:213) at org.apache.solr.util.SimplePostTool.postFiles(SimplePostTool.java:152 ) at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java:112) C:\solr\example\exampledocs> - Pinpoint customers who are looking for what you sell.
Re: can't post.sh/post.jar
Michael, You can start the process with more memory assigned to your JVM Like this : java -Xms512m -Xmx512m -jar post.jar *.xml This will start the process with an initial memory of 512 MB and a maximum memory use of 512 MB. This is a little explanation about the memory parameters -Xmsinitial java heap size -Xmxmaximum java heap size -Xmnthe size of the heap for the young generation Hope this helps, Thierry On 18/06/07, michael ravits <[EMAIL PROTECTED]> wrote: hello solrs! I get the following error on windows when trying to index an ~60mb xml file with post.jar. Also couldn't get post.sh to work - anyone successfully ran it on windows? C:\solr\example\exampledocs>java -jar post.jar flix.xml SimplePostTool: version 1.2 SimplePostTool: WARNING: Make sure your XML documents are encoded in UTF-8, othe r encodings are not currently supported SimplePostTool: POSTing files to http://localhost:8983/solr/update.. SimplePostTool: POSTing file flix.xml Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at sun.net.www.http.PosterOutputStream.write(Unknown Source) at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) at sun.nio.cs.StreamEncoder.write(Unknown Source) at java.io.OutputStreamWriter.write(Unknown Source) at org.apache.solr.util.SimplePostTool.pipe(SimplePostTool.java :281) at org.apache.solr.util.SimplePostTool.postData( SimplePostTool.java:247) at org.apache.solr.util.SimplePostTool.postFile( SimplePostTool.java:213) at org.apache.solr.util.SimplePostTool.postFiles( SimplePostTool.java:152 ) at org.apache.solr.util.SimplePostTool.main(SimplePostTool.java :112) C:\solr\example\exampledocs> - Pinpoint customers who are looking for what you sell.
