Re: highlighting error using 1.4rc

Mark, do you have more details on what kind of queries will make this bug show up? On Tue, Nov 3, 2009 at 5:33 AM, Mark Miller wrote: > Sorry - it was a bug in the backport from trunk to 2.9.1 - didn't > realize that code didn't get hit because we didn't pass a null field - > else the tests woul

Re: Match all terms in doc

On Nov 1, 2009, at 1:20 AM, AHMET ARSLAN wrote: Hi How do I restrict hits to documents containing all words (regardless of order) of a query in particular field? Suppose I have two documents with a field called name in my index: doc1 => name: Pink doc2 => name: Pink Floyd When querying for

Re: Problems downloading lucene 2.9.1

Heya Ryan... For me the big problem with adding http://people.apache.org/~mikemccand/staging-area/rc3_lucene2.9.1/maven/to my build config is that the artifact names of the interim release are the same as the final objects

How to integrate Solr into my project

Hi, I wish to intergrate Solr into my current working project. I've played around the Solr example and get it started in my tomcat. But the next step is HOW do i integrate that into my working project? You see, Lucence provides API and tutorial on what class i need to instanstiate in order to index

Re: weird behaviour while inserting records into solr

HI, am inserting the records one by one. 1st i create a solr input document, add it into solr, perform a commit. i loop this entire process for a million times. Regards, Raakhi On Wed, Oct 28, 2009 at 1:45 AM, Grant Ingersoll wrote: > > On Oct 26, 2009, at 1:14 AM, Rakhi Khatwani wrote: >

Re: Dismax and Standard Queries together

Hi Ram, You can add another field total ( catchall field ) and copy all other fields into this field ( using copyField directive ) http://wiki.apache.org/solr/SchemaXml#Copy_Fields and use this field in DisMax qf parameter, for example qf=business_name^2.0 category_name^1.0 sub_category_name^1.0

Re: How to integrate Solr into my project

is it a java project ? did you see this page http://wiki.apache.org/solr/Solrj ? On Tue, Nov 3, 2009 at 2:25 PM, Caroline Tan wrote: > Hi, > I wish to intergrate Solr into my current working project. I've played > around the Solr example and get it started in my tomcat. But the next step > is HOW

Re: How to integrate Solr into my project

Take a look at this - http://wiki.apache.org/solr/Solrj Cheers Avlesh On Tue, Nov 3, 2009 at 2:25 PM, Caroline Tan wrote: > Hi, > I wish to intergrate Solr into my current working project. I've played > around the Solr example and get it started in my tomcat. But the next step > is HOW do i int

Re: How to integrate Solr into my project

Ya, it's a Java projecti just browse this site you suggested... http://wiki.apache.org/solr/Solrj Which means, i declared the dependancy to solr-solrj and solr-core jars, have those jars added to my project lib and by following the Solrj tutorial, i will be able to even index a DB table into S

Re: Lucene FieldCache memory requirements

On Mon, Nov 2, 2009 at 9:27 PM, Fuad Efendi wrote: > I believe this is correct estimate: > >> C. [maxdoc] x [4 bytes ~ (int) Lucene Document ID] >> >>   same as >> [String1_Document_Count + ... + String10_Document_Count + ...] >> x [4 bytes per DocumentID] That's right. Except: as Mark said, you

RE: CPU utilization and query time high on Solr slave when snapshot install

Hi Walter, When the issue occurred, we did try to set autowarming off, but it did not solve the problem. The only thing which worked, was optimizing the slave index. But, what you say is logical and I will try it again. But, the basic question I have is, our solr index is not huge by any means.

tf*idf scoring

Hello list, I have a question about Lucene's calculation of tf*idf value. I first noticed that Solr's tf does not compare to tf values based on calculation elsewhere such as http://odin.himinbi.org/idf_to_item:item/comparing_tf%3Aidf_to_item% 3Aitem_similarity.xhtml or http://en.wikipedia.org/wik

Re: How to integrate Solr into my project

Hi Caroline, i think that you must take an overview tour ;-) , solrj is just a solr java client ... Some clues: - Define your own index schema (it's just like a SQL DDL) . - There are different ways to put docs in your index: - SolrJ (Solr clie

Re: How to integrate Solr into my project

2009/11/3 Licinio Fernández Maurelo > Hi Caroline, > > i think that you must take an overview tour ;-) , solrj is just a solr java > client ... > > Some clues: > > > - Define your own index schema > (it's just like a SQL DDL) . > - There are different wa

Re: How to integrate Solr into my project

Thanks guy.. ~caroLine On Tue, Nov 3, 2009 at 7:32 PM, Israel Ekpo wrote: > 2009/11/3 Licinio Fernández Maurelo > > > Hi Caroline, > > > > i think that you must take an overview tour ;-) , solrj is just a solr > java > > client ... > > > > Some clues: > > > > > > - Define your own index sche

Re: Indexing multiple entities

On Sun, Nov 1, 2009 at 10:34 AM, Christian López Espínola wrote: > On Sun, Nov 1, 2009 at 5:30 AM, Avlesh Singh wrote: >>> >>> The use case on DocumentObjectBinder is that I could override >>> toSolrInputDocument, and if field = ID, I could do: setField("id", >>> obj.getClass().getName() + obj.ge

Re: Indexing multiple entities

On Mon, Nov 2, 2009 at 11:07 AM, Chantal Ackermann wrote: >> I'm using a code generator for my entities, and I cannot modify the >> generation. >> I need to work out another option :( > > shouldn't code generators help development and not make it more complex and > difficult? oO Yeah, they do. Bu

Re: Getting update/extract RequestHandler to work under Tomcat

Try making it a non-Lazy loaded handler. Does that help? On Nov 2, 2009, at 4:37 PM, Glock, Thomas wrote: Hoping someone might help with getting /update/extract RequestHandler to work under Tomcat. Error 500 happens when trying to access http://localhost:8080/apache-solr-1.4-dev/update/ex

Re: tf*idf scoring

Inline below On Nov 3, 2009, at 2:30 AM, Markus Jelsma - Buyways B.V. wrote: Hello list, I have a question about Lucene's calculation of tf*idf value. I first noticed that Solr's tf does not compare to tf values based on calculation elsewhere such as http://odin.himinbi.org/idf_to_item:item/c

Re: Problems downloading lucene 2.9.1

Yeah, that would be useful. We could hook into the new Apache Repository stuff, but that will take some work. Also, I think 1.4 is in all likelihood a special case in terms of dealing with the Lucene release. -Grant On Nov 3, 2009, at 12:45 AM, Ian Ibbotson wrote: Heya Ryan... For me

DIH : RegexTransformer with groupNames requires all groups to be not empty?

Dear all, my DIH config contains the following directive for the RegexTransformer: (this is SOLR 1.4.0 RC downloaded yesterday from Grant's URL) It expects input of the kind (version A): Daniel Radcliffe|24897,1,1,Harry Potter It should also work with (version B): Daniel Radcliffe|24897,1,1,

DIH : RegexTransformer with groupNames requires all groups to be not empty?

follow-up: regex="([^\|]+)\|\d+,\d+,\d+,(.+)" is the version I chose after I had the following problems with regex="([^\|]+)\|\d+,\d+,\d+,(.*)" (changed * into + for the second group): The role field contained empty values even if I added a TrimFilterFactory with minimum length of 1. So, I ch

Re: tracking solr response time

>I didn't see where you said what Solr version you were using. below is the solr version info :- Solr Specification Version: 1.2.2008.07.22.15.48.39 Solr Implementation Version: 1.3-dev Lucene Specification Version: 2.3.1 Lucene Implementation Version: 2.3.1 629191 - buschmi - 2008-02-19 19:15:48

Re: solrj query size limit?

That was it. Didn't see that optional parameter - the POST works. Thanks! On Nov 3, 2009, at 1:57 AM, Avlesh Singh wrote: Did you hit the limit for maximum number of characters in a GET request? Cheers Avlesh On Tue, Nov 3, 2009 at 9:36 AM, Gregg Horan wrote: I'm constructing a quer

Re: tf*idf scoring

> > > > > > According to different algorithms, the tf for term c would be 3 / 1 = > > 0.33 instead of 1 returned by Solr. > > I don't follow. The TF (term frequency) is the number of times the > term c occurs in that particular document, i.e. 1 time. I see that above, and below, i made some

very slow add/commit time

Hi, I am experiencing a problem with an index of about 80 millions documents (41Gb). I am trying to update documents in this index using Solrj. When I do: solrServer.add(docs); //docs is a List that contains 1000 SolrInputDocument (takes 36sec) solrServer.commit(false,false); //either ne

Re: very slow add/commit time

How many MB have you set of cache on your solrconfig.xml? On Tue, Nov 3, 2009 at 12:24 PM, Marc Des Garets wrote: > Hi, > > > > I am experiencing a problem with an index of about 80 millions documents > (41Gb). I am trying to update documents in this index using Solrj. > > > > When I do: > > solr

RE: Getting update/extract RequestHandler to work under Tomcat

Thanks - Looked at it last night and I think the problem is that I need to compile the ExtractingRequestHandler classes/jar. I see the source - but no classes or jar that seems to fit the bill. I've had problems getting ant to build from the nightly trunk. I'm of the opinion I simply need

TermVectorComponent : Required / Optional Parameters

Hi all! Are these attributes required to make TermVectorComponent requests work? termPositions="true" termOffsets="true" I have quite a lot of fields with termVectors="true" (for facetting), but I don't get any results when requesting: .../solr/epg/select?q=*%3A*&version=2.2&start=0&rows=1 &i

RE: very slow add/commit time

If you mean ramBufferSizeMB, I have it set on 512. The maxBufferedDocs is commented. If you mean queryResultMaxDocsCached, it is set on 200 but is it used when indexing? -Original Message- From: Bruno [mailto:brun...@gmail.com] Sent: 03 November 2009 14:27 To: solr-user@lucene.apache.org

RE: Getting update/extract RequestHandler to work under Tomcat

: I see the source - but no classes or jar that seems to fit the bill. : : I've had problems getting ant to build from the nightly trunk. I'm of ... : If there is an existing jar of the ExtractingRequestHandler classes that : I might download - please point me to it. If you are downloading a

Re: very slow add/commit time

Try raising you ramBufferSize (it helped a lot when my team had performance issues) And also try checkin this link (helps a lot) http://wiki.apache.org/solr/SolrPerformanceFactors Regards On Tue, Nov 3, 2009 at 12:38 PM, Marc Des Garets wrote: > If you mean ramBufferSizeMB, I have it set on 512

DIH : RegexTransformer with groupNames requires all groups to be not empty?

Ok, I can confirm that the following configuration for RegexTransformer works as I would expect it: regex="[^\|]+\|\d+,\d+,\d+,(.+)" /> To the multivalued fields participant and role, values are only added if their corresponding regex matches. The following configuration does not add an

Re: Multifield query parser and phrase query behaviour from 1.3 to 1.4

: However, even when it's set to 'false' , the highlighting of a field : continues to work even if the search doesn't. : Does the highlighter use a different strategy to match the query terms : in the fields? if it has term vectors, it uses them, otherwise it re analyzes the stored fields. -Ho

Re: Getting update/extract RequestHandler to work under Tomcat

: The \contrib and \dist directories were copied directly below the : "webapps\apache-solr-1.4-dev" unchanged from the example. ...that doesn't sound right, they shouldn't be copied into webapps at all. can you show a full directory structure... : Im the catalina log I see all the "Adding spec

Re: tf*idf scoring

On Nov 3, 2009, at 5:54 AM, Markus Jelsma - Buyways B.V. wrote: I see, but why not return the true values of Lucene? I'm not sure what you mean by this. The TVC returns the term frequency and the document frequency and TF/DF as reported by Lucene. The actual raw values. What you are

Re: TermVectorComponent : Required / Optional Parameters

On Nov 3, 2009, at 6:37 AM, Chantal Ackermann wrote: Hi all! Are these attributes required to make TermVectorComponent requests work? termPositions="true" termOffsets="true" I have quite a lot of fields with termVectors="true" (for facetting), but I don't get any results when requesting:

RE: Lucene FieldCache memory requirements

Sorry Mike, Mark, I am confused again... Yes, I need some more memory for processing ("while FieldCache is being loaded"), obviously, but it was not main subject... With StringIndexCache, I have 10 arrays (cardinality of this field is 10) storing (int) Lucene Document ID. > Except: as Mark said

Re: apply a patch on solr

: --- org/apache/solr/handler/MoreLikeThisHandler.java (revision 772437) : +++ org/apache/solr/handler/MoreLikeThisHandler.java (working copy) : : now check out revision 772437 using the --revision switch in svn, patch : away, and then svn up to make sure everything merges cleanly. This is a : g

Re: TermVectorComponent : Required / Optional Parameters

Hi Grant, I'd be glad to help update the wiki. But just to make sure I'm understanding correctly: for TermVectorComponent to work correctly, all these three attributes (termVectors, termPositions, termOffsets) must be set to "true"? No matter how extensive the termVector request will look like

Highlighting is very slow

Hi everyone, I'm experimenting with highlighting for the first time, and it seems shockingly slow for some queries. For example, this query: http://server:8080/solr/select/?q=transferase&qt=dismax&version=2.2&start=0&rows=10&indent=on takes 313ms. But when I add highlighting: http://server:80

Highlighting is very slow

Hi everyone, I'm experimenting with highlighting for the first time, and it seems shockingly slow for some queries. For example, this query: http://server:8080/solr/select/?q=transferase&qt=dismax&version=2.2&start=0&rows=10&indent=on takes 313ms. But when I add highlighting: http://server:80

Re: adding and updating a lot of document to Solr, metadata extraction etc

On Mon, Nov 02, 2009 at 05:45:37PM -0800, Lance Norskog wrote: > About large XML files and http overhead: you can tell solr to load the > file directly from a file system. This will stream thousands of > documents in one XML file without loading everything in memory at > once. > > This is a new bo

Re: Date Facet Giving Count more than actual

: q=&facet=true&facet.date=daysForFilter&facet.date.start=2009-10-23T18:30:01Z&facet.date.gap=%2B1DAY&facet.date.end=2009-10-28T18:30:01Z : For example I get total 18 documents for my query, and the facet count for : date 2009-10-23T18:30:01Z is 11; whereas there are only 5 documents : containin

Re: logging options for 1.3

: Is there any way to get 1.3 Solr to use something other than java logging? Solr 1.3 is compiled directly against the JUL logging APIs, so no. : Am running solr inside tomcat and would like logging for solr to be directed : to one set of (rotated) log files and leave tomcat logging in its own l

Re: CPU utilization and query time high on Solr slave when snapshot install

Optimizing an index takes CPU, but if you are doing it on a machine dedicated to indexing, that does not matter. It will make queries faster. wunder On Nov 3, 2009, at 2:12 AM, biku...@sapient.com wrote: Hi Walter, When the issue occurred, we did try to set autowarming off, but it did n

Customizing Field Score (Multivalued Field)

We've had a customized score calculator for one of our fields that we developed when we were using Lucene instead of Solr (lucene 2.4). During our switch to Solr, we simply continued to use that code. However, as the version of Lucene used in Solr changed to 2.9, somewhere along the way (unfortun

Re: TermVectorComponent : Required / Optional Parameters

: Indeed, you must have stored positions and offsets for the TVC to return : positions and offsets. we should probably make the TermVectorCOmponent more resilient and actaully assert these things about the field (using the schema metadata) ... it can add warning/error info to the output if it's

Re: TermVectorComponent : Required / Optional Parameters

On Nov 3, 2009, at 7:59 AM, Chantal Ackermann wrote: Hi Grant, I'd be glad to help update the wiki. But just to make sure I'm understanding correctly: for TermVectorComponent to work correctly, all these three attributes (termVectors, termPositions, termOffsets) must be set to "true"? No

Re: Lock problems: Lock obtain timed out

: 02-Nov-2009 10:35:27 org.apache.solr.update.SolrIndexWriter finalize : SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates : a bug -- POSSIBLE RESOURCE LEAK!!! can you post some context showing what the logs look like just before these errors? I'm not sure what might be caus

Re: Highlighting is very slow

Hi, We had a similar case once (although not with those really long response times). Fixed by moving to JRE 1.6 and tuning garbage collection. Bye, Jaco. 2009/11/3 Andrew Clegg > > Hi everyone, > > I'm experimenting with highlighting for the first time, and it seems > shockingly slow for some

Re: Opaque replication failures

OK, I've solved this. For posterity: The master doesn't make anything available for replication unless you set replicateAfter="startup", or unless you set replicateAfter="commit" and then both add a document and execute a commit. If you don't do one of those, even manually clicking "Replicate No

Re: Re : Solr - Plugin : QParserPlugin is not working..

: plugin.query.parser.QueryParserPluginOne" in logs. I am sure that the : request handler with which this query parser plugin is linked is : working,Because I could find the results of System.out.println()s : (those included in requesthandler) in log, but not query parser : plugin's System.outs or

Re: MoreLikeThis support Dismax parameters

> > As i said: that may be what you're looking for (it's hard to tell based on > your email) but the other possibility is that you want to be able to > specify bq (and maybe bf) type parrams to influence the MLT portion of the > request (ie: apply a bias so docs matching a particular query/func are

Re: CPU utilization and query time high on Solr slave when snapshot install

Well thats what I get for typing on my iPhone when I'm not sure of my memory - Lance called me out on this - I put "Hmm...I think" because I wasn't positive if I remembered right, but I had thought auto warming just populates based on the old entries (probably because of the javadoc for the CacheRe

Re: Programmatically configuring SLF4J for Solr 1.4?

: it sorted. I'm using a DirectSolrConnection embedded in a JRuby : application, and everything works great, except I can't seem to get it to do : anything except log to the console. I've tried pointing : 'java.util.logging.config.file' to a properties file, as well as specifying how have you tr

RE: Getting update/extract RequestHandler to work under Tomcat

Follow-up - This is now working (sadly I'm not sure exactly why!) but I've successfully used curl (under windows) and the following examples to parse content curl http://localhost:8080/apache-solr-1.4-dev/update/extract?extractOnly=tru e --data-binary @curl-config.pdf -H "Content-type:applicati

Re: TermsComponent results don't change after documents removed from index

Bill Au wrote: Should the results of the TermsComponent change after documents have been removed from the index? I am thinking about using the prefix of TermsComponent to implement auto-suggest. But I noticed that the prefix counts in TermsComponent don't change after documents have been delete

lucid kstem group and artifact id to put in POM

Hello Right now we are using lucid Kstemmer and it works fine and the two jars required "lucid-kstem.jar" and "lucid-solr-kstem.jar" are present in our web app. i am trying to get hold of groupid and artifact so that i can plug it into maven to download this two files from POM. i searched the mave

Re: adding and updating a lot of document to Solr, metadata extraction etc

The DIH has improved a great deal from Solr 1.3 to 1.4. You will be much better off using the DIH from this. This is the current Solr release candidate binary: http://people.apache.org/~gsingers/solr/1.4.0/ On Tue, Nov 3, 2009 at 8:08 AM, Eugene Dzhurinsky wrote: > On Mon, Nov 02, 2009 at 05:45:

Re: Plugin Performance Issues

: : : : : : : : ... : only do indexing on the master server. However, with this schema in place : on the slaves, as well as our custom.jar in the solrHome/lib directory, we : run into these issues where the memory usage grows and grows without : explanation. .

Re: Spell check suggestion and correct way of implementation and some Questions

Thanks i included the buildOncommit and buildOnOptimize as true and indexed some documents and it automatically builds the dictionary. Are there any performance issues we should be aware of, with this approach. Rashid -- View this message in context: http://old.nabble.com/Spell-check-suggest

Re: can't find solr.xml

: I have downloaded apache-solr-1.3.0.tgz for Linux and don't see solr.xml. can : someone assist. example/multicore/solr.xml is an example solr.xml file demonstrating the Multi-Core support in solr 1.3. as mentioned on the wiki, you can run the multicore example using this command... java -Ds

Re: TermsComponent results don't change after documents removed from index

Thanks for pointing that out. The TermsComponent prefix query is running much faster than the facet prefix query. I guess there is yet another reason to optimize the index. Bill On Tue, Nov 3, 2009 at 5:09 PM, Koji Sekiguchi wrote: > Bill Au wrote: > >> Should the results of the TermsComponen

Re: question about collapse.type = adjacent

Hi Martijn, This clarifies it all for me. Thanks a lot! Michael Martijn v Groningen wrote: > > Hi Micheal, > > Field collapsing is basicly done in two steps. The first step is to > get the uncollapsed sorted (whether it is score or a field value) > documents and the second step is to apply

Re: apply a patch on solr

Perfect. This is what I need to know instead of patching 'in the dark'. Good thing SVN revision cuts across all files like a tag. Thanks Mike! Michael cambridgemike wrote: > > You can see what revision the patch was written for at the top of the > patch, > it will look like this: > > Index

Re: apply a patch on solr

Hmmm, perhaps I jumped the gun. I just looked over the field collapse patch for SOLR-236 and each file listed in the patch has its own revision #. E.g. from field-collapse-5.patch: --- src/java/org/apache/solr/core/SolrConfig.java (revision 824364) --- src/solrj/org/apache/solr/client/so

Re: apply a patch on solr

patch -p0 < /path/to/field-collapse-5.patch On Tue, Nov 3, 2009 at 7:48 PM, michael8 wrote: > > Hmmm, perhaps I jumped the gun.  I just looked over the field collapse patch > for SOLR-236 and each file listed in the patch has its own revision #. > > E.g. from field-collapse-5.patch: > --- src/jav

Highlighting performance between 1.3 and 1.4rc

Hi, The fix MarkM provided yesterday for the problem I reported encountering with the highlighter appears to be working--I installed the Lucene 2.9.1 rc4 artifacts. Now I'm running into an oddity regarding performance. Our integration test is running slower than it used to. I've placed some av

Re: apply a patch on solr

sorry got cut off, patch, then ant clean dist, will give you the modified solr war file, if it doesnt apply cleanly (which i dont think is currently the case), you can go back to the latest revision referenced in the patch, On Tue, Nov 3, 2009 at 8:17 PM, Joe Calderon wrote: > patch -p0 < /path/

Re: apply a patch on solr

Thanks, but my question here was not about the patch command itself (which I already know), but about simpler way (if any) to go about guaranteeing a proper patch with the right file revisions needed by the patch. Michael Joe Calderon-2 wrote: > > patch -p0 < /path/to/field-collapse-5.patch >

Re: Highlighting performance between 1.3 and 1.4rc

The 1.4 highlighter is Now slower if you have multi term queries or phrase queries. You can get the old behavior (which is faster) if you pass usePhraseHighlighter=false - but you will not get correct phrase highlighting and multi term queries won't highlight - eg prefix/ wildcard/range. -

Re: solr search

Thank you for your reply. I have corrected this error, but now I am getting this error -- HTTP Status 500 - Bad version number in .class file java.lang.UnsupportedClassVersionError: Bad version number in .class file at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.def

Sending file to Solr via HTTP POST

Hi, >From the Solr wiki on ExtractingRequestHandler tutorial, when it comes to the part to post file to Solr, it always uses the curl command, e.g. curl 'http://localhost:8983/*solr*/update/extract?literal.id=doc1&commit=true' -F myfi...@tutorial.html I have never used curl and i was thinking is

RE: Highlighting performance between 1.3 and 1.4rc

Thanks Mark, that did bring the time back down. I'll have to investigate a little more, and weigh the pros of each to determine which best suits are needs. Jake -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, November 03, 2009 11:23 PM To: solr-user@lu

Re: solr search

start with the examples in the download. That should help On Wed, Nov 4, 2009 at 11:14 AM, manishkbawne wrote: > > Thank you for your reply. I have corrected this error, but now I am getting > this error -- > > HTTP Status 500 - Bad version number in .class file > java.lang.UnsupportedClassVersio