help for preprocessing the query

2008-05-12 Thread Umar Shah
Hi, Due some requirement I need to transform the user queries before passing it to the standard handler in Solr, can anyone suggest me the best way to do this. I will need to use a transfomation class that would provide functions to process the input query 'qIn' and transform it to the resultant

Re: help for preprocessing the query

2008-05-12 Thread Koji Sekiguchi
Hi Umar, You may be able to preprocess your request parameter in your servlet filter. In the doFilter() method, you do: ServletRequest myRequest = new MyServletRequestWrapper( request ); : chain.doFilter( myRequest, response ); And you have MyServletRequestWrapper that extends ServletRequest

MultiLingual Search

2008-05-12 Thread Sachit P. Menon
Hello folks, My project requires having the same content (mostly) in multiple languages. Do I need to have different files for the same content in different languages? This will need indexing of every file whenever a new content appears? Or can we have a mapping of the content with all ot

Re: MultiLingual Search

2008-05-12 Thread Alexander Ramos Jardim
I would look toward some implementations: 1st: You could have one index for each language. Just record the preferred language in session and use it to select the index you are searching in. Pros: It is easy to add a new language, just create put another index instance online. Cons:It can become e

Re: Extending XmlRequestHandler

2008-05-12 Thread Alexander Ramos Jardim
Just continuing on my quest. How difficult it would be to make a RequestHandler that understands a given soap request? 2008/5/9 Alexander Ramos Jardim <[EMAIL PROTECTED]>: > Thanks, > > > To maybe save you from reinventing the wheel, when I asked a similar > > question a couple weeks back, hossma

RE: Solr hardware specs

2008-05-12 Thread dudes dudes
thanks all, they were all valuable information . btw: is there any e-book on Solr ? many thanks, ak > Date: Fri, 9 May 2008 13:45:05 -0700 > Subject: Re: Solr hardware specs > From: [EMAIL PROTECTED] > To: solr-user@lucene.apache.org > > And use a log

Re: MultiLingual Search

2008-05-12 Thread Norberto Meijome
On Mon, 12 May 2008 16:16:28 +0530 "Sachit P. Menon" <[EMAIL PROTECTED]> wrote: > My project requires having the same content (mostly) in multiple languages. hi Sachit, please search the archives of the list. this topic seems to come up twice a week or thereabouts :) You are of course encoura

Re: Extending XmlRequestHandler

2008-05-12 Thread Erik Hatcher
On May 12, 2008, at 8:31 AM, Alexander Ramos Jardim wrote: How difficult it would be to make a RequestHandler that understands a given soap request? I would implement this at the servlet API layer (or rather some SOAP toolkit, like Axis)... and wire in to Solr's API there. Erik

Re: Extending XmlRequestHandler

2008-05-12 Thread Alexander Ramos Jardim
So, Wouldn't you use a SoapRequestHandler? Would you use SolrJ to make the wiring? Or would you put the SOAP on the solr server side? 2008/5/12 Erik Hatcher <[EMAIL PROTECTED]>: > > On May 12, 2008, at 8:31 AM, Alexander Ramos Jardim wrote: > > > How difficult it would be to make a RequestHandle

Re: Extending XmlRequestHandler

2008-05-12 Thread Erik Hatcher
On May 12, 2008, at 9:18 AM, Alexander Ramos Jardim wrote: Wouldn't you use a SoapRequestHandler? First of all, *I* wouldn't really want to be caught coding up any kind of client or server SOAP call to Solr. Seems mostly ridiculous to me when Solr's response is malleable to practically any

Re: How Special Character '&' used in indexing

2008-05-12 Thread Ricky
Hi Mike, Thanx for your reply. I have got the answer to the question posted. I know people are donating time here. ASAP doesnt mean that am demanding them to reply fast. Please read the lines before you comment something(*Please kindly* reply ASAP). Am a newbie and with curiosity i have requested

Re: Extending XmlRequestHandler

2008-05-12 Thread Alexander Ramos Jardim
Erik, Thanks for the comments. But they raised some doubts in my mind. What I need to do is to integrate Solr to an environment that communicates via wsdl/SOAP. There will be lots of web services communicating to Solr. Solr will be used like a web service, so I need to make possible for the other

Re: Extending XmlRequestHandler

2008-05-12 Thread Shalin Shekhar Mangar
Performance wise, it would be best for your web services to communicate to Solr using SolrJ. I'm sure it would be better performance-wise than SOAP and you won't need to do anything custom with Solr. If you're using, Solr 1.3, you can have a *huge* performance boost by using the BinaryResponseParse

Re: Extending XmlRequestHandler

2008-05-12 Thread Erik Hatcher
On May 12, 2008, at 9:52 AM, Alexander Ramos Jardim wrote: I understood what you said about putting the SOAP at Solr. I agree. That's not smart. Now, I am thinking about the web service talking with an embedded Solr server. Is that you were talking about? Quite pleasantly you don't even real

Re: Extending XmlRequestHandler

2008-05-12 Thread Alexander Ramos Jardim
Nice. I will try that with Solr-1.3 as Shalin suggests. 2008/5/12 Erik Hatcher <[EMAIL PROTECTED]>: > > On May 12, 2008, at 9:52 AM, Alexander Ramos Jardim wrote: > > > I understood what you said about putting the SOAP at Solr. I agree. > > That's > > not smart. > > Now, I am thinking about the

Re: Missing content Stream

2008-05-12 Thread Ricky
Hi Hoss, * 1) Posting the exact same question twice because you didn't get a reply in the first 8 hours isn't going to encourage people to reply faster. best case scenerio: you waste people's time they could be spending reading another email; worst case scnerio: you irk people and put them in a ba

Re: Multicore and SolrResourceLoader

2008-05-12 Thread Grant Ingersoll
On May 10, 2008, at 1:03 PM, Chris Hostetter wrote: : I've been digging around in multicore and I am curious as to how to force a : reload of the sharedLib classloader. I can reload a given core, which : instantiates a new SolrResourceLoader for that core, but I want to be able to : re

Re: help for preprocessing the query

2008-05-12 Thread Umar Shah
On Mon, May 12, 2008 at 2:50 PM, Koji Sekiguchi <[EMAIL PROTECTED]> wrote: > Hi Umar, > > You may be able to preprocess your request parameter in your > servlet filter. In the doFilter() method, you do: > > ServletRequest myRequest = new MyServletRequestWrapper( request ); Thanks for your respon

Re: help for preprocessing the query

2008-05-12 Thread Shalin Shekhar Mangar
ServletRequest and ServletRequestWrapper are part of the Java servlet-api (not Solr). Basically, Koji is hinting at writing a ServletFilter implementation (again using servlet-api) and creating a wrapper ServletRequest which modifies the underlying request params which can then be used by Solr. On

Re: help for preprocessing the query

2008-05-12 Thread Umar Shah
On Mon, May 12, 2008 at 8:42 PM, Shalin Shekhar Mangar < [EMAIL PROTECTED]> wrote: > ServletRequest and ServletRequestWrapper are part of the Java servlet-api > (not Solr). Basically, Koji is hinting at writing a ServletFilter > implementation (again using servlet-api) and creating a wrapper > Ser

Re: help for preprocessing the query

2008-05-12 Thread Koji Sekiguchi
Shalin Shekhar Mangar write: ServletRequest and ServletRequestWrapper are part of the Java servlet-api (not Solr). Basically, Koji is hinting at writing a ServletFilter implementation (again using servlet-api) and creating a wrapper ServletRequest which modifies the underlying request params whic

Re: help for preprocessing the query

2008-05-12 Thread Grant Ingersoll
I haven't written one, but I _think_ you could just implement a QParser that does the transformation. See the LuceneQParser or the DismaxQParser. On May 12, 2008, at 4:59 AM, Umar Shah wrote: Hi, Due some requirement I need to transform the user queries before passing it to the standa

result limit / diversity with an OR query

2008-05-12 Thread s d
Hi,I have a query similar to: x OR y OR z and i want to know if there is a way to make sure i get 1 result with x, 1 result with y and one with z ? Alternatively, is it possible to achieve through facets? Thanks, S.

Re: result limit / diversity with an OR query

2008-05-12 Thread Francisco Sanmartin
the easy answer is: x AND y AND z . This will return ALL the documents containing x,y and z. But if you want also get the documents containin AT LEAST one of the three, try this: (x AND y AND z)^10 OR (x OR y OR z) (the idea is boosting the AND query) this way, the documents that "x and y

Re: help for preprocessing the query

2008-05-12 Thread Shalin Shekhar Mangar
You'll *not* write a servlet. You'll write implement the Filter interface http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/servlet/Filter.html In the doFilter method, you'll create a ServletRequestWrapper which changes the incoming param. Then you'll call chain.doFilter with the new request obj

Re: exceeded limit of maxWarmingSearchers

2008-05-12 Thread Sasha Voynow
Thanks for the advice. Unfortunately, my plan was to two have two instances both running as "masters" although one would only be a warm-standby for querying purposes. I just wanted a little bit of redundancy for the moment and I though a true master-slave setup would be overkill. Is it really probl

Re: Unlimited number of return documents?

2008-05-12 Thread Marc Bechler
Hi all, one possible use case could be to synchronize the index against a given database. E.g., assume that you have a filesystem that is indexed periodically. If files are deleted on this filesystem, they will not be deleted in the index. This way, you can get (e.g.) the complete content fro

Re: Loading performance slowdown at ~ 400K documents

2008-05-12 Thread Mike Klaas
Glad to hear it. Incidentally, lowering maxBufferedDocs will reduce peak memory consumption during indexing, at a cost of slower indexing throughput. -Mike On 11-May-08, at 3:41 AM, Tracy Flynn wrote: Thanks for the replies. For a completely different reason, I happened to look at the me

Re: result limit / diversity with an OR query

2008-05-12 Thread Mike Klaas
On 12-May-08, at 9:31 AM, s d wrote: Hi,I have a query similar to: x OR y OR z and i want to know if there is a way to make sure i get 1 result with x, 1 result with y and one with z ? The easiest way is to execute three separate queries: +x y z x +y z x y +z -Mike

Re: Unlimited number of return documents?

2008-05-12 Thread Shalin Shekhar Mangar
Hi Marc, Think about how one would go about implementing a manual database table to table synchronization. It may not be a good idea to iterate all rows from the target database and checking for existence in the source database to remove rows which were deleted in the source table. The best way to

single character terms in index - why?

2008-05-12 Thread Naomi Dushay
I'm experienced with Lucene, less so than SOLR. I am looking at two systems built on top of SOLR for a library discovery service: blacklight and vufind. I checked the raw lucene index using Luke and noticed that both of these indexes have single character terms in the index, such as "d" o

Re: Unlimited number of return documents?

2008-05-12 Thread Francisco Sanmartin
Just to know it, what where the thresholds were u got the exception? ( I want to know the order of magnitude, i know it depends on the machine and the config, but is just to know have an approximate idea). Thanks Pako Marc Bechler wrote: Hi all, one possible use case could be to synchronize

Re: single character terms in index - why?

2008-05-12 Thread Yonik Seeley
On Mon, May 12, 2008 at 4:13 PM, Naomi Dushay <[EMAIL PROTECTED]> wrote: > So I'm now asking: why would SOLR want single character terms? Solr, like Lucene, can be configured however you want. The example schema is just that - an example. But, there are many field types that might be intereste

JMX monitoring

2008-05-12 Thread Marshall Weir
Hi, I'm new to Solr and I've been attempting to get JMX monitoring working. I can get simple information by using the - Dcom.sun.management.jmxremote command line switch, but I'd like to get more useful statistics. I've been working on applying the SOLR-256 and jmx patch, but the original

indexing pdf documents

2008-05-12 Thread Cam Bazz
Hello, Before making a little program to extract the txt from my pdfs and feed it into solr with xml, I just wanted to check if solr has capability to digest pdf files apart from xml? Best Regards, -C.B.

Re: indexing pdf documents

2008-05-12 Thread Chris Harris
Solr does not have this support built in, but there's a patch for it: https://issues.apache.org/jira/browse/SOLR-284 On Mon, May 12, 2008 at 2:02 PM, Cam Bazz <[EMAIL PROTECTED]> wrote: > Hello, > > Before making a little program to extract the txt from my pdfs and feed it > into solr with xml,

Re: AND vs. OR query performance

2008-05-12 Thread Yonik Seeley
In general, AND will perform better than OR (because of skipping in the scorers). But if the number of documents matching the AND is close to that matching the OR query, then skipping doesn't gain you much and probably has a little more overhead. -Yonik On Sun, May 11, 2008 at 4:04 AM, Lars Kott

Re: AND vs. OR query performance

2008-05-12 Thread Lars Kotthoff
Thanks for the clarification. The behaviour I'm seeing is that OR queries are almost *twice* as performant as AND queries, so that's probably down to my specific setup/data. I'll try to investigate further. Lars On Mon, 12 May 2008 19:35:00 -0400 "Yonik Seeley" <[EMAIL PROTECTED]> wrote: > In ge

Selecting data with an order on string field causes slow commits from then on

2008-05-12 Thread David Stevenson
We have a table that has roughly 1M rows. If we run a query against the table and order by a string field that has a large number of unique values then subsequent commits of any other document takes much longer. If we don't run the query or if we order on a string field with very few unique value

Re: Selecting data with an order on string field causes slow commits from then on

2008-05-12 Thread Mike Klaas
This was answered yesterday on the list: http://www.nabble.com/Re%3A-exceeded-limit-of-maxWarmingSearchers-p17165631.html regards, -Mike On 12-May-08, at 6:12 PM, David Stevenson wrote: We have a table that has roughly 1M rows. If we run a query against the table and order by a string field

Field Grouping

2008-05-12 Thread oleg_gnatovskiy
Hello. I was wondering if there is a way to get solr to return fields with the same value for a particular field together. For example I might want to have all the documents with exactly the same name field all returned next to each other. Is this possible? Thanks! -- View this message in context

Re: Field Grouping

2008-05-12 Thread Yonik Seeley
On Mon, May 12, 2008 at 9:58 PM, oleg_gnatovskiy <[EMAIL PROTECTED]> wrote: > Hello. I was wondering if there is a way to get solr to return fields with > the same value for a particular field together. For example I might want to > have all the documents with exactly the same name field all ret

Re: Field Grouping

2008-05-12 Thread oleg_gnatovskiy
But I don't want the search results to be ranked based on that field. I only want all the documents with the same value grouped together... The way my system is set up, most documents will have that field empty. Thus, if Is rot by it, those documents that have a value will bubble to the top...

Re: JMX monitoring

2008-05-12 Thread Shalin Shekhar Mangar
Hi Marshall, I've uploaded a new patch which works off the current trunk. Let me know if you run into any problems with this. On Tue, May 13, 2008 at 2:36 AM, Marshall Weir <[EMAIL PROTECTED]> wrote: > Hi, > > I'm new to Solr and I've been attempting to get JMX monitoring working. I > can get si