search in related multiValued fields
Hi, I am stuck at the search in related multiValued fields. Suppose I have the following data: in-550803 Ibrahim EL AMRANI GFI OUTSOURCIA 2 2 com-550804 Mehdi LAHLOU Smart Development GFI 2 1 The "experience_company", "experience_years" are linked, so the value of the first "experience_company" is linked with the first "experience_years" and the second with the second and so on ... If I want, for example, to select the people whose fields "experience_company" equal "GFI", and "experience_years" equal 2, how to express the request? q = Thanks -- View this message in context: http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16120730.html Sent from the Solr - User mailing list archive at Nabble.com.
Smart Indexing for Better performance and functionality ??
Hi, I have the following use case. I could implement the solution but performance is affected. I need some smart ways of doing this. Use Case : Incoming data has two fields which have values like 'WAL MART STORES INC' and 'wal-mart-stores-inc'. Users can search the data either in 'walmart' 'wal mart' or 'wal-mart' also partially on any part of the name from the start of word like 'wal', 'walm' 'wal m' etc . I could get the solution by using two indexes, one as text field for the first field (wal mart ) column and sub word wal-mart-stores (with WordDelimiterFilterFactory filter). Is there a smart way of doing or any other techniques to boost the performance? I need to use them for a high traffic application where the response requirements are around 50 milli seconds. I have some control on modifying the incoming data and data set is around 100K records. Can someone suggest better ways of implementing. I can provide more information the tokens and filters I am using. Thanks Ravi -- View this message in context: http://www.nabble.com/Smart-Indexing-for-Better-performance-and-functionalitytp16121987p16121987.html Sent from the Solr - User mailing list archive at Nabble.com.
Changing Solr Query Syntax
Hi solr users, I need to change the query format for solr a little bit. How can I accomplish this. I don't wan to modify the underlying lucene query specification but just the way I query the index through the the GET http method in solr. Thanks a lot for your help. Ben
Re: Changing Solr Query Syntax
Hi Ben, It would be nice if you can tell us your use-case so that we can be more helpful. Why does the normal query syntax not work well for you? What are you trying to accomplish? Maybe there is an easier way. On Tue, Mar 18, 2008 at 8:17 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > Hi solr users, > I need to change the query format for solr a little bit. How can I > accomplish this. I don't wan to modify the underlying lucene query > specification but just the way I query the index through the the GET http > method in solr. > Thanks a lot for your help. > > Ben > -- Regards, Shalin Shekhar Mangar.
sort by index id descending?
Is there any way to sort by index id - descending? (by order of indexed) Thanks, Jae
Re: search in related multiValued fields
I have the same problem please help me :-( émile coué wrote: > > Hi, > > I am stuck at the search in related multiValued fields. > Suppose I have the following data: > > in-550803 > Ibrahim EL AMRANI > > GFI > OUTSOURCIA > > > 2 > 2 > > > > com-550804 > Mehdi LAHLOU > > Smart Development > GFI > > > 2 > 1 > > > > The "experience_company", "experience_years" are linked, so the value of > the first "experience_company" is linked with the first "experience_years" > and the second with the second and so on ... > > If I want, for example, to select the people whose fields > "experience_company" equal "GFI", and "experience_years" equal 2, how to > express the request? q = > > Thanks > -- View this message in context: http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16124015.html Sent from the Solr - User mailing list archive at Nabble.com.
Connection reset exception
Hi, I have been using solr for sometime now. The solr.war is deployed on a windows box on a Jboss server. I was getting a java.net.SocketException connection reset exception all of a sudden from yesterday morning. I am not sure as to what's happening cos there has been no change to the environment what so ever. This had happened to me once before and I had posted the same but failed to get any response back. As the server was a dev box, I could happily re start my jboss and make the exception disappear the last time. But I am worried now as this is occurring once again. The only change that I see that was different from 2 months back to now, is that due to development activities, the number of hits to the solr box has drastically reduced. But from this, there has been no change. I have checked the N/W connections and stuff like that from the box and they seem OK. Any help on this is really appreciated. Warm Regards, Sundar Sankarnarayanan Software Engineer @University of Phoenix
Re: Changing Solr Query Syntax
Shalin, thanks a lot for answering that fast. Use Case: I'm migrating from a proprietary index server (XYZ) to Solr. All my applications and my customer's applications relay on the query specification of XYZ. It would be hard to modify all those apps to use the Solr Query Syntax (although, it would be ideal, Sorl query is a lot superior than that of XYZ, but impractical). On Tue, Mar 18, 2008 at 9:50 AM, Shalin Shekhar Mangar < [EMAIL PROTECTED]> wrote: > Hi Ben, > > It would be nice if you can tell us your use-case so that we can be > more helpful. > > Why does the normal query syntax not work well for you? What are you > trying to accomplish? Maybe there is an easier way. > > On Tue, Mar 18, 2008 at 8:17 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > > Hi solr users, > > I need to change the query format for solr a little bit. How can I > > accomplish this. I don't wan to modify the underlying lucene query > > specification but just the way I query the index through the the GET > http > > method in solr. > > Thanks a lot for your help. > > > > Ben > > > > > > -- > Regards, > Shalin Shekhar Mangar. >
Re: Changing Solr Query Syntax
Hi Shalin, thanks a lot for answering that fast. Use Case: I'm migrating from a proprietary index server (XYZ) to Solr. All my applications and my customer's applications relay on the query specification of XYZ. It would be hard to modify all those apps to use the Solr Query Syntax (although, it would be ideal, Sorl query is a lot superior than that of XYZ). Basically I need to replace : with = ; + with / and = with : in the query syntax. Thank you. On Tue, Mar 18, 2008 at 9:50 AM, Shalin Shekhar Mangar < [EMAIL PROTECTED]> wrote: > Hi Ben, > > It would be nice if you can tell us your use-case so that we can be > more helpful. > > Why does the normal query syntax not work well for you? What are you > trying to accomplish? Maybe there is an easier way. > > On Tue, Mar 18, 2008 at 8:17 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > > Hi solr users, > > I need to change the query format for solr a little bit. How can I > > accomplish this. I don't wan to modify the underlying lucene query > > specification but just the way I query the index through the the GET > http > > method in solr. > > Thanks a lot for your help. > > > > Ben > > > > > > -- > Regards, > Shalin Shekhar Mangar. >
Re: Changing Solr Query Syntax
Hi Ben, If I had to do this, I would start by adding a custom javax.servlet.Filter into Solr. It should work fine since all you're doing is replacing characters in the q parameter for requests coming into /select handler. It's a bit hackish but that's exactly what you're trying to do :) Don't know if there's an alternate/easier way. On Tue, Mar 18, 2008 at 9:30 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > Hi Shalin, thanks a lot for answering that fast. > > > Use Case: > I'm migrating from a proprietary index server (XYZ) to Solr. All my > applications and my customer's applications relay on the query specification > of XYZ. It would be hard to modify all those apps to use the Solr Query > Syntax (although, it would be ideal, Sorl query is a lot superior than that > of XYZ). > > Basically I need to replace : with = ; + with / and = with : in the query > syntax. > > Thank you. > > > On Tue, Mar 18, 2008 at 9:50 AM, Shalin Shekhar Mangar < > [EMAIL PROTECTED]> wrote: > > > > > Hi Ben, > > > > It would be nice if you can tell us your use-case so that we can be > > more helpful. > > > > Why does the normal query syntax not work well for you? What are you > > trying to accomplish? Maybe there is an easier way. > > > > On Tue, Mar 18, 2008 at 8:17 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > > > Hi solr users, > > > I need to change the query format for solr a little bit. How can I > > > accomplish this. I don't wan to modify the underlying lucene query > > > specification but just the way I query the index through the the GET > > http > > > method in solr. > > > Thanks a lot for your help. > > > > > > Ben > > > > > > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > -- Regards, Shalin Shekhar Mangar.
Re: Changing Solr Query Syntax
Shalin, Thanks a lot. I'll do that. On Tue, Mar 18, 2008 at 11:13 AM, Shalin Shekhar Mangar < [EMAIL PROTECTED]> wrote: > Hi Ben, > > If I had to do this, I would start by adding a custom > javax.servlet.Filter into Solr. It should work fine since all you're > doing is replacing characters in the q parameter for requests coming > into /select handler. It's a bit hackish but that's exactly what > you're trying to do :) > > Don't know if there's an alternate/easier way. > > On Tue, Mar 18, 2008 at 9:30 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > > Hi Shalin, thanks a lot for answering that fast. > > > > > > Use Case: > > I'm migrating from a proprietary index server (XYZ) to Solr. All my > > applications and my customer's applications relay on the query > specification > > of XYZ. It would be hard to modify all those apps to use the Solr Query > > Syntax (although, it would be ideal, Sorl query is a lot superior than > that > > of XYZ). > > > > Basically I need to replace : with = ; + with / and = with : in the > query > > syntax. > > > > Thank you. > > > > > > On Tue, Mar 18, 2008 at 9:50 AM, Shalin Shekhar Mangar < > > [EMAIL PROTECTED]> wrote: > > > > > > > > > Hi Ben, > > > > > > It would be nice if you can tell us your use-case so that we can be > > > more helpful. > > > > > > Why does the normal query syntax not work well for you? What are you > > > trying to accomplish? Maybe there is an easier way. > > > > > > On Tue, Mar 18, 2008 at 8:17 PM, Ben Sanchez <[EMAIL PROTECTED]> > wrote: > > > > Hi solr users, > > > > I need to change the query format for solr a little bit. How can I > > > > accomplish this. I don't wan to modify the underlying lucene query > > > > specification but just the way I query the index through the the > GET > > > http > > > > method in solr. > > > > Thanks a lot for your help. > > > > > > > > Ben > > > > > > > > > > > > > > > > -- > > > Regards, > > > Shalin Shekhar Mangar. > > > > > > > > > -- > Regards, > Shalin Shekhar Mangar. >
Re: Changing Solr Query Syntax
The other option would be a custom QParserPlugin. -Yonik On Tue, Mar 18, 2008 at 12:18 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > Shalin, Thanks a lot. I'll do that. > > On Tue, Mar 18, 2008 at 11:13 AM, Shalin Shekhar Mangar < > > > [EMAIL PROTECTED]> wrote: > > > Hi Ben, > > > > If I had to do this, I would start by adding a custom > > javax.servlet.Filter into Solr. It should work fine since all you're > > doing is replacing characters in the q parameter for requests coming > > into /select handler. It's a bit hackish but that's exactly what > > you're trying to do :) > > > > Don't know if there's an alternate/easier way. > > > > On Tue, Mar 18, 2008 at 9:30 PM, Ben Sanchez <[EMAIL PROTECTED]> wrote: > > > Hi Shalin, thanks a lot for answering that fast. > > > > > > > > > Use Case: > > > I'm migrating from a proprietary index server (XYZ) to Solr. All my > > > applications and my customer's applications relay on the query > > specification > > > of XYZ. It would be hard to modify all those apps to use the Solr Query > > > Syntax (although, it would be ideal, Sorl query is a lot superior than > > that > > > of XYZ). > > > > > > Basically I need to replace : with = ; + with / and = with : in the > > query > > > syntax. > > > > > > Thank you. > > > > > > > > > On Tue, Mar 18, 2008 at 9:50 AM, Shalin Shekhar Mangar < > > > [EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > Hi Ben, > > > > > > > > It would be nice if you can tell us your use-case so that we can be > > > > more helpful. > > > > > > > > Why does the normal query syntax not work well for you? What are you > > > > trying to accomplish? Maybe there is an easier way. > > > > > > > > On Tue, Mar 18, 2008 at 8:17 PM, Ben Sanchez <[EMAIL PROTECTED]> > > wrote: > > > > > Hi solr users, > > > > > I need to change the query format for solr a little bit. How can I > > > > > accomplish this. I don't wan to modify the underlying lucene query > > > > > specification but just the way I query the index through the the > > GET > > > > http > > > > > method in solr. > > > > > Thanks a lot for your help. > > > > > > > > > > Ben > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > Shalin Shekhar Mangar. > > > > > > > > > > > > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > >
Re: Connection reset exception
Sundar, Most likely this is not related to Solr. I would look in JBoss logs and the logs for the servlet container that you are using with JBoss. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Sundar Sankaranarayanan <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, March 18, 2008 11:54:45 AM Subject: Connection reset exception Hi, I have been using solr for sometime now. The solr.war is deployed on a windows box on a Jboss server. I was getting a java.net.SocketException connection reset exception all of a sudden from yesterday morning. I am not sure as to what's happening cos there has been no change to the environment what so ever. This had happened to me once before and I had posted the same but failed to get any response back. As the server was a dev box, I could happily re start my jboss and make the exception disappear the last time. But I am worried now as this is occurring once again. The only change that I see that was different from 2 months back to now, is that due to development activities, the number of hits to the solr box has drastically reduced. But from this, there has been no change. I have checked the N/W connections and stuff like that from the box and they seem OK. Any help on this is really appreciated. Warm Regards, Sundar Sankarnarayanan Software Engineer @University of Phoenix
Re: Connection reset exception
Sundar, Most likely this is not related to Solr. I would look in JBoss logs and the logs for the servlet container that you are using with JBoss. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Sundar Sankaranarayanan <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, March 18, 2008 11:54:45 AM Subject: Connection reset exception Hi, I have been using solr for sometime now. The solr.war is deployed on a windows box on a Jboss server. I was getting a java.net.SocketException connection reset exception all of a sudden from yesterday morning. I am not sure as to what's happening cos there has been no change to the environment what so ever. This had happened to me once before and I had posted the same but failed to get any response back. As the server was a dev box, I could happily re start my jboss and make the exception disappear the last time. But I am worried now as this is occurring once again. The only change that I see that was different from 2 months back to now, is that due to development activities, the number of hits to the solr box has drastically reduced. But from this, there has been no change. I have checked the N/W connections and stuff like that from the box and they seem OK. Any help on this is really appreciated. Warm Regards, Sundar Sankarnarayanan Software Engineer @University of Phoenix
Re: Smart Indexing for Better performance and functionality ??
Yerraguntla, Which approaches have you tried so far? You want a query for "walmart" to match a document that, in its original input form contains "wal mart"? It sounds like you may want to try the n-gram approach with one of the NGram analyzers/factories. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Yerraguntla <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, March 18, 2008 10:43:07 AM Subject: Smart Indexing for Better performance and functionality ?? Hi, I have the following use case. I could implement the solution but performance is affected. I need some smart ways of doing this. Use Case : Incoming data has two fields which have values like 'WAL MART STORES INC' and 'wal-mart-stores-inc'. Users can search the data either in 'walmart' 'wal mart' or 'wal-mart' also partially on any part of the name from the start of word like 'wal', 'walm' 'wal m' etc . I could get the solution by using two indexes, one as text field for the first field (wal mart ) column and sub word wal-mart-stores (with WordDelimiterFilterFactory filter). Is there a smart way of doing or any other techniques to boost the performance? I need to use them for a high traffic application where the response requirements are around 50 milli seconds. I have some control on modifying the incoming data and data set is around 100K records. Can someone suggest better ways of implementing. I can provide more information the tokens and filters I am using. Thanks Ravi -- View this message in context: http://www.nabble.com/Smart-Indexing-for-Better-performance-and-functionalitytp16121987p16121987.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: search in related multiValued fields
Hi, Go to your Solr admin page and enter this in that big text area: experience_company:GFI AND experience_years:2 Then look at the URL after you've submitted the search. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: émile coué <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, March 18, 2008 9:02:42 AM Subject: search in related multiValued fields Hi, I am stuck at the search in related multiValued fields. Suppose I have the following data: in-550803 Ibrahim EL AMRANI GFI OUTSOURCIA 2 2 com-550804 Mehdi LAHLOU Smart Development GFI 2 1 The "experience_company", "experience_years" are linked, so the value of the first "experience_company" is linked with the first "experience_years" and the second with the second and so on ... If I want, for example, to select the people whose fields "experience_company" equal "GFI", and "experience_years" equal 2, how to express the request? q = Thanks -- View this message in context: http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16120730.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: search in related multiValued fields
Hi, Thank you Otis Gospodnetic, but experience_company:GFI AND experience_years:2 will give me both docs, it should return just the first. My need is : poeple having GFI in "experience_company" and 2 in "experience_years" where GFI and 2 are in same position in the vector. I've told : > The "experience_company", "experience_years" are linked, so the value of > the > first "experience_company" is linked with the first "experience_years" and > the second with the second and so on ... > Thanks Mehdi LAHLOU Otis Gospodnetic wrote: > > Hi, > > Go to your Solr admin page and enter this in that big text area: > > experience_company:GFI AND experience_years:2 > > Then look at the URL after you've submitted the search. > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > - Original Message > From: émile coué <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Sent: Tuesday, March 18, 2008 9:02:42 AM > Subject: search in related multiValued fields > > > Hi, > > I am stuck at the search in related multiValued fields. > Suppose I have the following data: > > in-550803 > Ibrahim EL AMRANI > > GFI > OUTSOURCIA > > > 2 > 2 > > > > com-550804 > Mehdi LAHLOU > > Smart Development > GFI > > > 2 > 1 > > > > The "experience_company", "experience_years" are linked, so the value of > the > first "experience_company" is linked with the first "experience_years" and > the second with the second and so on ... > > If I want, for example, to select the people whose fields > "experience_company" equal "GFI", and "experience_years" equal 2, how to > express the request? q = > > Thanks > -- > View this message in context: > http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16120730.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- View this message in context: http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16125917.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: sort by index id descending?
Finding the way how to sort by internal_docid desc. Thanks, Jae On Tue, Mar 18, 2008 at 11:41 AM, Jae Joo <[EMAIL PROTECTED]> wrote: > Is there any way to sort by index id - descending? (by order of indexed) > > Thanks, > Jae >
Re: search in related multiValued fields
Hello Mehdi, You might need to concatenate the fields and create a combined value such as GFI-2 instead of leaving them separate, or do both in the same document so you have the option of filtering by either. --j On Tue, Mar 18, 2008 at 10:09 AM, émile coué <[EMAIL PROTECTED]> wrote: > > Hi, > Thank you Otis Gospodnetic, but experience_company:GFI AND > experience_years:2 will give me both docs, it should return just the first. > My need is : poeple having GFI in "experience_company" and 2 in > "experience_years" where GFI and 2 are in same position in the vector. > > I've told : > > > > > The "experience_company", "experience_years" are linked, so the value of > > the > > first "experience_company" is linked with the first "experience_years" and > > the second with the second and so on ... > > > > Thanks > Mehdi LAHLOU > > > > > > Otis Gospodnetic wrote: > > > > Hi, > > > > Go to your Solr admin page and enter this in that big text area: > > > > experience_company:GFI AND experience_years:2 > > > > Then look at the URL after you've submitted the search. > > > > Otis > > -- > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message > > From: émile coué <[EMAIL PROTECTED]> > > To: solr-user@lucene.apache.org > > Sent: Tuesday, March 18, 2008 9:02:42 AM > > Subject: search in related multiValued fields > > > > > > Hi, > > > > I am stuck at the search in related multiValued fields. > > Suppose I have the following data: > > > > in-550803 > > Ibrahim EL AMRANI > > > > GFI > > OUTSOURCIA > > > > > > 2 > > 2 > > > > > > > > com-550804 > > Mehdi LAHLOU > > > > Smart Development > > GFI > > > > > > 2 > > 1 > > > > > > > > The "experience_company", "experience_years" are linked, so the value of > > the > > first "experience_company" is linked with the first "experience_years" and > > the second with the second and so on ... > > > > If I want, for example, to select the people whose fields > > "experience_company" equal "GFI", and "experience_years" equal 2, how to > > express the request? q = > > > > Thanks > > -- > > View this message in context: > > > http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16120730.html > > Sent from the Solr - User mailing list archive at Nabble.com. > > > > > > -- > View this message in context: > http://www.nabble.com/search-in-related-multiValued-fields-tp16120730p16125917.html > > > Sent from the Solr - User mailing list archive at Nabble.com. > >
RE: Connection reset exception
Hi Otis, Thanks for the response. The Jboss logs were the one that pointed out to the connection reset. What could be the problem here. Solr is the only things that is running on that box. These are the logs, if it can help. 2008-03-18 08:30:36,740 [STDERR:152] ERROR - Mar 18, 2008 8:30:36 AM org.apache.solr.core.SolrException log SEVERE: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.ja va:747) at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRe ad(InternalInputBuffer.java:777) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInpu tFilter.java:115) at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer. java:712) at org.apache.coyote.Request.doRead(Request.java:418) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java :284) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:404) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:299) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.j ava:192) at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411) at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183) at java.io.InputStreamReader.read(InputStreamReader.java:167) at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:2972) at org.xmlpull.mxp1.MXParser.more(MXParser.java:3026) at org.xmlpull.mxp1.MXParser.parseEndTag(MXParser.java:1671) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1172) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at org.xmlpull.mxp1.MXParser.nextTag(MXParser.java:1078) at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequest Handler.java:298) at org.apache.solr.handler.XmlUpdateRequestHandler.update(XmlUpdateRequestH andler.java:162) at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpd ateRequestHandler.java:84) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB ase.java:77) at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja va:191) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j ava:159) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilte r.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAs sociationValve.java:175) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.j ava:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnecti onValve.java:156) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1 48) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:86 9) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proc essConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint .java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorker Thread.java:112) at java.lang.Thread.run(Thread.java:595) Warm Regards, Sundar Sankarnarayanan Software Engineer @University of Phoenix -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 18, 2008 9:46 AM To: solr-user@lucene.apache.org Subject: Re: Connection reset exception Sundar, Most likely this is not related to Solr. I would look in JBoss logs and the logs for the servlet container that you are using with JBoss. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Sundar Sankaranarayanan <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, March 18, 2008 11:54:45 AM Subject: Connection reset exception Hi, I have been using solr for sometime now. The solr.war is deployed on a windows box on a Jboss server. I was getting a java.net.SocketException connection reset exception all of a su
Re: sorting on aggregate averages
: I have a problem of returning an list of results which is sorted on a : average of ranks returned from aggregates. : the qury would be something like ? : q=product:p1+product:p2+product:p3; sort score desc : To explain Supose I have documents with fields Product, Manufacturer, Rank : and I want to return the top manufacturers across products p1,p2,p3 with : highest average rank on these products. the topic of generating statistics on facet constraints has come up before ... but nothing for doing that is provided out of the box at the moment. while basic stats like the min/mean/median/stddev/max of a numeric facet field (in the context of a q/fq) would be relativeily straight forward to add to Solr's built in simple facet support; more complex types statistics (like hat you describe) would be difficult to implement in a way that would be generally reusable through simple query params ... however: it would probably be fairly straightfoward to implemnt domain specific stats like this directly in a custom plugin. The new SearchComponents framework available in the trunk would probably be an easy way to do this, allthough it's not very well documented at the moment. If you lok at the existing FacetComponent however, seeing how it generates facet counts, and extending it to know about your specific fields and generate the type of stats you want should be possible. -Hoss
Re: Smart way of indexing for Better performance
: I have the following use case. I could implement the solution but : performance is affected. I need some smart ways of doing this. : Use Case : : Incoming data has two fields which have values like 'WAL MART STORES INC' : and 'wal-mart-stores-inc'. : Users can search the data either in 'walmart' 'wal mart' or 'wal-mart' : also partially on any part of the name from the start of word like 'wal', : 'walm' 'wal m' etc . I could get the solution by using two indexes, one : as text field for the first field (wal mart ) column and sub word : wal-mart-stores (with WordDelimiterFilterFactory filter). there are lots of solutions that could work, all depending on what *else* you need to be able to match on besides just prefix queries where whitespace/punctuation are ignored. One example: using KeywordTokenizer, along with a PatternReplaceFilter that throws away non letter charagers and a LowercaseFilter and then issuing all your queries as PrefixQueries will get w* wa* wal* and walm* to all match "wal mart", "WALMART", "WAL-mart", etc but that won't let "mart" match a document contain "wal mart" .. but you can always use copyField and hit one field for the first type of query, and the other field for "normal" queries. depending on the nature of your data (ie: how many documents, how common certian prefixes are, etc...) you might get better performacne at the expense of a larger index if you use something like the EdgeNGramTokenFilter or EdgeNGramTokenizer to index all the prefixes of various sizes so you don't need to do a prefix query The bottom line: there are *lots* of options, you'll need to experimentto find the right solution that matches when you want to match, and doesn't when you don't -Hoss
Re: sorting on a multivalued field
On Mon, Mar 17, 2008 at 12:39 PM, Chris Hostetter <[EMAIL PROTECTED]> wrote: > > : It appears that adding sort functions would be done in Lucene, and not > > : in solr. I'm not sure I want to go down that path, so I'm wondering > : if there's a way to accomplish this with solr. From recent > : discussions, it sounds like I might be able to do this with some boost > : magic. Unfortunately, I haven't found any examples of boosting that > : seem close to what I want to do. > > I can't think of anyway to accomplish anything like this without writing > some custom Java code in Solr ... either some custom ValueSources to use > in FunctionQuery, or a custom Sort object. > the custom Sort object seems a bit more direct. I'm not very familiar with the solr source. Can you give me some idea of how to get started -- maybe this is now a better discussion for solr-dev . . . -- - Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind. -- Dr. Seuss
Re: Date Range Query + Fields
: For instance: A document with a start_date of 1/1/08 and an end_date : of 3/1/08 should still match for a search of the range 2/1/08 to : 2/2/08. if I'm understanding you correctly, what you want is something like... startDate:[* TO NOW+30DAY] AND endDate:[NOW TO *] ..that should give you all documents where the document's start/end range overlaps with the range of NOW to NOW+30DAYS. although to more explicit i would write that as... +startDate:[* TO NOW+30DAY] +endDate:[NOW TO *] ...and then i'd wonder if "30 days" is really what i want, or if i actually want a month, and i was just assuming 30 days, so i'd then use... +startDate:[* TO NOW+1MONTH] +endDate:[NOW TO *] ...and because millisecond granularity is almost never that important, i would then change it to... +startDate:[* TO NOW+1DAY/DAY+1MONTH] +endDate:[NOW/DAY TO *] -Hoss
Re: search in related multiValued fields
The topic of "related multivalued fields" has been discussed before. as Jayson describes there are tricks you can use to build a single field containing all the vlaues in such a way that you can query for combinations; but it would also be very possible to build a new type of phrase or span query that would know about multiple fields and ensure that a match on fieldA occured near the same "position" as matches on fieldB... http://www.nabble.com/related-multivalued-fields-to10288703.html#a10313384 http://www.nabble.com/One-item%2C-multiple-fields%2C-and-range-queries-to8308844.html#a8377712 ...but no one that i know of has attempted to implement the general purpose solution in Solr (it would be a non trivial amount of work: creating the new Lucnee Query classes, adding QueyParser syntax to take advantage of them, etc...) -Hoss
Solr question
Hi All: I am new to Solr and I am facing a question if solr would can be helpful in a project that I'm working on. The project is a client/server app that requires a client app to index the documents and send the results in rdf to server. The client needs to be smart enough to know when a new document has been added to a specified folder, index it and send the results in rdf/xml to the server. The server will be a web service which will parse the xml and store the metadata in the a database. The search will be conducted on the server and will return the results from the database which will be links to the documents on the client. The client , which is also running a webserver will take the request when the user clicks on the link to the document residing on the client. I believe lucene will be useful in this scenario and solr can be used as a web app. I would like to get any input on this architecture and would request any pointers if there is any app already doing something similar and how lucene/solr can be useful in this case. Thanks for your time. Best Regards Raghav Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs
Re: sort by index id descending?
: Is there any way to sort by index id - descending? (by order of indexed) Not that i can think of. Lucene already has support for it, so it would probably be a fairly simple patch if someone wanted to try to implement it, we just need some syntax to make the parameter parsing construct the right Sort object -- allthough I'm loath to add another "magic" field name like "score" since "docid" or "id" or anything else we can think of could easily conflict with a field name in someones schema. if we add something like this I'd want to add configuration to solrconfig.xml to determine what the "magic" field names for sorting by internal id and score should be. -Hoss
Re: Connection reset exception
Sundar - I can't tell. Could it be a timeout issue? Otis (Jetty fan who learned to dislike JBoss 5/6/7 years ago) -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Sundar Sankaranarayanan <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Tuesday, March 18, 2008 3:42:14 PM Subject: RE: Connection reset exception Hi Otis, Thanks for the response. The Jboss logs were the one that pointed out to the connection reset. What could be the problem here. Solr is the only things that is running on that box. These are the logs, if it can help. 2008-03-18 08:30:36,740 [STDERR:152] ERROR - Mar 18, 2008 8:30:36 AM org.apache.solr.core.SolrException log SEVERE: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.ja va:747) at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRe ad(InternalInputBuffer.java:777) at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInpu tFilter.java:115) at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer. java:712) at org.apache.coyote.Request.doRead(Request.java:418) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java :284) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:404) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:299) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.j ava:192) at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411) at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183) at java.io.InputStreamReader.read(InputStreamReader.java:167) at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:2972) at org.xmlpull.mxp1.MXParser.more(MXParser.java:3026) at org.xmlpull.mxp1.MXParser.parseEndTag(MXParser.java:1671) at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1172) at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093) at org.xmlpull.mxp1.MXParser.nextTag(MXParser.java:1078) at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequest Handler.java:298) at org.apache.solr.handler.XmlUpdateRequestHandler.update(XmlUpdateRequestH andler.java:162) at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpd ateRequestHandler.java:84) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB ase.java:77) at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja va:191) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j ava:159) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilte r.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:202) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:173) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.java:178) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAs sociationValve.java:175) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.j ava:74) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :105) at org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnecti onValve.java:156) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. java:107) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1 48) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:86 9) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proc essConnection(Http11BaseProtocol.java:664) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint .java:527) at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorker Thread.java:112) at java.lang.Thread.run(Thread.java:595) Warm Regards, Sundar Sankarnarayanan Software Engineer @University of Phoenix -Original Message- From: Otis Gospodnetic [mailto:[EMAIL PROTECTED] Sent: Tuesday, March 18, 2008 9:46 AM To: solr-user@lucene.apache.org Subject: Re: Connection reset exception Sundar, Most likely this is not related to Solr. I would look in JBoss logs and the logs for the servlet container that you are using with JBoss. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - O
introduction and help!
I normally wouldn't just signup to a list and post immediately but... I hope there are some Tomcat experts here. I'm trying to setup solr and tomcat. I get the following: INFO: HTMLManager: start: Starting web application at '/solr' Mar 19, 2008 12:57:26 AM org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() Mar 19, 2008 12:57:26 AM org.apache.solr.core.Config getInstanceDir INFO: No /solr/home in JNDI Mar 19, 2008 12:57:26 AM org.apache.solr.core.Config getInstanceDir INFO: Solr home defaulted to 'null' (could not find system property or JNDI) Mar 19, 2008 12:57:26 AM org.apache.solr.core.Config setInstanceDir INFO: Solr home set to 'solr/' Mar 19, 2008 12:57:26 AM org.apache.catalina.core.StandardContext filterStart SEVERE: Exception starting filter SolrRequestFilter I get that I should be specifying my solr home somewhere and some have suggested that it should be specified in $CATALINA_HOME/conf/Catalina/localhost Others have mentioned specifying in a solr.xml file placed in the $CATALINA_HOME/conf directory. Others have said that solr.war is a special webapp that should not be placed in the standard Tomcat webapps directory and JAVA_OPTS should have solr.solr.home set. Huh? Where? I tried setting it in the ENV before starting Tomcat to no avail. Sigh. Help! I'm running Tomcat 6.0.16 and SOLR 1.2.0 1. What the heck is a "conf/Catalina/localhost?" Is it a directory? A file? I have neither a Catalina or a localhost directory there. 2. Tomcat does absolutely nothing unless I put solr.war in its webapps directory. Then I can get it to fail on solr, at least. 3. I tried putting the following into my server.xml file (no effect): 4. I tried putting the same into my context.xml file but then it just overrode the context for the manager and ruined that whole thing for me. Honestly. Any help would be *much* appreciated but shouldn't Tomcat be the easiest way to run SOLR? Thank you, David
Tomcat 6.0 solr home not set
(Sorry for the double post - I committed one of my pet peeves - I didn't create a descriptive subject) I normally wouldn't just signup to a list and post immediately but... I hope there are some Tomcat experts here. I'm trying to setup solr and tomcat. I get the following: INFO: HTMLManager: start: Starting web application at '/solr' Mar 19, 2008 12:57:26 AM org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() Mar 19, 2008 12:57:26 AM org.apache.solr.core.Config getInstanceDir INFO: No /solr/home in JNDI Mar 19, 2008 12:57:26 AM org.apache.solr.core.Config getInstanceDir INFO: Solr home defaulted to 'null' (could not find system property or JNDI) Mar 19, 2008 12:57:26 AM org.apache.solr.core.Config setInstanceDir INFO: Solr home set to 'solr/' Mar 19, 2008 12:57:26 AM org.apache.catalina.core.StandardContext filterStart SEVERE: Exception starting filter SolrRequestFilter I get that I should be specifying my solr home somewhere and some have suggested that it should be specified in $CATALINA_HOME/conf/Catalina/localhost Others have mentioned specifying in a solr.xml file placed in the $CATALINA_HOME/conf directory. Others have said that solr.war is a special webapp that should not be placed in the standard Tomcat webapps directory and JAVA_OPTS should have solr.solr.home set. Huh? Where? I tried setting it in the ENV before starting Tomcat to no avail. Sigh. Help! I'm running Tomcat 6.0.16 and SOLR 1.2.0 1. What the heck is a "conf/Catalina/localhost?" Is it a directory? A file? I have neither a Catalina or a localhost directory there. 2. Tomcat does absolutely nothing unless I put solr.war in its webapps directory. Then I can get it to fail on solr, at least. 3. I tried putting the following into my server.xml file (no effect): 4. I tried putting the same into my context.xml file but then it just overrode the context for the manager and ruined that whole thing for me. Honestly. Any help would be *much* appreciated but shouldn't Tomcat be the easiest way to run SOLR? Thank you, David
Re: Tomcat 6.0 solr home not set
: Honestly. Any help would be *much* appreciated but shouldn't Tomcat be : the easiest way to run SOLR? The *easiest* way to get Solr up and running is to follow the example on the tutorial to start the Jetty server included with the release (the tutorial assumes you are using a nightly build, but the same thing works with the official releases)... http://lucene.apache.org/solr/tutorial.html cd example java -jar start.jar ...it really is that easy. If that works for you, and you want to try tomcat next, then you should start by understanding why you're seeing differnet docs talking about solr.solr.home system properties, and JNDI, and context files. Solr is very flexible in how you cna tell it where to find it's configuration files -- called the "Solr Home" -- just pick the one that meets your needs... http://wiki.apache.org/solr/SolrInstall If you look at the Solr wikipage on Tomcat... http://wiki.apache.org/solr/SolrTomcat ...you'll see a "Simple Example Install" which works a lot like the out of hte box jetty install -- it assumes the solr home dir is ./solr relative where you start tomcat from. (As written it uses tomcat 5.5.25 but the steps would be exactly the same for tomcat 6). For a more "permenant" Tomcat installation, I would suggest using JNDI ... which means creating a context file. See the "Configuring Solr Home with JNDI" section of the same wiki page. (NOTE: the JNDI section had some very missleading comments about Tomcat 6 which I just (re)moved) -Hoss
