RE: [ANN] Solr 1.4.1 Released
It appears the 1.4.1 version was deployed with a new maven groupId For eample, if you are trying to download solr-core, here are the differences between 1.4.0 and 1.4.1. 1.4.0 groupId: org.apache.solr artifactId: solr-core 1.4.1 groupId: org.apache.solr.solr artifactId:solr-core Was this change intentional or a mistake? If it was a mistake, can someone please fix it in maven's central repository? thanks, Jason -Original Message- From: Mark Miller [mailto:markrmil...@apache.org] Sent: Fri 6/25/2010 6:23 AM To: solr-user@lucene.apache.org; gene...@lucene.apache.org; annou...@apache.org Subject: [ANN] Solr 1.4.1 Released -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Apache Solr 1.4.1 has been released and is now available for public download! http://www.apache.org/dyn/closer.cgi/lucene/solr/ Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites. Solr is written in Java and runs as a standalone full-text search server within a servlet container such as Tomcat. Solr uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language. Solr's powerful external configuration allows it to be tailored to almost any type of application without Java coding, and it has an extensive plugin architecture when more advanced customization is required. Solr 1.4.1 is a bug fix release for Solr 1.4 that includes many Solr bug fixes as well as Lucene bug fixes from Lucene 2.9.3. See all of the CHANGES here: http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1/CHANGES.txt - - Mark Miller on behalf of the Solr team -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMJK3AAAoJED+/0YJ4eWrIrfAP/RLD7QvreOBFebICN/eiRzCH 1dHOt9Scn7qGQU4RvXZ8GQq37AuoRMgmgckntttFLCCD5w5A29/GxzyZbAoQDQ0B OkaHsYIcUuhbLq8QtlTjt+rK3gc6oxMoCRMJBS7DfUFUyROl6om4gpYAVem50qDy FfBdgRxp4VZ07E7VwmMvma03nSrKuvX0bwE8NXksaCAVsvkmi8Sh7aLMPPVHgsuD pbY8kB0hXCULJgs9ZAc2t6+T38+eV9wxJSeAktVlGAvNlYTavW2bxzF5wQk+kXCd DwGjdlU9/ebHdx3MHJyE0zXSl4rGFsy8zfh/ntk7UV7qklQ2jn5Ur18zLqv4vkb1 Ea78GpoqCZWlMGcRUSErtH33cGs4blo/kuJZj/VLrk6jxO4x4beUsAfRcM/YliJW Z6OuFtpcdVDjVl4aB2xbAMwDl2DXqgyNmlxs8vvqdRoDhN8wZ91raO0kkbrkzj1f 5gPD//Efx6RcrYtXAV3HKAwI7FLP8MhzFu1Y2FK2FY7DyFNmirad03+pB6bFs1xq ARU6pdeTYvv+PsWH3Keaw/L/nb0BYbU8R1sVhkvjm+S9gJ6cCcKJkeAkNgL+6QNm JPJ5VeXVFGVmwzQ5mE3j6qX1uDrEmLA2T5Dd7bssWtwveLoyfo0s7qezIfbRamnc T3iyCE6cuSU9CvCEqN+o =nBB9 -END PGP SIGNATURE-
REST calls
The solr docs say it is RESTful, yet it seems that it doesn't use http headers in a RESTful way. For example, it doesn't seem to use the Accept: request header to determine the media-type to be returned. Instead, it requires a query parameter to be used in the URL. Also, it doesn't seem to use return 304 Not Modified if the request header "if-modified-since" is used. Am I doing something wrong or is Solr not truly completely RESTful? thanks, Jason
URLDataSource
I would like to the URLDataSource to make RESTful calls to get content and only re-index when content changes. This means using http headers to make a request and using the response headers to determine when to make the request. For example, Request Headers: Accept: application/xml if-modified-since: timestamp Response Headers: Expires: timestamp Etag: etag In this case Solr would make a request or the specified media type by adding it to the accept header. Also, it would use a timestamp in the if-modified-since on requests after the first request. This timestamp would be the last time that indexing took place. So, we only want to index again if changes happened. The RESTful service would return content the first time contacted with the expires header, which would tell Solr when is the next time it should check for new content to be indexed. At that point the RESTful service could return 304 Not Modified or it could return new content. If it returns new content, it is indexed. Otherwise, Solr reads the new Expires header to see when it should make the next request. My question is whether or not there is anything in Solr that currently supports this or if I would have to implement this myself? I wasn't able to find anything. thanks, Jason
RE: REST calls
Using Accept headers is a pretty standard practice and so are conditional GETs. Quite easy to test with curl: curl -X GET -H "Accept:application/xml" http://solr.com/search curl -X GET -H "Accept:application/json" http://solr.com/search Jason -Original Message- From: Don Werve [mailto:d...@madwombat.com] Sent: Tuesday, June 29, 2010 9:40 PM To: solr-user@lucene.apache.org Subject: Re: REST calls 2010/6/27 Jason Chaffee > The solr docs say it is RESTful, yet it seems that it doesn't use http > headers in a RESTful way. For example, it doesn't seem to use the Accept: > request header to determine the media-type to be returned. Instead, it > requires a query parameter to be used in the URL. Also, it doesn't seem to > use return 304 Not Modified if the request header "if-modified-since" is > used. > The summary: Solr is restful, and does a very good job of it. The long version: There is no official 'REST' standard that dictates the behavior of the implementation; rather, REST is a set of guidelines on building APIs that are both discoverable and easily usable without having to resort to third-party libraries. Generally speaking, an application is RESTful if it provides an API that accepts arguments passed as HTTP form variables, returns results in an open format (XML, JSON, YAML, etc.), and respects certain semantics relating to HTTP verbs; e.g., GET/HEAD return the resource without modification, DELETEs are destructive, POST creates a resource, PUT alters it. Solr meets all of these requirements. The nature of the result format, and how to change it, is entirely up to the implementer. A common convention is to use a filename extension (.json, .xml) appended to the URL. It's less common to specify the request format as part of the query parameters (like Solr does), but not unheard of. And, to be honest, this is actually the first time I've heard of using the 'Accept' header to change the result format, as it makes it a lot harder to use a web browser, or command-line tools like curl or wget, to debug your API.
RE: REST calls
In that case, being able to use Accept headers and conditional GET's would make them more powerful and easier to use. The Accept header could be used, if present, otherwise use the query parameter. Or, vice versa. Also, conditional GET's are a big win when you know the data and results are not changing often. Jason -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Wednesday, June 30, 2010 7:12 AM To: solr-user@lucene.apache.org Subject: Re: REST calls Solr's APIs are described as "REST-like", and probably do qualify as "restful" the way the term is commonly used. I'm personally much more interested in making our APIs more powerful and easier to use, regardless of any REST purity tests. -Yonik http://www.lucidimagination.com
RE: REST calls
Two more jaxrs solutions: http://www.jboss.org/resteasy http://cxf.apache.org/docs/jax-rs.html However, I am not suggesting changing the core implementation. Just want to make it more powerful by utilizing headers. I can accept the other issues that have been mentioned as not RESTful. Also, I do plan to make patches for the issues I mentioned. I just wanted to know if I was missing anything or someone else already had contributed an extension. Jason -Original Message- From: Ryan McKinley [mailto:ryan...@gmail.com] Sent: Wednesday, June 30, 2010 3:07 PM To: solr-user@lucene.apache.org Subject: Re: REST calls If there is a real desire/need to make things "restful" in the official sense, it is worth looking at using a REST framework as the controller rather then the current solution. perhaps: http://www.restlet.org/ https://jersey.dev.java.net/ These would be cool since they encapsulate lots of the request plumbing work that it would be better if we could leverage more widely used approaches then support our own. That said, what we have is functional and powerful -- if you are concerned about people editing the index (with GET/POST or whatever) there are plenty of ways to solve this. ryan On Wed, Jun 30, 2010 at 5:31 PM, Lance Norskog wrote: > I've looked at the problem. It's fairly involved. It probably would > take several iterations. (But not as many as field collapsing :) > > On Wed, Jun 30, 2010 at 2:11 PM, Yonik Seeley > wrote: >> On Wed, Jun 30, 2010 at 4:55 PM, Lance Norskog wrote: >>> Apparently this is not ReStFuL It is IMVHO insane. >> >> Patches welcome... >> >> -Yonik >> http://www.lucidimagination.com >> > > > > -- > Lance Norskog > goks...@gmail.com >
RE: REST calls
Cool, I must have configured something wrong then, because it wasn't working for me. Thanks! -Original Message- From: Erik Hatcher [mailto:erik.hatc...@gmail.com] Sent: Wednesday, June 30, 2010 7:51 PM To: solr-user@lucene.apache.org Subject: Re: REST calls Solr has 304 support with the last-modified and etag headers. Erik On Jun 30, 2010, at 7:52 PM, Jason Chaffee wrote: > In that case, being able to use Accept headers and conditional GET's > would make them more powerful and easier to use. The Accept header > could be used, if present, otherwise use the query parameter. Or, > vice > versa. Also, conditional GET's are a big win when you know the data > and > results are not changing often. > > Jason > > -Original Message- > From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik > Seeley > Sent: Wednesday, June 30, 2010 7:12 AM > To: solr-user@lucene.apache.org > Subject: Re: REST calls > > Solr's APIs are described as "REST-like", and probably do qualify as > "restful" the way the term is commonly used. > > I'm personally much more interested in making our APIs more powerful > and easier to use, regardless of any REST purity tests. > > -Yonik > http://www.lucidimagination.com
DataImport problem
I am getting the following error with the DataImport and I am not sure why as I following the documentation. I am trying to use XPath and the URLDataSource but it fails to load the datasource. SEVERE: Full Import failed org.apache.solr.handler.dataimport.DataImportHandlerException: No dataSource :null available for entity :store Processing Document # 1 at org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:279) at org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:94) at org.apache.solr.handler.dataimport.XPathEntityProcessor.init(XPathEntityProcessor.java:78) at org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:71) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:319) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:331) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370) Sep 4, 2010 7:12:51 PM org.apache.solr.update.DirectUpdateHandler2 rollback Here is my dataconfig.xml: http://localhost:8080/app"; encoding="UTF-8" connectionTimeout="5000" readTimeout="6" />
multivalued fields in result
Is it possible to return multivalued files in the result? I would like to have a multivalued field that is stored and not indexed (I also copy the same field into another field where it is tokenized and indexed). I would then like all the values of this field returned in the result set. Is there a way to do this? If it is not possible, could someone elaborate why that is so that I may see if I can make it work. thanks, Jason
RE: multivalued fields in result
But it doesn't seem to be returning mulitvalued fields that are stored. It is returning all of the single value fields though. -Original Message- From: Markus Jelsma [mailto:markus.jel...@buyways.nl] Sent: Sat 9/11/2010 4:19 AM To: solr-user@lucene.apache.org Subject: RE: multivalued fields in result Yes, you'll get what is stored and asked for. -Original message----- From: Jason Chaffee Sent: Sat 11-09-2010 05:27 To: solr-user@lucene.apache.org; Subject: multivalued fields in result Is it possible to return multivalued files in the result? I would like to have a multivalued field that is stored and not indexed (I also copy the same field into another field where it is tokenized and indexed). I would then like all the values of this field returned in the result set. Is there a way to do this? If it is not possible, could someone elaborate why that is so that I may see if I can make it work. thanks, Jason
RE: multivalued fields in result
My schema.xml was fine. The problem was that my test queries weren't returning top 10 documents that had data in the fields. Once I increased the rows, I saw the results. Definitely user error. :) Thanks for help though. Jason -Original Message- From: Lance Norskog [mailto:goks...@gmail.com] Sent: Sun 9/12/2010 6:23 PM To: solr-user@lucene.apache.org Subject: Re: multivalued fields in result Also, the 'v' is capitalized: multiValued. (This is one reason why posting your schema helps.) Erick Erickson wrote: > Can we see your schema file? Because it sounds like you didn't > really declare your field multivalued="true" on the face of things. > > But if it is multivalued AND you changed it, did you reindex after > you changed the schema? > > Best > Erick > > On Sun, Sep 12, 2010 at 4:21 AM, Jason Chaffee wrote: > > >> But it doesn't seem to be returning mulitvalued fields that are stored. It >> is returning all of the single value fields though. >> >> >> -Original Message- >> From: Markus Jelsma [mailto:markus.jel...@buyways.nl] >> Sent: Sat 9/11/2010 4:19 AM >> To: solr-user@lucene.apache.org >> Subject: RE: multivalued fields in result >> >> Yes, you'll get what is stored and asked for. >> >> -Original message- >> From: Jason Chaffee >> Sent: Sat 11-09-2010 05:27 >> To: solr-user@lucene.apache.org; >> Subject: multivalued fields in result >> >> Is it possible to return multivalued files in the result? >> >> I would like to have a multivalued field that is stored and not indexed (I >> also copy the same field into another field where it is tokenized and >> indexed). I would then like all the values of this field returned in the >> result set. Is there a way to do this? >> >> If it is not possible, could someone elaborate why that is so that I may >> see if I can make it work. >> >> thanks, >> >> Jason >> >> >> >
using variables/properties in dataconfig.xml
Is it possible to use the same type of property configuration in
dataconfig.xml as is possible in solrconfig.xml?
I tried it and it didn't seem to work. For example,
${solr.data.dir:/opt/search/store/solr/data}
And in the dataconfig.xml, I would like to do this to configure the
baseUrl:
Thanks,
Jason
*:* query with dismax
I am using dismax and trying to use q=*:* to return all indexed documents. However, it is always returning 0 found. If I used the default select (not dismax) handler and try q=*:* then it returns all documents. There is nothing in the logs to indicate why this happening. Does anyone have any clues? Thanks, Jason
RE: *:* query with dismax
Can you shed some light on what you did to configure it to handle *:*? I have the same issue that I need it to work for faceting, but I do need the dismax abilities as well. -Original Message- From: Mark Mandel [mailto:mark.man...@gmail.com] Sent: Friday, May 06, 2011 4:30 PM To: solr-user@lucene.apache.org Subject: Re: *:* query with dismax This is exactly what should be happening, as the dismax parser doesn't understand regular query syntax (and for good reason too). This tripped me up as well when I first started using dismax. Solution for me was to comfigure the handler to use *:* when the query is empty, so that you can still get back a full result set if you need it, say for faceting. HTH Mark On May 7, 2011 9:22 AM, "Jason Chaffee" wrote: > I am using dismax and trying to use q=*:* to return all indexed > documents. However, it is always returning 0 found. > > > > If I used the default select (not dismax) handler and try q=*:* then it > returns all documents. > > > > There is nothing in the logs to indicate why this happening. > > > > Does anyone have any clues? > > > > Thanks, > > > > Jason >
RE: *:* query with dismax
This is what I thought, but for some reason I wasn't able to get it to work before. Thus, I sent the email. However, maybe I forgot to not specify q because it is working fine for me now. Thanks everyone. -Original Message- From: Rob Casson [mailto:rob.cas...@gmail.com] Sent: Fri 5/6/2011 7:34 PM To: solr-user@lucene.apache.org Subject: Re: *:* query with dismax it does seem a little weird, but q.alt will get what you want: http://wiki.apache.org/solr/DisMaxQParserPlugin#q.alt hth, rc On Fri, May 6, 2011 at 7:41 PM, Jason Chaffee wrote: > Can you shed some light on what you did to configure it to handle *:*? > I have the same issue that I need it to work for faceting, but I do need > the dismax abilities as well. > > -Original Message- > From: Mark Mandel [mailto:mark.man...@gmail.com] > Sent: Friday, May 06, 2011 4:30 PM > To: solr-user@lucene.apache.org > Subject: Re: *:* query with dismax > > This is exactly what should be happening, as the dismax parser doesn't > understand regular query syntax (and for good reason too). This tripped > me > up as well when I first started using dismax. > > Solution for me was to comfigure the handler to use *:* when the query > is > empty, so that you can still get back a full result set if you need it, > say > for faceting. > > HTH > > Mark > On May 7, 2011 9:22 AM, "Jason Chaffee" wrote: >> I am using dismax and trying to use q=*:* to return all indexed >> documents. However, it is always returning 0 found. >> >> >> >> If I used the default select (not dismax) handler and try q=*:* then > it >> returns all documents. >> >> >> >> There is nothing in the logs to indicate why this happening. >> >> >> >> Does anyone have any clues? >> >> >> >> Thanks, >> >> >> >> Jason >> >
How to query multiple fields with phrases
I am migrating from lucene to solr and I am not quite sure how do what I need to. I need to do a search that will search 3 different fields and combine the results. First, it needs to not break the phrase into tokens, but rather treat it is a phrase for one field. The other fields need to be parsed with their normal analyzers. In Lucene, I did the following: 1.Ran the query through my custom analyzer and then created a PhraseQuery and then did a search 2.Used a MultiFieldQueryParser for the other two fields and then did a search 3.Combined the results of (1) and (2) into a Set and returned them. How can do the same thing in Solr? Thanks. Jason
Query elevation based on field
Is it possible to do query elevation based on field? Basically, I would like to search the same term on three different fields: q=field1:term OR field2:term OR field3:term and I would like to sort the results by fourth field sort=field4+asc However, I would like to elevate all of field1 matches to be at the beginning, with those matches sorted ascending and the rest of the field2 and field3 matches sorted ascending. Is this possible? Thanks.
Is it possible to disable fieldNorm?
I would like to either disable fieldNorm in the scoring or make sure that it is the same for all documents. I am creating EdgeNGrams and that can cause the number of terms for a document to be variable, but I do not want it do affect the scoring for this field. Is there an easy way to do this? thanks, Jason
How to use dismax and boosting properly?
I am using dismax and I have configured to search 3 different fields with one field getting an extra boost so that I the results of that field are at the top of result set. Then, I sort the results by another field to get the ordering. My problem is that the scores are being skewed by the adding the scores from the different fields. What I really want is to have all matches in the boost field have an equal score and take precedence over matches from other fields. I want them to have the same score so that the sorting will sort them alphabetically. Therefore, the scores must be the same. Because the query is being found in all three fields with different number of occurrences some scores are being skewed in the boosted matches and it is putting them at the top of my results and alphabetically, they should be near the bottom. Here is an example, in case my explanation isn't clear: I have dismax with the following config: Field1^3.0 Field2^0.1 Field3^0.1 score desc, sortField asc Where sortField is the original keyword token, without any processing except for lowercase. Field1 (the boosted field) a at at& at&t a ab abe abeb abebo abeboo abebook abebooks Field2 a at at& at&t a at att a at at at & at & at & t at&t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t abebooks a ab abe abeb abebo abeboo abebook abebooks Field3 a at at& at&t a at att a at at at & at & at & t at&t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t abebooks a ab abe abeb abebo abeboo abebook abebooks The user types in the query 'a': Here is the debugQuery: 5.4186125 = (MATCH) sum of: 2.7147598 = (MATCH) max plus 0.1 times others of: 0.10907243 = (MATCH) weight(Field2:a^0.1 in 80), product of: 0.01970195 = queryWeight(Field2:a^0.1), product of: 0.1 = boost 3.1962826 = idf(docFreq=117, maxDocs=1061) 0.0616402 = queryNorm 5.5361238 = (MATCH) fieldWeight(Field2:a in 80), product of: 1.7320508 = tf(termFreq(Field2:a)=3) 3.1962826 = idf(docFreq=117, maxDocs=1061) 1.0 = fieldNorm(field=Field2, doc=80) 2.7038527 = (MATCH) weight(Field1:a^3.0 in 80), product of: 0.7071054 = queryWeight(Field1:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = queryNorm 3.8238325 = (MATCH) fieldWeight(Field1:a in 80), product of: 1.0 = tf(termFreq(Field1:a)=1) 3.8238325 = idf(docFreq=62, maxDocs=1061) 1.0 = fieldNorm(field= Field1, doc=80) 2.7038527 = (MATCH) weight(Field1:a^3.0 in 80), product of: 0.7071054 = queryWeight(Field1:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = queryNorm 3.8238325 = (MATCH) fieldWeight(Field1:a in 80), product of: 1.0 = tf(termFreq(Field1:a)=1) 3.8238325 = idf(docFreq=62, maxDocs=1061) 1.0 = fieldNorm(field= Field1, doc=80) 5.4140024 = (MATCH) sum of: 2.71015 = (MATCH) max plus 0.1 times others of: 0.062973 = (MATCH) weight(edgeNGramStandardField:a^0.1 in 138), product of: 0.01970195 = queryWeight(edgeNGramStandardField:a^0.1), product of: 0.1 = boost 3.1962826 = idf(docFreq=117, maxDocs=1061) 0.0616402 = queryNorm 3.1962826 = (MATCH) fieldWeight(edgeNGramStandardField:a in 138), product of: 1.0 = tf(termFreq(edgeNGramStandardField:a)=1) 3.1962826 = idf(docFreq=117, maxDocs=1061) 1.0 = fieldNorm(field=edgeNGramStandardField, doc=138) 2.7038527 = (MATCH) weight(edgeNGramKeywordField:a^3.0 in 138), product of: 0.7071054 = queryWeight(edgeNGramKeywordField:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = queryNorm 3.8238325 = (MATCH) fieldWeight(edgeNGramKeywordField:a in 138), product of: 1.0 = tf(termFreq(edgeNGramKeywordField:a)=1) 3.8238325 = idf(docFreq=62, maxDocs=1061) 1.0 = fieldNorm(field=edgeNGramKeywordField, doc=138) 2.7038527 = (MATCH) weight(edgeNGramKeywordField:a^3.0 in 138), product of: 0.7071054 = queryWeight(edgeNGramKeywordField:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = queryNorm 3.8238325 = (MATCH) fieldWeight(edgeNGramKeywordField:a in 138), product of: 1.0 = tf(termFreq(edgeNGramKeywordField:a)=1) 3.8238325 = idf(docFreq=62, maxDocs=1061) 1.0 = fieldNorm(field=edgeNGramKeywordField, doc=138) As you can see the Boosted field scores are identical, so I would like those results to come above anything
RE: How to use dismax and boosting properly?
I thought I tried that, but I guess I didn't restart Solr to pick up the configuration. That did the trick. Thanks! -Original Message- From: Nagelberg, Kallin [mailto:knagelb...@globeandmail.com] Sent: Thursday, February 25, 2010 1:10 PM To: 'solr-user@lucene.apache.org' Subject: RE: How to use dismax and boosting properly? Try setting the boost to 0 for the fields you don't want to contribute to the score. Kallin Nagelberg -Original Message- From: Jason Chaffee [mailto:jchaf...@ebates.com] Sent: Thursday, February 25, 2010 4:03 PM To: solr-user@lucene.apache.org Subject: How to use dismax and boosting properly? I am using dismax and I have configured to search 3 different fields with one field getting an extra boost so that I the results of that field are at the top of result set. Then, I sort the results by another field to get the ordering. My problem is that the scores are being skewed by the adding the scores from the different fields. What I really want is to have all matches in the boost field have an equal score and take precedence over matches from other fields. I want them to have the same score so that the sorting will sort them alphabetically. Therefore, the scores must be the same. Because the query is being found in all three fields with different number of occurrences some scores are being skewed in the boosted matches and it is putting them at the top of my results and alphabetically, they should be near the bottom. Here is an example, in case my explanation isn't clear: I have dismax with the following config: Field1^3.0 Field2^0.1 Field3^0.1 score desc, sortField asc Where sortField is the original keyword token, without any processing except for lowercase. Field1 (the boosted field) a at at& at&t a ab abe abeb abebo abeboo abebook abebooks Field2 a at at& at&t a at att a at at at & at & at & t at&t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t abebooks a ab abe abeb abebo abeboo abebook abebooks Field3 a at at& at&t a at att a at at at & at & at & t at&t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t at&t att at & t abebooks a ab abe abeb abebo abeboo abebook abebooks The user types in the query 'a': Here is the debugQuery: 5.4186125 = (MATCH) sum of: 2.7147598 = (MATCH) max plus 0.1 times others of: 0.10907243 = (MATCH) weight(Field2:a^0.1 in 80), product of: 0.01970195 = queryWeight(Field2:a^0.1), product of: 0.1 = boost 3.1962826 = idf(docFreq=117, maxDocs=1061) 0.0616402 = queryNorm 5.5361238 = (MATCH) fieldWeight(Field2:a in 80), product of: 1.7320508 = tf(termFreq(Field2:a)=3) 3.1962826 = idf(docFreq=117, maxDocs=1061) 1.0 = fieldNorm(field=Field2, doc=80) 2.7038527 = (MATCH) weight(Field1:a^3.0 in 80), product of: 0.7071054 = queryWeight(Field1:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = queryNorm 3.8238325 = (MATCH) fieldWeight(Field1:a in 80), product of: 1.0 = tf(termFreq(Field1:a)=1) 3.8238325 = idf(docFreq=62, maxDocs=1061) 1.0 = fieldNorm(field= Field1, doc=80) 2.7038527 = (MATCH) weight(Field1:a^3.0 in 80), product of: 0.7071054 = queryWeight(Field1:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = queryNorm 3.8238325 = (MATCH) fieldWeight(Field1:a in 80), product of: 1.0 = tf(termFreq(Field1:a)=1) 3.8238325 = idf(docFreq=62, maxDocs=1061) 1.0 = fieldNorm(field= Field1, doc=80) 5.4140024 = (MATCH) sum of: 2.71015 = (MATCH) max plus 0.1 times others of: 0.062973 = (MATCH) weight(edgeNGramStandardField:a^0.1 in 138), product of: 0.01970195 = queryWeight(edgeNGramStandardField:a^0.1), product of: 0.1 = boost 3.1962826 = idf(docFreq=117, maxDocs=1061) 0.0616402 = queryNorm 3.1962826 = (MATCH) fieldWeight(edgeNGramStandardField:a in 138), product of: 1.0 = tf(termFreq(edgeNGramStandardField:a)=1) 3.1962826 = idf(docFreq=117, maxDocs=1061) 1.0 = fieldNorm(field=edgeNGramStandardField, doc=138) 2.7038527 = (MATCH) weight(edgeNGramKeywordField:a^3.0 in 138), product of: 0.7071054 = queryWeight(edgeNGramKeywordField:a^3.0), product of: 3.0 = boost 3.8238325 = idf(docFreq=62, maxDocs=1061) 0.0616402 = q
keyword query tokenizer
I have the following configured for a particular field: I am using dismax and querying multiple fields and I expect the query to be parsed different for each field. For some reason, it is not kept as single token for this field's query. For example, the query "Apple Store" is being broken into two tokens, "apple" and "store". I would expect it to be "apple store". Does anyone have ideas of what might be going on here? Thanks, Jason
Re: keyword query tokenizer
Thanks, didn't realize that. On Mar 25, 2010, at 4:04 PM, "Ahmet Arslan" wrote: > I have the following configured for a > particular field: > > > > > > class="solr.KeywordTokenizerFactory" /> > > class="solr.LowerCaseFilterFactory" /> > > > > > > > > I am using dismax and querying multiple fields and I expect > the query to > be parsed different for each field. For some reason, > it is not kept as > single token for this field's query. For example, the > query "Apple > Store" is being broken into two tokens, "apple" and > "store". I would > expect it to be "apple store". > > > > Does anyone have ideas of what might be going on here? Before analysis phase, QueryParser splits on whitespace. You can alter this behavior by escaping whitespace with back slash. apple\ store
Re: keyword query tokenizer
I am curious as to why the query parser does any tokenizing? I would think you would want control/configure this with your analyzers? Does anyone know the answer to this. Is there a performance gain or something? Thanks, Jason On Mar 25, 2010, at 4:04 PM, "Ahmet Arslan" wrote: > I have the following configured for a > particular field: > > > > > > class="solr.KeywordTokenizerFactory" /> > > class="solr.LowerCaseFilterFactory" /> > > > > > > > > I am using dismax and querying multiple fields and I expect > the query to > be parsed different for each field. For some reason, > it is not kept as > single token for this field's query. For example, the > query "Apple > Store" is being broken into two tokens, "apple" and > "store". I would > expect it to be "apple store". > > > > Does anyone have ideas of what might be going on here? Before analysis phase, QueryParser splits on whitespace. You can alter this behavior by escaping whitespace with back slash. apple\ store
RE: keyword query tokenizer
Seems like a short-coming to me. I would rather it not parse it unless there is some delimiter to break it, other than white space since white space is used in phrases. Instead of this, "name:bob content:climate" Maybe use a comma, "name:bob,content:climate" "name:bob foo bar,content:climate control" Then, I can also pass phrases to those fields and allow the analyzers to handle the tokenizing. Jason -Original Message- From: Tommy Chheng [mailto:tommy.chh...@gmail.com] Sent: Thursday, March 25, 2010 8:25 PM To: solr-user@lucene.apache.org Subject: Re: keyword query tokenizer Multi-field searches is one reason of doing the tokenizing in the parser. Imagine if your query was "name:bob content:climate" The parser can tokenize the query into "name:bob", "content:climate" and pass each into their own analyzer. Tommy Chheng Programmer and UC Irvine Graduate Student Twitter @tommychheng http://tommy.chheng.com On 3/25/10 7:37 PM, Jason Chaffee wrote: > I am curious as to why the query parser does any tokenizing? I would > think you would want control/configure this with your analyzers? > > Does anyone know the answer to this. Is there a performance gain or > something? > > Thanks, > > Jason > > On Mar 25, 2010, at 4:04 PM, "Ahmet Arslan" wrote: > >> > I have the following configured for a >> > particular field: >> > >> > >> > >> > >> > >> > > > class="solr.KeywordTokenizerFactory" /> >> > >> > > > class="solr.LowerCaseFilterFactory" /> >> > >> > >> > >> > >> > >> > >> > >> > I am using dismax and querying multiple fields and I expect >> > the query to >> > be parsed different for each field. For some reason, >> > it is not kept as >> > single token for this field's query. For example, the >> > query "Apple >> > Store" is being broken into two tokens, "apple" and >> > "store". I would >> > expect it to be "apple store". >> > >> > >> > >> > Does anyone have ideas of what might be going on here? >> >> Before analysis phase, QueryParser splits on whitespace. You can >> alter this behavior by escaping whitespace with back slash. apple\ store >> >> >> >
RE: keyword query tokenizer
I tried escaping the whitespace, but no avail. It is still be broken into two tokens and the whitespace. Has anyone else tried this? -Original Message- From: Ahmet Arslan [mailto:iori...@yahoo.com] Sent: Thursday, March 25, 2010 4:05 PM To: solr-user@lucene.apache.org Subject: Re: keyword query tokenizer > I have the following configured for a > particular field: > > > > > > class="solr.KeywordTokenizerFactory" /> > > class="solr.LowerCaseFilterFactory" /> > > > > > > > > I am using dismax and querying multiple fields and I expect > the query to > be parsed different for each field. For some reason, > it is not kept as > single token for this field's query. For example, the > query "Apple > Store" is being broken into two tokens, "apple" and > "store". I would > expect it to be "apple store". > > > > Does anyone have ideas of what might be going on here? Before analysis phase, QueryParser splits on whitespace. You can alter this behavior by escaping whitespace with back slash. apple\ store
RE: keyword query tokenizer
Got it working, there was a typo. -Original Message- From: Jason Chaffee [mailto:jchaf...@ebates.com] Sent: Friday, March 26, 2010 1:05 PM To: solr-user@lucene.apache.org Subject: RE: keyword query tokenizer I tried escaping the whitespace, but no avail. It is still be broken into two tokens and the whitespace. Has anyone else tried this? -Original Message- From: Ahmet Arslan [mailto:iori...@yahoo.com] Sent: Thursday, March 25, 2010 4:05 PM To: solr-user@lucene.apache.org Subject: Re: keyword query tokenizer > I have the following configured for a > particular field: > > > > > > class="solr.KeywordTokenizerFactory" /> > > class="solr.LowerCaseFilterFactory" /> > > > > > > > > I am using dismax and querying multiple fields and I expect > the query to > be parsed different for each field. For some reason, > it is not kept as > single token for this field's query. For example, the > query "Apple > Store" is being broken into two tokens, "apple" and > "store". I would > expect it to be "apple store". > > > > Does anyone have ideas of what might be going on here? Before analysis phase, QueryParser splits on whitespace. You can alter this behavior by escaping whitespace with back slash. apple\ store
RE: keyword query tokenizer
Ahh, but that is exactly what I don't want the DisjunctionMaxQuery to do. I do not max scoring field per "word". Instead, I want it per "phrase" which may be a single word or multiple words. -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Friday, March 26, 2010 10:35 PM To: solr-user@lucene.apache.org Subject: Re: keyword query tokenizer : : I am curious as to why the query parser does any tokenizing? I would think : you would want control/configure this with your analyzers? : : Does anyone know the answer to this. Is there a performance gain or something? it's not about performance, it's about hte query parser syntax. whitespace is "markup" as far as the query parser is concerned -- just like +,-, etc.. whitespace characters are instructions for the query parsers. Essentially: unquoted whitespace is the markup that tells the query parser to create an "OR" query out of the "chunks" of input on either side of hte space (+ signifies MUST, - signifies PROHIBITED, but there is no markup to signify "SHOULD") Also: if the query parser didn't chunk on whitespace queries like this... aWord aField:anotherWord ...wouldn't work in the standard query parser. You may think "but i'm using dismax, why does dismax need to worry about that?" but the key to remember there is that if dismax didn't split on whitespace prior to analysis, it wouldn't be able to build the DisjunctionMaxQuery's that it uses to find the max scoring field per "word" (which is the whole point of hte parser). -Hoss
RE: keyword query tokenizer
I didn't know the quotes would work. I thought it had to be escaped and I wasn't too fond of that because you have to unescape in the analysis phase. Using quotes doesn't seem so bad to me. -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Monday, March 29, 2010 11:16 AM To: solr-user@lucene.apache.org Subject: RE: keyword query tokenizer : Ahh, but that is exactly what I don't want the DisjunctionMaxQuery to : do. I do not max scoring field per "word". Instead, I want it per : "phrase" which may be a single word or multiple words. then you need to quote your enitre "q" param. (or escape all the white space and meta characters) : You may think "but i'm using dismax, why does dismax need to worry about : : that?" but the key to remember there is that if dismax didn't split on : whitespace prior to analysis, it wouldn't be able to build the : DisjunctionMaxQuery's that it uses to find the max scoring field per : "word" (which is the whole point of hte parser). -Hoss
negative boost
Is it possible to give a negative in boost in dismax? For instance, field1^3 field2^0 field3^-0.1 Thanks, Jason
RE: negative boost
Unfortunately, my results aren't quite what I want unless I use 0 on the second field. Instead, if something matches in all the fields it is elevated to the top. I only want the first field match elevated to the top and I want all first field matches to have the same weight. Next, I want all field2 matches to have the same weight, and finally, I want all field3 matches to have the same weight. But I want field1 matches to be at the top, then field 2, and finally field3. I don't care if the term is all three fields or not. Does this make sense? -Original Message- From: Paul Libbrecht [mailto:p...@activemath.org] Sent: Monday, March 29, 2010 1:10 PM To: solr-user@lucene.apache.org Subject: Re: negative boost Jason, don't you want field1^3 * field2^1 * field3^0.9 ? As written in Lucene in action, it's all multiplied. So "negative boost" means "boost under 1" (and probably elsewhere) paul PS: take the log and you get this negative. Le 29-mars-10 à 21:08, Jason Chaffee a écrit : > Is it possible to give a negative in boost in dismax? For instance, > > > > field1^3 field2^0 field3^-0.1 > > > > Thanks, > > > > Jason >
RE: negative boost
I understand that it defeats the reason for dismax, at least the original reason for dismax. However, if I can do it this way without having to write my own handler because I need to search multiple fields and combine the results, then it is still preferable and thus another way to leverage dismax. Thanks for the tip. I will try it. Jason -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Monday, March 29, 2010 5:06 PM To: solr-user@lucene.apache.org Subject: RE: negative boost : Unfortunately, my results aren't quite what I want unless I use 0 on the : second field. Instead, if something matches in all the fields it is : elevated to the top. I only want the first field match elevated to the : top and I want all first field matches to have the same weight. Next, I : want all field2 matches to have the same weight, and finally, I want all : field3 matches to have the same weight. But I want field1 matches to be : at the top, then field 2, and finally field3. I don't care if the term : is all three fields or not. try qf=field1^1+field2^100+field3^1&tie=0 : Does this make sense? it does, but it kind of defeats the point of dismax. what i cited should help -- the key is to make the boosts vastly differnet scales, and eliminate the tiebreaker value. -Hoss
RE: negative boost
I think the key was change the tie to 0. I had it at 0.1. Getting exactly what I want now. Big thanks for the help. -Original Message- From: Jason Chaffee [mailto:jchaf...@ebates.com] Sent: Monday, March 29, 2010 5:20 PM To: solr-user@lucene.apache.org Subject: RE: negative boost I understand that it defeats the reason for dismax, at least the original reason for dismax. However, if I can do it this way without having to write my own handler because I need to search multiple fields and combine the results, then it is still preferable and thus another way to leverage dismax. Thanks for the tip. I will try it. Jason -Original Message- From: Chris Hostetter [mailto:hossman_luc...@fucit.org] Sent: Monday, March 29, 2010 5:06 PM To: solr-user@lucene.apache.org Subject: RE: negative boost : Unfortunately, my results aren't quite what I want unless I use 0 on the : second field. Instead, if something matches in all the fields it is : elevated to the top. I only want the first field match elevated to the : top and I want all first field matches to have the same weight. Next, I : want all field2 matches to have the same weight, and finally, I want all : field3 matches to have the same weight. But I want field1 matches to be : at the top, then field 2, and finally field3. I don't care if the term : is all three fields or not. try qf=field1^1+field2^100+field3^1&tie=0 : Does this make sense? it does, but it kind of defeats the point of dismax. what i cited should help -- the key is to make the boosts vastly differnet scales, and eliminate the tiebreaker value. -Hoss
RE: Evangelism
Netflix search is built with Solr. That seems like a fairly big and recognizable company. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Thursday, April 29, 2010 11:44 AM To: solr-user@lucene.apache.org Subject: Re: Evangelism This is a Lucene story, but may well apply... By the time I'd sent a request for assistance to the vendor of one of our search tools and received the reply "you didn't give us the right license number", I'd found Lucene, indexed part of my corpus and run successful searches against it. And had answers provided to me from the users list. Paying for support provides, I believe, a false sense of security. Once you sign up, you're at the mercy of the vendor for many things, among them: 1> releases are far apart 2> if the company gets purchased, all sorts of interesting things happen. Witness Microsoft buying FAST recently, then announcing they were not doing any more development on *nix platforms. 3> If the company does go out of business, you are stuck with binary code you can't compile/run/fix understand. 4> You are at the mercy of the next release for "really gotta have it now" changes. Unless you're willing to pay...er...a considerable sum to get a special fix, which may not even be an option. That said, not all open source products are great, I just happen to think that SOLR/Lucene is. Add to that that problems that are found are often fixed in a day or two, a record that no commercial package I've ever used has matched. Here's one technique you can use to sell it to management. Get a pilot up and running in, oh, say three days (ok, take a week). Try the same thing with commercial package X. Do not, under any circumstances, be satisfied with the powerpoint presentation from a commercial vendor . Require working code. Then evaluate ... Best Erick On Thu, Apr 29, 2010 at 2:24 PM, Nagelberg, Kallin < knagelb...@globeandmail.com> wrote: > I had a very hard time selling Solr to business folks. Most are of the mind > that if you're not paying for something it can't be any good. That might > also be why they refrain from posting 'powered by solr' on their website, as > if it might show them to be cheap. They are also fearful of lack of support > should you get hit by a bus. This might be remedied by recommending > professional services from a company such as lucid imagination. > > I think your best bet is to create a working demo with your data and show > them the performance. > > Cheers, > -Kallin Nagelberg > > > > -Original Message- > From: Israel Ekpo [mailto:israele...@gmail.com] > Sent: Thursday, April 29, 2010 2:19 PM > To: solr-user@lucene.apache.org > Subject: Re: Evangelism > > Their main search page has the "Powered by Solr" logo > > http://www.lucidimagination.com/search/ > > > > On Thu, Apr 29, 2010 at 2:18 PM, Israel Ekpo wrote: > > > Checkout Lucid Imagination > > > > http://www.lucidimagination.com/About-Search > > > > This should convince you. > > > > > > On Thu, Apr 29, 2010 at 2:10 PM, Daniel Baughman >wrote: > > > >> Hi I'm new to the list here, > >> > >> > >> > >> I'd like to steer someone in the direction of Solr, and I see the list > of > >> companies using solr, but none have a "power by solr" logo or anything. > >> > >> > >> > >> Does anyone have any great links with evidence to majorly successful > solr > >> projects? > >> > >> > >> > >> Thanks in advance, > >> > >> > >> > >> Dan B. > >> > >> > >> > >> > > > > > > -- > > "Good Enough" is not good enough. > > To give anything less than your best is to sacrifice the gift. > > Quality First. Measure Twice. Cut Once. > > http://www.israelekpo.com/ > > > > > > -- > "Good Enough" is not good enough. > To give anything less than your best is to sacrifice the gift. > Quality First. Measure Twice. Cut Once. > http://www.israelekpo.com/ >
RE: Evangelism
Forgot the link. http://www.lucidimagination.com/Community/Marketplace/Application-Showca se-Wiki/Netflix -Original Message- From: Jason Chaffee [mailto:jchaf...@ebates.com] Sent: Thursday, April 29, 2010 11:52 AM To: solr-user@lucene.apache.org Subject: RE: Evangelism Netflix search is built with Solr. That seems like a fairly big and recognizable company. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Thursday, April 29, 2010 11:44 AM To: solr-user@lucene.apache.org Subject: Re: Evangelism This is a Lucene story, but may well apply... By the time I'd sent a request for assistance to the vendor of one of our search tools and received the reply "you didn't give us the right license number", I'd found Lucene, indexed part of my corpus and run successful searches against it. And had answers provided to me from the users list. Paying for support provides, I believe, a false sense of security. Once you sign up, you're at the mercy of the vendor for many things, among them: 1> releases are far apart 2> if the company gets purchased, all sorts of interesting things happen. Witness Microsoft buying FAST recently, then announcing they were not doing any more development on *nix platforms. 3> If the company does go out of business, you are stuck with binary code you can't compile/run/fix understand. 4> You are at the mercy of the next release for "really gotta have it now" changes. Unless you're willing to pay...er...a considerable sum to get a special fix, which may not even be an option. That said, not all open source products are great, I just happen to think that SOLR/Lucene is. Add to that that problems that are found are often fixed in a day or two, a record that no commercial package I've ever used has matched. Here's one technique you can use to sell it to management. Get a pilot up and running in, oh, say three days (ok, take a week). Try the same thing with commercial package X. Do not, under any circumstances, be satisfied with the powerpoint presentation from a commercial vendor . Require working code. Then evaluate ... Best Erick On Thu, Apr 29, 2010 at 2:24 PM, Nagelberg, Kallin < knagelb...@globeandmail.com> wrote: > I had a very hard time selling Solr to business folks. Most are of the mind > that if you're not paying for something it can't be any good. That might > also be why they refrain from posting 'powered by solr' on their website, as > if it might show them to be cheap. They are also fearful of lack of support > should you get hit by a bus. This might be remedied by recommending > professional services from a company such as lucid imagination. > > I think your best bet is to create a working demo with your data and show > them the performance. > > Cheers, > -Kallin Nagelberg > > > > -Original Message- > From: Israel Ekpo [mailto:israele...@gmail.com] > Sent: Thursday, April 29, 2010 2:19 PM > To: solr-user@lucene.apache.org > Subject: Re: Evangelism > > Their main search page has the "Powered by Solr" logo > > http://www.lucidimagination.com/search/ > > > > On Thu, Apr 29, 2010 at 2:18 PM, Israel Ekpo wrote: > > > Checkout Lucid Imagination > > > > http://www.lucidimagination.com/About-Search > > > > This should convince you. > > > > > > On Thu, Apr 29, 2010 at 2:10 PM, Daniel Baughman >wrote: > > > >> Hi I'm new to the list here, > >> > >> > >> > >> I'd like to steer someone in the direction of Solr, and I see the list > of > >> companies using solr, but none have a "power by solr" logo or anything. > >> > >> > >> > >> Does anyone have any great links with evidence to majorly successful > solr > >> projects? > >> > >> > >> > >> Thanks in advance, > >> > >> > >> > >> Dan B. > >> > >> > >> > >> > > > > > > -- > > "Good Enough" is not good enough. > > To give anything less than your best is to sacrifice the gift. > > Quality First. Measure Twice. Cut Once. > > http://www.israelekpo.com/ > > > > > > -- > "Good Enough" is not good enough. > To give anything less than your best is to sacrifice the gift. > Quality First. Measure Twice. Cut Once. > http://www.israelekpo.com/ >
