SolrSharp and UTF-8, danish letters are messed up
Norwegian and danish letters like "ÆØÅ" are messed up when indexing using SolrSharp??? I've checked the C# code, and to me it looks right. If I instead write my documents to disk (in utf-8), uploads the file to my solr server and runs post.jar from the tutorial, everything works out correctly and the letters are not messed up. I have already modified my tomcat(6) server.xml with . What am I doing wrong? ...besides programming in c# ;) Kind regards, Peter
Getting SolrSharp to work
Version solrsharp-Dec-30-2007.zip
I had to make a few changes to get this version to work. Here is what I did.
--
// fix 1,. Solr server messed up Danish/Norwegian letters like "æøå" because
the HttpWbRequest object wasn't sending in utf-8 to Solr server
Class: SolrSearcher,
Method: public static HttpStatusCode WebPost(string url, byte[] bytesToPost,
ref string statusDescription)
Changed line
oRequest.ContentType = "text/xml";
To
oRequest.ContentType = "text/xml;charset=\"utf-8\"";
-
// fix 2, Could not get Schema.xml from server.. Permission denied.. or wrong
URL.
Class SolrSearcher,
Method private void SetSolrPaths()
Changed
this.SOLR_CONFIG = this.SOLR + "admin/get-file.jsp?file=solrconfig.xml";
To
this.SOLR_CONFIG = this.SOLR + "admin/file/?file=solrconfig.xml";
this.SOLR_SCHEMA = this.SOLR + "admin/get-file.jsp?file=schema.xml";
To
this.SOLR_SCHEMA = this.SOLR + "admin/file/?file=schema.xml";
---
// fix 3,. Added functionality. My schema had field with required=false and the
validation failed.
// if (solrService.SolrSchema.IsValidUpdateIndexDocument(doc)) // failed
//
1)
Class: SolrField
Method: public SolrField(XmlNode xnSolrField, SolrSchema solrSchema)
At Last line:
Before:
this.solrschema = solrschema;
After:
if (xnSolrField.Attributes["required"] != null)
{
this.required =
Convert.ToBoolean(xnSolrField.Attributes["required"].Value);
}
this.solrschema = solrSchema;
2) Added Property getter in SolrField class
private bool required = false;
public bool IsRequired
{
get { return this.required; }
}
3)
Add check for IsRequired in field loop
Class: SolrSchema
Method: public bool IsValidUpdateIndexDocument(UpdateIndexDocument
updateIndexDocument)
{
foreach (SolrField solrField in this.solrFields)
{
if ((!solrField.IsCopied) &&
(!solrField.IsDefaulted) &&
(!docfieldnames.Contains(solrField.Name) &&
solrField.IsRequired) // ADDED by Peter Thygesen
)
{
return false;
}
else
{
docfieldnames.Remove(solrField.Name);
}
}
}
// fix 4:
// removed unnecessary namespace declarations in posted xml
Class: IndexDocument
Method: public string SerializeToString()
After XmlSerializerNamespaces xsn = new XmlSerializerNamespaces();
Add line:
xsn.Add("", "");
Hope someone can use this.. I took me all day to figure out.
\Peter Thygesen
RE: Getting SolrSharp to work
Solr Server:
HW: IBM xSeries 300 (old pizzabox)
OS: Ubuntu Server 7.10
WEB: Tomcat 6.0.14
SOLR: Nightly build 2008-01-14
Thx,
Peter
-Original Message-
From: Jeff Rodenburg [mailto:[EMAIL PROTECTED]
Sent: 18. januar 2008 03:24
To: solr-user@lucene.apache.org
Subject: Re: Getting SolrSharp to work
Thanks for the updates Peter. Can you pass along your Solr instance
information?
thanks,
jeff r.
On Jan 17, 2008 7:22 AM, Peter Thygesen <[EMAIL PROTECTED]> wrote:
> Version solrsharp-Dec-30-2007.zip
>
> I had to make a few changes to get this version to work. Here is what I
> did.
>
> --
> // fix 1,. Solr server messed up Danish/Norwegian letters like "æøå"
> because the HttpWbRequest object wasn't sending in utf-8 to Solr server
> Class: SolrSearcher,
> Method: public static HttpStatusCode WebPost(string url, byte[]
> bytesToPost, ref string statusDescription)
> Changed line
> oRequest.ContentType = "text/xml";
> To
> oRequest.ContentType = "text/xml;charset=\"utf-8\"";
>
> -
> // fix 2, Could not get Schema.xml from server.. Permission denied.. or
> wrong URL.
> Class SolrSearcher,
> Method private void SetSolrPaths()
> Changed
> this.SOLR_CONFIG = this.SOLR + "admin/get-file.jsp?file=solrconfig.xml";
> To
> this.SOLR_CONFIG = this.SOLR + "admin/file/?file=solrconfig.xml";
>
> this.SOLR_SCHEMA = this.SOLR + "admin/get-file.jsp?file=schema.xml";
> To
> this.SOLR_SCHEMA = this.SOLR + "admin/file/?file=schema.xml";
>
> ---
> // fix 3,. Added functionality. My schema had field with required=false
> and the validation failed.
> // if (solrService.SolrSchema.IsValidUpdateIndexDocument(doc)) // failed
> //
> 1)
> Class: SolrField
> Method: public SolrField(XmlNode xnSolrField, SolrSchema solrSchema)
> At Last line:
>
> Before:
> this.solrschema = solrschema;
>
> After:
> if (xnSolrField.Attributes["required"] != null)
> {
> this.required = Convert.ToBoolean(xnSolrField.Attributes
> ["required"].Value);
> }
> this.solrschema = solrSchema;
>
> 2) Added Property getter in SolrField class
> private bool required = false;
> public bool IsRequired
> {
> get { return this.required; }
> }
>
> 3)
> Add check for IsRequired in field loop
> Class: SolrSchema
> Method: public bool IsValidUpdateIndexDocument(UpdateIndexDocument
> updateIndexDocument)
>{
>
>foreach (SolrField solrField in this.solrFields)
>{
>if ((!solrField.IsCopied) &&
>(!solrField.IsDefaulted) &&
>(!docfieldnames.Contains(solrField.Name) &&
> solrField.IsRequired) // ADDED by Peter Thygesen
>)
>{
>return false;
>}
>else
>{
>docfieldnames.Remove(solrField.Name);
>}
>}
> ....
>}
>
>
> // fix 4:
> // removed unnecessary namespace declarations in posted xml
> Class: IndexDocument
> Method: public string SerializeToString()
> After XmlSerializerNamespaces xsn = new XmlSerializerNamespaces();
> Add line:
> xsn.Add("", "");
>
>
> Hope someone can use this.. I took me all day to figure out.
>
> \Peter Thygesen
>
>
>
LIFO, Last In First Out, sort by date desc
In Solr Relevancy FAQ if found a "solution" for boosting newer document... However a newbie dumb ass like me can't get it to work.. :( I want to query solr and have it show me results by date desc. (date is a field) I suppose I can use the explicit sort.. but FAQ talks about boosting and I would like to try that too. I don't know what "qf", "mm" ect. I should set. This is what I tried, but didn't get "date desc" result. (I've tried other values and leaving out attributes, but soo far.. no luck) explicit body^1 headline^1 recip(rord(date),1,1000,1000) id,headline,date,score Hope someone has the time to help a newbie.. :| \Peter
Getting SolrSharp to work, Part 2
I wrote a small client in .Net which query Solr and dumps the result on
screen.. fantastic low-tech.. ;)
However I ran into new SolrSharp problems. My schema allows a particular
field to be multiValued, but if it only has one value, it will cause
SolrSharp fail in line 88 of Class: IndexFiledAttribute.
My SearchRecord property is an array (List) and line 88 tries to set my
property as if it was a string. The code should be corrected by checking
if the property is an array and not whether it has 1 value or more.
E.g. change line 85 to 085> if(!this.PropertyInfo.PropertyType.IsArray)
Original code (from class IndexFiledAttribute):
082> public void SetValue(SearchRecord searchRecord)
083> {
084> XmlNodeList xnlvalues =
searchRecord.XNodeRecord.SelectNodes(this.XnodeExpression);
085> if (xnlvalues.Count == 1) //single value
086> {
087> XmlNode xnodevalue = xnlvalues[0];
088> this.PropertyInfo.SetValue(searchRecord,
Convert.ChangeType(xnodevalue.InnerText, this.PropertyInfo.PropertyType)
, null);
089> }
090> else if (xnlvalues.Count > 1) //array
091> {
092> Type basetype =
this.PropertyInfo.PropertyType.GetElementType();
093> Array valueArray = Array.CreateInstance(basetype,
xnlvalues.Count);
094> for (int i = 0; i < xnlvalues.Count; i++)
095> {
096>
valueArray.SetValue(Convert.ChangeType(xnlvalues[i].InnerText,
basetype), i);
097> }
098> this.PropertyInfo.SetValue(searchRecord, valueArray, null);
099> }
100> }
My code (replace):
085>if(!this.PropertyInfo.PropertyType.IsArray) // single value
090>else // array
Cheers,
Peter Thygesen
-- hope to see you all at ApacheCon in Amsterdam :)
RE: Getting SolrSharp to work, Part 2
This patch covers the issues I wrote about in my previous mails "How to
get SolrSharp to work" and "How to get SolrSharp to work, part 2"
By the way should I post on this thread, or on CodePlex. When the topic
is "SolrSharp"? I don't mind adding a few more comments to the
discussion I already started on CodePlex.
\peter
-Original Message-
From: Jeff Rodenburg [mailto:[EMAIL PROTECTED]
Sent: 24. januar 2008 20:59
To: solr-user@lucene.apache.org
Subject: Re: Getting SolrSharp to work, Part 2
Hey Peter - if you could submit your changes as an svn patch, we could
apply
the update much faster.
thanks,
jeff
On Jan 23, 2008 2:42 AM, Peter Thygesen <[EMAIL PROTECTED]> wrote:
> I wrote a small client in .Net which query Solr and dumps the result
on
> screen.. fantastic low-tech.. ;)
>
> However I ran into new SolrSharp problems. My schema allows a
particular
> field to be multiValued, but if it only has one value, it will cause
> SolrSharp fail in line 88 of Class: IndexFiledAttribute.
>
> My SearchRecord property is an array (List) and line 88 tries to set
my
> property as if it was a string. The code should be corrected by
checking
> if the property is an array and not whether it has 1 value or more.
> E.g. change line 85 to 085>
if(!this.PropertyInfo.PropertyType.IsArray)
>
> Original code (from class IndexFiledAttribute):
> 082> public void SetValue(SearchRecord searchRecord)
> 083> {
> 084> XmlNodeList xnlvalues =
> searchRecord.XNodeRecord.SelectNodes(this.XnodeExpression);
> 085> if (xnlvalues.Count == 1) //single value
> 086> {
> 087> XmlNode xnodevalue = xnlvalues[0];
> 088> this.PropertyInfo.SetValue(searchRecord,
> Convert.ChangeType(xnodevalue.InnerText,
this.PropertyInfo.PropertyType)
> , null);
> 089> }
> 090> else if (xnlvalues.Count > 1) //array
> 091> {
> 092> Type basetype =
> this.PropertyInfo.PropertyType.GetElementType();
> 093> Array valueArray = Array.CreateInstance(basetype,
> xnlvalues.Count);
> 094> for (int i = 0; i < xnlvalues.Count; i++)
> 095> {
> 096>
> valueArray.SetValue(Convert.ChangeType(xnlvalues[i].InnerText,
> basetype), i);
> 097> }
> 098> this.PropertyInfo.SetValue(searchRecord, valueArray, null);
> 099> }
> 100> }
>
> My code (replace):
> 085>if(!this.PropertyInfo.PropertyType.IsArray) // single value
> 090>else // array
>
> Cheers,
> Peter Thygesen
>
> -- hope to see you all at ApacheCon in Amsterdam :)
>
>
>
RE: Getting SolrSharp to work, Part 2
Ups. Forgot to tell that the patch was uploaded on CodePlex
http://www.codeplex.com/solrsharp/SourceControl/PatchList.aspx
\peter
-Original Message-
From: Peter Thygesen
Sent: 25. januar 2008 13:17
To: solr-user@lucene.apache.org
Subject: RE: Getting SolrSharp to work, Part 2
This patch covers the issues I wrote about in my previous mails "How to
get SolrSharp to work" and "How to get SolrSharp to work, part 2"
By the way should I post on this thread, or on CodePlex. When the topic
is "SolrSharp"? I don't mind adding a few more comments to the
discussion I already started on CodePlex.
\peter
-Original Message-
From: Jeff Rodenburg [mailto:[EMAIL PROTECTED]
Sent: 24. januar 2008 20:59
To: solr-user@lucene.apache.org
Subject: Re: Getting SolrSharp to work, Part 2
Hey Peter - if you could submit your changes as an svn patch, we could
apply
the update much faster.
thanks,
jeff
On Jan 23, 2008 2:42 AM, Peter Thygesen <[EMAIL PROTECTED]> wrote:
> I wrote a small client in .Net which query Solr and dumps the result
on
> screen.. fantastic low-tech.. ;)
>
> However I ran into new SolrSharp problems. My schema allows a
particular
> field to be multiValued, but if it only has one value, it will cause
> SolrSharp fail in line 88 of Class: IndexFiledAttribute.
>
> My SearchRecord property is an array (List) and line 88 tries to set
my
> property as if it was a string. The code should be corrected by
checking
> if the property is an array and not whether it has 1 value or more.
> E.g. change line 85 to 085>
if(!this.PropertyInfo.PropertyType.IsArray)
>
> Original code (from class IndexFiledAttribute):
> 082> public void SetValue(SearchRecord searchRecord)
> 083> {
> 084> XmlNodeList xnlvalues =
> searchRecord.XNodeRecord.SelectNodes(this.XnodeExpression);
> 085> if (xnlvalues.Count == 1) //single value
> 086> {
> 087> XmlNode xnodevalue = xnlvalues[0];
> 088> this.PropertyInfo.SetValue(searchRecord,
> Convert.ChangeType(xnodevalue.InnerText,
this.PropertyInfo.PropertyType)
> , null);
> 089> }
> 090> else if (xnlvalues.Count > 1) //array
> 091> {
> 092> Type basetype =
> this.PropertyInfo.PropertyType.GetElementType();
> 093> Array valueArray = Array.CreateInstance(basetype,
> xnlvalues.Count);
> 094> for (int i = 0; i < xnlvalues.Count; i++)
> 095> {
> 096>
> valueArray.SetValue(Convert.ChangeType(xnlvalues[i].InnerText,
> basetype), i);
> 097> }
> 098> this.PropertyInfo.SetValue(searchRecord, valueArray, null);
> 099> }
> 100> }
>
> My code (replace):
> 085>if(!this.PropertyInfo.PropertyType.IsArray) // single value
> 090>else // array
>
> Cheers,
> Peter Thygesen
>
> -- hope to see you all at ApacheCon in Amsterdam :)
>
>
>
RE: Multiple Search in Solr
I'm also looking for a solution with multiple indices. Soo.. great, are you saying the patch doesn't work or what? And could you elaborate a little more on the "I have written the Lucene application.." What did you do? -Peter Thygesen -Original Message- From: Jae Joo [mailto:[EMAIL PROTECTED] Sent: 4. februar 2008 14:59 To: solr-user@lucene.apache.org Subject: RE: Multiple Search in Solr I have downloaded version 1.3 and built multiple indices. I could not find any way for multiple indices search at Solr level, I have written the Lucene application. It is working well. Jae Joo -Original Message- From: Niveen Nagy [mailto:[EMAIL PROTECTED] Sent: Monday, February 04, 2008 8:55 AM To: solr-user@lucene.apache.org Subject: Multiple Search in Solr Hello , I have a question concerning solr multiple indices. We have 4 solr indices in our system and we want to use distributed search (Multiple search) that searches in the four indices in parallel. We downloaded the latest code from svn and we applied the patch distributed.patch but we need more detailed description on how to use this patch and what changes should be applied to solr schema, and how these indices should be located. Another question here is could the steps be applied to our indices that was built using a version before applying the distributed patch. Thanks in advance. Best Regards, Niveen Nagy
