Issue with Facet Query
Hi, I am searching the indexes with facet query. Below is the query. q=Answer&version=2.2&start=0&rows=10&indent=on&qt=dismaxrequest&facet=true&facet.field=productPrice_product_str_s:[0%20TO%2020] It is giving me an exception saying: str name="exception">org.apache.solr.common.SolrException: undefined field productPrice_product_str_s:[0 TO 20] at org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:994) at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:152) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute Can someone please guide me, how to prevent this exception. I guess, I am missing some entries in some config file like solrConfig or schema. I would appreciate if someone can tell me the specific entries, I need to make in any config file. Thanks a lot. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22615577.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with Facet Query
On Fri, Mar 20, 2009 at 1:14 PM, dabboo wrote: > > Hi, > > I am searching the indexes with facet query. Below is the query. > > > q=Answer&version=2.2&start=0&rows=10&indent=on&qt=dismaxrequest&facet=true&facet.field=productPrice_product_str_s:[0%20TO%2020] > facet.field takes a field name. It does not accept queries. Use facet.query for getting count of a query. Use fq to restrict facets by a certain query. See http://wiki.apache.org/solr/SimpleFacetParameters -- Regards, Shalin Shekhar Mangar.
Re: Issue with Facet Query
Thanks a lot for this information. But is there any way, I can impose the range on the facet. for e.g. If I want to search the data between a specific range, how should I form my query. Do I need to make some entries some where. Thanks, Amit Garg Shalin Shekhar Mangar wrote: > > On Fri, Mar 20, 2009 at 1:14 PM, dabboo wrote: > >> >> Hi, >> >> I am searching the indexes with facet query. Below is the query. >> >> >> q=Answer&version=2.2&start=0&rows=10&indent=on&qt=dismaxrequest&facet=true&facet.field=productPrice_product_str_s:[0%20TO%2020] >> > > facet.field takes a field name. It does not accept queries. Use > facet.query > for getting count of a query. Use fq to restrict facets by a certain > query. > > See http://wiki.apache.org/solr/SimpleFacetParameters > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22615979.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
==> where are you seeing it as ""Solène" as opposed to the correct way of solène? I have "Solène" in my Mysql DATA BASE ! so i don't know if this is correct or not ? i gess that "Solène" is solène in UTF-8 ?! I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp, so when i try with solène everything is ok ! but when i try with Solène (like what i have in DB) analysis convert à in A delete ¨ so i get SolAne !!! I think that ISOLatin1AccentFilterFactory take only string with Charset ISO-8859-1 . So any solution to transform my string to ISO-8859-1 before indexing process. May be by creating transformer in DataImportHandler ? (Never code in java :( ) Thank you all. Koji Sekiguchi-2 wrote: > > aerox7 wrote: >> Hi, >> I have a mysql data base in UTF-8. I have a row with "Solène" (solène). >> I >> want to transforme this to solene, so i use Solr >> ISOLatin1AccentFilterFactory to perform this task but it dosn't work ?!! >> >> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to utf-8 >> so >> normaly ISOLatin1AccentFilterFactory have to replace the accent ... >> >> any ideas ? >> >> i use DataImportHandler. >> > > If a mapping rule "è" to "e" is always true in your field, you can try > to use MappingCharFilter > instead of ISOLatin1AccentFilter. Add the following line to > mapping-ISOLatin1Accent.txt: > > "è" => "e" > > and add the following fieldType: > > positionIncrementGap="100" > > > mapping="mapping-ISOLatin1Accent.txt"/> > > > > > MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build. > > Koji > > > > -- View this message in context: http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22616220.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with Facet Query
Shalin, thanks a lot. One quick question: Now, after putting the query in the way, you suggested, I am getting: - - 23315 But it is not returning me records. Do I need to enter this field entry in schema.xml to get the records or anywhere else. Thanks, Amit Garg Shalin Shekhar Mangar wrote: > > On Fri, Mar 20, 2009 at 1:49 PM, dabboo wrote: > >> >> Thanks a lot for this information. But is there any way, I can impose the >> range on the facet. >> for e.g. If I want to search the data between a specific range, how >> should >> I >> form my query. >> > > Use a filter query, fq=productPrice_product_str_s:[0 TO 20] > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22616536.html Sent from the Solr - User mailing list archive at Nabble.com.
Search transparently with Solr with multiple cores, different indexes, common response type
Hello all,
here I am with another question... :-)
I figured that I have to change approach to implement the requirements I
have :-(
Here it is what I have to index:
1) data "A" in an Oracle DB Table "A"
2) data "B" in an Oracle DB Table "B"
3) data "C" in different files
Data "A", "B", and "C" are slightly different, thus they are indexed
differently; obviously the client receives the search results for all data
types in a consistent/common format. The client application shall be able to
search among each or all data types ("A", "B", "C"). The order will be
configurable, like: return the first 5 from data "A", the first 10 from "B",
all "C".
At first I thought of using only one Solr with different datasources, and
one huge index, but I figured that delta imports would be very
hard/expensive/impossible.
Reading some other posts I thought that maybe a better approach would be as
following:
1) one Solr core for each data type (one for "A", one for "B", one for "C")
2) one index fora each data type, thus one document type for "A", one for
"B", and one for "C"
3) client applications shall be able to search on one or all cores
4) the cores shall return search results in a common XML format
5) search results shall be aggregated in a configurable way
Can you please tell me if this architecture is possible with Solr? Obviously
I am not looking for an "out-of-the-.box" solution, I just need to
understand what I have to develop myself and what is already available.
1) is a multicore architecture: I know it is possible and I tested that it
works great
2) same as above, no problems here :-)
3) I want to "hide" the different cores to the client application; the
client application should send the requests to one "guy" that parses the
request and forwards it to the cores. Is this a custom RequestHandler? Any
link (to the Wiki?) to understand better? Or is there anything already
available to achieve this?
4) The "guy" that parses the request and forwards it to the cores shall
aggregate and return results in a common XML format: is this a custom
ResponseHandler?
5) I know this is just my business logic :-)
Any thougts/warning/advice about this?
Thanks a lot in advance!
Giovanni
Re: Issue with Facet Query
On Fri, Mar 20, 2009 at 2:27 PM, dabboo wrote: > > Shalin, thanks a lot. One quick question: > > Now, after putting the query in the way, you suggested, I am getting: > > - > - > 23315 > > > > > > But it is not returning me records. Do I need to enter this field entry in > schema.xml to get the records or anywhere else. > facet.query returns the number of documents matching that query after applying any filters (fq) that you may have specified. Can you tell us your use-case? -- Regards, Shalin Shekhar Mangar.
Re: Issue with Facet Query
Thanks Shalin, thanks a lot. I appreciate your help in resolving this issue. Thanks, Amit Shalin Shekhar Mangar wrote: > > On Fri, Mar 20, 2009 at 2:27 PM, dabboo wrote: > >> >> Shalin, thanks a lot. One quick question: >> >> Now, after putting the query in the way, you suggested, I am getting: >> >> - >> - >> 23315 >> >> >> >> >> >> But it is not returning me records. Do I need to enter this field entry >> in >> schema.xml to get the records or anywhere else. >> > > facet.query returns the number of documents matching that query after > applying any filters (fq) that you may have specified. > > Can you tell us your use-case? > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22616724.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Special character indexing
Hi Shalin,
Thanks for the suggestion. I tried following code, (not sure about the
exact usage)
CommonsHttpSolrServer ess = new
CommonsHttpSolrServer("http://localhost:8983/solr";);
ess.setRequestWriter(new BinaryRequestWriter());
SolrInputDocument solrdoc = new SolrInputDocument();
solrdoc.addField("id", "Kimi");
solrdoc.addField("name", "03 Kimi Räikkönen ");
ess.add(solrdoc);
But got following exception on the server
WARNING: The @Deprecated SolrUpdateServlet does not accept query parameters:
wt=javabin
If you are using solrj, make sure to register a request handler to /update
rather then use this servlet.
Add: to
your solrconfig.xml
Mar 20, 2009 3:14:48 PM org.apache.solr.common.SolrException log
SEVERE: Error processing "legacy" update
command:com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-
CHAR, code 1))
at [row,col {unknown-source}]: [1,1]
at
com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:675)
at
com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:660)
at
com.ctc.wstx.sr.BasicStreamReader.readSpacePrimary(BasicStreamReader.java:4916)
at
com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2003)
at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1069)
at
org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:148)
at
org.apache.solr.handler.XmlUpdateRequestHandler.doLegacyUpdate(XmlUpdateRequestHandler.java:393)
at
org.apache.solr.servlet.SolrUpdateServlet.doPost(SolrUpdateServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1098)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:295)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:835)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:723)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Thanks in advance for help.
Siddharth
-Original Message-
From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com]
Sent: Friday, March 20, 2009 10:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Special character indexing
On Fri, Mar 20, 2009 at 10:17 AM, Gargate, Siddharth wrote:
> I tried with Jetty but the same issue. Just a guess, but looks like
> the fix for SOLR-973 might have introduced this issue.
>
I'm not sure how SOLR-973 can cause this issue. Can you try using the
BinaryRequestWriter and see if it succeeds?
http://wiki.apache.org/solr/Solrj#head-ddc28af4033350481a3cbb27bc1d25bffd801af0
--
Regards,
Shalin Shekhar Mangar.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
Hi,
My guess is that *although* your DB is in UTF-8, the database engine sends
you the rows in ISO-Latin1, so before doing *anything* after receiving the
data, you should transcode from ISO-Latin1 to UTF-8 and then send that to
SolR. I'm no Java expert, but in perl (MySQL DB in utf-8) I have to do with
any row:
$row=decode("iso-8859-1",$row);
... and before building the xml to invoque and add document to SolR:
$row=encode("utf8",$row);
On Fri, Mar 20, 2009 at 10:55 AM, aerox7 wrote:
>
> I add :
> "è" => "e" to mapping-ISOLatin1Accent.txt
>
> and add the following fieldType:
>
> positionIncrementGap="100" >
>
> mapping="mapping-ISOLatin1Accent.txt"/>
>
>
>
>
> By still have the same probleme ! it's only work when i store ISO string
> into UTF-8 data base (ex: store solène not solène) :,(
>
>
>
>
> aerox7 wrote:
> >
> > ==> where are you seeing it as ""Solène" as opposed to the
> > correct way of solène?
> >
> > I have "Solène" in my Mysql DATA BASE ! so i don't know if this is
> > correct or not ? i gess that "Solène" is solène in UTF-8 ?!
> >
> > I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp, so
> > when i try with solène everything is ok ! but when i try with Solène
> > (like what i have in DB) analysis convert à in A delete ¨ so i get SolAne
> > !!!
> >
> > I think that ISOLatin1AccentFilterFactory take only string with Charset
> > ISO-8859-1 .
> >
> > So any solution to transform my string to ISO-8859-1 before indexing
> > process. May be by creating transformer in DataImportHandler ? (Never
> code
> > in java :( )
> >
> > Thank you all.
> >
> >
> > Koji Sekiguchi-2 wrote:
> >>
> >> aerox7 wrote:
> >>> Hi,
> >>> I have a mysql data base in UTF-8. I have a row with "Solène"
> (solène).
> >>> I
> >>> want to transforme this to solene, so i use Solr
> >>> ISOLatin1AccentFilterFactory to perform this task but it dosn't work
> ?!!
> >>>
> >>> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to
> utf-8
> >>> so
> >>> normaly ISOLatin1AccentFilterFactory have to replace the accent ...
> >>>
> >>> any ideas ?
> >>>
> >>> i use DataImportHandler.
> >>>
> >>
> >> If a mapping rule "è" to "e" is always true in your field, you can try
> >> to use MappingCharFilter
> >> instead of ISOLatin1AccentFilter. Add the following line to
> >> mapping-ISOLatin1Accent.txt:
> >>
> >> "è" => "e"
> >>
> >> and add the following fieldType:
> >>
> >> >> positionIncrementGap="100" >
> >>
> >> >> mapping="mapping-ISOLatin1Accent.txt"/>
> >>
> >>
> >>
> >>
> >> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build.
> >>
> >> Koji
> >>
> >>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22617278.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
“I may not believe in myself, but I believe in what I'm doing.”
-- Jimmy Page
Re: Issue with Facet Query
On Fri, Mar 20, 2009 at 1:49 PM, dabboo wrote: > > Thanks a lot for this information. But is there any way, I can impose the > range on the facet. > for e.g. If I want to search the data between a specific range, how should > I > form my query. > Use a filter query, fq=productPrice_product_str_s:[0 TO 20] -- Regards, Shalin Shekhar Mangar.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
I add : "è" => "e" to mapping-ISOLatin1Accent.txt and add the following fieldType: By still have the same probleme ! it's only work when i store ISO string into UTF-8 data base (ex: store solène not solène) :,( aerox7 wrote: > > ==> where are you seeing it as ""Solène" as opposed to the > correct way of solène? > > I have "Solène" in my Mysql DATA BASE ! so i don't know if this is > correct or not ? i gess that "Solène" is solène in UTF-8 ?! > > I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp, so > when i try with solène everything is ok ! but when i try with Solène > (like what i have in DB) analysis convert à in A delete ¨ so i get SolAne > !!! > > I think that ISOLatin1AccentFilterFactory take only string with Charset > ISO-8859-1 . > > So any solution to transform my string to ISO-8859-1 before indexing > process. May be by creating transformer in DataImportHandler ? (Never code > in java :( ) > > Thank you all. > > > Koji Sekiguchi-2 wrote: >> >> aerox7 wrote: >>> Hi, >>> I have a mysql data base in UTF-8. I have a row with "Solène" (solène). >>> I >>> want to transforme this to solene, so i use Solr >>> ISOLatin1AccentFilterFactory to perform this task but it dosn't work ?!! >>> >>> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to utf-8 >>> so >>> normaly ISOLatin1AccentFilterFactory have to replace the accent ... >>> >>> any ideas ? >>> >>> i use DataImportHandler. >>> >> >> If a mapping rule "è" to "e" is always true in your field, you can try >> to use MappingCharFilter >> instead of ISOLatin1AccentFilter. Add the following line to >> mapping-ISOLatin1Accent.txt: >> >> "è" => "e" >> >> and add the following fieldType: >> >> > positionIncrementGap="100" > >> >> > mapping="mapping-ISOLatin1Accent.txt"/> >> >> >> >> >> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build. >> >> Koji >> >> >> >> > > -- View this message in context: http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22617278.html Sent from the Solr - User mailing list archive at Nabble.com.
how can I check field which are indexed but not stored?
Hi I've an issue, I've some data which come up but I've applied a filtre on it and it shouldnt, when I check in my database mysql I've obviously the document which has been updated so I will like to see how it is in solr. if I do : /solr/video/select?q=id:8582006 I will just see field which has been stored. Is there a way to see how data are indexed for other field of my schema which are not stored but indexed. Like a bit in the console dataimporthandler, which with verbose activated I can see every field of my schema. Otherwise what would you reckon in this case, a document which has not been updated ? how can I sort it out? Thanks a lot guys for your excellent help -- View this message in context: http://www.nabble.com/how-can-I-check-field-which-are-indexed-but-not-stored--tp22617914p22617914.html Sent from the Solr - User mailing list archive at Nabble.com.
FW: Special character indexing
Thanks Shalin,
Adding BinaryUpdateRequestHandler solved the issue. Thank you very much.
Just one query, shouldn't XmlUpdateRequestHandler also work for these
characters? I saw another user mentioning the same issue and it was working
with DirectXmlRequest.
-Original Message-
From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com]
Sent: Friday, March 20, 2009 3:58 PM
To: solr-user@lucene.apache.org
Subject: Re: Special character indexing
On Fri, Mar 20, 2009 at 3:19 PM, Gargate, Siddharth wrote:
> Hi Shalin,
>Thanks for the suggestion. I tried following code, (not sure
> about the exact usage)
>
>CommonsHttpSolrServer ess = new CommonsHttpSolrServer("
> http://localhost:8983/solr";);
>ess.setRequestWriter(new BinaryRequestWriter());
>SolrInputDocument solrdoc = new SolrInputDocument();
>solrdoc.addField("id", "Kimi");
>solrdoc.addField("name", "03 Kimi Räikkönen ");
>ess.add(solrdoc);
>
> But got following exception on the server
>
> WARNING: The @Deprecated SolrUpdateServlet does not accept query
> parameters: wt=javabin
> If you are using solrj, make sure to register a request handler to
> /update rather then use this servlet.
> Add: class="solr.XmlUpdateRequestHandler" > to your solrconfig.xml
>
Yes, you need to add the following to your solrconfig.xml
--
Regards,
Shalin Shekhar Mangar.
Re: Combination of solr.xml and solrconfig.xml
Erik Hatcher schrieb: Unfortunately Solr doesn't offer that kind of reusability (yet). But you could just go old school and use XML entity includes to pull in a fragment of a file, and externalize the common pieces that way. Not as pretty, but oughta work. one last little question ;-) http://java.sun.com/dtd/web-app_2_3.dtd"; [ "/var/lib/tomcat5.5/webapps/solr/default_solrconfig.xml"> ]> Is there a chance to set the "home directory" using a variable ? For example an unix enviroment variable ? Greets -Ralf-
Re: alternative lucene directories support
Otis, The fact is that some code instantiates FSDirectory indirectly by using deprecated constructors. I provided a patch here https://issues.apache.org/jira/browse/SOLR-465 but I don't have rights to re-open the issue. Also there is logic in Solr which is tied to file system usage even if file system index is not used: - SolrCore chechs index existence by looking at file system directory existense which is incorrect in the case of non-fs directory - Spell checker has FSDirectory hard-code IMO this code is ought to be changed too. I'm ready to contribute all these changes if it's appropriate. Is it better to write to dev maillist for that? On Thu, Mar 19, 2009 at 8:58 PM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > > My quick grep of the sources and scan of the results doesn't see any > problematic areas, but if you see some places that still need a fix, yes, > please reopen the issue and submit the patch. Do you also plan on > submitting the actual alternative Directory impl? > > $ ffjg FSDire | egrep 'SolrIndexW|SolrCore|UpdateH' > ./src/java/org/apache/solr/core/SolrCore.java:import > org.apache.lucene.store.FSDirectory; > ./src/java/org/apache/solr/core/SolrCore.java://return new > SolrIndexSearcher(this, schema, "main", > IndexReader.open(FSDirectory.getDirectory(getIndexDir()), readOnly), true, > false); > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message > > From: Andrey Klochkov > > To: solr-user@lucene.apache.org > > Sent: Thursday, March 19, 2009 10:22:57 AM > > Subject: alternative lucene directories support > > > > Hi all > > > > We want to use Solr with lucene Directory implementation which places > index > > into Coherence data grid. > > I fact I managed to run Solr in such configuration although I had to > patch > > it. > > I think that the issue about alternate directories support (SOLR-465) > should > > be re-opened because there are some places in source code where > FSDirectory > > hard-coding is still present (SolrCore, SolrIndexWriter and > UpdateHandler). > > I can provide a patch to fix it. > > > > WDYT? > > > > -- > > Andrew Klochkov > > -- Andrew Klochkov
Facet Query Results Issue
Hi, this is my facet query. facet.field=productPrice_product_str_s&facet.query=productPrice_product_str_s:[0%20TO%20100] This is my query and these are results, I am getting: 202 57 14 11 10 9 9 7 6 5 4 2 2 2 2 2 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 It is only returning results, which are having values started with 2, 3, 4 or some other integer instead of only 1. It is not returning records in which value is >10 and <100. Please suggest. thanks, Amit -- View this message in context: http://www.nabble.com/Facet-Query-Results-Issue-tp22617883p22617883.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with Facet Query
Hi Shalin, One more thing, facet.field=productPrice_product_str_s&facet.query=productPrice_product_str_s:[0%20TO%20100] This is my query and these are results, I am getting: 202 57 14 11 10 9 9 7 6 5 4 2 2 2 2 2 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 It is only returning results, which are having values started with 2, 3, 4 or some other integer instead of only 1. It is not returning records in which value is >10 and <100. Please suggest. thanks, Amit dabboo wrote: > > Thanks Shalin, thanks a lot. I appreciate your help in resolving this > issue. > > Thanks, > Amit > > Shalin Shekhar Mangar wrote: >> >> On Fri, Mar 20, 2009 at 2:27 PM, dabboo wrote: >> >>> >>> Shalin, thanks a lot. One quick question: >>> >>> Now, after putting the query in the way, you suggested, I am getting: >>> >>> - >>> - >>> 23315 >>> >>> >>> >>> >>> >>> But it is not returning me records. Do I need to enter this field entry >>> in >>> schema.xml to get the records or anywhere else. >>> >> >> facet.query returns the number of documents matching that query after >> applying any filters (fq) that you may have specified. >> >> Can you tell us your use-case? >> >> -- >> Regards, >> Shalin Shekhar Mangar. >> >> > > -- View this message in context: http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22617745.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Special character indexing
On Fri, Mar 20, 2009 at 3:19 PM, Gargate, Siddharth wrote:
> Hi Shalin,
>Thanks for the suggestion. I tried following code, (not sure about
> the exact usage)
>
>CommonsHttpSolrServer ess = new CommonsHttpSolrServer("
> http://localhost:8983/solr";);
>ess.setRequestWriter(new BinaryRequestWriter());
>SolrInputDocument solrdoc = new SolrInputDocument();
>solrdoc.addField("id", "Kimi");
>solrdoc.addField("name", "03 Kimi Räikkönen ");
>ess.add(solrdoc);
>
> But got following exception on the server
>
> WARNING: The @Deprecated SolrUpdateServlet does not accept query
> parameters: wt=javabin
> If you are using solrj, make sure to register a request handler to /update
> rather then use this servlet.
> Add:
> to your solrconfig.xml
>
Yes, you need to add the following to your solrconfig.xml
--
Regards,
Shalin Shekhar Mangar.
Re: how can I check field which are indexed but not stored?
On Fri, 2009-03-20 at 03:41 -0700, sunnyfr wrote: > Hi > > I've an issue, I've some data which come up but I've applied a filtre on it > and it shouldnt, when I check in my database mysql I've obviously the > document which has been updated so I will like to see how it is in solr. > > if I do : /solr/video/select?q=id:8582006 I will just see field which has > been stored. Is there a way to see how data are indexed for other field of > my schema which are not stored but indexed. /solr/admin/luke will show you a lot of information concering stored and indexed fields. Hope this is what you meant. > > Like a bit in the console dataimporthandler, which with verbose activated I > can see every field of my schema. > > Otherwise what would you reckon in this case, a document which has not been > updated ? how can I sort it out? > > Thanks a lot guys for your excellent help
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
I'm using DataImportHandler to send my data to Solr ! so you mean it possible
to apply a transformer in db-config.xml with a perl script ?
Óscar Marín Miró wrote:
>
> Hi,
>
> My guess is that *although* your DB is in UTF-8, the database engine sends
> you the rows in ISO-Latin1, so before doing *anything* after receiving the
> data, you should transcode from ISO-Latin1 to UTF-8 and then send that to
> SolR. I'm no Java expert, but in perl (MySQL DB in utf-8) I have to do
> with
> any row:
>
> $row=decode("iso-8859-1",$row);
>
> ... and before building the xml to invoque and add document to SolR:
>
> $row=encode("utf8",$row);
>
> On Fri, Mar 20, 2009 at 10:55 AM, aerox7 wrote:
>
>>
>> I add :
>> "è" => "e" to mapping-ISOLatin1Accent.txt
>>
>> and add the following fieldType:
>>
>> > positionIncrementGap="100" >
>>
>>> mapping="mapping-ISOLatin1Accent.txt"/>
>>
>>
>>
>>
>> By still have the same probleme ! it's only work when i store ISO string
>> into UTF-8 data base (ex: store solène not solène) :,(
>>
>>
>>
>>
>> aerox7 wrote:
>> >
>> > ==> where are you seeing it as ""Solène" as opposed to the
>> > correct way of solène?
>> >
>> > I have "Solène" in my Mysql DATA BASE ! so i don't know if this is
>> > correct or not ? i gess that "Solène" is solène in UTF-8 ?!
>> >
>> > I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp,
>> so
>> > when i try with solène everything is ok ! but when i try with Solène
>> > (like what i have in DB) analysis convert à in A delete ¨ so i get
>> SolAne
>> > !!!
>> >
>> > I think that ISOLatin1AccentFilterFactory take only string with Charset
>> > ISO-8859-1 .
>> >
>> > So any solution to transform my string to ISO-8859-1 before indexing
>> > process. May be by creating transformer in DataImportHandler ? (Never
>> code
>> > in java :( )
>> >
>> > Thank you all.
>> >
>> >
>> > Koji Sekiguchi-2 wrote:
>> >>
>> >> aerox7 wrote:
>> >>> Hi,
>> >>> I have a mysql data base in UTF-8. I have a row with "Solène"
>> (solène).
>> >>> I
>> >>> want to transforme this to solene, so i use Solr
>> >>> ISOLatin1AccentFilterFactory to perform this task but it dosn't work
>> ?!!
>> >>>
>> >>> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to
>> utf-8
>> >>> so
>> >>> normaly ISOLatin1AccentFilterFactory have to replace the accent
>> ...
>> >>>
>> >>> any ideas ?
>> >>>
>> >>> i use DataImportHandler.
>> >>>
>> >>
>> >> If a mapping rule "è" to "e" is always true in your field, you can
>> try
>> >> to use MappingCharFilter
>> >> instead of ISOLatin1AccentFilter. Add the following line to
>> >> mapping-ISOLatin1Accent.txt:
>> >>
>> >> "è" => "e"
>> >>
>> >> and add the following fieldType:
>> >>
>> >> > >> positionIncrementGap="100" >
>> >>
>> >> > >> mapping="mapping-ISOLatin1Accent.txt"/>
>> >> > class="solr.CharStreamAwareWhitespaceTokenizerFactory"/>
>> >>
>> >>
>> >>
>> >> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly
>> build.
>> >>
>> >> Koji
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22617278.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> “I may not believe in myself, but I believe in what I'm doing.”
>
> -- Jimmy Page
>
>
--
View this message in context:
http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22618085.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: FW: Special character indexing
On Fri, Mar 20, 2009 at 4:13 PM, Gargate, Siddharth wrote: > Thanks Shalin, > > Adding BinaryUpdateRequestHandler solved the issue. Thank you very much. > > Just one query, shouldn't XmlUpdateRequestHandler also work for these > characters? I saw another user mentioning the same issue and it was working > with DirectXmlRequest. > It should. I'll run a few tests to see where is the problem. -- Regards, Shalin Shekhar Mangar.
Re: Issue with Facet Query
On Fri, Mar 20, 2009 at 4:00 PM, dabboo wrote: > > Hi Shalin, > > One more thing, > > > facet.field=productPrice_product_str_s&facet.query=productPrice_product_str_s:[0%20TO%20100] > > This is my query and these are results, I am getting: > > 202 > 57 > 14 > 11 > 10 > 9 > 9 > 7 > 6 > 5 > 4 > 2 > 2 > 2 > 2 > 2 > 1 > 1 > 1 > 1 > 1 > 1 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > > It is only returning results, which are having values started with 2, 3, 4 > or some other integer instead of only 1. It is not returning records in > which value is >10 and <100. > Please do not send a duplicate mails. It will not help you get an answer faster. If you need to filter results to a specific range then you should use filter queries through the fq parameter: fq=productPrice_product_str_s:[0%20TO%20100] -- Regards, Shalin Shekhar Mangar.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
What I mean is that unless "solène" travels to Solr in strict UTF-8,
mapping-ISOLatin1Accent won't do anything, and posibly your DB query returns
data in ISO-Latin1 (I always have this issue with UTF8-Mysql), so unless you
transcode your data from Latin1 to UTF8 before sending it to SolR,
mapping-ISOLatin1Accent won't know how to interpret it.
Does it make any sense? :P
On Fri, Mar 20, 2009 at 11:53 AM, aerox7 wrote:
>
> I'm using DataImportHandler to send my data to Solr ! so you mean it
> possible
> to apply a transformer in db-config.xml with a perl script ?
>
>
> Óscar Marín Miró wrote:
> >
> > Hi,
> >
> > My guess is that *although* your DB is in UTF-8, the database engine
> sends
> > you the rows in ISO-Latin1, so before doing *anything* after receiving
> the
> > data, you should transcode from ISO-Latin1 to UTF-8 and then send that to
> > SolR. I'm no Java expert, but in perl (MySQL DB in utf-8) I have to do
> > with
> > any row:
> >
> > $row=decode("iso-8859-1",$row);
> >
> > ... and before building the xml to invoque and add document to SolR:
> >
> > $row=encode("utf8",$row);
> >
> > On Fri, Mar 20, 2009 at 10:55 AM, aerox7 wrote:
> >
> >>
> >> I add :
> >> "è" => "e" to mapping-ISOLatin1Accent.txt
> >>
> >> and add the following fieldType:
> >>
> >> >> positionIncrementGap="100" >
> >>
> >> >> mapping="mapping-ISOLatin1Accent.txt"/>
> >>
> >>
> >>
> >>
> >> By still have the same probleme ! it's only work when i store ISO string
> >> into UTF-8 data base (ex: store solène not solène) :,(
> >>
> >>
> >>
> >>
> >> aerox7 wrote:
> >> >
> >> > ==> where are you seeing it as ""Solène" as opposed to the
> >> > correct way of solène?
> >> >
> >> > I have "Solène" in my Mysql DATA BASE ! so i don't know if this is
> >> > correct or not ? i gess that "Solène" is solène in UTF-8 ?!
> >> >
> >> > I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp,
> >> so
> >> > when i try with solène everything is ok ! but when i try with Solène
> >> > (like what i have in DB) analysis convert à in A delete ¨ so i get
> >> SolAne
> >> > !!!
> >> >
> >> > I think that ISOLatin1AccentFilterFactory take only string with
> Charset
> >> > ISO-8859-1 .
> >> >
> >> > So any solution to transform my string to ISO-8859-1 before indexing
> >> > process. May be by creating transformer in DataImportHandler ? (Never
> >> code
> >> > in java :( )
> >> >
> >> > Thank you all.
> >> >
> >> >
> >> > Koji Sekiguchi-2 wrote:
> >> >>
> >> >> aerox7 wrote:
> >> >>> Hi,
> >> >>> I have a mysql data base in UTF-8. I have a row with "Solène"
> >> (solène).
> >> >>> I
> >> >>> want to transforme this to solene, so i use Solr
> >> >>> ISOLatin1AccentFilterFactory to perform this task but it dosn't work
> >> ?!!
> >> >>>
> >> >>> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to
> >> utf-8
> >> >>> so
> >> >>> normaly ISOLatin1AccentFilterFactory have to replace the accent
> >> ...
> >> >>>
> >> >>> any ideas ?
> >> >>>
> >> >>> i use DataImportHandler.
> >> >>>
> >> >>
> >> >> If a mapping rule "è" to "e" is always true in your field, you can
> >> try
> >> >> to use MappingCharFilter
> >> >> instead of ISOLatin1AccentFilter. Add the following line to
> >> >> mapping-ISOLatin1Accent.txt:
> >> >>
> >> >> "è" => "e"
> >> >>
> >> >> and add the following fieldType:
> >> >>
> >> >> >> >> positionIncrementGap="100" >
> >> >>
> >> >> >> >> mapping="mapping-ISOLatin1Accent.txt"/>
> >> >> >> class="solr.CharStreamAwareWhitespaceTokenizerFactory"/>
> >> >>
> >> >>
> >> >>
> >> >> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly
> >> build.
> >> >>
> >> >> Koji
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22617278.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > “I may not believe in myself, but I believe in what I'm doing.”
> >
> > -- Jimmy Page
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22618085.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
“I may not believe in myself, but I believe in what I'm doing.”
-- Jimmy Page
Re: Issue with Facet Query
I am using this query only but I am getting the same results. facet=true&facet.field=productPrice_product_str_s&fq=productPrice_product_str_s:[1%20TO%20100] - - 202 57 14 11 10 9 9 7 6 5 4 2 2 2 2 2 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 It still is not showing up the other values. Do I need to make any entry in schema or solrConfig xml files. Do I need to convert the string into numeric values etc etc. Please suggest. Thanks, Amit Shalin Shekhar Mangar wrote: > > On Fri, Mar 20, 2009 at 4:00 PM, dabboo wrote: > >> >> Hi Shalin, >> >> One more thing, >> >> >> facet.field=productPrice_product_str_s&facet.query=productPrice_product_str_s:[0%20TO%20100] >> >> This is my query and these are results, I am getting: >> >> 202 >> 57 >> 14 >> 11 >> 10 >> 9 >> 9 >> 7 >> 6 >> 5 >> 4 >> 2 >> 2 >> 2 >> 2 >> 2 >> 1 >> 1 >> 1 >> 1 >> 1 >> 1 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> >> It is only returning results, which are having values started with 2, 3, >> 4 >> or some other integer instead of only 1. It is not returning records in >> which value is >10 and <100. >> > > Please do not send a duplicate mails. It will not help you get an answer > faster. > > If you need to filter results to a specific range then you should use > filter > queries through the fq parameter: > > fq=productPrice_product_str_s:[0%20TO%20100] > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22618714.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with Facet Query
And you'll need to re-index once you make the schema change. On Fri, Mar 20, 2009 at 5:24 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > What is the type of the productPrice_product_str field? I'm guessing that > it is a string type. > > Since it is a float value and you need range search, you should change this > to a 'sfloat' or 'sdouble' in your schema.xml > > > On Fri, Mar 20, 2009 at 5:11 PM, dabboo wrote: > >> >> I am using this query only but I am getting the same results. >> >> >> >> facet=true&facet.field=productPrice_product_str_s&fq=productPrice_product_str_s:[1%20TO%20100] >> >> >> - >> - >> 202 >> 57 >> 14 >> 11 >> 10 >> 9 >> 9 >> 7 >> 6 >> 5 >> 4 >> 2 >> 2 >> 2 >> 2 >> 2 >> 1 >> 1 >> 1 >> 1 >> 1 >> 1 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> 0 >> >> It still is not showing up the other values. Do I need to make any entry >> in >> schema or solrConfig xml files. Do I need to convert the string into >> numeric >> values etc etc. >> >> Please suggest. >> >> Thanks, >> Amit >> >> >> Shalin Shekhar Mangar wrote: >> > >> > On Fri, Mar 20, 2009 at 4:00 PM, dabboo wrote: >> > >> >> >> >> Hi Shalin, >> >> >> >> One more thing, >> >> >> >> >> >> >> facet.field=productPrice_product_str_s&facet.query=productPrice_product_str_s:[0%20TO%20100] >> >> >> >> This is my query and these are results, I am getting: >> >> >> >> 202 >> >> 57 >> >> 14 >> >> 11 >> >> 10 >> >> 9 >> >> 9 >> >> 7 >> >> 6 >> >> 5 >> >> 4 >> >> 2 >> >> 2 >> >> 2 >> >> 2 >> >> 2 >> >> 1 >> >> 1 >> >> 1 >> >> 1 >> >> 1 >> >> 1 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> 0 >> >> >> >> It is only returning results, which are having values started with 2, >> 3, >> >> 4 >> >> or some other integer instead of only 1. It is not returning records in >> >> which value is >10 and <100. >> >> >> > >> > Please do not send a duplicate mails. It will not help you get an answer >> > faster. >> > >> > If you need to filter results to a specific range then you should use >> > filter >> > queries through the fq parameter: >> > >> > fq=productPrice_product_str_s:[0%20TO%20100] >> > >> > -- >> > Regards, >> > Shalin Shekhar Mangar. >> > >> > >> >> -- >> View this message in context: >> http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22618714.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > -- Regards, Shalin Shekhar Mangar.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
A got you :)
Sorry. Correct, I use a Perl client. But sorry to say, I don't use
DataImportHandler. I just make the queries to the DB, filter the results,
and build the solr XML 'by hand' at the perl script :(
On Fri, Mar 20, 2009 at 1:04 PM, aerox7 wrote:
>
> Yes ! i completely understand the problem. I'm just asking about your
> solution to resolvre this problem.
>
> I gess that you use Solar PERL Client to index your DATABASE. for my case i
> use DataImportHandler, so to only solution that i have with this is to
> create a transformer for DataImportHandler and try to convert my row from
> latin to UTF-8. (see
>
> http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9
> )
>
> So i just wanna know if you use DataImportHandler two with a perl script
> like a transformer ?
>
>
> Óscar Marín Miró wrote:
> >
> > What I mean is that unless "solène" travels to Solr in strict UTF-8,
> > mapping-ISOLatin1Accent won't do anything, and posibly your DB query
> > returns
> > data in ISO-Latin1 (I always have this issue with UTF8-Mysql), so unless
> > you
> > transcode your data from Latin1 to UTF8 before sending it to SolR,
> > mapping-ISOLatin1Accent won't know how to interpret it.
> >
> > Does it make any sense? :P
> >
> > On Fri, Mar 20, 2009 at 11:53 AM, aerox7 wrote:
> >
> >>
> >> I'm using DataImportHandler to send my data to Solr ! so you mean it
> >> possible
> >> to apply a transformer in db-config.xml with a perl script ?
> >>
> >>
> >> Óscar Marín Miró wrote:
> >> >
> >> > Hi,
> >> >
> >> > My guess is that *although* your DB is in UTF-8, the database engine
> >> sends
> >> > you the rows in ISO-Latin1, so before doing *anything* after receiving
> >> the
> >> > data, you should transcode from ISO-Latin1 to UTF-8 and then send that
> >> to
> >> > SolR. I'm no Java expert, but in perl (MySQL DB in utf-8) I have to do
> >> > with
> >> > any row:
> >> >
> >> > $row=decode("iso-8859-1",$row);
> >> >
> >> > ... and before building the xml to invoque and add document to SolR:
> >> >
> >> > $row=encode("utf8",$row);
> >> >
> >> > On Fri, Mar 20, 2009 at 10:55 AM, aerox7
> wrote:
> >> >
> >> >>
> >> >> I add :
> >> >> "è" => "e" to mapping-ISOLatin1Accent.txt
> >> >>
> >> >> and add the following fieldType:
> >> >>
> >> >> >> >> positionIncrementGap="100" >
> >> >>
> >> >> >> >> mapping="mapping-ISOLatin1Accent.txt"/>
> >> >> class="solr.CharStreamAwareWhitespaceTokenizerFactory"/>
> >> >>
> >> >>
> >> >>
> >> >> By still have the same probleme ! it's only work when i store ISO
> >> string
> >> >> into UTF-8 data base (ex: store solène not solène) :,(
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> aerox7 wrote:
> >> >> >
> >> >> > ==> where are you seeing it as ""Solène" as opposed to the
> >> >> > correct way of solène?
> >> >> >
> >> >> > I have "Solène" in my Mysql DATA BASE ! so i don't know if this is
> >> >> > correct or not ? i gess that "Solène" is solène in UTF-8 ?!
> >> >> >
> >> >> > I'vz tryed analysis in
> >> http://localhost:8983/solr/admin/analysis.jsp,
> >> >> so
> >> >> > when i try with solène everything is ok ! but when i try with
> >> Solène
> >> >> > (like what i have in DB) analysis convert à in A delete ¨ so i get
> >> >> SolAne
> >> >> > !!!
> >> >> >
> >> >> > I think that ISOLatin1AccentFilterFactory take only string with
> >> Charset
> >> >> > ISO-8859-1 .
> >> >> >
> >> >> > So any solution to transform my string to ISO-8859-1 before
> indexing
> >> >> > process. May be by creating transformer in DataImportHandler ?
> >> (Never
> >> >> code
> >> >> > in java :( )
> >> >> >
> >> >> > Thank you all.
> >> >> >
> >> >> >
> >> >> > Koji Sekiguchi-2 wrote:
> >> >> >>
> >> >> >> aerox7 wrote:
> >> >> >>> Hi,
> >> >> >>> I have a mysql data base in UTF-8. I have a row with "Solène"
> >> >> (solène).
> >> >> >>> I
> >> >> >>> want to transforme this to solene, so i use Solr
> >> >> >>> ISOLatin1AccentFilterFactory to perform this task but it dosn't
> >> work
> >> >> ?!!
> >> >> >>>
> >> >> >>> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat
> to
> >> >> utf-8
> >> >> >>> so
> >> >> >>> normaly ISOLatin1AccentFilterFactory have to replace the accent
> >> >> ...
> >> >> >>>
> >> >> >>> any ideas ?
> >> >> >>>
> >> >> >>> i use DataImportHandler.
> >> >> >>>
> >> >> >>
> >> >> >> If a mapping rule "è" to "e" is always true in your field, you
> can
> >> >> try
> >> >> >> to use MappingCharFilter
> >> >> >> instead of ISOLatin1AccentFilter. Add the following line to
> >> >> >> mapping-ISOLatin1Accent.txt:
> >> >> >>
> >> >> >> "è" => "e"
> >> >> >>
> >> >> >> and add the following fieldType:
> >> >> >>
> >> >> >> >> >> >> positionIncrementGap="100" >
> >> >> >>
> >> >> >> >> >> >> mapping="mapping-ISOLatin1Accent.txt"/>
> >> >> >> >> >> class="solr.CharStreamAwareWhitespaceTokenizerFactory"/>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> MappingCharFilter and mapping-ISOLatin1Accent.txt are in
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
On Fri, Mar 20, 2009 at 5:34 PM, aerox7 wrote: > > Yes ! i completely understand the problem. I'm just asking about your > solution to resolvre this problem. > > I gess that you use Solar PERL Client to index your DATABASE. for my case i > use DataImportHandler, so to only solution that i have with this is to > create a transformer for DataImportHandler and try to convert my row from > latin to UTF-8. (see > > http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9 > ) > > So i just wanna know if you use DataImportHandler two with a perl script > like a transformer ? > No, but you can use any language which is available on the Java VM. For example, Javascript (available by default on JDK6), JRuby, Jython, Groovy, BeanShell etc. But you may not need to do so much. Look at http://www.mysqlperformanceblog.com/2009/03/17/converting-character-sets/ -- Regards, Shalin Shekhar Mangar.
Re: Facet Query Results Issue
On Mar 20, 2009, at 6:39 AM, dabboo wrote: this is my facet query. facet .field =productPrice_product_str_s&facet.query=productPrice_product_str_s: [0%20TO%20100] This is my query and these are results, I am getting: It is only returning results, which are having values started with 2, 3, 4 or some other integer instead of only 1. It is not returning records in which value is >10 and <100. Please suggest. If you want the counts filtered, use fq (instead of or in addition to facet.query). facet.query/facet.field are for generating counts for documents that match q/fq parameters, but do not themselves filter. Erik
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
My DATABASE is already in UTF-8 (Collation and Charset). I already set Tomcat connector to UTF-8, and Mysql default charset to UTF-8 How to force mysql to send on UTF-8 (Or may be i have to do this for TomCat ?) i'm going crazy... :) Shalin Shekhar Mangar wrote: > > On Fri, Mar 20, 2009 at 5:34 PM, aerox7 wrote: > >> >> Yes ! i completely understand the problem. I'm just asking about your >> solution to resolvre this problem. >> >> I gess that you use Solar PERL Client to index your DATABASE. for my case >> i >> use DataImportHandler, so to only solution that i have with this is to >> create a transformer for DataImportHandler and try to convert my row from >> latin to UTF-8. (see >> >> http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9 >> ) >> >> So i just wanna know if you use DataImportHandler two with a perl script >> like a transformer ? >> > > No, but you can use any language which is available on the Java VM. For > example, Javascript (available by default on JDK6), JRuby, Jython, Groovy, > BeanShell etc. > > But you may not need to do so much. Look at > http://www.mysqlperformanceblog.com/2009/03/17/converting-character-sets/ > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22619285.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Issue with Facet Query
What is the type of the productPrice_product_str field? I'm guessing that it is a string type. Since it is a float value and you need range search, you should change this to a 'sfloat' or 'sdouble' in your schema.xml On Fri, Mar 20, 2009 at 5:11 PM, dabboo wrote: > > I am using this query only but I am getting the same results. > > > > facet=true&facet.field=productPrice_product_str_s&fq=productPrice_product_str_s:[1%20TO%20100] > > > - > - > 202 > 57 > 14 > 11 > 10 > 9 > 9 > 7 > 6 > 5 > 4 > 2 > 2 > 2 > 2 > 2 > 1 > 1 > 1 > 1 > 1 > 1 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > > It still is not showing up the other values. Do I need to make any entry in > schema or solrConfig xml files. Do I need to convert the string into > numeric > values etc etc. > > Please suggest. > > Thanks, > Amit > > > Shalin Shekhar Mangar wrote: > > > > On Fri, Mar 20, 2009 at 4:00 PM, dabboo wrote: > > > >> > >> Hi Shalin, > >> > >> One more thing, > >> > >> > >> > facet.field=productPrice_product_str_s&facet.query=productPrice_product_str_s:[0%20TO%20100] > >> > >> This is my query and these are results, I am getting: > >> > >> 202 > >> 57 > >> 14 > >> 11 > >> 10 > >> 9 > >> 9 > >> 7 > >> 6 > >> 5 > >> 4 > >> 2 > >> 2 > >> 2 > >> 2 > >> 2 > >> 1 > >> 1 > >> 1 > >> 1 > >> 1 > >> 1 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> 0 > >> > >> It is only returning results, which are having values started with 2, 3, > >> 4 > >> or some other integer instead of only 1. It is not returning records in > >> which value is >10 and <100. > >> > > > > Please do not send a duplicate mails. It will not help you get an answer > > faster. > > > > If you need to filter results to a specific range then you should use > > filter > > queries through the fq parameter: > > > > fq=productPrice_product_str_s:[0%20TO%20100] > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > > > > -- > View this message in context: > http://www.nabble.com/Issue-with-Facet-Query-tp22615577p22618714.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- Regards, Shalin Shekhar Mangar.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
Yes ! i completely understand the problem. I'm just asking about your
solution to resolvre this problem.
I gess that you use Solar PERL Client to index your DATABASE. for my case i
use DataImportHandler, so to only solution that i have with this is to
create a transformer for DataImportHandler and try to convert my row from
latin to UTF-8. (see
http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9)
So i just wanna know if you use DataImportHandler two with a perl script
like a transformer ?
Óscar Marín Miró wrote:
>
> What I mean is that unless "solène" travels to Solr in strict UTF-8,
> mapping-ISOLatin1Accent won't do anything, and posibly your DB query
> returns
> data in ISO-Latin1 (I always have this issue with UTF8-Mysql), so unless
> you
> transcode your data from Latin1 to UTF8 before sending it to SolR,
> mapping-ISOLatin1Accent won't know how to interpret it.
>
> Does it make any sense? :P
>
> On Fri, Mar 20, 2009 at 11:53 AM, aerox7 wrote:
>
>>
>> I'm using DataImportHandler to send my data to Solr ! so you mean it
>> possible
>> to apply a transformer in db-config.xml with a perl script ?
>>
>>
>> Óscar Marín Miró wrote:
>> >
>> > Hi,
>> >
>> > My guess is that *although* your DB is in UTF-8, the database engine
>> sends
>> > you the rows in ISO-Latin1, so before doing *anything* after receiving
>> the
>> > data, you should transcode from ISO-Latin1 to UTF-8 and then send that
>> to
>> > SolR. I'm no Java expert, but in perl (MySQL DB in utf-8) I have to do
>> > with
>> > any row:
>> >
>> > $row=decode("iso-8859-1",$row);
>> >
>> > ... and before building the xml to invoque and add document to SolR:
>> >
>> > $row=encode("utf8",$row);
>> >
>> > On Fri, Mar 20, 2009 at 10:55 AM, aerox7 wrote:
>> >
>> >>
>> >> I add :
>> >> "è" => "e" to mapping-ISOLatin1Accent.txt
>> >>
>> >> and add the following fieldType:
>> >>
>> >> > >> positionIncrementGap="100" >
>> >>
>> >>> >> mapping="mapping-ISOLatin1Accent.txt"/>
>> >>
>> >>
>> >>
>> >>
>> >> By still have the same probleme ! it's only work when i store ISO
>> string
>> >> into UTF-8 data base (ex: store solène not solène) :,(
>> >>
>> >>
>> >>
>> >>
>> >> aerox7 wrote:
>> >> >
>> >> > ==> where are you seeing it as ""Solène" as opposed to the
>> >> > correct way of solène?
>> >> >
>> >> > I have "Solène" in my Mysql DATA BASE ! so i don't know if this is
>> >> > correct or not ? i gess that "Solène" is solène in UTF-8 ?!
>> >> >
>> >> > I'vz tryed analysis in
>> http://localhost:8983/solr/admin/analysis.jsp,
>> >> so
>> >> > when i try with solène everything is ok ! but when i try with
>> Solène
>> >> > (like what i have in DB) analysis convert à in A delete ¨ so i get
>> >> SolAne
>> >> > !!!
>> >> >
>> >> > I think that ISOLatin1AccentFilterFactory take only string with
>> Charset
>> >> > ISO-8859-1 .
>> >> >
>> >> > So any solution to transform my string to ISO-8859-1 before indexing
>> >> > process. May be by creating transformer in DataImportHandler ?
>> (Never
>> >> code
>> >> > in java :( )
>> >> >
>> >> > Thank you all.
>> >> >
>> >> >
>> >> > Koji Sekiguchi-2 wrote:
>> >> >>
>> >> >> aerox7 wrote:
>> >> >>> Hi,
>> >> >>> I have a mysql data base in UTF-8. I have a row with "Solène"
>> >> (solène).
>> >> >>> I
>> >> >>> want to transforme this to solene, so i use Solr
>> >> >>> ISOLatin1AccentFilterFactory to perform this task but it dosn't
>> work
>> >> ?!!
>> >> >>>
>> >> >>> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to
>> >> utf-8
>> >> >>> so
>> >> >>> normaly ISOLatin1AccentFilterFactory have to replace the accent
>> >> ...
>> >> >>>
>> >> >>> any ideas ?
>> >> >>>
>> >> >>> i use DataImportHandler.
>> >> >>>
>> >> >>
>> >> >> If a mapping rule "è" to "e" is always true in your field, you can
>> >> try
>> >> >> to use MappingCharFilter
>> >> >> instead of ISOLatin1AccentFilter. Add the following line to
>> >> >> mapping-ISOLatin1Accent.txt:
>> >> >>
>> >> >> "è" => "e"
>> >> >>
>> >> >> and add the following fieldType:
>> >> >>
>> >> >> > >> >> positionIncrementGap="100" >
>> >> >>
>> >> >> > >> >> mapping="mapping-ISOLatin1Accent.txt"/>
>> >> >> > >> class="solr.CharStreamAwareWhitespaceTokenizerFactory"/>
>> >> >>
>> >> >>
>> >> >>
>> >> >> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly
>> >> build.
>> >> >>
>> >> >> Koji
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22617278.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > “I may not believe in myself, but I believe in what I'm doing.”
>> >
>> > -- Jimmy Page
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22618085.html
>> Sent from the
Error in identifying the primary key
Hi, I am new to Solr. I am trying to index SQL table rows. I am getting the below error. Can anyone help me in resolving this issue. Mar 20, 2009 6:03:38 PM org.apache.solr.handler.dataimport.DataImporter verifyWithSchema INFO: id is a required field in SolrSchema . But not found in DataConfig Mar 20, 2009 6:03:38 PM org.apache.solr.handler.dataimport.DataImportHandler inform SEVERE: Exception while loading DataImporter org.apache.solr.handler.dataimport.DataImportHandlerException: There are errors in the Schema The field :age present in DataConfig does not have a counterpart in Solr Schema The field :firstname present in DataConfig does not have a counterpart in Solr Schema The field :lastName present in DataConfig does not have a counterpart in Solr Schema at org.apache.solr.handler.dataimport.DataImporter.(DataImporter.java:108) at org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:95) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388) at org.apache.solr.core.SolrCore.(SolrCore.java:571) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:121) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4222) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:740) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:831) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:720) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1150) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022) at org.apache.catalina.core.StandardHost.start(StandardHost.java:736) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) at org.apache.catalina.core.StandardService.start(StandardService.java:448) at org.apache.catalina.core.StandardServer.start(StandardServer.java:700) at org.apache.catalina.startup.Catalina.start(Catalina.java:552) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433) Mar 20, 2009 6:03:38 PM org.apache.solr.servlet.SolrDispatchFilter init SEVERE: Could not start SOLR. Check solr/home property org.apache.solr.common.SolrException: FATAL: Could not create importer. DataImporter config invalid at org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:103) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388) at org.apache.solr.core.SolrCore.(SolrCore.java:571) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:121) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) Thanks
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
Hi, Maybe this info is handy for you: http://dev.mysql.com/doc/refman/5.0/en/charset-connection.html The fact is Mysql can have UTF8 in its storage engine (or defined by database), as you have, but the *connection* to the mysql client, can be set to latin1. In fact, here are my character_set variables: character_set_client = latin1 character_set_connection = latin1 character_set_database = utf8 character_set_filesystem = binary character_set_results = latin1 character_set_server = latin1 character_set_system = utf8 character_sets_dir = /usr/share/mysql/charsets/ As you see, the database is in utf8, *but* the client, connection, results and server, expects latin1. You can see this variables through a mysql console, just typing: $ mysql -u user -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8114 Server version: 5.0.32-Debian_7etch5-log Debian etch distribution Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> SHOW VARIABLES LIKE 'character_set%'; +--++ | Variable_name| Value | +--++ | character_set_client | latin1 | | character_set_connection | latin1 | | character_set_database | latin1 | | character_set_filesystem | binary | | character_set_results| latin1 | | character_set_server | latin1 | | character_set_system | utf8 | | character_sets_dir | /usr/share/mysql/charsets/ | +--++ 8 rows in set (0.00 sec) and change them like this: mysql> SET character_set_client = utf8; Query OK, 0 rows affected (0.00 sec) mysql> SHOW VARIABLES LIKE 'character_set%'; +--++ | Variable_name| Value | +--++ | character_set_client | utf8 | | character_set_connection | latin1 | | character_set_database | latin1 | | character_set_filesystem | binary | | character_set_results| latin1 | | character_set_server | latin1 | | character_set_system | utf8 | | character_sets_dir | /usr/share/mysql/charsets/ | +--++ 8 rows in set (0.00 sec) So... maybe after setting all variables that are set to latin1 to utf8 can solve your problem? If they are set to latin1, of course ;) If this is not the problem, hell, we escaped from work just for a few minutes :P On Fri, Mar 20, 2009 at 1:25 PM, aerox7 wrote: > > My DATABASE is already in UTF-8 (Collation and Charset). > > I already set Tomcat connector to UTF-8, and Mysql default charset to > UTF-8 How to force mysql to send on UTF-8 (Or may be i have to do this > for TomCat ?) > > i'm going crazy... :) > > > Shalin Shekhar Mangar wrote: > > > > On Fri, Mar 20, 2009 at 5:34 PM, aerox7 wrote: > > > >> > >> Yes ! i completely understand the problem. I'm just asking about your > >> solution to resolvre this problem. > >> > >> I gess that you use Solar PERL Client to index your DATABASE. for my > case > >> i > >> use DataImportHandler, so to only solution that i have with this is to > >> create a transformer for DataImportHandler and try to convert my row > from > >> latin to UTF-8. (see > >> > >> > http://wiki.apache.org/solr/DataImportHandler#head-27fcc2794bd71f7d727104ffc6b99e194bdb6ff9 > >> ) > >> > >> So i just wanna know if you use DataImportHandler two with a perl script > >> like a transformer ? > >> > > > > No, but you can use any language which is available on the Java VM. For > > example, Javascript (available by default on JDK6), JRuby, Jython, > Groovy, > > BeanShell etc. > > > > But you may not need to do so much. Look at > > > http://www.mysqlperformanceblog.com/2009/03/17/converting-character-sets/ > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > > > > -- > View this message in context: > http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22619285.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- “I may not believe in myself, but I believe in what I'm doing.” -- Jimmy Page
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
Usually, when I see characters like this, it means you aren't viewing/ handling the UTF-8 correctly when bringing it into Java. I would first check that your DB or JDBC driver is getting the chars out right. It may even be the case that they did not go into the DB correctly in the first place. On Mar 20, 2009, at 4:36 AM, aerox7 wrote: ==> where are you seeing it as ""Solène" as opposed to the correct way of solène? I have "Solène" in my Mysql DATA BASE ! so i don't know if this is correct or not ? i gess that "Solène" is solène in UTF-8 ?! I'vz tryed analysis in http://localhost:8983/solr/admin/ analysis.jsp, so when i try with solène everything is ok ! but when i try with Solène (like what i have in DB) analysis convert à in A delete ¨ so i get SolAne !!! I think that ISOLatin1AccentFilterFactory take only string with Charset ISO-8859-1 . So any solution to transform my string to ISO-8859-1 before indexing process. May be by creating transformer in DataImportHandler ? (Never code in java :( ) Thank you all. Koji Sekiguchi-2 wrote: aerox7 wrote: Hi, I have a mysql data base in UTF-8. I have a row with "Solène" (solène). I want to transforme this to solene, so i use Solr ISOLatin1AccentFilterFactory to perform this task but it dosn't work ?!! i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to utf-8 so normaly ISOLatin1AccentFilterFactory have to replace the accent ... any ideas ? i use DataImportHandler. If a mapping rule "è" to "e" is always true in your field, you can try to use MappingCharFilter instead of ISOLatin1AccentFilter. Add the following line to mapping-ISOLatin1Accent.txt: "è" => "e" and add the following fieldType: class="solr.CharStreamAwareWhitespaceTokenizerFactory"/> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build. Koji -- View this message in context: http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22616220.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solrj : probleme with utf-8 content
Hi,
I have that problem to. But I notice that it only happens if I send my data
via solrj. If I send it via the solr-ruby gem, everything is fine
(http://wiki.apache.org/solr/solr-ruby).
Here is my jruby script:
---
require 'rubygems'
require 'solr'
require 'rexml/document'
include Java
def send_via_solrj(text, url)
doc = org.apache.solr.common.SolrInputDocument.new
doc.addField('id', '1')
doc.addField('text', text)
server = org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.new(url)
server.add(doc);
server.commit();
end
def send_via_gem(text, url)
solr_doc = Solr::Document.new
solr_doc['id'] = '2'
solr_doc['text'] = text
options = {
:autocommit => :on
}
conn = Solr::Connection.new(url, options)
conn.add(solr_doc)
end
host = 'localhost'
port = ''
path = '/solr/core0'
url = "http://#{host}:#{port}#{path}";
text = "eaiou with circumflexes: êâîôû"
send_via_solrj(text, url)
send_via_gem(text, url)
puts "done!"
---
If I watch the http messages with tcpmon, I see that the data sent via solrj
is encoded in cp1252 while the data sent via the gem is utf-8.
Anyone has an idea of how we can configure sorlj to send in utf-8?
Thanks in advance.
Walid ABDELKABIR wrote:
>
> when executing this code I got in my index the field "includes" with this
> value : "? ? ?" :
> ---
> String content ="eaiou with circumflexes: êâîôû";
> SolrInputDocument doc = new SolrInputDocument();
> doc.addField( "id", "123", 1.0f );
> doc.addField( "includes", content, 1.0f );
> server.add( doc );
> ---
>
> but this code works fine :
>
> ---
> String addContent = ""
> +"123 name="includes">eaiou with circumflexes:âîôû"
> +"";
> DirectXmlRequest up = new DirectXmlRequest( "/update", addContent );
> server.request( up );
> ---
>
> thanks for help
>
>
--
View this message in context:
http://www.nabble.com/solrj-%3A-probleme-with-utf-8-content-tp22577377p22620317.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: delta-import commit=false doesn't seems to work
Thanks I gave more information there : http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-td22601442.html thanks a lot Paul Noble Paul നോബിള് नोब्ळ् wrote: > > sorry, the whole thing was commented . I did not notice that. I'll > look into that > > 2009/3/20 Noble Paul നോബിള് नोब्ळ् : >> you have set autoCommit every x minutes . it must have invoked commit >> automatically >> >> >> On Thu, Mar 19, 2009 at 4:17 PM, sunnyfr wrote: >>> >>> Hi, >>> >>> Even if I hit command=delta-import&commit=false&optimize=false >>> I still have commit set in my logs and sometimes even optimize=true, >>> >>> About optimize I wonder if it comes from commitment too close and one is >>> not >>> done, but still I don't know really. >>> >>> Any idea? >>> >>> Thanks a lot, >>> -- >>> View this message in context: >>> http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22620439.html Sent from the Solr - User mailing list archive at Nabble.com.
Unknown FieldType: 'string' used in QueryElevationComponent
Hi, I am having below schema.xml, I did not define any string field. But I am getting the below error when I start Tomcat, Can anyone please suggest me what is the issue here. WARNING: No queryConverter defined, using default converter Mar 20, 2009 7:31:55 PM org.apache.solr.core.QuerySenderListener newSearcher INFO: QuerySenderListener sending requests to searc...@fe135d main Mar 20, 2009 7:31:55 PM org.apache.solr.servlet.SolrDispatchFilter init SEVERE: Could not start SOLR. Check solr/home property org.apache.solr.common.SolrException: Unknown FieldType: 'string' used in QueryElevationComponent at org.apache.solr.handler.component.QueryElevationComponent.inform(QueryElevationComponent.java:151) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388) at org.apache.solr.core.SolrCore.(SolrCore.java:571) person_id all
Re: how can I check field which are indexed but not stored?
Cool I was just having a look on it but it doesn't seem to show up field which are not stored just tried : /admin/luke?id=8582006&fl=description but it doesn't seems to work :( It find this id but show up stored field. Did I do a mistake ? thanks a lot Markus Jelsma - Buyways B.V. wrote: > > > > On Fri, 2009-03-20 at 03:41 -0700, sunnyfr wrote: > >> Hi >> >> I've an issue, I've some data which come up but I've applied a filtre on >> it >> and it shouldnt, when I check in my database mysql I've obviously the >> document which has been updated so I will like to see how it is in solr. >> >> if I do : /solr/video/select?q=id:8582006 I will just see field which has >> been stored. Is there a way to see how data are indexed for other field >> of >> my schema which are not stored but indexed. > > > /solr/admin/luke > will show you a lot of information concering stored and indexed fields. > > Hope this is what you meant. > > >> >> Like a bit in the console dataimporthandler, which with verbose activated >> I >> can see every field of my schema. >> >> Otherwise what would you reckon in this case, a document which has not >> been >> updated ? how can I sort it out? >> >> Thanks a lot guys for your excellent help > > -- View this message in context: http://www.nabble.com/how-can-I-check-field-which-are-indexed-but-not-stored--tp22617914p22621773.html Sent from the Solr - User mailing list archive at Nabble.com.
q.alt and highlights
Is there any way to activate highlights using q.alt of dismax? I have hl well configurated and working for normal q in the field "content" (in the solr.xml). For q.alt, I try to do: http://localhost:8080/solr/select/?q=&q.alt=my_id:475836&start=0&rows=10&hl=true But no highlight is showed... Any advice? -- View this message in context: http://www.nabble.com/q.alt-and-highlights-tp22621774p22621774.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Compound word search (maybe DisMaxQueryPaser problem)
Oh my ... thinking even more about it, I have to admit you're right :) But that leaves me somewhat clueless again. So I'll just try and share my thoughts on this. Maybe someone will read this and can point me to a possible solution ... or tell me where I'm wrong. Say we have a schema with fields f1, f2 and f3. And the user queries for "a b c" (without the quotes). What I would expect as resulting query would be (leaving out the details like tie, boosting etc.): ((f1:a OR f2:a OR f3:a) AND (f1:b OR f2:b OR f3:b) AND (f1:c OR f2:c OR f3:c)) OR ((f1:ab OR f2:ab OR f3:ab) AND (f1:c f2:c f3:c)) OR ((f1:a OR f2:a OR f3:a) AND (f1:bc f2:bc f3:bc)) (possibly also f1:abc OR f2:abc .. and/or f1:a b c OR f2:a b c etc. ) So every possibility of how to write compound words is covered. But then there is the problem that there are fields that require exact matching (something like EAN, manufacturer code or product serial number. Unfortunately these can contain whitespaces etc. So a b c can also be a valid manufacturer code which sould match as a whole). So I modeled the fields in the schema accordingly: making exact match fields string and add ShingleFilter and WordDelimiterFiler for content fields. And I thought the fields analyzer stack would take care of how to process the user input. But when I pass the user query as phrase to the DisMax Handler (so that every field gets to see the whole user query and can tokenize and shingle it) I get a query like this: (f1:a b c) OR (f2:a OR f2:ab OR f2:b OR f2:bc OR f2:c) OR (f3:a OR f3:ab OR f3:b OR f3:bc OR f3:c) which apparently is not what I need as it also would find for example documents that only contain a or b etc. When using phrase fields this query is just added to the normal query and therefore the query fails to find the compound words. Also using the FieldQuery Analyzer does not yield the desired results as the parsed queries as a matter of fact look like the phrase queries from the DisMax parser. I tried dozends of variations and I'm still pretty sure that there must be a way to get this working. It doesn't look that hard. But for now I will settle this for the weekend :) Have a nice weekend all and thanks in advance for any comments or replies. Tobi Chris Hostetter schrieb: : Many thanks for your explanation. That really helped me a lot in understanding : DisMax - and finally I realized that DisMax is not at all what I need. : Actually I do not want results where "blue" is in one field and "tooth" in : another (imagine you search for a notebook with blue tooth and get some blue : products that accidentally have tooth in some field). except that if you use the "pf" param as well, a search for... blue tooth can score products where "blue tooth" appears in one field higher then products where "blue" apears in one field and "tooth" appears in another field. The approach you are describing might give you you better precisions (ie: less total results) but it will have a loss in precision, a query like this... blue tooth notebook ...probably won't be able to find documents matching the terms "product_type:notebook features:blue features:tooth" ... but dismax can. -Hoss
Re: Null pointer exception on use of ImportDataHandler (useSolrAddSchema="true")
thanks,
I applied the patch in SOLR-1077 and this is now fixed for me (i
updated the bug w/ a comment)
sam keen
2009/3/19 Noble Paul നോബിള് नोब्ळ् :
> it is a bug , I have raised an issue
>
> https://issues.apache.org/jira/browse/SOLR-1077
>
> On Fri, Mar 20, 2009 at 4:41 AM, Sam Keen wrote:
>> guess I spoke to soon. The above setup (with Shalin's fix) works for
>> a mock run of 2 records. But when I try it with the production data
>> of about 450 records, I get this error.
>>
>> again, any help is greatly appreciated
>>
>> sam keen
>>
>> Mar 19, 2009 3:59:20 PM
>> org.apache.solr.handler.dataimport.DataImporter doFullImport
>> INFO: Starting Full Import
>> Mar 19, 2009 3:59:20 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/dataimport params={command=full-import}
>> status=0 QTime=6
>> Mar 19, 2009 3:59:20 PM org.apache.solr.update.DirectUpdateHandler2 deleteAll
>> INFO: [] REMOVING ALL DOCUMENTS FROM INDEX
>> Mar 19, 2009 3:59:20 PM
>> org.apache.solr.handler.dataimport.HttpDataSource getData
>> INFO: Created URL to: http://local.smwe.com/factsheets/feed
>> Mar 19, 2009 3:59:36 PM
>> org.apache.solr.handler.dataimport.DataImporter doFullImport
>> SEVERE: Full Import failed
>> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException:
>> Index: 3, Size: 3
>> at
>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:85)
>> at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:226)
>> at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:180)
>> at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:163)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
>> at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
>> at
>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
>> at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
>> at java.util.ArrayList.RangeCheck(ArrayList.java:546)
>> at java.util.ArrayList.get(ArrayList.java:321)
>> at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.readRow(XPathEntityProcessor.java:266)
>> at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.access$100(XPathEntityProcessor.java:53)
>> at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor$1.handle(XPathEntityProcessor.java:229)
>> at
>> org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:149)
>> at
>> org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:174)
>> at
>> org.apache.solr.handler.dataimport.XPathRecordReader$Node.parse(XPathRecordReader.java:174)
>> at
>> org.apache.solr.handler.dataimport.XPathRecordReader$Node.access$000(XPathRecordReader.java:89)
>> at
>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:82)
>> ... 9 more
>>
>>
>>
>>
>> On Wed, Mar 18, 2009 at 2:58 PM, Sam Keen wrote:
>>> that worked perfectly Shalin. thanks so much for your help!
>>>
>>> sam keen
>>>
>>>
>>> On Wed, Mar 18, 2009 at 1:15 PM, Shalin Shekhar Mangar
>>> wrote:
On Thu, Mar 19, 2009 at 1:29 AM, Sam Keen wrote:
>
> What I am now attempting to do is leverage 'useSolrAddSchema="true"' .
> I have a URL the responds with a well formatted solr add xml (I'm able
> to add it by POSTing). But when I try to add it using
> http://localhost:8983/solr/dataimport?command=full-import i get a null
> pointer exception.
You need to use XPathEntityProcessor. If you do not specify a processor,
the
default is SqlEntityProcessor (used for DB imports).
Add the attribute processor="XPathEntityProcessor" to the entity and try.
--
Regards,
Shalin Shekhar Mangar.
>>>
>>
>
>
>
> --
> --Noble Paul
>
JVM exception_access_violation
I'm running Solr on Tomcat 6.0.18 with Java 6 update 7 on Windows 2003 64 bit. Over the past month or so, my JVM has crashed twice with the error below. Has anyone experienced this? My system is not heavily loaded, and the crash seems to coincide with an update (via DIH). I'm running trunk code from late January. Note that I update my index ~50 times per day, and this crash has happened twice in the past month (so 2 of 1500 updates seem to have triggered the crash). This Windows deployment is for demos, so I'm not too concerned about it. Interestingly, my production deployment is on a 64 bit Linux system (same versions of everything) and I haven't been able to reproduce the bug there. # # An unexpected error has been detected by Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x080e51c3, pid=4404, tid=956 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b23 mixed mode windows-amd64) # Problematic frame: # V [jvm.dll+0xe51c3] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # --- T H R E A D --- Current thread (0x01de2000): GCTaskThread [stack: 0x,0x] [id=956] siginfo: ExceptionCode=0xc005, reading address 0x Registers: EAX=0x3000, EBX=0x01e40330, ECX=0x000184b49821, EDX=0x000184b4b580 ESP=0x07cff9b0, EBP=0x, ESI=0x000184b4b580, EDI=0x0935 EIP=0x080e51c3, EFLAGS=0x00010206 Top of Stack: (sp=0x07cff9b0) 0x07cff9b0: 01e40330 0x07cff9c0: 000184b4dd88 0935 0x07cff9d0: 08464b08 01dbbdc0 0x07cff9e0: 01dbf190 8a65 0x07cff9f0: 2f5b4000 0002015f 0x07cffa00: 0002 01dbf2f0 0x07cffa10: 01e40330 01dbf430 0x07cffa20: 01dbf4f0 000201602d18 0x07cffa30: 07effa00 07cffb40 0x07cffa40: 0x07cffa50: 0830484d 0x07cffa60: 0002015f 0002 0x07cffa70: 0048 0001 0x07cffa80: 0001 00bb8501 0x07cffa90: 01dbf378 080ea807 0x07cffaa0: 07cffb40 07cffb40 Instructions: (pc=0x080e51c3) 0x080e51b3: 4c 8d 44 24 20 48 8b d6 48 8b 41 10 48 83 c1 10 0x080e51c3: ff 90 c0 01 00 00 44 8b 1d 08 f2 44 00 45 85 db Stack: [0x,0x], sp=0x07cff9b0, free space=127998k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xe51c3] [error occurred during error reporting (printing native stack), id 0xc005] --- P R O C E S S --- Java Threads: ( => current thread ) 0x10286c00 JavaThread "Thread-135" daemon [_thread_blocked, id=4892, stack(0x1169,0x1179)] 0x10285400 JavaThread "http-8084-10" daemon [_thread_blocked, id=5108, stack(0x1201,0x1211)] 0x10287400 JavaThread "http-8084-9" daemon [_thread_blocked, id=1772, stack(0x149a,0x14aa)] 0x1028a400 JavaThread "http-8084-8" daemon [_thread_blocked, id=1656, stack(0x11f1,0x1201)] 0x01dc2c00 JavaThread "http-8084-7" daemon [_thread_blocked, id=2056, stack(0x11e1,0x11f1)] 0x10288400 JavaThread "http-8084-6" daemon [_thread_blocked, id=4792, stack(0x11d1,0x11e1)] 0x10286800 JavaThread "MultiThreadedHttpConnectionManager cleanup" daemon [_thread_blocked, id=3792, stack(0x1251,0x1261)] 0x0f6e8400 JavaThread "http-8084-5" daemon [_thread_blocked, id=3540, stack(0x11c1,0x11d1)] 0x0f6e7800 JavaThread "http-8084-4" daemon [_thread_blocked, id=4048, stack(0x11b1,0x11c1)] 0x0f6e8000 JavaThread "http-8084-3" daemon [_thread_blocked, id=1932, stack(0x1159,0x1169)] 0x0f6e7000 JavaThread "http-8084-2" daemon [_thread_blocked, id=996, stack(0x1149,0x1159)] 0x01dc6000 JavaThread "http-8084-1" daemon [_thread_blocked, id=4924, stack(0x1139,0x1149)] 0x01dc5800 JavaThread "TP-Monitor" daemon [_thread_blocked, id=2288, stack(0x1121,0x1131)] 0x01dc5400 JavaThread "TP-Processor4" daemon [_thread_in_native, id=4588, stack(0x,0x1121)] 0x01dc4c00 JavaThread "TP-Processor3" daemon [_thread_blocked, id=652, stack(0x1101,0x)] 0x01dc4400
Re: stop word search
Hi Erik, I have now commented the query time stopword analyzer .I restarted the server.But now when i search for a stop word ,i am getting results. We had earlier indexed the content with the stop word analyzer.I dont think we need to reindex after commentting the query analyzer,right? This field is a text field with the defaul analyzer. Please let me know if i have missed something here. Regards Sujatha On 3/17/09, Erick Erickson wrote: > > Well, by definition, using an analyzer that removes stopwords > *should* do this at query time. This assumes that you used > an analyzer that removed stopwords at index and query time. > The stopwords are not in the index. > > You can get the behavior you expect by using an analyzer at > query time that does NOT remove stopwords, and one at > indexing time that *does* remove stopwords. Gut I'm having a > hard time imagining that this would result in a good user experience. > > I mean anytime that you had a stopword in the query where the > stopword was required, no results would be returned. Which would > be hard to explain to a user > > What is it you're trying to accomplish? > > Best > Erick > > > > On Tue, Mar 17, 2009 at 7:40 AM, revas wrote: > > > Hi, > > > > I have a query like this > > > > content:the AND iuser_id:5 > > > > which means return all docs of user id 5 which have the word "the" in > > content .Since 'the' is a stop word ,this query executes as just user_id > :5 > > inspite of the "AND" clause ,Whereas the expected result here is since > > there > > is no result for "the " ,no results shloud be returned. > > > > Am i missing anythin here? > > > > Regards > > >
Re: Error in identifying the primary key
for all the fields mentioned in data-config.xml there should be a counterpart in schema.xml anyway that is relaxed in the latest nightly On Fri, Mar 20, 2009 at 6:26 PM, radha c wrote: > Hi, > > I am new to Solr. I am trying to index SQL table rows. > I am getting the below error. Can anyone help me in resolving this issue. > > Mar 20, 2009 6:03:38 PM org.apache.solr.handler.dataimport.DataImporter > verifyWithSchema > INFO: id is a required field in SolrSchema . But not found in DataConfig > Mar 20, 2009 6:03:38 PM org.apache.solr.handler.dataimport.DataImportHandler > inform > SEVERE: Exception while loading DataImporter > org.apache.solr.handler.dataimport.DataImportHandlerException: There are > errors in the Schema > The field :age present in DataConfig does not have a counterpart in Solr > Schema > The field :firstname present in DataConfig does not have a counterpart in > Solr Schema > The field :lastName present in DataConfig does not have a counterpart in > Solr Schema > > at > org.apache.solr.handler.dataimport.DataImporter.(DataImporter.java:108) > at > org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:95) > at > org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388) > at org.apache.solr.core.SolrCore.(SolrCore.java:571) > at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:121) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302) > at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4222) > at > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760) > at > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:740) > at > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544) > at > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:831) > at > org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:720) > at > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490) > at > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1150) > at > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311) > at > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120) > at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022) > at > org.apache.catalina.core.StandardHost.start(StandardHost.java:736) > at > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014) > at > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) > at > org.apache.catalina.core.StandardService.start(StandardService.java:448) > at > org.apache.catalina.core.StandardServer.start(StandardServer.java:700) > at org.apache.catalina.startup.Catalina.start(Catalina.java:552) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295) > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433) > Mar 20, 2009 6:03:38 PM org.apache.solr.servlet.SolrDispatchFilter init > SEVERE: Could not start SOLR. Check solr/home property > org.apache.solr.common.SolrException: FATAL: Could not create importer. > DataImporter config invalid > at > org.apache.solr.handler.dataimport.DataImportHandler.inform(DataImportHandler.java:103) > at > org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:388) > at org.apache.solr.core.SolrCore.(SolrCore.java:571) > at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:121) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) > > Thanks > -- --Noble Paul
Re: delta-import commit=false doesn't seems to work
just hit the DIH without any command and you may be able to see the status of the last import. It can tell you whether a commit/optimize was performed On Fri, Mar 20, 2009 at 7:07 PM, sunnyfr wrote: > > Thanks I gave more information there : > http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-td22601442.html > > thanks a lot Paul > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> sorry, the whole thing was commented . I did not notice that. I'll >> look into that >> >> 2009/3/20 Noble Paul നോബിള് नोब्ळ् : >>> you have set autoCommit every x minutes . it must have invoked commit >>> automatically >>> >>> >>> On Thu, Mar 19, 2009 at 4:17 PM, sunnyfr wrote: Hi, Even if I hit command=delta-import&commit=false&optimize=false I still have commit set in my logs and sometimes even optimize=true, About optimize I wonder if it comes from commitment too close and one is not done, but still I don't know really. Any idea? Thanks a lot, -- View this message in context: http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >>> >>> -- >>> --Noble Paul >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- > View this message in context: > http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22620439.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul
DIH data-config loading
I'm trying to load or delete entities in data-config in runtime, changing the data-config.xml file, reload and delete or full-import as needed.My question is: does data-config gets loaded into memory in runtime an reload only, that is, can I change the file while solr is importing or deleting data? Another question: to delete documents, a different handler from import is used (update), is it problematic to delete documents from a determinate entity while importing? Thanks in advance, Rui Pereira
Re: delta-import commit=false doesn't seems to work
Like you can see, I did that and I've no information in my DIH but you can notice in my logs and even my segments that and optimize is fired alone automaticly? Noble Paul നോബിള് नोब्ळ् wrote: > > just hit the DIH without any command and you may be able to see the > status of the last import. It can tell you whether a commit/optimize > was performed > > On Fri, Mar 20, 2009 at 7:07 PM, sunnyfr wrote: >> >> Thanks I gave more information there : >> http://www.nabble.com/Problem-for-replication-%3A-segment-optimized-automaticly-td22601442.html >> >> thanks a lot Paul >> >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> sorry, the whole thing was commented . I did not notice that. I'll >>> look into that >>> >>> 2009/3/20 Noble Paul നോബിള് नोब्ळ् : you have set autoCommit every x minutes . it must have invoked commit automatically On Thu, Mar 19, 2009 at 4:17 PM, sunnyfr wrote: > > Hi, > > Even if I hit command=delta-import&commit=false&optimize=false > I still have commit set in my logs and sometimes even optimize=true, > > About optimize I wonder if it comes from commitment too close and one > is > not > done, but still I don't know really. > > Any idea? > > Thanks a lot, > -- > View this message in context: > http://www.nabble.com/delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22597630p22597630.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22620439.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Re%3A-delta-import-commit%3Dfalse-doesn%27t-seems-to-work-tp22614216p22625149.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solrj : probleme with utf-8 content
do you know if your java file is encoded with utf-8? sometimes it will be encoded as something different and that can cause funny problems.. On Mar 18, 2009, at 7:46 AM, Walid ABDELKABIR wrote: when executing this code I got in my index the field "includes" with this value : "? ? ?" : --- String content ="eaiou with circumflexes: êâîôû"; SolrInputDocument doc = new SolrInputDocument(); doc.addField( "id", "123", 1.0f ); doc.addField( "includes", content, 1.0f ); server.add( doc ); --- but this code works fine : --- String addContent = "" +"123eaiou with circumflexes:âîôû" +""; DirectXmlRequest up = new DirectXmlRequest( "/update", addContent ); server.request( up ); --- thanks for help
Re: DIH data-config loading
On Fri, Mar 20, 2009 at 10:57 PM, Rui Pereira wrote: > I'm trying to load or delete entities in data-config in runtime, changing > the data-config.xml file, reload and delete or full-import as needed.My > question is: does data-config gets loaded into memory in runtime an reload > only, that is, can I change the file while solr is importing or deleting > data? it is safe to edit the data-config.xml . The reload happens only only if you issue the command=reload-config > Another question: to delete documents, a different handler from import is > used (update), is it problematic to delete documents from a determinate > entity while importing? Solr does not have an issue , but be aware that the commit may be happening after the import and if that is OK for your data then it should be OK > > Thanks in advance, > Rui Pereira > -- --Noble Paul
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
May be there is an issue with the recent changes with SOLR-973 I have given a new patch on SOLR-973 aerox ,is it possible to confirm if that is the problem On Fri, Mar 20, 2009 at 6:52 PM, Grant Ingersoll wrote: > Usually, when I see characters like this, it means you aren't > viewing/handling the UTF-8 correctly when bringing it into Java. I would > first check that your DB or JDBC driver is getting the chars out right. It > may even be the case that they did not go into the DB correctly in the first > place. > > On Mar 20, 2009, at 4:36 AM, aerox7 wrote: > >> >> ==> where are you seeing it as ""Solène" as opposed to the >> correct way of solène? >> >> I have "Solène" in my Mysql DATA BASE ! so i don't know if this is >> correct >> or not ? i gess that "Solène" is solène in UTF-8 ?! >> >> I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp, so >> when i try with solène everything is ok ! but when i try with Solène >> (like >> what i have in DB) analysis convert à in A delete ¨ so i get SolAne !!! >> >> I think that ISOLatin1AccentFilterFactory take only string with Charset >> ISO-8859-1 . >> >> So any solution to transform my string to ISO-8859-1 before indexing >> process. May be by creating transformer in DataImportHandler ? (Never code >> in java :( ) >> >> Thank you all. >> >> >> Koji Sekiguchi-2 wrote: >>> >>> aerox7 wrote: Hi, I have a mysql data base in UTF-8. I have a row with "Solène" (solène). I want to transforme this to solene, so i use Solr ISOLatin1AccentFilterFactory to perform this task but it dosn't work ?!! i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to utf-8 so normaly ISOLatin1AccentFilterFactory have to replace the accent ... any ideas ? i use DataImportHandler. >>> >>> If a mapping rule "è" to "e" is always true in your field, you can try >>> to use MappingCharFilter >>> instead of ISOLatin1AccentFilter. Add the following line to >>> mapping-ISOLatin1Accent.txt: >>> >>> "è" => "e" >>> >>> and add the following fieldType: >>> >>> >> positionIncrementGap="100" > >>> >>> >> mapping="mapping-ISOLatin1Accent.txt"/> >>> >>> >>> >>> >>> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build. >>> >>> Koji >>> >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22616220.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > -- --Noble Paul
Re: solrj : probleme with utf-8 content
SOLR-973 seems to have caused the problem On Fri, Mar 20, 2009 at 11:01 PM, Ryan McKinley wrote: > do you know if your java file is encoded with utf-8? > > sometimes it will be encoded as something different and that can cause funny > problems.. > > > On Mar 18, 2009, at 7:46 AM, Walid ABDELKABIR wrote: > >> when executing this code I got in my index the field "includes" with this >> value : "? ? ?" : >> --- >> String content ="eaiou with circumflexes: êâîôû"; >> SolrInputDocument doc = new SolrInputDocument(); >> doc.addField( "id", "123", 1.0f ); >> doc.addField( "includes", content, 1.0f ); >> server.add( doc ); >> --- >> >> but this code works fine : >> >> --- >> String addContent = "" >> +"123> name="includes">eaiou with circumflexes:âîôû" >> +""; >> DirectXmlRequest up = new DirectXmlRequest( "/update", addContent ); >> server.request( up ); >> --- >> >> thanks for help > > -- --Noble Paul
Re: Page-Rank algorithm
Victor, Solr knows nothing about hyperlinks, web pages, and such. Solr doesn't even have a web crawler. You should ask on nutch-u...@lucene... mailing list instead. The answer there will be positive. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: "Huang, Zijian(Victor)" > To: solr-user@lucene.apache.org > Sent: Thursday, March 19, 2009 5:55:36 PM > Subject: Page-Rank algorithm > > Hi, >Do you guys know if there is some versions of the page-rank algorithm > already implemented in Solr(Lucene)? If not, how hard is it to > implement. I am trying to improve the ranking relevance for Solr. > > Thanks > > > Vic
Re: solrj : probleme with utf-8 content
yes, now it works fine with the trunk sources thanks! Noble Paul നോബിള് नोब्ळ् wrote: > > SOLR-973 seems to have caused the problem > > On Fri, Mar 20, 2009 at 11:01 PM, Ryan McKinley wrote: >> do you know if your java file is encoded with utf-8? >> >> sometimes it will be encoded as something different and that can cause >> funny >> problems.. >> >> >> On Mar 18, 2009, at 7:46 AM, Walid ABDELKABIR wrote: >> >>> when executing this code I got in my index the field "includes" with >>> this >>> value : "? ? ?" : >>> --- >>> String content ="eaiou with circumflexes: êâîôû"; >>> SolrInputDocument doc = new SolrInputDocument(); >>> doc.addField( "id", "123", 1.0f ); >>> doc.addField( "includes", content, 1.0f ); >>> server.add( doc ); >>> --- >>> >>> but this code works fine : >>> >>> --- >>> String addContent = "" >>> +"123>> name="includes">eaiou with circumflexes:âîôû" >>> +""; >>> DirectXmlRequest up = new DirectXmlRequest( "/update", addContent ); >>> server.request( up ); >>> --- >>> >>> thanks for help >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/solrj-%3A-probleme-with-utf-8-content-tp22577377p22627715.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: stop word search
Yes, you do need to reindex after removing the stopword filter from the configuration. When you indexed the first time using the stopword filter, the words were NOT indexed, so they won't be found now that they're getting through the query analyzer. Best Erick On Fri, Mar 20, 2009 at 1:02 PM, revas wrote: > Hi Erik, > > I have now commented the query time stopword analyzer .I restarted the > server.But now when i search for a stop word ,i am getting results. > > We had earlier indexed the content with the stop word analyzer.I dont think > we need to reindex after commentting the query analyzer,right? > > This field is a text field with the defaul analyzer. > > Please let me know if i have missed something here. > > Regards > Sujatha > > > On 3/17/09, Erick Erickson wrote: > > > > Well, by definition, using an analyzer that removes stopwords > > *should* do this at query time. This assumes that you used > > an analyzer that removed stopwords at index and query time. > > The stopwords are not in the index. > > > > You can get the behavior you expect by using an analyzer at > > query time that does NOT remove stopwords, and one at > > indexing time that *does* remove stopwords. Gut I'm having a > > hard time imagining that this would result in a good user experience. > > > > I mean anytime that you had a stopword in the query where the > > stopword was required, no results would be returned. Which would > > be hard to explain to a user > > > > What is it you're trying to accomplish? > > > > Best > > Erick > > > > > > > > On Tue, Mar 17, 2009 at 7:40 AM, revas wrote: > > > > > Hi, > > > > > > I have a query like this > > > > > > content:the AND iuser_id:5 > > > > > > which means return all docs of user id 5 which have the word "the" in > > > content .Since 'the' is a stop word ,this query executes as just > user_id > > :5 > > > inspite of the "AND" clause ,Whereas the expected result here is since > > > there > > > is no result for "the " ,no results shloud be returned. > > > > > > Am i missing anythin here? > > > > > > Regards > > > > > >
Re: Stemming in Solr
: Can someone please let me know how to implement stemming in solr. I am : particularly looking of the changes, I might need to do in the config files : and also if I need to use some already supplied libraries/factories etc etc. i would start by searching the wiki and email archives for stemming... http://wiki.apache.org/solr/?action=fullsearch&context=180&value=stemming&fullsearch=Text -Hoss
Re: Special Characters search in solr
: Yes, I did and below is my debugQuery result. before you even look at the debug section, look at the params section in the responseHeader... : Colo� the raw value Solr is getting from your servlet container doesn't match what you think you are sending... : It is actually converting "Coloèr" to "Colo�" and hence not searching. It is ...i'm guessing that either your servlet container is missconfigured for dealing with UTF-8 characters, or your client code is doing something not quite right ... untill you get that value you expect to see coming back in that responseHeader, there's no point in fiddling with your schema. -Hoss
Re: Issue with Facet Query
: I am using this query only but I am getting the same results. : : : facet=true&facet.field=productPrice_product_str_s&fq=productPrice_product_str_s:[1%20TO%20100] ... : It still is not showing up the other values. Do I need to make any entry in : schema or solrConfig xml files. Do I need to convert the string into numeric : values etc etc. ... : >> It is only returning results, which are having values started with 2, 3, : >> 4 : >> or some other integer instead of only 1. It is not returning records in : >> which value is >10 and <100. your "fq" param is saying you only want docs matching values between 1 and 100, you seem to be using a string type, so it's not going to match anything starting with a character other then a "1" ... if it doens't match any docs with values like "23" then the facet counts for "23" are going to be 0 as well. reading between the lines, i think you missunderstood Shalin about 10 messages ago ... "fq" is for providing a *filter* query, it restricts the results of your entire query. facet.query is for faceting on an arbitrary query (which can be a range query) if you search for 'ipod' and you want to get back *all* the documents that match, but you also want to know how many of those have a price between $10 and $100 use a facet.query. if you search for 'ipod' and you want to get back *only* the documents that have a price between $10 and $100 use an fq. ...but either way: yes, convert to a numeric field type so that your ranges will actually work properly. -Hoss
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
Hi, I've cheked MySql conf with "mysql> SHOW VARIABLES LIKE 'character_set%'; " : all character_set are in UTF-8. I think that dataimporter get data in ISO. so the i just write a custom transformer to change the row's charset from iso to utf and now it work. --> Noble Paul : I use SOLR 1.4 Nighty 2009-03-18 build. i have to download the last one to apply your patch ? Noble Paul നോബിള് नोब्ळ् wrote: > > May be there is an issue with the recent changes with SOLR-973 > I have given a new patch on SOLR-973 > aerox ,is it possible to confirm if that is the problem > > > On Fri, Mar 20, 2009 at 6:52 PM, Grant Ingersoll > wrote: >> Usually, when I see characters like this, it means you aren't >> viewing/handling the UTF-8 correctly when bringing it into Java. I would >> first check that your DB or JDBC driver is getting the chars out right. >> It >> may even be the case that they did not go into the DB correctly in the >> first >> place. >> >> On Mar 20, 2009, at 4:36 AM, aerox7 wrote: >> >>> >>> ==> where are you seeing it as ""Solène" as opposed to the >>> correct way of solène? >>> >>> I have "Solène" in my Mysql DATA BASE ! so i don't know if this is >>> correct >>> or not ? i gess that "Solène" is solène in UTF-8 ?! >>> >>> I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp, so >>> when i try with solène everything is ok ! but when i try with Solène >>> (like >>> what i have in DB) analysis convert à in A delete ¨ so i get SolAne !!! >>> >>> I think that ISOLatin1AccentFilterFactory take only string with Charset >>> ISO-8859-1 . >>> >>> So any solution to transform my string to ISO-8859-1 before indexing >>> process. May be by creating transformer in DataImportHandler ? (Never >>> code >>> in java :( ) >>> >>> Thank you all. >>> >>> >>> Koji Sekiguchi-2 wrote: aerox7 wrote: > > Hi, > I have a mysql data base in UTF-8. I have a row with "Solène" > (solène). > I > want to transforme this to solene, so i use Solr > ISOLatin1AccentFilterFactory to perform this task but it dosn't work > ?!! > > i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to > utf-8 > so > normaly ISOLatin1AccentFilterFactory have to replace the accent > ... > > any ideas ? > > i use DataImportHandler. > If a mapping rule "è" to "e" is always true in your field, you can try to use MappingCharFilter instead of ISOLatin1AccentFilter. Add the following line to mapping-ISOLatin1Accent.txt: "è" => "e" and add the following fieldType: >>> positionIncrementGap="100" > >>> mapping="mapping-ISOLatin1Accent.txt"/> MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build. Koji >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22616220.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22633051.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Problem with UTF-8 and Solr ISOLatin1AccentFilterFactory
I guess you can mention a JdbcDataSource property characterEncoding="UTF8" and it should help On Sat, Mar 21, 2009 at 10:58 AM, aerox7 wrote: > > Hi, > I've cheked MySql conf with "mysql> SHOW VARIABLES LIKE 'character_set%'; " > : all character_set are in UTF-8. > > I think that dataimporter get data in ISO. so the i just write a custom > transformer to change the row's charset from iso to utf and now it work. > > --> Noble Paul : I use SOLR 1.4 Nighty 2009-03-18 build. i have to download > the last one to apply your patch ? > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> May be there is an issue with the recent changes with SOLR-973 >> I have given a new patch on SOLR-973 >> aerox ,is it possible to confirm if that is the problem >> >> >> On Fri, Mar 20, 2009 at 6:52 PM, Grant Ingersoll >> wrote: >>> Usually, when I see characters like this, it means you aren't >>> viewing/handling the UTF-8 correctly when bringing it into Java. I would >>> first check that your DB or JDBC driver is getting the chars out right. >>> It >>> may even be the case that they did not go into the DB correctly in the >>> first >>> place. >>> >>> On Mar 20, 2009, at 4:36 AM, aerox7 wrote: >>> ==> where are you seeing it as ""Solène" as opposed to the correct way of solène? I have "Solène" in my Mysql DATA BASE ! so i don't know if this is correct or not ? i gess that "Solène" is solène in UTF-8 ?! I'vz tryed analysis in http://localhost:8983/solr/admin/analysis.jsp, so when i try with solène everything is ok ! but when i try with Solène (like what i have in DB) analysis convert à in A delete ¨ so i get SolAne !!! I think that ISOLatin1AccentFilterFactory take only string with Charset ISO-8859-1 . So any solution to transform my string to ISO-8859-1 before indexing process. May be by creating transformer in DataImportHandler ? (Never code in java :( ) Thank you all. Koji Sekiguchi-2 wrote: > > aerox7 wrote: >> >> Hi, >> I have a mysql data base in UTF-8. I have a row with "Solène" >> (solène). >> I >> want to transforme this to solene, so i use Solr >> ISOLatin1AccentFilterFactory to perform this task but it dosn't work >> ?!! >> >> i gess that "Solène" is "solène" in UTF-8 ?! i also set tomcat to >> utf-8 >> so >> normaly ISOLatin1AccentFilterFactory have to replace the accent >> ... >> >> any ideas ? >> >> i use DataImportHandler. >> > > If a mapping rule "è" to "e" is always true in your field, you can try > to use MappingCharFilter > instead of ISOLatin1AccentFilter. Add the following line to > mapping-ISOLatin1Accent.txt: > > "è" => "e" > > and add the following fieldType: > > positionIncrementGap="100" > > > mapping="mapping-ISOLatin1Accent.txt"/> > > > > > MappingCharFilter and mapping-ISOLatin1Accent.txt are in nightly build. > > Koji > > > > -- View this message in context: http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22616220.html Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- > View this message in context: > http://www.nabble.com/Problem-with-UTF-8-and-Solr-ISOLatin1AccentFilterFactory-tp22607642p22633051.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul
