Re: solrj and CLOSE_WAIT's

[Ian Holsman]

Ryan McKinley wrote:

not sure if it is something we can do better or part of HttpClient...

From:
http://www.nabble.com/CLOSE_WAIT-td19959428.html

it seems to suggest you may want to call:
con.closeIdleConnections(0L);

But if you are creating a new MultiThreadedHttpConnectionManager for 
each request, is seems odd you would have to explicitly close the 
connection for each request.


What happens if you try using a SimpleHttpConnectionManager rather 
then a MultiThreadedHttpConnectionManager?  You can explicitly pass in:
 

I was thinking the same thing when i saw the other constructor.

I've modified the code to call the 'simple' version and will let it run
for an hour or three to make sure it works and doesn't exhibit the
behavior, so far it looks good and there are no CLOSE_WAITs (or
FIN_WAIT2's) showing up for longer than a couple of seconds. (according
to netstat -tn)

I'd petition we go back to the 'stupid' version by default that just
does what it is supposed to do, and leave the other one for 'experts'. I
can't even see how to tell the multi-threaded version to close itself
nicely ;(




to:
public CommonsHttpSolrServer(URL baseURL, HttpClient client, 
ResponseParser parser, boolean useMultiPartPost) {


if that fixes things, it is a bit disturbing, but something we should 
look into.


ryan

solr1.3 dismax qf/bf boolean

[sunnyfr]

Hi,

I try to find an equilibrium between all my boost, but I'm a bit confused
between qf and bf
so I put :

&qf=title^10

then I try to bring back in first or boost a lot a boolean field when it's
true :
&bf=isbest^10 or 100 or more doesn't bring back in head of list the best one
...???

same for another boolean that I need to boost.

I have to boost as well the recent one so I did :
&bf=recip(rord(created),1,1000,1000)^20

I would like to boost as well the video which are more view then the others
so I tried :
&bf=pow(nb_views,10)

I read the wiki, but I'm still a bit confused.
And have to say that result are not good and even sometime bring me back in
head of list a title which even didn't contain words that I look for, like
for q=tom jerry (my schema contain OR as default maybe I should put AND and
manage properly mm property) and the first video that I have doesn't
contrain this words.

?qt=dismax&fl=*&bf=isbest^10&qf=title^30&qf=title_en^20&bf=recip(rord(created),1,1000,1000)^20&qf=description^1&bf=pow(stat_views,10)&debugQuery=true&q=tom
jerry

I checked my debug but I just don't get why really it doesn't bring me back
exactly the same words in first .. I know I boost other fields but it should
bring back up them in first.?
Sorry I'm confusing,

Thanks a lot for your help
-- 
View this message in context: 
http://www.nabble.com/solr1.3-dismax-qf-bf-boolean-tp20498185p20498185.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solrj queries

[Erik Hatcher]

AND must be entirely capitalized to set clauses on both sides as  
_required_.


Erik


On Nov 14, 2008, at 12:44 AM, Raghunandan Rao wrote:


Yes.
But is that how we do in Solrj by setting SolrQuery("text:Right And
title:go").

Thanks a lot.

-Original Message-
From: Ryan McKinley [mailto:[EMAIL PROTECTED]
Sent: Friday, November 14, 2008 10:55 AM
To: solr-user@lucene.apache.org
Subject: Re: Solrj queries


On Nov 14, 2008, at 12:14 AM, Raghunandan Rao wrote:


Hi,

Can anybody let me know how to do logical searches using Solrj. I  
have

indexed the data and now I want to do search based on fields using
Solrj
as I have to retrieve in the form of beans.



For eg.

I have two fields, First name and last name

I have to retrieve only names which match both of the above fields.



q.setQuery( "first:value1 AND last:value2" );

It seem you are asking for:
http://lucene.apache.org/java/2_4_0/queryparsersyntax.html

and not really solrj related...

ryan

Re: solr1.3 dismax qf/bf boolean

[Erik Hatcher]

On Nov 14, 2008, at 5:44 AM, sunnyfr wrote:

I read the wiki, but I'm still a bit confused.
And have to say that result are not good and even sometime bring me  
back in
head of list a title which even didn't contain words that I look  
for, like
for q=tom jerry (my schema contain OR as default maybe I should put  
AND and

manage properly mm property) and the first video that I have doesn't
contrain this words.

? 
qt 
= 
dismax 
&fl=*&bf=isbest^10&qf=title^30&qf=title_en^20&bf=recip(rord(created), 
1,1000,1000)^20&qf=description^1&bf=pow(stat_views, 
10)&debugQuery=true&q=tom

jerry

I checked my debug but I just don't get why really it doesn't bring  
me back
exactly the same words in first .. I know I boost other fields but  
it should

bring back up them in first.?
Sorry I'm confusing,


qf must be single valued append all of those together into a  
single qf with a space separator (URL encoded):


   &qf=title^30+title_en^20+description

Same with bf, it is single-valued so you have to combine everything  
into a single parameter.


Erik

troubles with delta import

[Marc Sturlese]

Hey there, I am using dataimport with full-import successfully but there's no
way do make it work with delta-import. Aparently solr doesn't show any error
but it does not do what it is supose to.
I thing the problme is with dataimport.properties because it is never
updated. I have it placed in the same folder as solrconfig.xml and
schema.xml and the writing permissions are set propertly. What makes me
doubt is that couldn't find anywhere to tell solr the path of this file.
Don't know if solr is suposed to find it automatically.

My data-config.xml looks like this:









*I have in the rows of the table a timestamp field called dt_last_modified

Other thing that can't exactly understant is why i have to put the query and
delta-query... why just with deltaquery (with more fields in the select) is
not enough?

After the ejecution everything seems to go ok (even with the debug and
verbose mode) but no docs have changed and dataimport.properties is not
updated...

Any suggestion? Have done many tests but no way...

-- 
View this message in context: 
http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: troubles with delta import

[Shalin Shekhar Mangar]

Hi Marc,

Did you do a full-import first? If not, no value for last import time is
written and the delta query may fail. We should fix this to use a sane
default so that people do not need to full import first.

You need to put both because we support both full and delta, both of which
need different kinds of queries and we cannot decide what you are going to
use.

On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese <[EMAIL PROTECTED]>wrote:

>
> Hey there, I am using dataimport with full-import successfully but there's
> no
> way do make it work with delta-import. Aparently solr doesn't show any
> error
> but it does not do what it is supose to.
> I thing the problme is with dataimport.properties because it is never
> updated. I have it placed in the same folder as solrconfig.xml and
> schema.xml and the writing permissions are set propertly. What makes me
> doubt is that couldn't find anywhere to tell solr the path of this file.
> Don't know if solr is suposed to find it automatically.
>
> My data-config.xml looks like this:
> 
> url="jdbc:mysql://path_db" user="user" password="pwd"/>
>
>
>
>
>
>
> 
> *I have in the rows of the table a timestamp field called dt_last_modified
>
> Other thing that can't exactly understant is why i have to put the query
> and
> delta-query... why just with deltaquery (with more fields in the select) is
> not enough?
>
> After the ejecution everything seems to go ok (even with the debug and
> verbose mode) but no docs have changed and dataimport.properties is not
> updated...
>
> Any suggestion? Have done many tests but no way...
>
> --
> View this message in context:
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: DataImportHandler, custom properties

[DODMax]

> http://wiki.apache.org/solr/DataImportHandler#head-071ff018f44ecbdb1cf55afc4c2a857f44ea1ea4
> 
> 
> 
> ...
> 
> 

The problem is that "namedesc" must be an existing column and I have no idea
of column name in advance.

To be more clear, what I want is to be able to retrive information which are
not in database from the index. For example to store source database in a
kind of custom property and have it back with results.  
-- 
View this message in context: 
http://www.nabble.com/DataImportHandler%2C-custom-properties-tp20482190p20498600.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: troubles with delta import

[Noble Paul നോബിള്‍ नोब्ळ्]

On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese <[EMAIL PROTECTED]> wrote:
>
> Hey there, I am using dataimport with full-import successfully but there's no
> way do make it work with delta-import. Aparently solr doesn't show any error
> but it does not do what it is supose to.
> I thing the problme is with dataimport.properties because it is never
> updated. I have it placed in the same folder as solrconfig.xml and
> schema.xml and the writing permissions are set propertly. What makes me
> doubt is that couldn't find anywhere to tell solr the path of this file.
> Don't know if solr is suposed to find it automatically.
>
> My data-config.xml looks like this:
> 
> url="jdbc:mysql://path_db" user="user" password="pwd"/>
>
>
>
>
>
>
> 
> *I have in the rows of the table a timestamp field called dt_last_modified
>
> Other thing that can't exactly understant is why i have to put the query and
> delta-query... why just with deltaquery (with more fields in the select) is
> not enough?
deltaQuery is used to identify the changed rows.
after the changed rows are identified , it uses the query, or
deltaImportQuery to do the actual import
>
> After the ejecution everything seems to go ok (even with the debug and
> verbose mode) but no docs have changed and dataimport.properties is not
> updated...
>
> Any suggestion? Have done many tests but no way...
>
> --
> View this message in context: 
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



-- 
--Noble Paul

Re: troubles with delta import

[Marc Sturlese]

Hey Shalin,
I have tryied 2 methods:
1-First doing a full-import and after a delta-import
2.-Start directly with the delta-import.

In any of both cases the date of the import.properties file is updated. I
have it placed in the same folder as schema.xml,data-config.xml and
solrconfig.xml (is where I think is must be placed acording to what I
understand in the wiki). Is it correct?
Is the only thing that I think maybe I am missing...
Thanks in advance


Shalin Shekhar Mangar wrote:
> 
> Hi Marc,
> 
> Did you do a full-import first? If not, no value for last import time is
> written and the delta query may fail. We should fix this to use a sane
> default so that people do not need to full import first.
> 
> You need to put both because we support both full and delta, both of which
> need different kinds of queries and we cannot decide what you are going to
> use.
> 
> On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese
> <[EMAIL PROTECTED]>wrote:
> 
>>
>> Hey there, I am using dataimport with full-import successfully but
>> there's
>> no
>> way do make it work with delta-import. Aparently solr doesn't show any
>> error
>> but it does not do what it is supose to.
>> I thing the problme is with dataimport.properties because it is never
>> updated. I have it placed in the same folder as solrconfig.xml and
>> schema.xml and the writing permissions are set propertly. What makes me
>> doubt is that couldn't find anywhere to tell solr the path of this file.
>> Don't know if solr is suposed to find it automatically.
>>
>> My data-config.xml looks like this:
>> 
>>> url="jdbc:mysql://path_db" user="user" password="pwd"/>
>>
>>
>>
>>
>>
>>
>> 
>> *I have in the rows of the table a timestamp field called
>> dt_last_modified
>>
>> Other thing that can't exactly understant is why i have to put the query
>> and
>> delta-query... why just with deltaquery (with more fields in the select)
>> is
>> not enough?
>>
>> After the ejecution everything seems to go ok (even with the debug and
>> verbose mode) but no docs have changed and dataimport.properties is not
>> updated...
>>
>> Any suggestion? Have done many tests but no way...
>>
>> --
>> View this message in context:
>> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/troubles-with-delta-import-tp20498449p20500510.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Multi field queries

[sunnyfr]

Thanks Hoss,
and about boosting boolean value or even number of view, does it work this
way :

boolean value :
&qf=best_one^10

Number of view:
pow(nb_views,0.01)^20

Otherwise about searching two words and have in first title who have exactly
this two words in first it's just about mm parameter?

Thanks a lot Hoss


hossman wrote:
> 
> 
> : Documents in my solr index has three fields, name, content and summary.
> : Suppose the user query be, "java sky democratic". I want the resulting
> : documents to have all the terms in the query ( "java sky democratic") in
> : either name, content or the summary (for example i.e., java and sky is
> in the
> : content and democratic is in the summary).
> 
> take a look at the "dismax" request handler.  it is designed explicitly 
> for this purpose.  
> 
>   http://wiki.apache.org/solr/DisMaxRequestHandler
> 
> (NOTE: if you want all the input words to be required, set mm=100%)
> 
> 
> -Hoss
> 
> 
> 

-- 
View this message in context: 
http://www.nabble.com/Multi-field-queries-tp15499246p20500795.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: troubles with delta import

[Shalin Shekhar Mangar]

Mark, even if the dataimport.properties is not present, DIH will create it.

Are you seeing any exceptions in the log or in the debug mode?

On Fri, Nov 14, 2008 at 7:03 PM, Marc Sturlese <[EMAIL PROTECTED]>wrote:

>
> Hey Shalin,
> I have tryied 2 methods:
> 1-First doing a full-import and after a delta-import
> 2.-Start directly with the delta-import.
>
> In any of both cases the date of the import.properties file is updated. I
> have it placed in the same folder as schema.xml,data-config.xml and
> solrconfig.xml (is where I think is must be placed acording to what I
> understand in the wiki). Is it correct?
> Is the only thing that I think maybe I am missing...
> Thanks in advance
>
>
> Shalin Shekhar Mangar wrote:
> >
> > Hi Marc,
> >
> > Did you do a full-import first? If not, no value for last import time is
> > written and the delta query may fail. We should fix this to use a sane
> > default so that people do not need to full import first.
> >
> > You need to put both because we support both full and delta, both of
> which
> > need different kinds of queries and we cannot decide what you are going
> to
> > use.
> >
> > On Fri, Nov 14, 2008 at 4:35 PM, Marc Sturlese
> > <[EMAIL PROTECTED]>wrote:
> >
> >>
> >> Hey there, I am using dataimport with full-import successfully but
> >> there's
> >> no
> >> way do make it work with delta-import. Aparently solr doesn't show any
> >> error
> >> but it does not do what it is supose to.
> >> I thing the problme is with dataimport.properties because it is never
> >> updated. I have it placed in the same folder as solrconfig.xml and
> >> schema.xml and the writing permissions are set propertly. What makes me
> >> doubt is that couldn't find anywhere to tell solr the path of this file.
> >> Don't know if solr is suposed to find it automatically.
> >>
> >> My data-config.xml looks like this:
> >> 
> >> >> url="jdbc:mysql://path_db" user="user" password="pwd"/>
> >>
> >>
> >>
> >>
> >>
> >>
> >> 
> >> *I have in the rows of the table a timestamp field called
> >> dt_last_modified
> >>
> >> Other thing that can't exactly understant is why i have to put the query
> >> and
> >> delta-query... why just with deltaquery (with more fields in the select)
> >> is
> >> not enough?
> >>
> >> After the ejecution everything seems to go ok (even with the debug and
> >> verbose mode) but no docs have changed and dataimport.properties is not
> >> updated...
> >>
> >> Any suggestion? Have done many tests but no way...
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20498449.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20500510.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: troubles with delta import

[Marc Sturlese]

Hey,
That's the weird thing... in the log everything seems to work fine:

Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImportHandler
processConfiguration
INFO: Processing configuration from solrconfig.xml:
{config=/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml}
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
INFO: Data Configuration loaded successfully
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
INFO: Starting Delta Import
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity homes_tbl_ads with URL:
jdbc:mysql://localhost/path_db
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 11
Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DocBuilder
execute
INFO: Time taken = 0:0:0.47
Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr_web path=/dataimport
params={verbose=true&command=delta-import&debug=on} status=0 QTime=130 
Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr_web path=/dataimport params={command=show-config}
status=0 QTime=0 

I am calling the dataimport this way:
http://...dataimport?command=full-import&debug=on&verbose=true
http://...dataimport?command=delta-import&debug=on&verbose=true

In delta-import I am getting this aoutput with the verbose debug:

...
delta-import
debug

...
lst name="statusMessages">
1
10
0
2008-11-14 15:12:46
0:0:0.47


It also shows the changes in the rows in the output of the verbose debug but
nothing change in the index when I check it with Luke.
I keep thinking that something is wrong coz the import.properties it is not
being created... but can't find why :(

solrconfig.xml:
 


/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml

  

data-config.xml:











Thanks a lot



-- 
View this message in context: 
http://www.nabble.com/troubles-with-delta-import-tp20498449p20501450.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: troubles with delta import

[Shalin Shekhar Mangar]

Ok I found the problem.

In debug mode, DataImportHandler does not commit documents since it is meant
for debugging only. If you want to do a commit, add commit=true as a request
parameter.

On Fri, Nov 14, 2008 at 7:56 PM, Marc Sturlese <[EMAIL PROTECTED]>wrote:

>
> Hey,
> That's the weird thing... in the log everything seems to work fine:
>
> Nov 14, 2008 3:12:46 PM
> org.apache.solr.handler.dataimport.DataImportHandler
> processConfiguration
> INFO: Processing configuration from solrconfig.xml:
>
> {config=/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml}
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
> loadDataConfig
> INFO: Data Configuration loaded successfully
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> INFO: Starting Delta Import
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
> call
> INFO: Creating a connection for entity homes_tbl_ads with URL:
> jdbc:mysql://localhost/path_db
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
> call
> INFO: Time taken for getConnection(): 11
> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DocBuilder
> execute
> INFO: Time taken = 0:0:0.47
> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr_web path=/dataimport
> params={verbose=true&command=delta-import&debug=on} status=0 QTime=130
> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr_web path=/dataimport params={command=show-config}
> status=0 QTime=0
>
> I am calling the dataimport this way:
> http://...dataimport?command=full-import&debug=on&verbose=true
> http://...dataimport?command=delta-import&debug=on&verbose=true
>
> In delta-import I am getting this aoutput with the verbose debug:
>
> ...
> delta-import
> debug
> 
> ...
> lst name="statusMessages">
> 1
> 10
> 0
> 2008-11-14 15:12:46
> 0:0:0.47
> 
>
> It also shows the changes in the rows in the output of the verbose debug
> but
> nothing change in the index when I check it with Luke.
> I keep thinking that something is wrong coz the import.properties it is not
> being created... but can't find why :(
>
> solrconfig.xml:
>   class="org.apache.solr.handler.dataimport.DataImportHandler"
> default="false">
>
>
>
> name="config">/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml
>
>  
>
> data-config.xml:
>
> 
> url="jdbc:mysql://localhost/trovit_es" user="root" password=""/>
>
>
>
>
>
>
> 
>
> Thanks a lot
>
>
>
> --
> View this message in context:
> http://www.nabble.com/troubles-with-delta-import-tp20498449p20501450.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: troubles with delta import

[Marc Sturlese]

Hey Shalin!
Now at least I am getting some errors in the log file :D... Hope now I will
be able to find the problem.
Thanks for everything!


Shalin Shekhar Mangar wrote:
> 
> Ok I found the problem.
> 
> In debug mode, DataImportHandler does not commit documents since it is
> meant
> for debugging only. If you want to do a commit, add commit=true as a
> request
> parameter.
> 
> On Fri, Nov 14, 2008 at 7:56 PM, Marc Sturlese
> <[EMAIL PROTECTED]>wrote:
> 
>>
>> Hey,
>> That's the weird thing... in the log everything seems to work fine:
>>
>> Nov 14, 2008 3:12:46 PM
>> org.apache.solr.handler.dataimport.DataImportHandler
>> processConfiguration
>> INFO: Processing configuration from solrconfig.xml:
>>
>> {config=/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml}
>> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
>> loadDataConfig
>> INFO: Data Configuration loaded successfully
>> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>> INFO: Starting Delta Import
>> Nov 14, 2008 3:12:46 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Creating a connection for entity homes_tbl_ads with URL:
>> jdbc:mysql://localhost/path_db
>> Nov 14, 2008 3:12:46 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Time taken for getConnection(): 11
>> Nov 14, 2008 3:12:46 PM org.apache.solr.handler.dataimport.DocBuilder
>> execute
>> INFO: Time taken = 0:0:0.47
>> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr_web path=/dataimport
>> params={verbose=true&command=delta-import&debug=on} status=0 QTime=130
>> Nov 14, 2008 3:12:46 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr_web path=/dataimport params={command=show-config}
>> status=0 QTime=0
>>
>> I am calling the dataimport this way:
>> http://...dataimport?command=full-import&debug=on&verbose=true
>> http://...dataimport?command=delta-import&debug=on&verbose=true
>>
>> In delta-import I am getting this aoutput with the verbose debug:
>>
>> ...
>> delta-import
>> debug
>> 
>> ...
>> lst name="statusMessages">
>> 1
>> 10
>> 0
>> 2008-11-14 15:12:46
>> 0:0:0.47
>> 
>>
>> It also shows the changes in the rows in the output of the verbose debug
>> but
>> nothing change in the index when I check it with Luke.
>> I keep thinking that something is wrong coz the import.properties it is
>> not
>> being created... but can't find why :(
>>
>> solrconfig.xml:
>>  > class="org.apache.solr.handler.dataimport.DataImportHandler"
>> default="false">
>>
>>
>>>
>> name="config">/opt/netbeans-5.5.1/enterprise3/apache-tomcat-5.5.17/bin/solr/conf/data-config.xml
>>
>>  
>>
>> data-config.xml:
>>
>> 
>>> url="jdbc:mysql://localhost/trovit_es" user="root" password=""/>
>>
>>
>>
>>
>>
>>
>> 
>>
>> Thanks a lot
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/troubles-with-delta-import-tp20498449p20501450.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 
> 

-- 
View this message in context: 
http://www.nabble.com/troubles-with-delta-import-tp20498449p20502269.html
Sent from the Solr - User mailing list archive at Nabble.com.

questions about Solr connection methods

[Matt Mitchell]

I'm implementing connection adapters in ruby/jruby and wondering how all of
the different solr "connection" classes relate.

Is the only difference between EmbeddedSolrServer and DirectSolrConnection,
that EmbeddedSolrServer provides some higher level methods for adding,
deleting etc.? Or is there something else happening underneath the covers?
If the higher level methods in EmbeddedSolrServer aren't really of use to
me, would it be better to use the simpler DirectSolrConnection?

Does DirectSolrConnection support multicore?

Thanks,
Matt

RE: STATS functions ....

[souravm]

Thanks Erik.

Regards,
Sourav

-Original Message-
From: Erik Hatcher [mailto:[EMAIL PROTECTED] 
Sent: Thursday, November 13, 2008 6:29 PM
To: solr-user@lucene.apache.org
Subject: Re: STATS functions 

StatsComponent is in trunk now, committed after the 1.3 release.  
 - note the "Status" and the "Subversion Commits" stuff there (just  
trying to help folks help themselves track this stuff :).

I suspect the code could be pulled from that issue and applied to 1.3  
release, even JAR'ing it up separately, and tossing it in as a  
"plugin".  We probably should be creating all these sorts of goodies  
and independent modules of code that aren't "core", but that gets  
fuzzy to say what's core and what isn't too.

Erik


On Nov 13, 2008, at 8:26 PM, souravm wrote:

>
> Hi,
>
> As I understand the STATS functions (Min, Max, Average, Standard  
> Deviation etc.) would be available in Solr 1.4.
>
> Just wondering if they are already there in the latest trunk. Else  
> can anyone suggest any other tool which can be used with Solr 1.3 to  
> achieve this requirement ?
>
> Regards,
> Sourav
>
>  CAUTION - Disclaimer *
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION  
> intended solely
> for the use of the addressee(s). If you are not the intended  
> recipient, please
> notify the sender by e-mail and delete the original message.  
> Further, you are not
> to copy, disclose, or distribute this e-mail or its contents to any  
> other person and
> any such actions are unlawful. This e-mail may contain viruses.  
> Infosys has taken
> every reasonable precaution to minimize this risk, but is not liable  
> for any damage
> you may sustain as a result of any virus in this e-mail. You should  
> carry out your
> own virus checks before opening the e-mail or attachment. Infosys  
> reserves the
> right to monitor and review the content of all messages sent to or  
> from this e-mail
> address. Messages sent to or from this e-mail address may be stored  
> on the
> Infosys e-mail system.
> ***INFOSYS End of Disclaimer INFOSYS***

Newbe! Trying to run solr-1.3.0 under tomcat. Please help

[Fergus McMenemie]

Hello all, 

Further to various messages. I just cannot get solr 1.3 to launch
under OS X with tomcat. Solr 1.2 works fine with tomcat and I am
OK with 1.3 under jetty.


I have tried tomcat-5.5.20 and 5.2.27. I have tried solr
1.3.0 plus the nightly build. I have tried under OS X 10.5
and 10.4 (leopard and tiger) all fail as follows. I also 
tried cutting and pasting the instructions from:-
http://wiki.apache.org/solr/SolrTomcat

Here is what I see on the browser. When I try to access 
http://localhost:8080/solr


At 14:26 + 14/11/08, Fergus McMenemie wrote:
>HTTP Status 500 - Severe errors in solr configuration. 
> Check your log files for more detailed information on what may be wrong.
> If you want solr to continue after configuration errors, change: 
> false in null 
> - 
>java.lang.RuntimeException: java.lang.NoSuchMethodError: 
>org.apache.lucene.index.IndexReader.open(Lorg/apache/lucene/store/Directory;Z)Lorg/apache/lucene/index/IndexReader;
> at org at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1065)
> at org at org.apache.solr.core.SolrCore.(SolrCore.java:553)
> at org at 
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:120)
> at org at 
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
> at org at 
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221)
> at org at 
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302)
> at org at 
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78)
> at org at 
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635)
> at org at 
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4222)
> at org at 
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760)
> at org at 
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:740)
> at org at 
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544)
> at org at 
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:626)
> at org at 
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553)
> at org at 
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:488)
> at org at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1150)
> at org at 
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
> at org at 
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
> at org at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022)
> at org at org.apache.catalina.core.StandardHost.start(StandardHost.java:736)
> at org at 
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014)
> at org at 
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> at org at 
> org.apache.catalina.core.StandardService.start(StandardService.java:448)
> at org at 
> org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
> at org at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:585)
> at org at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
> at org at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
>Caused by: java.lang.NoSuchMethodError: 
>org.apache.lucene.index.IndexReader.open(Lorg/apache/lucene/store/Directory;Z)Lorg/apache/lucene/index/IndexReader;
> at org at 
> org.apache.solr.search.SolrIndexSearcher.(SolrIndexSearcher.java:109)
> at org at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1055) ... 
> 30 more 


Here is a dump from tomcat/logs/catalina.out. It suggests there
is something wrong with my solr/home property, however you can
see that earlier on it seemed ok with this property.

At 14:26 + 14/11/08, Fergus McMenemie wrote:
>Nov 14, 2008 4:55:33 AM org.apache.catalina.core.AprLifecycleListener 
>lifecycleEvent INFO:
> The Apache Tomcat Native library which allows optimal performance in 
> production environments was not found on the java.library.path: 
> /usr/local/bin:.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
>Nov 14, 2008 4:55:33 AM org.apache.coyote.http11.Http11BaseProtocol init INFO: 
>Initializing Coyote HTTP/1.1 on http-8080
>Nov 14, 2008 4:55:33 AM org.apache.catalina.startup.Catalina load INFO: 
>Initialization processed in 575 ms
>Nov 14, 2008 4:55:33 AM org.apache.catalina.core.StandardService start INFO: 
>Starting service Catalina
>Nov 14, 2008 4:55:33 AM org.apache.catalina.core.Standar

Re: Newbe! Trying to run solr-1.3.0 under tomcat. Please help

[Erik Hatcher]

Fergus,

I just downloaded Tomcat 5.5.27, put a solr.xml file in conf/Catalina/ 
localhost with the following:


  debug="0" crossContext="true" >
 

  

And Solr started up just fine and it's admin, etc worked as expected.

Oh, and on Mac OS X (of course!), version 10.5.5.

Erik

On Nov 14, 2008, at 12:17 PM, Fergus McMenemie wrote:


Hello all,

Further to various messages. I just cannot get solr 1.3 to launch
under OS X with tomcat. Solr 1.2 works fine with tomcat and I am
OK with 1.3 under jetty.


I have tried tomcat-5.5.20 and 5.2.27. I have tried solr
1.3.0 plus the nightly build. I have tried under OS X 10.5
and 10.4 (leopard and tiger) all fail as follows. I also
tried cutting and pasting the instructions from:-
http://wiki.apache.org/solr/SolrTomcat

Here is what I see on the browser. When I try to access
http://localhost:8080/solr


At 14:26 + 14/11/08, Fergus McMenemie wrote:

HTTP Status 500 - Severe errors in solr configuration.
Check your log files for more detailed information on what may be  
wrong.
If you want solr to continue after configuration errors, change:  
false in null

-
java.lang.RuntimeException: java.lang.NoSuchMethodError:  
org.apache.lucene.index.IndexReader.open(Lorg/apache/lucene/store/ 
Directory;Z)Lorg/apache/lucene/index/IndexReader;
at org at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java: 
1065)

at org at org.apache.solr.core.SolrCore.(SolrCore.java:553)
at org at org.apache.solr.core.CoreContainer 
$Initializer.initialize(CoreContainer.java:120)
at org at  
org 
.apache 
.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at org at  
org 
.apache 
.catalina 
.core 
.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221)
at org at  
org 
.apache 
.catalina 
.core 
.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java: 
302)
at org at  
org 
.apache 
.catalina 
.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78)
at org at  
org 
.apache 
.catalina.core.StandardContext.filterStart(StandardContext.java:3635)
at org at  
org.apache.catalina.core.StandardContext.start(StandardContext.java: 
4222)
at org at  
org 
.apache 
.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760)
at org at  
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java: 
740)
at org at  
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544)
at org at  
org 
.apache 
.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:626)
at org at  
org 
.apache 
.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553)
at org at  
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java: 
488)
at org at  
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1150)
at org at  
org 
.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java: 
311)
at org at  
org 
.apache 
.catalina 
.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
at org at  
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022)
at org at  
org.apache.catalina.core.StandardHost.start(StandardHost.java:736)
at org at  
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014)
at org at  
org.apache.catalina.core.StandardEngine.start(StandardEngine.java: 
443)
at org at  
org.apache.catalina.core.StandardService.start(StandardService.java: 
448)
at org at  
org.apache.catalina.core.StandardServer.start(StandardServer.java: 
700)
at org at org.apache.catalina.startup.Catalina.start(Catalina.java: 
552)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at  
sun 
.reflect 
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at  
sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25) at  
java.lang.reflect.Method.invoke(Method.java:585)
at org at  
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java: 
433)
Caused by: java.lang.NoSuchMethodError:  
org.apache.lucene.index.IndexReader.open(Lorg/apache/lucene/store/ 
Directory;Z)Lorg/apache/lucene/index/IndexReader;
at org at  
org 
.apache.solr.search.SolrIndexSearcher.(SolrIndexSearcher.java: 
109)
at org at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java: 
1055) ... 30 more



Here is a dump from tomcat/logs/catalina.out. It suggests there
is something wrong with my solr/home property, however you can
see that earlier on it seemed ok with this property.

At 14:26 + 14/11/08, Fergus McMenemie wrote:
Nov 14, 2008 4:55:33 AM  
org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO:
The Apache Tomcat Native library which allows optimal performance  
in production environments was not found on the java.library.path: / 
usr/local/bin:.:/Library/Java/Extensions:/System/Library/Java/ 
Extensions:/usr/lib/java
Nov 14, 2008 4:55:33 AM org.apache.coyote.http11.Htt

Re: Newbe! Trying to run solr-1.3.0 under tomcat. Please help

[Erik Hatcher]

To be fair, my first message was about Solr trunk + Tomcat 5.5.27, but  
I just tried it by pointing to a Solr 1.3.0 official release and it  
worked fine as well.


Erik

On Nov 14, 2008, at 12:30 PM, Erik Hatcher wrote:


Fergus,

I just downloaded Tomcat 5.5.27, put a solr.xml file in conf/ 
Catalina/localhost with the following:


 debug="0" crossContext="true" >


 

And Solr started up just fine and it's admin, etc worked as expected.

Oh, and on Mac OS X (of course!), version 10.5.5.

Erik

On Nov 14, 2008, at 12:17 PM, Fergus McMenemie wrote:


Hello all,

Further to various messages. I just cannot get solr 1.3 to launch
under OS X with tomcat. Solr 1.2 works fine with tomcat and I am
OK with 1.3 under jetty.


I have tried tomcat-5.5.20 and 5.2.27. I have tried solr
1.3.0 plus the nightly build. I have tried under OS X 10.5
and 10.4 (leopard and tiger) all fail as follows. I also
tried cutting and pasting the instructions from:-
http://wiki.apache.org/solr/SolrTomcat

Here is what I see on the browser. When I try to access
http://localhost:8080/solr


At 14:26 + 14/11/08, Fergus McMenemie wrote:

HTTP Status 500 - Severe errors in solr configuration.
Check your log files for more detailed information on what may be  
wrong.
If you want solr to continue after configuration errors, change:  
false in null

-
java.lang.RuntimeException: java.lang.NoSuchMethodError:  
org.apache.lucene.index.IndexReader.open(Lorg/apache/lucene/store/ 
Directory;Z)Lorg/apache/lucene/index/IndexReader;
at org at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java: 
1065)

at org at org.apache.solr.core.SolrCore.(SolrCore.java:553)
at org at org.apache.solr.core.CoreContainer 
$Initializer.initialize(CoreContainer.java:120)
at org at  
org 
.apache 
.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at org at  
org 
.apache 
.catalina 
.core 
.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221)
at org at  
org 
.apache 
.catalina 
.core 
.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java: 
302)
at org at  
org 
.apache 
.catalina 
.core.ApplicationFilterConfig.(ApplicationFilterConfig.java: 
78)
at org at  
org 
.apache 
.catalina.core.StandardContext.filterStart(StandardContext.java: 
3635)
at org at  
org 
.apache.catalina.core.StandardContext.start(StandardContext.java: 
4222)
at org at  
org 
.apache 
.catalina.core.ContainerBase.addChildInternal(ContainerBase.java: 
760)
at org at  
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java: 
740)
at org at  
org.apache.catalina.core.StandardHost.addChild(StandardHost.java: 
544)
at org at  
org 
.apache 
.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:626)
at org at  
org 
.apache 
.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:553)
at org at  
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java: 
488)
at org at  
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1150)
at org at  
org 
.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java: 
311)
at org at  
org 
.apache 
.catalina 
.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
at org at  
org.apache.catalina.core.ContainerBase.start(ContainerBase.java: 
1022)
at org at  
org.apache.catalina.core.StandardHost.start(StandardHost.java:736)
at org at  
org.apache.catalina.core.ContainerBase.start(ContainerBase.java: 
1014)
at org at  
org.apache.catalina.core.StandardEngine.start(StandardEngine.java: 
443)
at org at  
org 
.apache.catalina.core.StandardService.start(StandardService.java: 
448)
at org at  
org.apache.catalina.core.StandardServer.start(StandardServer.java: 
700)
at org at org.apache.catalina.startup.Catalina.start(Catalina.java: 
552)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at  
sun 
.reflect 
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at  
sun 
.reflect 
.DelegatingMethodAccessorImpl 
.invoke(DelegatingMethodAccessorImpl.java:25) at  
java.lang.reflect.Method.invoke(Method.java:585)
at org at  
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org at  
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
Caused by: java.lang.NoSuchMethodError:  
org.apache.lucene.index.IndexReader.open(Lorg/apache/lucene/store/ 
Directory;Z)Lorg/apache/lucene/index/IndexReader;
at org at  
org 
.apache 
.solr.search.SolrIndexSearcher.(SolrIndexSearcher.java:109)
at org at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java: 
1055) ... 30 more



Here is a dump from tomcat/logs/catalina.out. It suggests there
is something wrong with my solr/home property, however you can
see that earlier on it seemed ok with this property.

At 14:26 + 14/11/08, Fergus McMenemie wrote:
Nov 14, 2008 4:55:33 AM  
org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO:
The Apache Tomcat Native library wh

Fatal exception in solr 1.3+ replication

[William Pierce]

Folks:

I am using the nightly build of 1.3 as of Oct 23 so as to use the replication 
handler.   I am running on windows 2003 server with tomcat 6.0.14.   Everything 
was running fine until I noticed that certain updated records were not showing 
up on the slave.  Further investigation showed me that the failures have indeed 
been occurring since early this morning with a fatal exceptionhere is a 
segment of the tomcat log:
  INFO: Total time taken for download : 0 secs
  Nov 14, 2008 5:34:24 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
  INFO: Conf files are not downloaded or are in sync
  Nov 14, 2008 5:34:24 AM org.apache.solr.update.DirectUpdateHandler2 commit
  INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
  Nov 14, 2008 5:34:24 AM org.apache.solr.handler.ReplicationHandler doSnapPull
  SEVERE: SnapPull failed 
  org.apache.solr.common.SolrException: Snappull failed : 
   at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:278)
   at 
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.java:208)
   at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:121)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)
  Caused by: java.lang.RuntimeException: 
org.apache.lucene.store.AlreadyClosedException: this Directory is closed
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1037)
   at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:350)
   at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:353)
   at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:265)
   ... 11 more
  Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is 
closed
   at org.apache.lucene.store.Directory.ensureOpen(Directory.java:220)
   at org.apache.lucene.store.FSDirectory.list(FSDirectory.java:320)
   at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:533)
   at 
org.apache.lucene.index.SegmentInfos.readCurrentVersion(SegmentInfos.java:366)
   at 
org.apache.lucene.index.DirectoryIndexReader.isCurrent(DirectoryIndexReader.java:188)
   at 
org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:124)
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1016)
   ... 14 more
  Nov 14, 2008 5:38:52 AM org.apache.solr.update.DirectUpdateHandler2 commit

Any ideas, anyone?

-- Bill

Programatic way to know when an optimize is finished?

[Phillip Farber]

I'd like to automate my indexing processes.  Is there a slick method to 
know when an optimize on an index has completed?


Thanks,

Phil

Re: Programatic way to know when an optimize is finished?

[Alexander Ramos Jardim]

You could intercept the optimize method execution with an advisor?

2008/11/14 Phillip Farber <[EMAIL PROTECTED]>

>
> I'd like to automate my indexing processes.  Is there a slick method to
> know when an optimize on an index has completed?
>
> Thanks,
>
> Phil
>



-- 
Alexander Ramos Jardim

Re: Programatic way to know when an optimize is finished?

[Shalin Shekhar Mangar]

You can add a SolrEventListener to the postOptimize hook.

On Fri, Nov 14, 2008 at 11:34 PM, Phillip Farber <[EMAIL PROTECTED]> wrote:

>
> I'd like to automate my indexing processes.  Is there a slick method to
> know when an optimize on an index has completed?
>
> Thanks,
>
> Phil
>



-- 
Regards,
Shalin Shekhar Mangar.

Spellcheck.onlyMorePopular

[Jeff Newburn]

I am trying to get the onlyMorePopular variable to function correctly.  I
have tried adding both spellchecker.onlyMorePopular as well as
sp.query.onlyMorePopular yet neither of these seem to change the spelling
suggestion response.  I am not sure if I simply do not understand what it is
intended to do or if I am doing something wrong. Below is some of the
response showing this:


?

0
19
?

true
true
true
nice


+

?

?

?

1
0
4
?

Nike


Nike




-Jeff

Re: DataImportHandler not indexing all the records

[Giri]

Hi Noble,
thanks for the help, here are the details: the field "id" is unique, when I
did a select distinct(id), it returned 1 million rows.

---
db-data-config.xml
note: I limit the resultset to 1 million in the select query
---













 



-
in the solr Schema.xml:


   









   
   

   
   


   
   
   
   
   
   
   
   
   
 


I run the index via  firefox browser using
http://localhost:8080/solr/dataimport?command=full-import
I checked the status using
http://localhost:8080/solr/dataimport?command=status
initially the status increased steadily, but after reaching 613071, the
status stayed for a while (as below), and then it displayed the completed
message :


-

0
1

-

-

db-data-config.xml


status
busy
A command is still running...
-

0:3:24.266
1
613071
613070
0
2008-11-14 12:12:16

-

This response format is experimental.  It is likely to change in the future.



---

>>NOTE: this is the status result after it completed
---


-

0
1

-

-

db-data-config.xml


status
idle

-

1
615360
0
2008-11-14 12:12:16
-

Indexing completed. Added/Updated: 615360 documents. Deleted 0 documents.

2008-11-14 12:16:32
2008-11-14 12:16:32
0:4:16.154

-

This response format is experimental.  It is likely to change in the future.



-

here is the full solr scehma.xml content:





  





























 


  





  





  

  




  
  
  
  
  
  
  
  
  
  
  
  




  






  

 
 
   










   
   


   
   


   
   
   
   
   
   
   
   
   
 

 
 id

 
 text

 
 

  



 
 


-


On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള്‍ नोब्ळ् <
[EMAIL PROTECTED]> wrote:

> the fact that it got committed in the end suggests there was no error in
> between
>
> look at the status url and see the no:of rows returned etc.
>
> It gives a clue as to what would have really happened. or you can
> paste your dataconfig and status xmls and we may be able to suggest
> something
>
> On Thu, Nov 13, 2008 at 9:26 AM, Giri <[EMAIL PROTECTED]> wrote:
> > Hi Noble,
> >
> > thanks for reply, my comments are below
> >
> >>>why is the id field multivalued?
> > I was just trying various options, yes, this ID is unique, and I check
> for
> > duplicates, when I did a distinct (id) query to the MySQL database, it
> > returned almost 2 million.
> >
> >>> look at the status host:post/dataimport gives you the status
> > I constantly checked the status  using the  dataimport URL,  the status
> was
> > increased upto 600K records, then it stopped increasing, then took few
> > minutes to commit the indexed data.
> >
> >
> > On Tue, Nov 11, 2008 at 11:35 PM, Noble Paul നോബിള്‍ नोब्ळ् <
> > [EMAIL PROTECTED]> wrote:
> >
> >> why is the id field multivalued? is there a uniqueKey in the schema ?
> >> Are you sure there are no duplicates?
> >>
> >> look at the status host:post/dataimport gives you the status
> >> it can give you some clue
> >>
> >> --Noble
> >>
> >>
> >> On Wed, Nov 12, 2008 at 4:53 AM, Giri <[EMAIL PROTECTED]> wrote:
> >> > Hi,
> >> >
> >> > I have about ~ 2 million records in a mySQL database table (about 9
> >> fields
> >> > from a single table), and I am trying to load it to the solr using
> >> > DataImportHandler using the command=full-import option. it only
> indexed
> >> > about 615360 records out of 2 millions.
> >> >
> >> > here is my db-data-config.xml
> >> > 
> >> > >> > url="jdbc:mysql://localhost:3306/mydb" user="ua" password="pw"
> batchSize
> >> > ="-1"/>
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > 
> >> >
> >> > 
> >> >
> >> > and in my solr schema.xml, i define these fields as:
> >> >
> >> > >> > multiValued="true"/>
> >> > >> > multiValued="true" require

Re: Fatal exception in solr 1.3+ replication

[Mark Miller]

Looks like there might be an issue with the reopen - I'm not seeing what 
it could be offhand though. Have to find what could be closing a 
Directory unexpectedly...I'll try to take a further look over the weekend.


- Mark

William Pierce wrote:

Folks:

I am using the nightly build of 1.3 as of Oct 23 so as to use the replication 
handler.   I am running on windows 2003 server with tomcat 6.0.14.   Everything 
was running fine until I noticed that certain updated records were not showing 
up on the slave.  Further investigation showed me that the failures have indeed 
been occurring since early this morning with a fatal exceptionhere is a 
segment of the tomcat log:
  INFO: Total time taken for download : 0 secs
  Nov 14, 2008 5:34:24 AM org.apache.solr.handler.SnapPuller fetchLatestIndex
  INFO: Conf files are not downloaded or are in sync
  Nov 14, 2008 5:34:24 AM org.apache.solr.update.DirectUpdateHandler2 commit
  INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
  Nov 14, 2008 5:34:24 AM org.apache.solr.handler.ReplicationHandler doSnapPull
  SEVERE: SnapPull failed 
  org.apache.solr.common.SolrException: Snappull failed : 
   at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:278)

   at 
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.java:208)
   at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:121)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
   at java.lang.Thread.run(Thread.java:619)
  Caused by: java.lang.RuntimeException: 
org.apache.lucene.store.AlreadyClosedException: this Directory is closed
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1037)
   at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:350)
   at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:353)
   at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:265)
   ... 11 more
  Caused by: org.apache.lucene.store.AlreadyClosedException: this Directory is 
closed
   at org.apache.lucene.store.Directory.ensureOpen(Directory.java:220)
   at org.apache.lucene.store.FSDirectory.list(FSDirectory.java:320)
   at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:533)
   at 
org.apache.lucene.index.SegmentInfos.readCurrentVersion(SegmentInfos.java:366)
   at 
org.apache.lucene.index.DirectoryIndexReader.isCurrent(DirectoryIndexReader.java:188)
   at 
org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:124)
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1016)
   ... 14 more
  Nov 14, 2008 5:38:52 AM org.apache.solr.update.DirectUpdateHandler2 commit

Any ideas, anyone?

-- Bill
  

delete oddity...

[Dan A. Dickey]

Note to whomever writes documentation:
It would be nice to note in the documentation for doing a 
that the  has to be exactly , and not the whatever
that is specified in the schema.xml configuration file.  For doing deletes
by id, of course.  :)

The wiki at http://wiki.apache.org/solr/UpdateXmlMessages?highlight=%28delete%29
was just a little misleading, at least for me.
-Dan

-- 
Dan A. Dickey
[EMAIL PROTECTED]

SAVVIS, Inc.
Built to Respond

Re: XsltResponseWriter date handling

[Chris Hostetter]

: for pragmatic "the desired output is straightforward XML" reasons.  I need to
: take a Solr date value in "2008-10-30T03:28:10.000Z" format and output it as a
: GMT formatted date like "Oct 30 2008 03:28:10 GMT-0600".
: 
: Anyone got the incantation handy to make such a conversion in XSLT?

I can't think of anyway to do that since, 03:28:10.000Z is not equivilent 
to 03:28:10 GMT-0600 ... GMT-0600 is a very differnet timezone then UTC.

assuming you mean that you want a way to go from a UTC date in the 
canonical format to an alternate format in a specified timezone, then i 
think you need to use some of the XSL dateTime functions i've seen 
documented here and there -- but i'm pretty sure they require XSL 2.0 
(which probably means loading a new XSL library into solr)...

http://www.w3schools.com/Xpath/xpath_functions.asp#datetime

If you're okay leaving things in UTC, i've seen a lot of people use 
substring to extract/reformat the constituent parts.



-Hoss

Re: delete oddity...

[Dan A. Dickey]

On Friday 14 November 2008 15:11:46 Dan A. Dickey wrote:
> Note to whomever writes documentation:
> It would be nice to note in the documentation for doing a 
> that the  has to be exactly , and not the whatever
> that is specified in the schema.xml configuration file.  For doing deletes
> by id, of course.  :)
> 
> The wiki at 
> http://wiki.apache.org/solr/UpdateXmlMessages?highlight=%28delete%29
> was just a little misleading, at least for me.

Ok, hold on a sec... let m
e clarify just a bit.

In my schema.xml I have something like:
myID

The xml:
fred
fails.

The xml:
fred
works.
-Dan

-- 
Dan A. Dickey
[EMAIL PROTECTED]

SAVVIS, Inc.
Built to Respond

Re: delete oddity...

[Erik Hatcher]

On Nov 14, 2008, at 4:11 PM, Dan A. Dickey wrote:

Note to whomever writes documentation:


We all do :)

Feel free to create a wiki account and edit the page if you like.   
Much appreciated in fact!


You're right ... that's a confusing oddity about the  command.

Erik




It would be nice to note in the documentation for doing a 
that the  has to be exactly , and not the whatever
that is specified in the schema.xml configuration file.  For doing  
deletes

by id, of course.  :)

The wiki at http://wiki.apache.org/solr/UpdateXmlMessages?highlight=%28delete%29
was just a little misleading, at least for me.
-Dan

--
Dan A. Dickey
[EMAIL PROTECTED]

SAVVIS, Inc.
Built to Respond

Re: XsltResponseWriter date handling

[Erik Hatcher]

Yeah, leaving it as UTC was fine.  My date vernacular was off in my  
original question, sorry.


Actually I was able to solve the problem a different way.  I'm  
generating a SIMILE Timeline XML output from Solr, and it has a date- 
time-format option.  Setting that to "iso8601" was all it took,  
leaving the Solr date output as-is.  Substring extraction would have  
done the trick just fine though.


I'll be contributing these bits back to contrib/velocity or as part of  
SolrJS when I get things running nicely.


Erik

On Nov 14, 2008, at 4:50 PM, Chris Hostetter wrote:

: for pragmatic "the desired output is straightforward XML"  
reasons.  I need to
: take a Solr date value in "2008-10-30T03:28:10.000Z" format and  
output it as a

: GMT formatted date like "Oct 30 2008 03:28:10 GMT-0600".
:
: Anyone got the incantation handy to make such a conversion in XSLT?

I can't think of anyway to do that since, 03:28:10.000Z is not  
equivilent
to 03:28:10 GMT-0600 ... GMT-0600 is a very differnet timezone then  
UTC.


assuming you mean that you want a way to go from a UTC date in the
canonical format to an alternate format in a specified timezone,  
then i

think you need to use some of the XSL dateTime functions i've seen
documented here and there -- but i'm pretty sure they require XSL 2.0
(which probably means loading a new XSL library into solr)...

http://www.w3schools.com/Xpath/xpath_functions.asp#datetime

If you're okay leaving things in UTC, i've seen a lot of people use
substring to extract/reformat the constituent parts.



-Hoss

RE: DataImportHandler, custom properties

[Lance Norskog]

These are what you may be asking:
1)
Do you wish to read records from the database that are already indexed, and
you want to change the fields found and leave the rest of the Solr document?
This would certainly be a worthwhile feature; there is a separate project to
add 'altering existing documents' in the Jira.

2)
Do you wish to pull both field names and values from the database and add
those to the document? Using the Solr wildcard field name feature? One
design pattern that inspired the Solr wildcard field name feature was to
index products that each have some unique metadata (memory chip has speed
etc., digital camera has resolution etc.) but the total number of tag names
is in the thousands. It was not practical to write a schema.xml with
thousands of field names. It is also not feasible to write a DIH config
handler with thousands of fieldname/xpath combinations.  This would also be
a worthwhile addition to DIH.

Or are you trying to solve another problem?

Lance


-Original Message-
From: DODMax [mailto:[EMAIL PROTECTED] 
Sent: Friday, November 14, 2008 3:20 AM
To: solr-user@lucene.apache.org
Subject: Re: DataImportHandler, custom properties




> http://wiki.apache.org/solr/DataImportHandler#head-071ff018f44ecbdb1cf
> 55afc4c2a857f44ea1ea4
> 
>   column="namedesc" template="hello${e.name},${eparent.surname}" /> ...
> 
> 

The problem is that "namedesc" must be an existing column and I have no idea
of column name in advance.

To be more clear, what I want is to be able to retrive information which are
not in database from the index. For example to store source database in a
kind of custom property and have it back with results.  
-- 
View this message in context:
http://www.nabble.com/DataImportHandler%2C-custom-properties-tp20482190p2049
8600.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Programatic way to know when an optimize is finished?

[Lance Norskog]

The 'optimize' http command blocks. If you script your automation, you can
just call the http and then the next command in the script runs after the
optimize finishes. Hours later, in our case.

Lance

-Original Message-
From: Phillip Farber [mailto:[EMAIL PROTECTED] 
Sent: Friday, November 14, 2008 10:04 AM
To: solr-user@lucene.apache.org
Subject: Programatic way to know when an optimize is finished?


I'd like to automate my indexing processes.  Is there a slick method to know
when an optimize on an index has completed?

Thanks,

Phil

Re: Newbie Question - getting search results from dataimport request handler

[Chris Hostetter]

: > Is here a bug in DIH that caused these unrecognized fields to be ignored,
: > or is it possible the errors were logged (by DUH2 maybe? ... it's been a
: > while since i looked at the update code) but DIH didn't notice them and
: > reported success anyway?
: 
: 
: If the data contains a field name which is not defined in the schema.xml,
: then DIH ignores it. This is a very common use-case where you may want to
: process intermediate data and add it to a completely new field. For example,
: if you have first-name and last-name coming in from DB and you want to
: combine them into a new field "name" with TemplateTransformer.

A i see.  so it's a feature that sometimes acts like a bug :)

it might be worth considering a new @attribute for  to indicate 
that they are going to be used purely as "component" fields (ie: your 
first-name/last-name example) and then have DIH pass all non-component 
fields along and error if undefined in the schema just like other updating 
RequestHandlers do.

either that, or require that people declaure indexed="false" 
stored="false" fields in the schema for these intermediate component 
fields so that we can properly warn then when DIH is getting data it 
doesn't know what to do with -- protecting people from field name typos 
and returning errors instead of silently ignoring unexpected input is 
fairly important behavir -- especially for new users.

-Hoss

Re: Spellcheck.onlyMorePopular

[Grant Ingersoll]

Try spellcheck.onlyMorePopular.


On Nov 14, 2008, at 1:53 PM, Jeff Newburn wrote:

I am trying to get the onlyMorePopular variable to function  
correctly.  I

have tried adding both spellchecker.onlyMorePopular as well as
sp.query.onlyMorePopular yet neither of these seem to change the  
spelling
suggestion response.  I am not sure if I simply do not understand  
what it is

intended to do or if I am doing something wrong. Below is some of the
response showing this:


?

0
19
?

true
true
true
nice


+

?

?

?

1
0
4
?

Nike


Nike




-Jeff


--
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ

Re: DataImportHandler not indexing all the records

[Noble Paul നോബിള്‍ नोब्ळ्]

There is no obvious problem

I can be reasonably sure that
the query

select * from climatedata.ws_record limit 100

would have fetched only  615360 rows.
This is a very reliable pice of information
615360

On Sat, Nov 15, 2008 at 12:41 AM, Giri <[EMAIL PROTECTED]> wrote:
> Hi Noble,
> thanks for the help, here are the details: the field "id" is unique, when I
> did a select distinct(id), it returned 1 million rows.
>
> ---
> db-data-config.xml
> note: I limit the resultset to 1 million in the select query
> ---
> 
> url="jdbc:mysql://localhost:3306/climatedata" user="user" password="pw"
> batchSize ="-1"/>
>
>
>
>
>
>
>
>
>
>
>
> 
>
> 
>
> -
> in the solr Schema.xml:
> 
> 
>multiValued="false"/>
> multiValued="true" required="false"/>
> multiValued="true" required="false"/>
> multiValued="true" required="false"/>
> indexed="true" stored="true"  required="false"/>
> indexed="true" stored="true"  required="false"/>
> multiValued="true"/>
> multiValued="true"/>
> multiValued="true"/>
>
>   
>multiValued="true" required="false"/>
>
>   
>required="false"/>
>
>
>   
>   
>   
>   
>   
>   
>   
>   
>   
>  
>
> 
> I run the index via  firefox browser using
> http://localhost:8080/solr/dataimport?command=full-import
> I checked the status using
> http://localhost:8080/solr/dataimport?command=status
> initially the status increased steadily, but after reaching 613071, the
> status stayed for a while (as below), and then it displayed the completed
> message :
> 
> 
> -
> 
> 0
> 1
> 
> -
> 
> -
> 
> db-data-config.xml
> 
> 
> status
> busy
> A command is still running...
> -
> 
> 0:3:24.266
> 1
> 613071
> 613070
> 0
> 2008-11-14 12:12:16
> 
> -
> 
> This response format is experimental.  It is likely to change in the future.
> 
> 
>
> ---
>
>>>NOTE: this is the status result after it completed
> ---
>
> 
> -
> 
> 0
> 1
> 
> -
> 
> -
> 
> db-data-config.xml
> 
> 
> status
> idle
> 
> -
> 
> 1
> 615360
> 0
> 2008-11-14 12:12:16
> -
> 
> Indexing completed. Added/Updated: 615360 documents. Deleted 0 documents.
> 
> 2008-11-14 12:16:32
> 2008-11-14 12:16:32
> 0:4:16.154
> 
> -
> 
> This response format is experimental.  It is likely to change in the future.
> 
> 
>
> -
>
> here is the full solr scehma.xml content:
> 
> 
> 
>
> 
>  
>
>
>
>
>
>
> sortMissingLast="true"/>
>
>
>
>
>
>
>
>
>
>
>
> sortMissingLast="true"/>
> sortMissingLast="true"/>
> sortMissingLast="true"/>
> sortMissingLast="true"/>
>
>
>
>
>
>
>
> 
>
> positionIncrementGap="100">
>  
>
>
>
>
>
>  
>
>
>
>
> positionIncrementGap="100">
>  
>
>  
>
>
>
> positionIncrementGap="100">
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>
>
>
> positionIncrementGap="100" >
>  
>
> ignoreCase="true" expand="false"/>
>
> generateWordParts="0" generateNumberParts="0" catenateWords="1"
> catenateNumbers="1" catenateAll="0"/>
>
> protected="protwords.txt"/>
>  
>
>  
>  
>   
> multiValued="false"/>
> multiValued="true" required="false"/>
> multiValued="true" required="false"/>
> multiValued="true" required="false"/>
> indexed="true" stored="true"  required="false"/>
> indexed="true" stored="true"  required="false"/>
> multiValued="true"/>
> multiValued="true"/>
> multiValued="true"/>
>
>   
>multiValued="true" required="false"/>
>
>
>   
>required="false"/>
>
>
>   
>   
>   
>   
>   
>   
>   
>   
>   
>  
>
>  
>  id
>
>  
>  text
>
>  
>  
>
>  
>
>
>
>  
>  
>
> 
> -
>
>
> On Wed, Nov 12, 2008 at 11:01 PM, Noble Paul നോബിള്‍ नोब्ळ् <
> [EMAIL PROTECTED]> wrote:
>
>> the fact that it got committed in the end suggests there was no error in
>> between
>>
>> look at the status url and see the no:of rows returned etc.
>>
>> It gives a clue as to what would have really happened. or you

Re: DataImportHandler, custom properties

[Noble Paul നോബിള്‍ नोब्ळ्]

On Sat, Nov 15, 2008 at 5:19 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
> These are what you may be asking:
> 1)
> Do you wish to read records from the database that are already indexed, and
> you want to change the fields found and leave the rest of the Solr document?
> This would certainly be a worthwhile feature; there is a separate project to
> add 'altering existing documents' in the Jira.
>
SOLR-139 and SOLR-828 are the issues currently tracking this. They are
n owhere close to a resolution AFAIK
> 2)
> Do you wish to pull both field names and values from the database and add
> those to the document? Using the Solr wildcard field name feature? One
> design pattern that inspired the Solr wildcard field name feature was to
> index products that each have some unique metadata (memory chip has speed
> etc., digital camera has resolution etc.) but the total number of tag names
> is in the thousands. It was not practical to write a schema.xml with
> thousands of field names. It is also not feasible to write a DIH config
> handler with thousands of fieldname/xpath combinations.  This would also be
> a worthwhile addition to DIH
DIH supports wild card fields . You do not have to put the .field
definition in data-config.

But you will have to write a transformer to add a column into the
returned row with the correct name.
not:There was a bug in 1.3 release , take a more recent version of DIH.
>
> Or are you trying to solve another problem?
>
> Lance
>
>
> -Original Message-
> From: DODMax [mailto:[EMAIL PROTECTED]
> Sent: Friday, November 14, 2008 3:20 AM
> To: solr-user@lucene.apache.org
> Subject: Re: DataImportHandler, custom properties
>
>
>
>
>> http://wiki.apache.org/solr/DataImportHandler#head-071ff018f44ecbdb1cf
>> 55afc4c2a857f44ea1ea4
>>
>>  > column="namedesc" template="hello${e.name},${eparent.surname}" /> ...
>> 
>>
>
> The problem is that "namedesc" must be an existing column and I have no idea
> of column name in advance.
>
> To be more clear, what I want is to be able to retrive information which are
> not in database from the index. For example to store source database in a
> kind of custom property and have it back with results.
> --
> View this message in context:
> http://www.nabble.com/DataImportHandler%2C-custom-properties-tp20482190p2049
> 8600.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>



-- 
--Noble Paul

Re: Newbie Question - getting search results from dataimport request handler

[Noble Paul നോബിള്‍ नोब्ळ्]

On Sat, Nov 15, 2008 at 6:33 AM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:
>
> : > Is here a bug in DIH that caused these unrecognized fields to be ignored,
> : > or is it possible the errors were logged (by DUH2 maybe? ... it's been a
> : > while since i looked at the update code) but DIH didn't notice them and
> : > reported success anyway?
> :
> :
> : If the data contains a field name which is not defined in the schema.xml,
> : then DIH ignores it. This is a very common use-case where you may want to
> : process intermediate data and add it to a completely new field. For example,
> : if you have first-name and last-name coming in from DB and you want to
> : combine them into a new field "name" with TemplateTransformer.
>
> A i see.  so it's a feature that sometimes acts like a bug :)
>
> it might be worth considering a new @attribute for  to indicate
> that they are going to be used purely as "component" fields (ie: your
> first-name/last-name example) and then have DIH pass all non-component
> fields along and error if undefined in the schema just like other updating
> RequestHandlers do.
>
> either that, or require that people declaure indexed="false"
> stored="false" fields in the schema for these intermediate component
> fields so that we can properly warn then when DIH is getting data it
> doesn't know what to do with -- protecting people from field name typos
> and returning errors instead of silently ignoring unexpected input is
> fairly important behavir -- especially for new users.
>
Actually it is done by DIH . When the dataconfig is loaded DIH reports
these information on the console. though it is limited , it helps to a
certain extent
> -Hoss
>
>



-- 
--Noble Paul

Re: Fatal exception in solr 1.3+ replication

[Mark Miller]

Hey William, sorry about the trouble. I have to look at this further, 
but I think the issue is fixed if you grab the latest trunk build. 
Solr-465 should inadvertently fix things - before that patch, a 
deprecated constructor for solrsearcher was being called - this 
constructor caused the underlying IndexReader to close its own 
Directory, and since IndexReaders are reopened, we don't want that.


Mark Miller wrote:
Looks like there might be an issue with the reopen - I'm not seeing 
what it could be offhand though. Have to find what could be closing a 
Directory unexpectedly...I'll try to take a further look over the 
weekend.


- Mark

William Pierce wrote:

Folks:

I am using the nightly build of 1.3 as of Oct 23 so as to use the 
replication handler.   I am running on windows 2003 server with 
tomcat 6.0.14.   Everything was running fine until I noticed that 
certain updated records were not showing up on the slave.  Further 
investigation showed me that the failures have indeed been occurring 
since early this morning with a fatal exceptionhere is a segment 
of the tomcat log:

  INFO: Total time taken for download : 0 secs
  Nov 14, 2008 5:34:24 AM org.apache.solr.handler.SnapPuller 
fetchLatestIndex

  INFO: Conf files are not downloaded or are in sync
  Nov 14, 2008 5:34:24 AM org.apache.solr.update.DirectUpdateHandler2 
commit

  INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
  Nov 14, 2008 5:34:24 AM org.apache.solr.handler.ReplicationHandler 
doSnapPull
  SEVERE: SnapPull failed   org.apache.solr.common.SolrException: 
Snappull failed :at 
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:278)
   at 
org.apache.solr.handler.ReplicationHandler.doSnapPull(ReplicationHandler.java:208) 


   at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:121)
   at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at 
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) 


   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) 

   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) 

   at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) 

   at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) 

   at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) 


   at java.lang.Thread.run(Thread.java:619)
  Caused by: java.lang.RuntimeException: 
org.apache.lucene.store.AlreadyClosedException: this Directory is closed

   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1037)
   at 
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:350) 


   at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:353)
   at 
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:265)

   ... 11 more
  Caused by: org.apache.lucene.store.AlreadyClosedException: this 
Directory is closed

   at org.apache.lucene.store.Directory.ensureOpen(Directory.java:220)
   at org.apache.lucene.store.FSDirectory.list(FSDirectory.java:320)
   at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:533) 

   at 
org.apache.lucene.index.SegmentInfos.readCurrentVersion(SegmentInfos.java:366) 

   at 
org.apache.lucene.index.DirectoryIndexReader.isCurrent(DirectoryIndexReader.java:188) 

   at 
org.apache.lucene.index.DirectoryIndexReader.reopen(DirectoryIndexReader.java:124) 


   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1016)
   ... 14 more
  Nov 14, 2008 5:38:52 AM org.apache.solr.update.DirectUpdateHandler2 
commit


Any ideas, anyone?

-- Bill
  

Re: solrj and CLOSE_WAIT's

[Ian Holsman]

Ryan McKinley wrote:

not sure if it is something we can do better or part of HttpClient...

From:
http://www.nabble.com/CLOSE_WAIT-td19959428.html

it seems to suggest you may want to call:
con.closeIdleConnections(0L);

But if you are creating a new MultiThreadedHttpConnectionManager for 
each request, is seems odd you would have to explicitly close the 
connection for each request.


What happens if you try using a SimpleHttpConnectionManager rather 
then a MultiThreadedHttpConnectionManager?  You can explicitly pass in:

  new HttpClient( new SimpleHttpConnectionManager()  )
to:
public CommonsHttpSolrServer(URL baseURL, HttpClient client, 
ResponseParser parser, boolean useMultiPartPost) {


if that fixes things, it is a bit disturbing, but something we should 
look into.


ryan




Hi Ryan.
the problem is with the HttpClient class.
the SimpleHttpConnectionManager by default hangs onto a connection "just 
in case".


It needs to be called like:
  CommonsHttpSolrServer solrServer;
  SimpleHttpConnectionManager cm = new 
SimpleHttpConnectionManager(true);

  HttpClient httpClient = new HttpClient(cm);

   solrServer = new CommonsHttpSolrServer(solrURL, httpClient);
   solrServer.optimize();
   cm.shutdown();


I'm not sure the 'shutdown' method is required, but the code now has 
*NO* CLOSE_WAITS after running for 3-4 hours.
I'll stick up a jira with makes 'SimpleHttpConnectionManager(true)' the 
default.


regards
Ian