Problem - Help me with DataImport
Good Morning, First, sorry for my poor english. I trying to index “blogs” (rss) to my solr, so I´m using a dataImportHandler for this. I can´t index the date and I don´t no how to index static values (constant) in a Field. When I make a “Full Import” it doesn´t index the docs; if I delete the line of date, it´s work. When I debug with verbose it shows me the right information. Below you can see my dataImportHandler: http://locademiaz.wordpress.com/feed/"; processor="XPathEntityProcessor" transformer="DateFormatTransformer" forEach="/rss/channel/item"> I appreciate your help. Thank you very much. Matias.
Re: Problem - Help me with DataImport
Stefan, Thank you very much! It works perfect... Any idea for the other question? Someone? Matias. 2011/3/2 Stefan Matheis > Matias, > > for indexing constant/static values .. try > http://wiki.apache.org/solr/DataImportHandler#TemplateTransformer > > Regards > Stefan > > On Wed, Mar 2, 2011 at 2:46 PM, Matias Alonso > wrote: > > Good Morning, > > > > > > First, sorry for my poor english. > > > > > > I trying to index “blogs” (rss) to my solr, so I´m using a > dataImportHandler > > for this. > > > > I can´t index the date and I don´t no how to index static values > (constant) > > in a Field. > > > > When I make a “Full Import” it doesn´t index the docs; if I delete the > line > > of date, it´s work. > > > > When I debug with verbose it shows me the right information. > > > > > > Below you can see my dataImportHandler: > > > > > > > > > > > > > > > > > > > >> > > pk="link" > > > > url=" > > http://locademiaz.wordpress.com/feed/"; > > > > > > processor="XPathEntityProcessor" > > > > > > transformer="DateFormatTransformer" > > > > > > forEach="/rss/channel/item"> > > > > > > dateTimeFormat="EEE, > > d MMM HH:mm:ss Z" /> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I appreciate your help. > > > > Thank you very much. > > > > > > > > Matias. > > >
Help -DIH (mail)
Hi everyone! I’m trying to index mails into solr through DHI (based on the “example-DIH”). For this I´m using my personal email from gmail, but I can´t index. Configuration in Data-config .xml: When I debug and verbose I get the following messange (only first lines): org.apache.solr.handler.dataimport.DataImportHandlerException: Connection failed Processing Document # 1 at I appreciate your help! Regards, Matias.
Re: Help -DIH (mail)
Thks Gora,
I forgot "s". Now, doesn´t have error, but not index.
This´s the answer in my command line (windows :( ).
..
..
..
protocol : imaps
host : imap.gmail.com
folders :
Recibidos,recibidos,RECIBIDOS,inbox.InBox,INBOX,Mail,MAIL,mail,CORREO,
correo,Correo
recurse : false
exclude : []
include : []
batchSize : 100
fetchSize : 32768
read timeout : 6
conection timeout : 3
custom filter :
fetch mail since : Sat Jan 01 00:00:00 GFT 2011
04/03/2011 14:05:40 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=/solr path=/select
params={command=status&qt=/dataimport} st
atus=0 QTime=0
04/03/2011 14:05:40 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=/solr path=/select
params={command=show-config&qt=/dataimpor
t} status=0 QTime=0
04/03/2011 14:05:40 org.apache.solr.handler.dataimport.MailEntityProcessor
conne
ctToMailBox
INFO: Connected to mailbox
04/03/2011 14:05:40 org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {} 0 0
04/03/2011 14:05:40 org.apache.solr.handler.dataimport.DocBuilder execute
INFO: Time taken = 0:0:3.31
04/03/2011 14:05:48 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=/solr path=/select
params={command=status&qt=/dataimport} st
atus=0 QTime=0
04/03/2011 14:05:48 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=/solr path=/select
params={command=show-config&qt=/dataimpor
t} status=0 QTime=0
Matias.
2011/3/4 Gora Mohanty
> On Fri, Mar 4, 2011 at 9:20 PM, Matias Alonso
> wrote:
> > Hi everyone!
> >
> >
> > I’m trying to index mails into solr through DHI (based on the
> > “example-DIH”). For this I´m using my personal email from gmail, but I
> can´t
> > index.
>
> Have not used the MailEntityProcessor with Gmail, but some
> points below:
>
> > Configuration in Data-config .xml:
> >
> >
> >
> >
> >
> > >
> > user="m...@gmail.com"
> ^ I presume that you have put in your actual
>email address here.
> [...]
> > protocol="imap"/>
> ^ Shouldn't this be imaps, at least as
> per http://wiki.apache.org/solr/MailEntityProcessor
>
> Regards,
> Gora
>
Re: Help -DIH (mail)
Thks Peter, Yes, gmail gives me imaps (i understood that). So, I tried what you mention but I had get the original mesange I posted. Matias. 2011/3/4 Peter Sturge > Hi, > > You need to put your password in as well. You should use protocol="imap" > unless your gmail is set for imaps (I don't believe the free gmail gives > you > this). > > user="u...@mydomain.com" > password="userpwd" > host="imap.mydomain.com" > include="" > exclude="" > processor="MailEntityProcessor" > protocol="imap" > /> > > HTH > Peter > > > > On Fri, Mar 4, 2011 at 4:42 PM, Gora Mohanty wrote: > > > On Fri, Mar 4, 2011 at 9:20 PM, Matias Alonso > > wrote: > > > Hi everyone! > > > > > > > > > I’m trying to index mails into solr through DHI (based on the > > > “example-DIH”). For this I´m using my personal email from gmail, but I > > can´t > > > index. > > > > Have not used the MailEntityProcessor with Gmail, but some > > points below: > > > > > Configuration in Data-config .xml: > > > > > > > > > > > > > > > > > > > > > > > user="m...@gmail.com" > > ^ I presume that you have put in your actual > >email address here. > > [...] > > > protocol="imap"/> > > ^ Shouldn't this be imaps, at least as > > per http://wiki.apache.org/solr/MailEntityProcessor > > > > Regards, > > Gora > > >
Re: Help -DIH (mail)
2011/3/4 Peter Sturge > Hi Matias, > > Can you post your data-config.xml? (with disquised names/credentials) > > Thanks, > Peter > > > On Fri, Mar 4, 2011 at 5:13 PM, Matias Alonso >wrote: > > > Thks Peter, > > > > Yes, gmail gives me imaps (i understood that). So, I tried what you > mention > > but I had get the original mesange I posted. > > > > Matias. > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > Hi, > > > > > > You need to put your password in as well. You should use > protocol="imap" > > > unless your gmail is set for imaps (I don't believe the free gmail > gives > > > you > > > this). > > > > > > > > user="u...@mydomain.com" > > > password="userpwd" > > > host="imap.mydomain.com" > > > include="" > > > exclude="" > > > processor="MailEntityProcessor" > > > protocol="imap" > > > /> > > > > > > HTH > > > Peter > > > > > > > > > > > > On Fri, Mar 4, 2011 at 4:42 PM, Gora Mohanty > wrote: > > > > > > > On Fri, Mar 4, 2011 at 9:20 PM, Matias Alonso < > matiasgalo...@gmail.com > > > > > > > wrote: > > > > > Hi everyone! > > > > > > > > > > > > > > > I’m trying to index mails into solr through DHI (based on the > > > > > “example-DIH”). For this I´m using my personal email from gmail, > but > > I > > > > can´t > > > > > index. > > > > > > > > Have not used the MailEntityProcessor with Gmail, but some > > > > points below: > > > > > > > > > Configuration in Data-config .xml: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > user="m...@gmail.com" > > > > ^ I presume that you have put in your actual > > > >email address here. > > > > [...] > > > > > protocol="imap"/> > > > > ^ Shouldn't this be imaps, at least as > > > > per http://wiki.apache.org/solr/MailEntityProcessor > > > > > > > > Regards, > > > > Gora > > > > > > > > > >
Re: Help -DIH (mail)
Hi Peter, I test with deltaFetch="false", but doesn´t work :( I'm using "DataImportHandler Development Console" to index ( http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimport); I'm working with "example-DIH". thks... 2011/3/4 Peter Sturge > Hi Matias, > > I haven't seen it in the posts, but I may have missed it -- what is the > import command you're sending? > Something like: > http://localhost:8983/solr/db/dataimport?command=full-import > > Can you also test it with deltaFetch="false". I seem to remember having > some > problems with delta in the MailEntityProcessor. > > > > On Fri, Mar 4, 2011 at 6:29 PM, Matias Alonso >wrote: > > > > > > >> user="myem...@gmail.com" > > password="mypassword" > > host="imap.gmail.com" > > fetchMailsSince="2011-01-01 00:00:00" > > deltaFetch="true" > > include="" > > exclude="" > > recurse="false" > > > > > > > folders="Recibidos,recibidos,RECIBIDOS,inbox.InBox,INBOX,Mail,MAIL,mail,CORREO,correo,Correo" > > includeContent="true" > > processAttachments="false" > > includeOtherUserFolders="false" > > includeSharedFolders="false" > > batchSize="100" > > processor="MailEntityProcessor" > > protocol="imaps" /> > > > > > > > > 2011/3/4 Peter Sturge > > > > > Hi Matias, > > > > > > Can you post your data-config.xml? (with disquised names/credentials) > > > > > > Thanks, > > > Peter > > > > > > > > > On Fri, Mar 4, 2011 at 5:13 PM, Matias Alonso > > >wrote: > > > > > > > Thks Peter, > > > > > > > > Yes, gmail gives me imaps (i understood that). So, I tried what you > > > mention > > > > but I had get the original mesange I posted. > > > > > > > > Matias. > > > > > > > > > > > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > > > > > Hi, > > > > > > > > > > You need to put your password in as well. You should use > > > protocol="imap" > > > > > unless your gmail is set for imaps (I don't believe the free gmail > > > gives > > > > > you > > > > > this). > > > > > > > > > > > > > > user="u...@mydomain.com" > > > > > password="userpwd" > > > > > host="imap.mydomain.com" > > > > > include="" > > > > > exclude="" > > > > > processor="MailEntityProcessor" > > > > > protocol="imap" > > > > > /> > > > > > > > > > > HTH > > > > > Peter > > > > > > > > > > > > > > > > > > > > On Fri, Mar 4, 2011 at 4:42 PM, Gora Mohanty > > > wrote: > > > > > > > > > > > On Fri, Mar 4, 2011 at 9:20 PM, Matias Alonso < > > > matiasgalo...@gmail.com > > > > > > > > > > > wrote: > > > > > > > Hi everyone! > > > > > > > > > > > > > > > > > > > > > I’m trying to index mails into solr through DHI (based on the > > > > > > > “example-DIH”). For this I´m using my personal email from > gmail, > > > but > > > > I > > > > > > can´t > > > > > > > index. > > > > > > > > > > > > Have not used the MailEntityProcessor with Gmail, but some > > > > > > points below: > > > > > > > > > > > > > Configuration in Data-config .xml: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > user="m...@gmail.com" > > > > > > ^ I presume that you have put in your actual > > > > > >email address here. > > > > > > [...] > > > > > > > protocol="imap"/> > > > > > > ^ Shouldn't this be imaps, at least as > > > > > > per http://wiki.apache.org/solr/MailEntityProcessor > > > > > > > > > > > > Regards, > > > > > > Gora > > > > > > > > > > > > > > > > > > > > >
Re: Help -DIH (mail)
Hi Peter, >From "DataImportHandler Development Console" I made a full-import, but didn´t work. Now, I execute " http://localhost:8983/solr/mail/dataimport?command=full-import"; but nothing happends; no index; no errors. thks... Matias. 2011/3/4 Peter Sturge > Hi Mataias, > > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimportaccesses > the dataimport handler, but you need to tell it to do something by > sending a command: > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimport > ?command=full-import< > http://localhost:8983/solr/db/dataimport?command=full-import> > > If you haven't already, have a look at: > > > http://www.lucidimagination.com/search/out?u=http%3A%2F%2Fwiki.apache.org%2Fsolr%2FDataImportHandler > > It gives very thorough and useful advice on getting the DIH working. > > Peter > > > > On Fri, Mar 4, 2011 at 6:59 PM, Matias Alonso >wrote: > > > Hi Peter, > > > > I test with deltaFetch="false", but doesn´t work :( > > I'm using "DataImportHandler Development Console" to index ( > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimport > ); > > I'm working with "example-DIH". > > > > thks... > > > > > > > > 2011/3/4 Peter Sturge > > > > > Hi Matias, > > > > > > I haven't seen it in the posts, but I may have missed it -- what is the > > > import command you're sending? > > > Something like: > > > http://localhost:8983/solr/db/dataimport?command=full-import > > > > > > Can you also test it with deltaFetch="false". I seem to remember having > > > some > > > problems with delta in the MailEntityProcessor. > > > > > > > > > > > > On Fri, Mar 4, 2011 at 6:29 PM, Matias Alonso > > >wrote: > > > > > > > > > > > > > > >> > > user="myem...@gmail.com" > > > > password="mypassword" > > > > host="imap.gmail.com" > > > > fetchMailsSince="2011-01-01 00:00:00" > > > > deltaFetch="true" > > > > include="" > > > > exclude="" > > > > recurse="false" > > > > > > > > > > > > > > > > > > folders="Recibidos,recibidos,RECIBIDOS,inbox.InBox,INBOX,Mail,MAIL,mail,CORREO,correo,Correo" > > > > includeContent="true" > > > > processAttachments="false" > > > > includeOtherUserFolders="false" > > > > includeSharedFolders="false" > > > > batchSize="100" > > > > processor="MailEntityProcessor" > > > > protocol="imaps" /> > > > > > > > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > > > > > Hi Matias, > > > > > > > > > > Can you post your data-config.xml? (with disquised > names/credentials) > > > > > > > > > > Thanks, > > > > > Peter > > > > > > > > > > > > > > > On Fri, Mar 4, 2011 at 5:13 PM, Matias Alonso < > > matiasgalo...@gmail.com > > > > > >wrote: > > > > > > > > > > > Thks Peter, > > > > > > > > > > > > Yes, gmail gives me imaps (i understood that). So, I tried what > you > > > > > mention > > > > > > but I had get the original mesange I posted. > > > > > > > > > > > > Matias. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > You need to put your password in as well. You should use > > > > > protocol="imap" > > > > > > > unless your gmail is set for imaps (I don't believe the free > > gmail > > > > > gives > > > > > > > you > > > > > > > this). > > > > > > > > > > > > > > > > > > > > user="u...@mydomain.com" > > > > > > &g
Re: Help -DIH (mail)
I´m using the trunk. Thanks Peter for your preoccupation! Matias. 2011/3/4 Peter Sturge > Hi Matias, > > What version of Solr are you using? Are you running any patches (maybe > SOLR-2245)? > > Thanks, > Peter > > > > On Fri, Mar 4, 2011 at 8:25 PM, Matias Alonso >wrote: > > > Hi Peter, > > > > From "DataImportHandler Development Console" I made a full-import, but > > didn´t work. > > > > Now, I execute " > > http://localhost:8983/solr/mail/dataimport?command=full-import"; but > > nothing > > happends; no index; no errors. > > > > thks... > > > > Matias. > > > > > > > > 2011/3/4 Peter Sturge > > > > > Hi Mataias, > > > > > > > > > > > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimportaccesses > > > the dataimport handler, but you need to tell it to do something by > > > sending a command: > > > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimport > > > ?command=full-import< > > > http://localhost:8983/solr/db/dataimport?command=full-import> > > > > > > If you haven't already, have a look at: > > > > > > > > > > > > http://www.lucidimagination.com/search/out?u=http%3A%2F%2Fwiki.apache.org%2Fsolr%2FDataImportHandler > > > > > > It gives very thorough and useful advice on getting the DIH working. > > > > > > Peter > > > > > > > > > > > > On Fri, Mar 4, 2011 at 6:59 PM, Matias Alonso > > >wrote: > > > > > > > Hi Peter, > > > > > > > > I test with deltaFetch="false", but doesn´t work :( > > > > I'm using "DataImportHandler Development Console" to index ( > > > > > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimport > > > ); > > > > I'm working with "example-DIH". > > > > > > > > thks... > > > > > > > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > > > > > Hi Matias, > > > > > > > > > > I haven't seen it in the posts, but I may have missed it -- what is > > the > > > > > import command you're sending? > > > > > Something like: > > > > > http://localhost:8983/solr/db/dataimport?command=full-import > > > > > > > > > > Can you also test it with deltaFetch="false". I seem to remember > > having > > > > > some > > > > > problems with delta in the MailEntityProcessor. > > > > > > > > > > > > > > > > > > > > On Fri, Mar 4, 2011 at 6:29 PM, Matias Alonso < > > matiasgalo...@gmail.com > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > > >> > > > > user="myem...@gmail.com" > > > > > > password="mypassword" > > > > > > host="imap.gmail.com" > > > > > > fetchMailsSince="2011-01-01 00:00:00" > > > > > > deltaFetch="true" > > > > > > include="" > > > > > > exclude="" > > > > > > recurse="false" > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > folders="Recibidos,recibidos,RECIBIDOS,inbox.InBox,INBOX,Mail,MAIL,mail,CORREO,correo,Correo" > > > > > > includeContent="true" > > > > > > processAttachments="false" > > > > > > includeOtherUserFolders="false" > > > > > > includeSharedFolders="false" > > > > > > batchSize="100" > > > > > > processor="MailEntityProcessor" > > > > > > protocol="imaps" /> > > > > > > > > > > > > > > > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > > > > > > > > > Hi Matias, > > > > > > > > > > > > > > Can you post your data-config.xml? (with disquised > > > names/credentials) > > &
Re: Help -DIH (mail)
Hi Peter, When I execute the commands you mentioned, nothing happend. Below I show you the comands executed and the answered of they. Sorry, but I don´t know how to enable the log; my jre is by default. Rememeber I´m running the example-DIH (trunk\solr\example\example-DIH\solr); java -Dsolr.solr.home="./example-DIH/solr/" -jar start.jar. Import: http://localhost:8983/solr/mail/dataimport?command=status< http://localhost:8983/solr/mail/dataimport?command=full-import> - 0 15 - - data-config.xml - full-import<http://localhost:8983/solr/mail/dataimport?command=full-import> idle - This response format is experimental. It is likely to change in the future. Status: http://localhost:8983/solr/mail/dataimport?command=status< http://localhost:8983/solr/mail/dataimport?command=full-import> - 0 0 - - data-config.xml - status<http://localhost:8983/solr/mail/dataimport?command=full-import> idle - This response format is experimental. It is likely to change in the future. Thank you for your help. Matias. 2011/3/4 Peter Sturge > Can you try this: > > Issue a full import command like this: > > http://localhost:8983/solr/dataimport?command=full-import< > http://localhost:8983/solr/db/dataimport?command=full-import> > > (There is no core name here - if you're using a core name (db?), then add > that in between solr/ and /dataimport) > > then, run: > http://localhost:8983/solr/dataimport?command=status< > http://localhost:8983/solr/db/dataimport?command=full-import> > > This will show the results of the previous import. Has it been rolled-back? > If so, there might be something in the log if it's enabled (see your jre's > lib/logging.properties file). > (you won't see any errors unless you run the status command - that's where > they're stored) > > HTH > Peter > > > > > On Sat, Mar 5, 2011 at 12:46 AM, Matias Alonso >wrote: > > > I´m using the trunk. > > > > Thanks Peter for your preoccupation! > > > > Matias. > > > > > > > > 2011/3/4 Peter Sturge > > > > > Hi Matias, > > > > > > What version of Solr are you using? Are you running any patches (maybe > > > SOLR-2245)? > > > > > > Thanks, > > > Peter > > > > > > > > > > > > On Fri, Mar 4, 2011 at 8:25 PM, Matias Alonso > > >wrote: > > > > > > > Hi Peter, > > > > > > > > From "DataImportHandler Development Console" I made a full-import, > but > > > > didn´t work. > > > > > > > > Now, I execute " > > > > http://localhost:8983/solr/mail/dataimport?command=full-import"; but > > > > nothing > > > > happends; no index; no errors. > > > > > > > > thks... > > > > > > > > Matias. > > > > > > > > > > > > > > > > 2011/3/4 Peter Sturge > > > > > > > > > Hi Mataias, > > > > > > > > > > > > > > > > > > > > > > > > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimportaccesses > > > > > the dataimport handler, but you need to tell it to do something by > > > > > sending a command: > > > > > > > > > http://localhost:8983/solr/mail/admin/dataimport.jsp?handler=/dataimport > > > > > ?command=full-import< > > > > > http://localhost:8983/solr/db/dataimport?command=full-import> > > > > > > > > > > If you haven't already, have a look at: > > > > > > > > > > > > > > > > > > > > > > > > > http://www.lucidimagination.com/search/out?u=http%3A%2F%2Fwiki.apache.org%2Fsolr%2FDataImportHandler > > > > > > > > > > It gives very thorough and useful advice on getting the DIH > working. > > > > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > On Fri, Mar 4, 2011 at 6:59 PM, Matias Alonso < > > matiasgalo...@gmail.com > > > > > >wrote: > > > > > > > > > > > Hi Peter, > > > > > > > > > > > > I test with deltaFetch="false", but doesn´t work :( > > > > > > I'm using "DataImportHandler Development Console" to index ( > > > > > > > > > > > > http://localhost:8983/solr/mail/admin/
Re: Help -DIH (mail)
1 11:52:03 org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@1cee792 main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
09/03/2011 11:52:03 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener sending requests to Searcher@1cee792 main
09/03/2011 11:52:03 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=null path=null
params={start=0&event=newSearcher&q=solr&rows=10} hits=0 status=0 QTime=0
09/03/2011 11:52:03 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=null path=null
params={start=0&event=newSearcher&q=rocks&rows=10} hits=0 status=0 QTime=0
09/03/2011 11:52:03 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=null path=null
params={event=newSearcher&q=static+newSearcher+warming+query+from+solrconfig.xml}
hits=0 status=0 QTime=0
09/03/2011 11:52:03 org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
09/03/2011 11:52:03 org.apache.solr.core.SolrCore registerSearcher
INFO: [mail] Registered new searcher Searcher@1cee792 main
09/03/2011 11:52:03 org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@9a18a0 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=5,evictions=0,size=5,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
09/03/2011 11:52:03 org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport.properties
09/03/2011 11:52:03 org.apache.solr.handler.dataimport.SolrWriter persist
INFO: Wrote last indexed time to dataimport.properties
09/03/2011 11:52:03 org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {deleteByQuery=*:*,optimize=} 0 0
09/03/2011 11:52:03 org.apache.solr.handler.dataimport.DocBuilder execute
INFO: Time taken = 0:0:2.359
09/03/2011 11:54:58 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=/solr path=/dataimport params={command=status} status=0
QTime=0
Thks,
Matias.
2011/3/9 Peter Sturge
> Hi,
>
> You've included some output in your message, so I presume something
> *did* happen when you ran the 'status' command (but it might not be
> what you wanted to happen :-)
>
> If you run:
> http://localhost:8983/solr/mail/dataimport?command=status
>
> and you get something like this back:
> idle
>
>
>
> It means that no full-import or delta-import has been run during the
> life of the JVM Solr session.
>
> You should try running:
>http://localhost:8983/solr/mail/dataimport?command=full-import
>
> Then run:
> http://localhost:8983/solr/mail/dataimport?command=status
>
> to see the status of the full-import (busy, idle, error, rolled back etc.)
>
> You can enable java logging by editing your JRE's lib/logging.properties
> file.
>
> Something like this should give you some log files:
> handlers= java.util.logging.FileHandler
> .level= INFO
> java.util.logging.FileHandler.pattern = ./logs/mylogs%d.log
> java.util.logging.FileeHandler.level = INFO
> java.util.logging.FileHandler.limit = 50
> java.util.logging.FileHandler.count = 1
> java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
>
> NOTE: Make sure the 'logs' folder exists (in your $cwd) before you
> start, or you'll get an error.
>
> HTH
> Peter
>
>
>
>
> On Wed, Mar 9, 2011 at 12:47 PM, Matias Alonso
> wrote:
> > Hi Peter,
> >
> > When I execute the commands you mentioned, nothing happend.
> > Below I show you the comands executed and the answered of they.
> > Sorry, but I don´t know how to enable the log; my jre is by default.
> > Rememeber I´m running the example-DIH
> (trunk\solr\example\example-DIH\solr);
> > java -Dsolr.solr.home="./example-DIH/solr/" -jar start.jar.
> >
> >
> >
> > Import:
> > http://localhost:8983/solr/mail/dataimport?command=status<
> > http://localhost:8983/solr/mail/dataimport?command=full-import>
> >
> >
> > -
> >
> > 0
> > 15
> >
> > -
&
Re: Help -DIH (mail)
Log:
09/03/2011 11:54:58 org.apache.solr.core.SolrCore execute
INFO: [mail] webapp=/solr path=/dataimport params={command=status} status=0
QTime=0
XML
-
0
0
-
-
data-config.xml
status
idle
-
0
0
0
2011-03-09 11:52:01
-
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
2011-03-09 11:52:03
2011-03-09 11:52:03
0
0:0:2.359
-
This response format is experimental. It is likely to change in the future.
Thks,
Matias.
2011/3/9 Peter Sturge
> Hi,
>
> When you ran the status command, what was the output?
>
>
> On Wed, Mar 9, 2011 at 2:55 PM, Matias Alonso
> wrote:
> > Peter,
> >
> > You´re right; may be I expose wrong because of my english.
> > I done everything you told me. I think that no find the "folder" when
> index.
> > What you thinking about?
> > Below I show to you part of the log.
> >
> >
> >
> > 09/03/2011 11:52:01 org.apache.solr.core.SolrCore execute
> > INFO: [mail] webapp=/solr path=/dataimport params={command=full-import}
> > status=0 QTime=0
> > 09/03/2011 11:52:01 org.apache.solr.handler.dataimport.DataImporter
> > doFullImport
> > INFO: Starting Full Import
> > 09/03/2011 11:52:01 org.apache.solr.handler.dataimport.SolrWriter
> > readIndexerProperties
> > INFO: Read dataimport.properties
> > 09/03/2011 11:52:01 org.apache.solr.update.DirectUpdateHandler2 deleteAll
> > INFO: [mail] REMOVING ALL DOCUMENTS FROM INDEX
> > 09/03/2011 11:52:01 org.apache.solr.core.SolrDeletionPolicy onInit
> > INFO: SolrDeletionPolicy.onInit: commits:num=1
> >commit{dir=D:\Search
> >
> Plugtree\trunk\solr\example\example-DIH\solr\mail\data\index,segFN=segments_c,version=1298912662799,generation=12,filenames=[segments_c]
> > 09/03/2011 11:52:01 org.apache.solr.core.SolrDeletionPolicy updateCommits
> > INFO: newest commit = 1298912662799
> > 09/03/2011 11:52:01
> org.apache.solr.handler.dataimport.MailEntityProcessor
> > logConfig
> > INFO: user : myem...@gmail.com
> > pwd : mypass
> > protocol : imaps
> > host : imap.gmail.com
> > folders :
> >
> Recibidos,recibidos,RECIBIDOS,inbox.InBox,INBOX,Mail,MAIL,mail,CORREO,correo,Correo,Mail,mail,MAIL
> > recurse : false
> > exclude : []
> > include : []
> > batchSize : 100
> > fetchSize : 32768
> > read timeout : 6
> > conection timeout : 3
> > custom filter :
> > fetch mail since : Thu Mar 03 00:00:00 GFT 2011
> >
> > 09/03/2011 11:52:03
> org.apache.solr.handler.dataimport.MailEntityProcessor
> > connectToMailBox
> > INFO: Connected to mailbox
> > 09/03/2011 11:52:03 org.apache.solr.handler.dataimport.DocBuilder finish
> > INFO: Import completed successfully
> > 09/03/2011 11:52:03 org.apache.solr.update.DirectUpdateHandler2 commit
> > INFO: start
> >
> commit(optimize=true,waitFlush=false,waitSearcher=true,expungeDeletes=false)
> > 09/03/2011 11:52:03 org.apache.solr.core.SolrDeletionPolicy onCommit
> > INFO: SolrDeletionPolicy.onCommit: commits:num=2
> >commit{dir=D:\Search
> >
> Plugtree\trunk\solr\example\example-DIH\solr\mail\data\index,segFN=segments_c,version=1298912662799,generation=12,filenames=[segments_c]
> >commit{dir=D:\Search
> >
> Plugtree\trunk\solr\example\example-DIH\solr\mail\data\index,segFN=segments_d,version=1298912662800,generation=13,filenames=[segments_d]
> > 09/03/2011 11:52:03 org.apache.solr.core.SolrDeletionPolicy updateCommits
> > INFO: newest commit = 1298912662800
> > 09/03/2011 11:52:03 org.apache.solr.search.SolrIndexSearcher
> > INFO: Opening Searcher@1cee792 main
> > 09/03/2011 11:52:03 org.apache.solr.update.DirectUpdateHandler2 commit
> > INFO: end_commit_flush
> > 09/03/2011 11:52:03 org.apache.solr.search.SolrIndexSearcher warm
> > INFO: autowarming Searcher@1cee792 main from Searcher@9a18a0 main
> >
> >
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > 09/03/2011 11:52:03 org.apache.solr.search.SolrIndexSearcher warm
> > INFO: autowarming result for Searcher@1cee792 main
> >
> >
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
> > 09/03/2011 11:52:03 org.apache.solr.search.SolrIndexSearcher warm
> > INFO: autowarming Searcher@1cee792 main from Searcher@9a18a0 main
> >
> >
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTim
email - DIH
Hi, I’m using Data Import Handler for index emails. The problem is that nota ll the emails was indexed When I do a full import. Someone have any idea? Regards, -- Matias.
Re: email - DIH
Thank you very much for your answer Erick. My apologies for the previous email; my problem is that I don´t speak English very well and I´m new in the world of mailing list. The problem is that I´m indexing emails throw Data import Handler using Gmail with imaps; I do this for search on email list in the future. The emails are indexed partiality and I can´t found the problem of why don´t index all of the emails. Below I show you de configuration of my DIH. The date of my emails is later to “2010-01-01 00:00:00”. I´ve done a full import and no errors were found, but in the status I saw that was added 28 documents, and in the console, I found 35 messanges. Below I show you the status screen, first, and then part of the console output. Status: 0 1 data-config.xml status idle 0 28 0 2011-03-22 15:55:12 Indexing completed. Added/Updated: 28 documents. Deleted 0 documents. 2011-03-22 15:55:20 2011-03-22 15:55:20 28 0:0:8.520 This response format is experimental. It is likely to change in the future. …” Mar 22, 2011 3:55:14 PM org.apache.solr.handler.dataimport.MailEntityProcessor connectToMailBox INFO: Connected to mailbox Mar 22, 2011 3:55:15 PM org.apache.solr.handler.dataimport.MailEntityProcessor$FolderIterator next INFO: Opened folder : inbox Mar 22, 2011 3:55:15 PM org.apache.solr.handler.dataimport.MailEntityProcessor$FolderIterator next INFO: Added its children to list : Mar 22, 2011 3:55:15 PM org.apache.solr.handler.dataimport.MailEntityProcessor$FolderIterator next INFO: NO children : Mar 22, 2011 3:55:16 PM org.apache.solr.handler.dataimport.MailEntityProcessor$MessageIterator INFO: Total messages : 35 Mar 22, 2011 3:55:16 PM org.apache.solr.handler.dataimport.MailEntityProcessor$MessageIterator INFO: Search criteria applied. Batching disabled Mar 22, 2011 3:55:19 PM org.apache.solr.handler.dataimport.DocBuilder finish INFO: Import completed successfully “… Regards, Matias. 2011/3/22 Erick Erickson > Not unless you provide a lot more data. Have you > inspected the Solr logs and seen any anomalies? > > Please review: > http://wiki.apache.org/solr/UsingMailingLists > > Best > Erick > > On Mon, Mar 21, 2011 at 3:56 PM, Matias Alonso > wrote: > > Hi, > > > > > > I’m using Data Import Handler for index emails. > > > > The problem is that nota ll the emails was indexed When I do a full > import. > > > > Someone have any idea? > > > > > > Regards, > > > > -- > > Matias. > > >
Re: email - DIH
Hi Gora, I appreciate your help. I´ve done what you said but if omit "fetchMailsSince" "full-import" doesn´t work. This´s the messenge on the console ..."SEVERE: Full Import failed:org.apache.solr.handler.dataimport.DataImportHandlerException: Invalid value for fetchMailSince: Processing Document # 1"... The email I use for this was created at the begin of this month. Regards, Matias. 2011/3/23 Gora Mohanty > On Tue, Mar 22, 2011 at 9:38 PM, Matias Alonso > wrote: > [...] > > The problem is that I´m indexing emails throw Data import Handler using > > Gmail with imaps; I do this for search on email list in the future. The > > emails are indexed partiality and I can´t found the problem of why don´t > > index all of the emails. > [...] > > I´ve done a full import and no errors were found, but in the status I saw > > that was added 28 documents, and in the console, I found 35 messanges. > [...] > > > INFO: Total messages : 35 > > > > Mar 22, 2011 3:55:16 PM > > org.apache.solr.handler.dataimport.MailEntityProcessor$MessageIterator > > > > > > INFO: Search criteria applied. Batching disabled > [...] > > The above seems to indicate that the MailEntityProcessor does find > all 35 messages, but indexes only 28. Are you sure that all 35 are > since 2010-01-01 00:00:00? Could you try without fetchMailsSince? > > Regards, > Gora > -- Matias.
Re: email - DIH
Hi Gora, Also, all the emails were received after that date. Regards, Matias. 2011/3/23 Gora Mohanty > On Tue, Mar 22, 2011 at 9:38 PM, Matias Alonso > wrote: > [...] > > The problem is that I´m indexing emails throw Data import Handler using > > Gmail with imaps; I do this for search on email list in the future. The > > emails are indexed partiality and I can´t found the problem of why don´t > > index all of the emails. > [...] > > I´ve done a full import and no errors were found, but in the status I saw > > that was added 28 documents, and in the console, I found 35 messanges. > [...] > > > INFO: Total messages : 35 > > > > Mar 22, 2011 3:55:16 PM > > org.apache.solr.handler.dataimport.MailEntityProcessor$MessageIterator > > > > > > INFO: Search criteria applied. Batching disabled > [...] > > The above seems to indicate that the MailEntityProcessor does find > all 35 messages, but indexes only 28. Are you sure that all 35 are > since 2010-01-01 00:00:00? Could you try without fetchMailsSince? > > Regards, > Gora > -- Matias.
