Re: Can find solr jars in WEB-INF/lib directory
On Mar 4, 2009, at 12:47 AM, The Flight Captain wrote: Thanks for the DirectSolrConnection tip, that eliminated the need for parsing the XML as a string, and all that code need to instantiate a container. I am now Using SolrCore.getSolrCore(). This method is deprecated though, what is the standard way of instaniating a SoreCore to pass it as a parameter to DirectSolrConnection? The DirectSolrConnection( String instanceDir, String dataDir, String loggingPath ) constructor has this code: CoreContainer cores = new CoreContainer(new SolrResourceLoader(instanceDir)); SolrConfig solrConfig = new SolrConfig(instanceDir, SolrConfig.DEFAULT_CONF_FILE, null); CoreDescriptor dcore = new CoreDescriptor(cores, "", solrConfig.getResourceLoader().getInstanceDir()); IndexSchema indexSchema = new IndexSchema(solrConfig, instanceDir+"/conf/schema.xml", null); core = new SolrCore( null, dataDir, solrConfig, indexSchema, dcore); Erik
Re: Search with Date field
Normally you will need to filer by dates... you can do it in two ways... http://fq=date_field[NOW-7DAY TO NOW] http://...fq=date_field[2007-01-01T21:21:21 TO 2009-01-01T13:13:13] I strongly recomend you to use /DAY in the first example. The search will be faster as it is omitting hour-min-sec. http://fq=date_field[NOW/DAY-7DAY TO NOW/DAY] Is this what you where asking for? solruser wrote: > > Can you reply with example of solr request? > > Amit > > On Mar 3, 2009, at 11:08 PM, dabboo wrote: > >> >> Hi, >> >> Date search is not happening in solr. I am giving the field name and >> date >> value as the request but it is not returning any value. >> >> Please suggest. >> >> Thanks, >> Amit Garg >> -- >> View this message in context: >> http://www.nabble.com/Search-with-Date-field-tp22324779p22324779.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > -- View this message in context: http://www.nabble.com/Search-with-Date-field-tp22324779p22326871.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Very long commit time.
Great,
It went down to less than 10 secs now :)
What I don't really understand is that my autowarmCount were pretty
low ( like 128 ) and still the autowarming of the caches were very
slow.
Can you explain more why it can be that slow ?
Cheers !
Jerome.
On Tue, Mar 3, 2009 at 8:00 PM, Yonik Seeley wrote:
> Looks like cache autowarming.
> If you have statically defined warming queries in solrconfig.xml, you
> could try setting autowarmCount=0 for all the caches.
>
> -Yonik
> http://www.lucidimagination.com
>
>
> On Tue, Mar 3, 2009 at 2:37 PM, Jérôme Etévé wrote:
>> Dear solr fans,
>>
>> I have a solr index of roughly 8M docs and I have here a little
>> problem when I commit some insertion into it.
>>
>> The insert itself is very fast, but my commit takes 163 seconds.
>>
>> Here's the solr trace the commit leaves:
>>
>> INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true)
>> 03-Mar-2009 20:20:35 org.apache.solr.search.SolrIndexSearcher
>> INFO: Opening searc...@7de212f9 main
>> 03-Mar-2009 20:20:35 org.apache.solr.update.DirectUpdateHandler2 commit
>> INFO: end_commit_flush
>> 03-Mar-2009 20:20:35 org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming searc...@7de212f9 main from searc...@732d8b11 main
>>
>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=16,evictions=0,size=16,warmupTime=71641,cumulative_lookups=90,cumulative_hits=68,cumulative_hitratio=0.75,cumulative_inserts=22,cumulative_evictions=0}
>> 03-Mar-2009 20:21:52 org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for searc...@7de212f9 main
>>
>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=16,evictions=0,size=16,warmupTime=76905,cumulative_lookups=90,cumulative_hits=68,cumulative_hitratio=0.75,cumulative_inserts=22,cumulative_evictions=0}
>> 03-Mar-2009 20:21:52 org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming searc...@7de212f9 main from searc...@732d8b11 main
>>
>> queryResultCache{lookups=24,hits=24,hitratio=1.00,inserts=32,evictions=0,size=32,warmupTime=82406,cumulative_lookups=6310,cumulative_hits=268,cumulative_hitratio=0.04,cumulative_inserts=6041,cumulative_evictions=5522}
>> 03-Mar-2009 20:23:17 org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for searc...@7de212f9 main
>>
>> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=32,evictions=0,size=32,warmupTime=85591,cumulative_lookups=6310,cumulative_hits=268,cumulative_hitratio=0.04,cumulative_inserts=6041,cumulative_evictions=5522}
>> 03-Mar-2009 20:23:17 org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming searc...@7de212f9 main from searc...@732d8b11 main
>>
>> documentCache{lookups=720,hits=710,hitratio=0.98,inserts=40,evictions=0,size=40,warmupTime=0,cumulative_lookups=415308,cumulative_hits=283661,cumulative_hitratio=0.68,cumulative_inserts=131647,cumulative_evictions=131105}
>> 03-Mar-2009 20:23:17 org.apache.solr.search.SolrIndexSearcher warm
>> INFO: autowarming result for searc...@7de212f9 main
>>
>> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=415308,cumulative_hits=283661,cumulative_hitratio=0.68,cumulative_inserts=131647,cumulative_evictions=131105}
>> 03-Mar-2009 20:23:17 org.apache.solr.core.QuerySenderListener newSearcher
>> INFO: QuerySenderListener sending requests to searc...@7de212f9 main
>>
>> // Then the few warm up queries defined in solrconfig.xml
>>
>> INFO: Closing searc...@732d8b11 main
>>
>> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=16,evictions=0,size=16,warmupTime=71641,cumulative_lookups=90,cumulative_hits=68,cumulative_hitratio=0.75,cumulative_inserts=22,cumulative_evictions=0}
>>
>> queryResultCache{lookups=24,hits=24,hitratio=1.00,inserts=32,evictions=0,size=32,warmupTime=82406,cumulative_lookups=6310,cumulative_hits=268,cumulative_hitratio=0.04,cumulative_inserts=6041,cumulative_evictions=5522}
>>
>> documentCache{lookups=720,hits=710,hitratio=0.98,inserts=40,evictions=0,size=40,warmupTime=0,cumulative_lookups=415308,cumulative_hits=283661,cumulative_hitratio=0.68,cumulative_inserts=131647,cumulative_evictions=131105}
>> 03-Mar-2009 20:23:18 org.apache.solr.update.processor.LogUpdateProcessor
>> finish
>> INFO: {commit=} 0 163189
>> 03-Mar-2009 20:23:18 org.apache.solr.core.SolrCore execute
>> INFO: [jobs] webapp=/cjsolr path=/update/ params={} status=0 QTime=163189
>>
>>
>> I'm sure I'm doing something wrong. Does this 163 seconds commit time
>> have to do with the commit parameters :
>> (optimize=false,waitFlush=false,waitSearcher=true) ??
>>
>> Thanks for any help.
>>
>> Cheers !!
>>
>> Jerome.
>>
>> --
>> Jerome Eteve.
>>
>> Chat with me live at http://www.eteve.net
>>
>> jer...@eteve.net
>>
>
--
Jerome Eteve.
Chat with me live at http://www.eteve.net
jer...@eteve.net
Re: datadir issue for multicore on latest trunk
it also ignore dataDir directive in solr.xml, in fact adding: doesn't change the behavior. this seems a bug introduced somewhere after 2nd february any clue? On Tue, Mar 3, 2009 at 5:56 PM, Walter Ferrara wrote: > there is a strange behavior which seems to affect hudson today (March 3rd) > build but not (for example) hudson February 2th build. > Basically when I start the multicore enviroment, it just create datadir in > the current path. > To replicate: > 1. download latest trunk > 2. go to example directory > > $ ls > README.txt example-DIH exampledocs logs solr start.jar > work > etc exampleAnalysis lib multicore start.bat webapps > > $ java -Dsolr.solr.home=multicore -jar start.jar > (then kill/sleep the process) > > $ ls > README.txt etc exampledocs multicore start.jar > core0 example-DIH lib solr webapps > core1 exampleAnalysis logs start.bat work > > you see core0 and core1 directory where they should not be :-); solr-1041 > doesn't fix in this case. > > ciao, > Walter > >
Re: datadir issue for multicore on latest trunk
> is not honored automatically. the tag has to be present in solrconfig.xml and it should use this value. but you can specify it as follows then it should be fine. can you just paste the log messages as solr starts --Noble On Wed, Mar 4, 2009 at 4:15 PM, Walter Ferrara wrote: > it also ignore dataDir directive in solr.xml, in fact adding: > > > > doesn't change the behavior. > > this seems a bug introduced somewhere after 2nd february > any clue? > > > On Tue, Mar 3, 2009 at 5:56 PM, Walter Ferrara wrote: > >> there is a strange behavior which seems to affect hudson today (March 3rd) >> build but not (for example) hudson February 2th build. >> Basically when I start the multicore enviroment, it just create datadir in >> the current path. >> To replicate: >> 1. download latest trunk >> 2. go to example directory >> >> $ ls >> README.txt example-DIH exampledocs logs solr start.jar >> work >> etc exampleAnalysis lib multicore start.bat webapps >> >> $ java -Dsolr.solr.home=multicore -jar start.jar >> (then kill/sleep the process) >> >> $ ls >> README.txt etc exampledocs multicore start.jar >> core0 example-DIH lib solr webapps >> core1 exampleAnalysis logs start.bat work >> >> you see core0 and core1 directory where they should not be :-); solr-1041 >> doesn't fix in this case. >> >> ciao, >> Walter >> >> > -- --Noble Paul
Custom Request Handler
Hi, Can someone please tell me if we can write our own Custom Request Handler and Custom Solr Core? I want to call execute method of custom solrcore class instead of SolrCore class. Actually, I have to modify getQParser method of SolrCore class, so that it shud start returning my custom plugin. Please suggest. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Custom-Request-Handler-tp22328365p22328365.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: datadir issue for multicore on latest trunk
using: doesn't work either here the output: 2009-03-04 12:50:54.890::INFO: Logging to STDERR via org.mortbay.log.StdErrLog 2009-03-04 12:50:54.968::INFO: jetty-6.1.3 4-mar-2009 12.50.55 org.apache.solr.servlet.SolrDispatchFilter init INFO: SolrDispatchFilter.init() 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: JNDI not configured for solr (NoInitialContextEx) 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader locateInstanceDir INFO: using system property solr.solr.home: multicore 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer$Initializer initialize INFO: looking for solr.xml: d:\DEV\apache-solr-2009-03-03_08-06-53\example\multicore\solr.xml 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader INFO: Solr home set to 'multicore/' 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader INFO: Solr home set to 'multicore\core0/' 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader 4-mar-2009 12.50.55 org.apache.solr.core.SolrConfig INFO: Loaded SolrConfig: solrconfig.xml 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: Reading Solr Schema 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: Schema name=example core zero 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created string: org.apache.solr.schema.StrField 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: default search field is name 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: query parser default operator is OR 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: unique key field: id 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore INFO: [core0] Opening new SolrCore at multicore\core0/, dataDir=core0//multicore/core0/ 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore INFO: JMX monitoring not detected for core: core0 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener INFO: [core0] Searching for listeners: //listen...@event="firstSearcher"] 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener INFO: [core0] Searching for listeners: //listen...@event="newSearcher"] 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore initIndex AVVERTENZA: [core0] Solr index directory 'core0\multicore\core0\index' doesn't exist. Creating new index... 4-mar-2009 12.50.55 org.apache.solr.update.SolrIndexWriter getDirectory AVVERTENZA: No lockType configured for core0//multicore/core0/index/ assuming 'simple' 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created standard: org.apache.solr.handler.StandardRequestHandler 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created /update: org.apache.solr.handler.XmlUpdateRequestHandler 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers 4-mar-2009 12.50.55 org.apache.solr.search.SolrIndexSearcher INFO: Opening searc...@1e57e8f main 4-mar-2009 12.50.55 org.apache.solr.update.DirectUpdateHandler2$CommitTracker INFO: AutoCommit: disabled 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform INFO: Adding component:org.apache.solr.handler.component.querycompon...@19a32e0 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform INFO: Adding component:org.apache.solr.handler.component.facetcompon...@8238f4 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform INFO: Adding component:org.apache.solr.handler.component.morelikethiscompon...@16925b0 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform INFO: Adding component:org.apache.solr.handler.component.highlightcompon...@297ffb 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform INFO: Adding component:org.apache.solr.handler.component.statscompon...@914f6a 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform INFO: Adding debug component:org.apache.solr.handler.component.debugcompon...@1f4cbee 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer register INFO: registering core: core0 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore registerSearcher INFO: [core0] Registered new searcher searc...@1e57e8f main 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader INFO: Solr home set to 'multicore\core1/' 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader createClassLoader INFO: Reusing parent classloader 4-mar-2009 12.50.55 org.apache.solr.core.SolrConfig INFO: Loaded SolrConfig: solrconfig.xml 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: Reading Solr Schema 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema INFO: Schema name=example cor
Re: datadir issue for multicore on latest trunk
On Wed, Mar 4, 2009 at 5:24 PM, Walter Ferrara wrote: > using: > > > > > doesn't work either dataDir="/multicore/core0" means the path is absolute. where did it create? > > here the output: > > 2009-03-04 12:50:54.890::INFO: Logging to STDERR via > org.mortbay.log.StdErrLog > 2009-03-04 12:50:54.968::INFO: jetty-6.1.3 > 4-mar-2009 12.50.55 org.apache.solr.servlet.SolrDispatchFilter init > INFO: SolrDispatchFilter.init() > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > locateInstanceDir > INFO: JNDI not configured for solr (NoInitialContextEx) > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > locateInstanceDir > INFO: using system property solr.solr.home: multicore > 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer$Initializer > initialize > INFO: looking for solr.xml: > d:\DEV\apache-solr-2009-03-03_08-06-53\example\multicore\solr.xml > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > INFO: Solr home set to 'multicore/' > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > createClassLoader > INFO: Reusing parent classloader > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > INFO: Solr home set to 'multicore\core0/' > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > createClassLoader > INFO: Reusing parent classloader > 4-mar-2009 12.50.55 org.apache.solr.core.SolrConfig > INFO: Loaded SolrConfig: solrconfig.xml > 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > INFO: Reading Solr Schema > 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > INFO: Schema name=example core zero > 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created string: org.apache.solr.schema.StrField > 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > INFO: default search field is name > 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > INFO: query parser default operator is OR > 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > INFO: unique key field: id > 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore > INFO: [core0] Opening new SolrCore at multicore\core0/, > dataDir=core0//multicore/core0/ > 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore > INFO: JMX monitoring not detected for core: core0 > 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener > INFO: [core0] Searching for listeners: //listen...@event="firstSearcher"] > 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener > INFO: [core0] Searching for listeners: //listen...@event="newSearcher"] > 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore initIndex > AVVERTENZA: [core0] Solr index directory 'core0\multicore\core0\index' > doesn't exist. Creating new index... > 4-mar-2009 12.50.55 org.apache.solr.update.SolrIndexWriter getDirectory > AVVERTENZA: No lockType configured for core0//multicore/core0/index/ > assuming 'simple' > 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created standard: org.apache.solr.handler.StandardRequestHandler > 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created /update: org.apache.solr.handler.XmlUpdateRequestHandler > 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers > 4-mar-2009 12.50.55 org.apache.solr.search.SolrIndexSearcher > INFO: Opening searc...@1e57e8f main > 4-mar-2009 12.50.55 > org.apache.solr.update.DirectUpdateHandler2$CommitTracker > INFO: AutoCommit: disabled > 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform > INFO: Adding > component:org.apache.solr.handler.component.querycompon...@19a32e0 > 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform > INFO: Adding > component:org.apache.solr.handler.component.facetcompon...@8238f4 > 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform > INFO: Adding > component:org.apache.solr.handler.component.morelikethiscompon...@16925b0 > 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform > INFO: Adding > component:org.apache.solr.handler.component.highlightcompon...@297ffb > 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform > INFO: Adding > component:org.apache.solr.handler.component.statscompon...@914f6a > 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform > INFO: Adding debug > component:org.apache.solr.handler.component.debugcompon...@1f4cbee > 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer register > INFO: registering core: core0 > 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore registerSearcher > INFO: [core0] Registered new searcher searc...@1e57e8f main > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > INFO: Solr home set to 'multicore\core1/' > 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > crea
Re: datadir issue for multicore on latest trunk
looks like a bug. must reopen the issue On Wed, Mar 4, 2009 at 5:26 PM, Noble Paul നോബിള് नोब्ळ् wrote: > On Wed, Mar 4, 2009 at 5:24 PM, Walter Ferrara wrote: >> using: >> >> >> >> >> doesn't work either > > dataDir="/multicore/core0" means the path is absolute. > where did it create? > >> >> here the output: >> >> 2009-03-04 12:50:54.890::INFO: Logging to STDERR via >> org.mortbay.log.StdErrLog >> 2009-03-04 12:50:54.968::INFO: jetty-6.1.3 >> 4-mar-2009 12.50.55 org.apache.solr.servlet.SolrDispatchFilter init >> INFO: SolrDispatchFilter.init() >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> locateInstanceDir >> INFO: JNDI not configured for solr (NoInitialContextEx) >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> locateInstanceDir >> INFO: using system property solr.solr.home: multicore >> 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer$Initializer >> initialize >> INFO: looking for solr.xml: >> d:\DEV\apache-solr-2009-03-03_08-06-53\example\multicore\solr.xml >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> INFO: Solr home set to 'multicore/' >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> createClassLoader >> INFO: Reusing parent classloader >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> INFO: Solr home set to 'multicore\core0/' >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> createClassLoader >> INFO: Reusing parent classloader >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrConfig >> INFO: Loaded SolrConfig: solrconfig.xml >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> INFO: Reading Solr Schema >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> INFO: Schema name=example core zero >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load >> INFO: created string: org.apache.solr.schema.StrField >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> INFO: default search field is name >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> INFO: query parser default operator is OR >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> INFO: unique key field: id >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore >> INFO: [core0] Opening new SolrCore at multicore\core0/, >> dataDir=core0//multicore/core0/ >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore >> INFO: JMX monitoring not detected for core: core0 >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener >> INFO: [core0] Searching for listeners: //listen...@event="firstSearcher"] >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener >> INFO: [core0] Searching for listeners: //listen...@event="newSearcher"] >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore initIndex >> AVVERTENZA: [core0] Solr index directory 'core0\multicore\core0\index' >> doesn't exist. Creating new index... >> 4-mar-2009 12.50.55 org.apache.solr.update.SolrIndexWriter getDirectory >> AVVERTENZA: No lockType configured for core0//multicore/core0/index/ >> assuming 'simple' >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load >> INFO: created standard: org.apache.solr.handler.StandardRequestHandler >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load >> INFO: created /update: org.apache.solr.handler.XmlUpdateRequestHandler >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader load >> INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers >> 4-mar-2009 12.50.55 org.apache.solr.search.SolrIndexSearcher >> INFO: Opening searc...@1e57e8f main >> 4-mar-2009 12.50.55 >> org.apache.solr.update.DirectUpdateHandler2$CommitTracker >> INFO: AutoCommit: disabled >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform >> INFO: Adding >> component:org.apache.solr.handler.component.querycompon...@19a32e0 >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform >> INFO: Adding >> component:org.apache.solr.handler.component.facetcompon...@8238f4 >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform >> INFO: Adding >> component:org.apache.solr.handler.component.morelikethiscompon...@16925b0 >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform >> INFO: Adding >> component:org.apache.solr.handler.component.highlightcompon...@297ffb >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform >> INFO: Adding >> component:org.apache.solr.handler.component.statscompon...@914f6a >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler inform >> INFO: Adding debug >> component:org.apache.solr.handler.component.debugcompon...@1f4cbee >> 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer register >> INFO: registering core: core0 >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore registerSearcher >> INFO: [core0] Registere
Stemming in Solr
Hi, I am trying to implement stemming in solr. If user searches for walk then all the records which have walk, walking, walks, walked etc should display. Please suggest. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Stemming-in-Solr-tp22328850p22328850.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Stemming in Solr
Hi, did you check Snowball stemmers (http://snowball.tartarus.org/)? Regards, Lukas On Wed, Mar 4, 2009 at 1:12 PM, dabboo wrote: > > Hi, > > I am trying to implement stemming in solr. If user searches for walk then > all the records which have walk, walking, walks, walked etc should display. > > Please suggest. > > Thanks, > Amit Garg > -- > View this message in context: > http://www.nabble.com/Stemming-in-Solr-tp22328850p22328850.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- http://blog.lukas-vlcek.com/
Re: Stemming in Solr
May be you can also check http://wiki.apache.org/solr/AnalyzersTokenizersTokenFiltersStemming is discussed there... On Wed, Mar 4, 2009 at 1:18 PM, Lukáš Vlček wrote: > Hi, > did you check Snowball stemmers (http://snowball.tartarus.org/)? > > Regards, > Lukas > > > On Wed, Mar 4, 2009 at 1:12 PM, dabboo wrote: > >> >> Hi, >> >> I am trying to implement stemming in solr. If user searches for walk then >> all the records which have walk, walking, walks, walked etc should >> display. >> >> Please suggest. >> >> Thanks, >> Amit Garg >> -- >> View this message in context: >> http://www.nabble.com/Stemming-in-Solr-tp22328850p22328850.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > http://blog.lukas-vlcek.com/ > -- http://blog.lukas-vlcek.com/
Re: datadir issue for multicore on latest trunk
tried with but no luck, the dataDir parameter seems ignored, no matter what is written there On Wed, Mar 4, 2009 at 12:58 PM, Noble Paul നോബിള് नोब्ळ् < noble.p...@gmail.com> wrote: > looks like a bug. must reopen the issue > > On Wed, Mar 4, 2009 at 5:26 PM, Noble Paul നോബിള് नोब्ळ् > wrote: > > On Wed, Mar 4, 2009 at 5:24 PM, Walter Ferrara > wrote: > >> using: > >> > >> > >> > >> > >> doesn't work either > > > > dataDir="/multicore/core0" means the path is absolute. > > where did it create? > > > >> > >> here the output: > >> > >> 2009-03-04 12:50:54.890::INFO: Logging to STDERR via > >> org.mortbay.log.StdErrLog > >> 2009-03-04 12:50:54.968::INFO: jetty-6.1.3 > >> 4-mar-2009 12.50.55 org.apache.solr.servlet.SolrDispatchFilter init > >> INFO: SolrDispatchFilter.init() > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > >> locateInstanceDir > >> INFO: JNDI not configured for solr (NoInitialContextEx) > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > >> locateInstanceDir > >> INFO: using system property solr.solr.home: multicore > >> 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer$Initializer > >> initialize > >> INFO: looking for solr.xml: > >> d:\DEV\apache-solr-2009-03-03_08-06-53\example\multicore\solr.xml > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > >> INFO: Solr home set to 'multicore/' > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > >> createClassLoader > >> INFO: Reusing parent classloader > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > >> INFO: Solr home set to 'multicore\core0/' > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader > >> createClassLoader > >> INFO: Reusing parent classloader > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrConfig > >> INFO: Loaded SolrConfig: solrconfig.xml > >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > >> INFO: Reading Solr Schema > >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > >> INFO: Schema name=example core zero > >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader > load > >> INFO: created string: org.apache.solr.schema.StrField > >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > >> INFO: default search field is name > >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > >> INFO: query parser default operator is OR > >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema > >> INFO: unique key field: id > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore > >> INFO: [core0] Opening new SolrCore at multicore\core0/, > >> dataDir=core0//multicore/core0/ > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore > >> INFO: JMX monitoring not detected for core: core0 > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener > >> INFO: [core0] Searching for listeners: > //listen...@event="firstSearcher"] > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener > >> INFO: [core0] Searching for listeners: //listen...@event="newSearcher"] > >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore initIndex > >> AVVERTENZA: [core0] Solr index directory 'core0\multicore\core0\index' > >> doesn't exist. Creating new index... > >> 4-mar-2009 12.50.55 org.apache.solr.update.SolrIndexWriter getDirectory > >> AVVERTENZA: No lockType configured for core0//multicore/core0/index/ > >> assuming 'simple' > >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader > load > >> INFO: created standard: org.apache.solr.handler.StandardRequestHandler > >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader > load > >> INFO: created /update: org.apache.solr.handler.XmlUpdateRequestHandler > >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader > load > >> INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers > >> 4-mar-2009 12.50.55 org.apache.solr.search.SolrIndexSearcher > >> INFO: Opening searc...@1e57e8f main > >> 4-mar-2009 12.50.55 > >> org.apache.solr.update.DirectUpdateHandler2$CommitTracker > >> INFO: AutoCommit: disabled > >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler > inform > >> INFO: Adding > >> component:org.apache.solr.handler.component.querycompon...@19a32e0 > >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler > inform > >> INFO: Adding > >> component:org.apache.solr.handler.component.facetcompon...@8238f4 > >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler > inform > >> INFO: Adding > >> > component:org.apache.solr.handler.component.morelikethiscompon...@16925b0 > >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler > inform > >> INFO: Adding > >> component:org.apache.solr.handler.component.highlightcompon...@297ffb > >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler > inform > >> INFO: Adding > >> component:org.apach
Re: Custom Request Handler
Can you explain the problem you are having? Is there a bug in the SolrCore that you are working around? Or, are you just trying to plugin your own Query Parser? If that is the case, there is a way to do that already. You might find http://wiki.apache.org/solr/ SolrPlugins helpful. -Grant On Mar 4, 2009, at 6:42 AM, dabboo wrote: Hi, Can someone please tell me if we can write our own Custom Request Handler and Custom Solr Core? I want to call execute method of custom solrcore class instead of SolrCore class. Actually, I have to modify getQParser method of SolrCore class, so that it shud start returning my custom plugin. Please suggest. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Custom-Request-Handler-tp22328365p22328365.html Sent from the Solr - User mailing list archive at Nabble.com. -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene: http://www.lucidimagination.com/search
Re: Very long commit time.
On Wed, Mar 4, 2009 at 5:25 AM, Jérôme Etévé wrote: > Great, > > It went down to less than 10 secs now :) > What I don't really understand is that my autowarmCount were pretty > low ( like 128 ) and still the autowarming of the caches were very > slow. > > Can you explain more why it can be that slow ? One possibility is a lack of physical memory available to the OS for caching reads on both the old index and the new index. This would cause all of the queries to be slower if they ended up doing real disk IO for each query/filter being warmed. -Yonik http://www.lucidimagination.com
Tomcat5 + Solr. Problems in deploying the Webapp
Hi all, I am trying to setup a solr instance with Tomcat5 on a Fedora10 machine. Here is what I did, 1.) Copy the apache-solr-nightly.war to webapps/solr.war 2.) Set solr.solr.home in tomcat.conf 3.) Use the Manager interface of tomcat to deploy the webapp But, while doing so, I get the following exceptions. Mar 4, 2009 6:55:09 PM org.apache.catalina.core.StandardContext filterStart SEVERE: Exception starting filter SolrRequestFilter java.lang.NoClassDefFoundError: Could not initialize class org.apache.solr.core.SolrConfig at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:76) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4222) at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1173) at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:549) at org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:105) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:636) What am I missing? If it matters I am running the nightly build from March 3 2009. Thanks and Regards Sudharshan S Blog : http://www.sudharsh.wordpress.com IRC : Sup3rkiddo @ Freenode, Gimpnet
How to search the database tables using solr.
Hi, I am working in a software concern. We are having some R&D base work like making use of solr search tool for searching our Oracle database content. I read through the documnets provided in the site, there I found the solr supports only the document search and we have to provide the document in the xml with the whole content that we need to search. But Is there any way to index our database content and search the indexed database content. Any help will be appriciated. Thanks.
Re: How to search the database tables using solr.
On Wed, Mar 4, 2009 at 7:32 PM, Radha C. wrote: > Hi, > > I am working in a software concern. We are having some R&D base work like > making use of solr search tool for searching our Oracle database > content. I read through the documnets provided in the site, there I found > the solr supports only the document search and we have to provide the > document in the xml with the whole content that we need to search. But Is > there any way to index our database content and search the indexed database > content. > Take a look at DataImportHandler. http://wiki.apache.org/solr/DataImportHandler -- Regards, Shalin Shekhar Mangar.
MoreLikeThis filtering
Is it possible to filter similarities found by the MLT component/ handler? Something like mlt.fq=site_id:86? We have 32 cores in our Solr install, and some of those cores have up to 8 sites indexed in them. Typically those cores will have one very large site with a few hundred thousand indexed documents, and lots of small sites with significantly less documents indexed. We're looking to implement a MLT component for our sites but want the similar stories to be only for a specific site (not all sites in the core). Is there a way to do something like this, or will we have to make mods (I'm not seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? /solr/dsteiger/mlt?q=story_id:188665+AND+site_id:86&mlt.fq=site_id:86 (We have all all of our other defaults set up in the handler config.) Thanks. --- Doug Steigerwald Software Developer McClatchy Interactive dsteigerw...@mcclatchyinteractive.com
Re: MoreLikeThis filtering
Doug, does the good old 'fq' not work with MLT? It should... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Doug Steigerwald > To: solr-user@lucene.apache.org > Sent: Wednesday, March 4, 2009 9:20:40 AM > Subject: MoreLikeThis filtering > > Is it possible to filter similarities found by the MLT component/handler? > Something like mlt.fq=site_id:86? > > We have 32 cores in our Solr install, and some of those cores have up to 8 > sites > indexed in them. Typically those cores will have one very large site with a > few > hundred thousand indexed documents, and lots of small sites with > significantly > less documents indexed. > > We're looking to implement a MLT component for our sites but want the similar > stories to be only for a specific site (not all sites in the core). > > Is there a way to do something like this, or will we have to make mods (I'm > not > seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? > > /solr/dsteiger/mlt?q=story_id:188665+AND+site_id:86&mlt.fq=site_id:86 > > (We have all all of our other defaults set up in the handler config.) > > Thanks. > --- > Doug Steigerwald > Software Developer > McClatchy Interactive > dsteigerw...@mcclatchyinteractive.com
RE: How to search the database tables using solr.
Thanks Shalin, We just stepped on solr. This information is very much useful for me. But before that I want some clear details about where to start.. I want to test this in my local environment, so I need some basic information about how to start using this ( database and solr ). Do you have some information on this? Thanks in advance. _ From: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Sent: Wednesday, March 04, 2009 7:48 PM To: solr-user@lucene.apache.org; cra...@ceiindia.com Subject: Re: How to search the database tables using solr. On Wed, Mar 4, 2009 at 7:32 PM, Radha C. wrote: Hi, I am working in a software concern. We are having some R&D base work like making use of solr search tool for searching our Oracle database content. I read through the documnets provided in the site, there I found the solr supports only the document search and we have to provide the document in the xml with the whole content that we need to search. But Is there any way to index our database content and search the indexed database content. Take a look at DataImportHandler. http://wiki.apache.org/solr/DataImportHandler -- Regards, Shalin Shekhar Mangar.
Re: MoreLikeThis filtering
'fq' seems to only work with finding the documents with your original query, not for filtering the similar documents. Doug On Mar 4, 2009, at 9:28 AM, Otis Gospodnetic wrote: Doug, does the good old 'fq' not work with MLT? It should... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Doug Steigerwald To: solr-user@lucene.apache.org Sent: Wednesday, March 4, 2009 9:20:40 AM Subject: MoreLikeThis filtering Is it possible to filter similarities found by the MLT component/ handler? Something like mlt.fq=site_id:86? We have 32 cores in our Solr install, and some of those cores have up to 8 sites indexed in them. Typically those cores will have one very large site with a few hundred thousand indexed documents, and lots of small sites with significantly less documents indexed. We're looking to implement a MLT component for our sites but want the similar stories to be only for a specific site (not all sites in the core). Is there a way to do something like this, or will we have to make mods (I'm not seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? /solr/dsteiger/mlt?q=story_id:188665+AND+site_id:86&mlt.fq=site_id:86 (We have all all of our other defaults set up in the handler config.) Thanks. --- Doug Steigerwald Software Developer McClatchy Interactive dsteigerw...@mcclatchyinteractive.com
Re: MoreLikeThis filtering
Hm. I checked out a clean Solr 1.3.0 and indexed the example docs and set up a simple MLT handler the example queries on the Wiki work fine (fq can filter out docs). Our build has a slight change to QueryComponent so another query isn't done when we use localsolr+field collapsing, but that change doesn't look like it would make a difference. It just conditionally sets rb.setNeedDocSet() to true or false. Will run some tests on a clean fresh build of Solr to see if it's our build. Doug On Mar 4, 2009, at 9:28 AM, Otis Gospodnetic wrote: Doug, does the good old 'fq' not work with MLT? It should... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Doug Steigerwald To: solr-user@lucene.apache.org Sent: Wednesday, March 4, 2009 9:20:40 AM Subject: MoreLikeThis filtering Is it possible to filter similarities found by the MLT component/ handler? Something like mlt.fq=site_id:86? We have 32 cores in our Solr install, and some of those cores have up to 8 sites indexed in them. Typically those cores will have one very large site with a few hundred thousand indexed documents, and lots of small sites with significantly less documents indexed. We're looking to implement a MLT component for our sites but want the similar stories to be only for a specific site (not all sites in the core). Is there a way to do something like this, or will we have to make mods (I'm not seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? /solr/dsteiger/mlt?q=story_id:188665+AND+site_id:86&mlt.fq=site_id:86 (We have all all of our other defaults set up in the handler config.) Thanks. --- Doug Steigerwald Software Developer McClatchy Interactive dsteigerw...@mcclatchyinteractive.com
Re: MoreLikeThis filtering
Sorry. The examples on the wiki aren't working with the 'fq' to filter the similarities. It just filters the actual queries. http://localhost:8983/solr/mlt?q=id:SP2514N&mlt.fl=manu,cat&mlt.mindf=1&mlt.mintf=1&fq=popularity:6&mlt.displayTerms=details&mlt=true The popularity of the doc found is 6, and trying to use 'fq=popularity: 6' brings back similarities with a popularity other than 6. Doug On Mar 4, 2009, at 10:39 AM, Doug Steigerwald wrote: Hm. I checked out a clean Solr 1.3.0 and indexed the example docs and set up a simple MLT handler the example queries on the Wiki work fine (fq can filter out docs). Our build has a slight change to QueryComponent so another query isn't done when we use localsolr +field collapsing, but that change doesn't look like it would make a difference. It just conditionally sets rb.setNeedDocSet() to true or false. Will run some tests on a clean fresh build of Solr to see if it's our build. Doug On Mar 4, 2009, at 9:28 AM, Otis Gospodnetic wrote: Doug, does the good old 'fq' not work with MLT? It should... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Doug Steigerwald To: solr-user@lucene.apache.org Sent: Wednesday, March 4, 2009 9:20:40 AM Subject: MoreLikeThis filtering Is it possible to filter similarities found by the MLT component/ handler? Something like mlt.fq=site_id:86? We have 32 cores in our Solr install, and some of those cores have up to 8 sites indexed in them. Typically those cores will have one very large site with a few hundred thousand indexed documents, and lots of small sites with significantly less documents indexed. We're looking to implement a MLT component for our sites but want the similar stories to be only for a specific site (not all sites in the core). Is there a way to do something like this, or will we have to make mods (I'm not seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? /solr/dsteiger/mlt?q=story_id:188665+AND+site_id:86&mlt.fq=site_id: 86 (We have all all of our other defaults set up in the handler config.) Thanks. --- Doug Steigerwald Software Developer McClatchy Interactive dsteigerw...@mcclatchyinteractive.com
Re: Very long commit time.
On Wed, Mar 4, 2009 at 1:21 PM, Yonik Seeley wrote: > On Wed, Mar 4, 2009 at 5:25 AM, Jérôme Etévé wrote: >> Great, >> >> It went down to less than 10 secs now :) >> What I don't really understand is that my autowarmCount were pretty >> low ( like 128 ) and still the autowarming of the caches were very >> slow. >> >> Can you explain more why it can be that slow ? > > One possibility is a lack of physical memory available to the OS for > caching reads on both the old index and the new index. This would > cause all of the queries to be slower if they ended up doing real disk > IO for each query/filter being warmed. Strange, we've got plenty of memory on this box and the swap is zero. But well, I'm happy we went around the problem. What's your experience with commits with ~10M docs ( and ~128 autowarming count in caches ) ? Cheers. Jerome. -- Jerome Eteve. Chat with me live at http://www.eteve.net jer...@eteve.net
Re: indexing while optimizing
Upgraded to solr 1.3, and also moved it to a different machine (more CPU the rest is the same). All the problems seem to have gone away... Should have upgraded from the start. On Fri, Feb 27, 2009 at 2:50 PM, Laimonas Simutis wrote: > Hey, > > my SOLR setup looks like the following: > > server running apache-tomcat with solr1.2, index size is about 1G (a > bit more than 4 million documents). > > I have another machine that basically every minute or so sends some > documents to be indexed. I have autocommit turned on with maxDocs: > 5000, maxTime: 30ms. > > Also on the server cron job runs twice per day to optimize the index. > And sometimes it happens that the index messages come at the time of > optimize running. I know that running optimize on an index that is > about to have documents added or deleted is not that useful, but it > did help eliminate "too many file handles open" problem. > > Is it bad that I try to index when the optimize is running? I do see > failures on the client side from time to time, but the messages get > resent and indexed eventually. > > One recurring problem is that once per 36 hours or so SOLR server > becomes really unresponsive, just spinning crazy on CPU and it is all > in java (solr) process. When I try to shut down apache, apache goes > down but the java process is left running. I am trying to pin point > where the problem is, and wonder if my indexing-commit is not right. > The box is solely dedicated for solr, so there is really nothing else > running on it. > > Any pointers or observations appreciated. > > thanks, > > L >
Re: How to search the database tables using solr.
On Wed, Mar 4, 2009 at 7:51 PM, Radha C. wrote: > Thanks Shalin, > > We just stepped on solr. This information is very much useful for me. But > before that I want some clear details about where to start.. > I want to test this in my local environment, so I need some basic > information about how to start using this ( database and solr ). Do you > have > some information on this? > I think the easiest way is to start using Solr is with the embedded jetty container. Modify the example/conf/schema.xml file and add your own fields etc. Read through the DataImportHandler wiki page and at the example/example-DIH directory in the solr zip/tarball. If you have a specific doubt/question, ask on the list. -- Regards, Shalin Shekhar Mangar.
Re: Tomcat5 + Solr. Problems in deploying the Webapp
Hi, Have you looked at this page: http://wiki.apache.org/solr/SolrTomcat It almost sounds like you're deploying twice? Putting the solr.war in webapps would be one way, and the other would be a context config file + using the web manager. If you're using the config/context, then don't put the solr.war in webapps, tomcat should do that for you after deploying with the manager. Matt On Wed, Mar 4, 2009 at 8:55 AM, Sudharshan S wrote: > Hi all, > I am trying to setup a solr instance with Tomcat5 on a Fedora10 > machine. Here is what I did, > > 1.) Copy the apache-solr-nightly.war to webapps/solr.war > 2.) Set solr.solr.home in tomcat.conf > 3.) Use the Manager interface of tomcat to deploy the webapp > > But, while doing so, I get the following exceptions. > > Mar 4, 2009 6:55:09 PM org.apache.catalina.core.StandardContext filterStart > SEVERE: Exception starting filter SolrRequestFilter > java.lang.NoClassDefFoundError: Could not initialize class > org.apache.solr.core.SolrConfig >at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:76) >at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:221) >at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:302) >at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:78) >at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3635) >at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4222) >at > org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1173) >at > org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:549) >at > org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:105) >at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) >at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) >at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) >at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) >at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) >at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) >at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) >at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) >at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) >at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) >at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) >at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) >at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) >at java.lang.Thread.run(Thread.java:636) > > > What am I missing? If it matters I am running the nightly build from > March 3 2009. > > Thanks and Regards > Sudharshan S > Blog : http://www.sudharsh.wordpress.com > IRC : Sup3rkiddo @ Freenode, Gimpnet >
Re: Very long commit time.
On Wed, Mar 4, 2009 at 11:04 AM, Jérôme Etévé wrote: > Strange, we've got plenty of memory on this box and the swap is zero. > But well, I'm happy we went around the problem. What's your experience > with commits with ~10M docs ( and ~128 autowarming count in caches ) ? Probably depends somewhat on the nature of the filters and queries. It does sound a little slow, so perhaps they could be optimized in some way. If you have enough memory, the autowarming queries shouldn't be slower than the normal queries, so you should be able to use your logs to find the slow queries and try to figure out why they are slower. -Yonik http://www.lucidimagination.com
Re: Tomcat5 + Solr. Problems in deploying the Webapp
Hi Matt, On Wed, Mar 4, 2009 at 9:52 PM, Matt Mitchell wrote: > Hi, > > Have you looked at this page: http://wiki.apache.org/solr/SolrTomcat Yes. I mimicked the SimpleInstall method > > It almost sounds like you're deploying twice? Putting the solr.war in > webapps would be one way, and the other would be a context config file + > using the web manager. If you're using the config/context, then don't put > the solr.war in webapps, tomcat should do that for you after deploying with > the manager. Pardon me, but my experience with Tomcat is only a few hours old. From the Manager interface (manager/html) to be exact, I found out that the solr instance was not running, Hence clicked 'start'. The logs showed me the aforementioned exception. I guess using config/context is optional? since reading from the archives and your post, it seems that a config/context occurence and putting solr.war in webapps are mutually exclusive. If it helps, I have the following configuration option set in tomcat5.cont JAVA_OPTS="$JAVA_OPTS -Dcatalina.ext.dirs=$CATALINA_HOME/shared/lib:$CATALINA_HOME/common/lib -Dsolr.solr.home=/home/sudharsh/Projects/solr/solr" Regards Sudharshan S Blog : http://www.sudharsh.wordpress.com IRC : Sup3rkiddo @ Freenode, Gimpnet
Custom Field Type
Hello, I have a multivalued field in my schema of type text_ws, values are of the form #int #int I need to be able to query on the first and sort on the second, this does not seem to be enabled out of the box I looked around for a while and it seems the recommended approach is to create a custom field type and implement this logic in the getSortField method But since the field is multivalued, I need to sort by the value i am searching for, so i need access to the current query being executed. Also, i can't seem to figure out the correct -classpath to give to javac for it to find the packages needed to create the class file (my java is a bit rusty to say the least) Thanks, Fouad
Re: Custom Field Type
On Wed, Mar 4, 2009 at 12:24 PM, Fouad Mardini wrote: > I have a multivalued field in my schema of type text_ws, values are of the > form #int #int > I need to be able to query on the first and sort on the second, this does > not seem to be enabled out of the box Can you put the two numbers in separate fields for this purpose? If you can't do it from the indexer, a schema with copyField in conjunction with PatternTokenizerFactory could do it. -Yonik http://www.lucidimagination.com
Re: How to search the database tables using solr.
Also take a look at LuSql: http://lab.cisti-icist.nrc-cnrc.gc.ca/cistilabswiki/index.php/LuSql 2009/3/4 Shalin Shekhar Mangar : > On Wed, Mar 4, 2009 at 7:32 PM, Radha C. wrote: > >> Hi, >> >> I am working in a software concern. We are having some R&D base work like >> making use of solr search tool for searching our Oracle database >> content. I read through the documnets provided in the site, there I found >> the solr supports only the document search and we have to provide the >> document in the xml with the whole content that we need to search. But Is >> there any way to index our database content and search the indexed database >> content. >> > > Take a look at DataImportHandler. > > http://wiki.apache.org/solr/DataImportHandler > -- > Regards, > Shalin Shekhar Mangar. > -- -
Re: Multilanguage
17 feb 2009 kl. 21.26 skrev Grant Ingersoll: I believe Karl Wettin submitted a Lucene patch for a Language guesser: http://issues.apache.org/jira/browse/LUCENE-826 but it is marked as won't fix. The test case of LUCENE-1039 is a language classifier. I've use patch to detect languages of user queries (where I know the text contains text that is rather simple to classify as as specific language). karl
Re: MoreLikeThis filtering
Hah. Sorry, I'm really out of it today. The MoreLikeThisComponent doesn't seem to work for filtering using fq, but the MoreLikeThisHandler does. Problem solved, we'll just use the handler instead of a component. Doug On Mar 4, 2009, at 11:02 AM, Doug Steigerwald wrote: Sorry. The examples on the wiki aren't working with the 'fq' to filter the similarities. It just filters the actual queries. http://localhost:8983/solr/mlt?q=id:SP2514N&mlt.fl=manu,cat&mlt.mindf=1&mlt.mintf=1&fq=popularity:6&mlt.displayTerms=details&mlt=true The popularity of the doc found is 6, and trying to use 'fq=popularity:6' brings back similarities with a popularity other than 6. Doug On Mar 4, 2009, at 10:39 AM, Doug Steigerwald wrote: Hm. I checked out a clean Solr 1.3.0 and indexed the example docs and set up a simple MLT handler the example queries on the Wiki work fine (fq can filter out docs). Our build has a slight change to QueryComponent so another query isn't done when we use localsolr +field collapsing, but that change doesn't look like it would make a difference. It just conditionally sets rb.setNeedDocSet() to true or false. Will run some tests on a clean fresh build of Solr to see if it's our build. Doug On Mar 4, 2009, at 9:28 AM, Otis Gospodnetic wrote: Doug, does the good old 'fq' not work with MLT? It should... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Doug Steigerwald To: solr-user@lucene.apache.org Sent: Wednesday, March 4, 2009 9:20:40 AM Subject: MoreLikeThis filtering Is it possible to filter similarities found by the MLT component/ handler? Something like mlt.fq=site_id:86? We have 32 cores in our Solr install, and some of those cores have up to 8 sites indexed in them. Typically those cores will have one very large site with a few hundred thousand indexed documents, and lots of small sites with significantly less documents indexed. We're looking to implement a MLT component for our sites but want the similar stories to be only for a specific site (not all sites in the core). Is there a way to do something like this, or will we have to make mods (I'm not seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? /solr/dsteiger/mlt?q=story_id:188665+AND+site_id: 86&mlt.fq=site_id:86 (We have all all of our other defaults set up in the handler config.) Thanks. --- Doug Steigerwald Software Developer McClatchy Interactive dsteigerw...@mcclatchyinteractive.com
Re: MoreLikeThis filtering
I posted a while back with this problem and I've finally got it working using the following method: in solrconfig.xml: id,title 0 then when making the request, I do a normal search for the item with the necessary filter query (in my case discontinued:false) and set the qt=mlt to activate the mlt handler. So in your case: http://localhost:8983/solr/select?q=id:SP2514N&mlt.fl=manu,cat&mlt.mindf=1&mlt.mintf=1&fq=popularity:6&mlt.displayTerms=details&mlt=true&qt=mlt I had to use qt because I'm using a library to access solr and it doesn't include support for alternative handlers at the path level. Regards, Andrew Ingram Doug Steigerwald wrote: Sorry. The examples on the wiki aren't working with the 'fq' to filter the similarities. It just filters the actual queries. http://localhost:8983/solr/mlt?q=id:SP2514N&mlt.fl=manu,cat&mlt.mindf=1&mlt.mintf=1&fq=popularity:6&mlt.displayTerms=details&mlt=true The popularity of the doc found is 6, and trying to use 'fq=popularity:6' brings back similarities with a popularity other than 6. Doug On Mar 4, 2009, at 10:39 AM, Doug Steigerwald wrote: Hm. I checked out a clean Solr 1.3.0 and indexed the example docs and set up a simple MLT handler the example queries on the Wiki work fine (fq can filter out docs). Our build has a slight change to QueryComponent so another query isn't done when we use localsolr+field collapsing, but that change doesn't look like it would make a difference. It just conditionally sets rb.setNeedDocSet() to true or false. Will run some tests on a clean fresh build of Solr to see if it's our build. Doug On Mar 4, 2009, at 9:28 AM, Otis Gospodnetic wrote: Doug, does the good old 'fq' not work with MLT? It should... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Doug Steigerwald To: solr-user@lucene.apache.org Sent: Wednesday, March 4, 2009 9:20:40 AM Subject: MoreLikeThis filtering Is it possible to filter similarities found by the MLT component/handler? Something like mlt.fq=site_id:86? We have 32 cores in our Solr install, and some of those cores have up to 8 sites indexed in them. Typically those cores will have one very large site with a few hundred thousand indexed documents, and lots of small sites with significantly less documents indexed. We're looking to implement a MLT component for our sites but want the similar stories to be only for a specific site (not all sites in the core). Is there a way to do something like this, or will we have to make mods (I'm not seeing anything jump out at me in the Solr 1.3.0 or Lucene 2.4.0 code)? /solr/dsteiger/mlt?q=story_id:188665+AND+site_id:86&mlt.fq=site_id:86 (We have all all of our other defaults set up in the handler config.) Thanks. --- Doug Steigerwald Software Developer McClatchy Interactive dsteigerw...@mcclatchyinteractive.com
Re: DataImportHandler and delta-import question
Thanks,
I set up a another test instance of solr and ran a full import within
the DIH Development Console. I examined the query and found that
last_index_time is not getting set in the query. Yet the value does
get updated after a full import completes (outside of the development
console). Is there some place that I need to set the path to the
dataimport.properties file?
On Tue, Mar 3, 2009 at 8:03 PM, Noble Paul നോബിള് नोब्ळ्
wrote:
> I do not see anything wrong with this .It should have worked . Can you
> check that dataimport.properties is created (by DIH) in the conf
> directory? . check the content?
>
>
> are you sure that the query
>
> select DId from 2_Doc where ModifiedDate > '${dataimporter.last_index_time}'
>
> works with a date format -MM-dd HH:mm:ss . This is the format
> which DIH sends the date in . If the format is wrong you may need to
> format it using a dateformat function.
>
> see here
>
> http://wiki.apache.org/solr/DataImportHandler#head-5675e913396a42eb7c6c5d3c894ada5dadbb62d7
>
>
> The trunk DIH can work with Solr1.3 (you may need to put the DIH jar
> and slf4j). Can
> - Show quoted text -
> On Wed, Mar 4, 2009 at 3:53 AM, Garafola Timothy
> wrote:
>> I'm using solr 1.3 and am trying to get a delta-import with the DIH.
>> Recently the wiki, http://wiki.apache.org/solr/DataImportHandler, was
>> updated explaining that delta import is a 1.4 feature now but it was
>> still possible get a delta using the full import example here,
>> http://wiki.apache.org/solr/DataImportHandlerFaq#fullimportdelta. I
>> tried this but each time I run DIH, it reimports all rows and updates.
>>
>> Below is my data-config.xml. I set rootEntity to false and issued
>> command=full-import&clean=false&optimize=false through DIH. Am I
>> doing something wrong here or is the DataImportHandlerFaq incorrect?
>>
>>
>> > url="jdbc:mysql://pencil-somewhere.com:2/SomeDB" user="someUser"
>> password="somePassword"/>
>>
>> > query = "select DId from 2_Doc where
>> ModifiedDate > '${dataimporter.last_index_time}'
>> and DocType != 'Research Articles'">
>> > transformer="RegexTransformer"
>> query = "SELECT d.DId, d.SiteId,
>> d.DocTitle, d.DocURL, d.DocDesc,
>> d.DocType, d.Tags, d.Source,
>> d.Last90DaysRFIsPercent,
>> d.ModifiedDate, d.DocGuid, d.Author,
>> i.Industry FROM 2_Doc d LEFT
>> OUTER JOIN tmp_DocIndustry i
>> ON (d.DocId=i.DocId AND
>> d.SiteId=i.SiteId) where d.DocType != 'Research articles'
>> and d.DId = '${item.DId}' and
>> d.ModifiedDate > '${dataimporter.last_index_time}'">
>>
>>
>>
>> > ="DocTitle"/>
>>
>>
>> > regex="^(.{0,800})\b.*$" sourceColName="DocDesc"/>
>>
>> > splitBy=";" sourceColName="Tags"/>
>>
>> > "Last90DaysRFIsPercent" name ="Last90DaysRFIsPercent"/>
>> > ="ModifiedDate"/>
>>
>>
>> > ="Industry" sourceColName="Industry"/>
>>
>>
>>
>>
>>
>> Thanks,
>> -Tim
>>
>
>
>
> --
> --Noble Paul
>
--
-Tim
Possible bug with sdouble?
Hi everyone! So it seems like I ran into a bug with sdouble. I have a document with two fields. field1 is double when I index I send for both of them the same value (7300) and when I'm querying I'm getting the following 7300 7.3E7 What do you think? Why am I getting different notations? BTW I'm using solr 1.2 Thanks! Jonathan
Re: Possible bug with sdouble?
I'll do the obvious one first. :) Have you tried Solr 1.3 or, even better, 1.4-dev nightly? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Jonathan Ariel > To: solr-user@lucene.apache.org > Sent: Wednesday, March 4, 2009 2:56:03 PM > Subject: Possible bug with sdouble? > > Hi everyone! > So it seems like I ran into a bug with sdouble. > I have a document with two fields. > field1 is double > > > stored="true" omitNorms="true"/> > > indexed="true" stored="true" omitNorms="true"/> > > > > when I index I send for both of them the same value (7300) and when I'm > querying I'm getting the following > > 7300 > 7.3E7 > > What do you think? Why am I getting different notations? > > BTW I'm using solr 1.2 > > Thanks! > > Jonathan
Re: Possible bug with sdouble?
On Wed, Mar 4, 2009 at 2:56 PM, Jonathan Ariel wrote: > Hi everyone! > So it seems like I ran into a bug with sdouble. > I have a document with two fields. > field1 is double > > stored="true" omitNorms="true"/> > indexed="true" stored="true" omitNorms="true"/> > > when I index I send for both of them the same value (7300) and when I'm > querying I'm getting the following > > 7300 > 7.3E7 > > What do you think? Why am I getting different notations? The straight "double" type indexes things as a string w/o doing any sort of normalization. So a value of 100.000 will be indexed and stored as 100.000. "sdouble" needs to parse the float, and it gets more normalized as a byproduct. If anything, one could consider lack of normalization a bug with double, not sdouble. -Yonik http://www.lucidimagination.com
commit / new searcher delay?
I'm doing some testing of a solr master/slave config and find that, after syncing my slave, I need to sleep for about 400ms after commit to "see" the new index state. i.e. if I don't sleep, and I execute a query, I get results that reflect the prior state of the index. I suspect this has something to do with waiting for the searcher to warm and switch over (?). Though, I'm confused because when I print out /solr/admin/registry.jsp, the hashcode of the Searcher changes immediately (as the commit docs say, the commit operation blocks by default until a new searcher is in place). I've tried turning off all caching, to no effect. Anyone have any idea what could be going on here? Ideally, would be an operation that blocks until the exact moment when the new searcher is in place and is actually serving based on the new index state... -Steve
use () in the query string
Hello, In my case, my query of "id_s_i_s_nm:(om_B00114162K*)" returned nothing but query "id_s_i_s_nm:om_B00114162K*" returned the right result. What's the difference between using () or not. Thanks a lot, Kevin
Re: commit / new searcher delay?
On Thu, Mar 5, 2009 at 6:06 AM, Steve Conover wrote: > I'm doing some testing of a solr master/slave config and find that, > after syncing my slave, I need to sleep for about 400ms after commit > to "see" the new index state. i.e. if I don't sleep, and I execute a > query, I get results that reflect the prior state of the index. > How are you sending the commit? You should use commit with waitSearcher=true and waitFlush=true so that it blocks until the new searcher becomes available for querying. -- Regards, Shalin Shekhar Mangar.
Re: DataImportHandler and delta-import question
the dataimport.properties is created only after one successful import
.so it is available only from second import onwards. probably you can
create one manually and put it in the conf dir.
On Thu, Mar 5, 2009 at 12:52 AM, Garafola Timothy wrote:
> Thanks,
>
> I set up a another test instance of solr and ran a full import within
> the DIH Development Console. I examined the query and found that
> last_index_time is not getting set in the query. Yet the value does
> get updated after a full import completes (outside of the development
> console). Is there some place that I need to set the path to the
> dataimport.properties file?
>
> On Tue, Mar 3, 2009 at 8:03 PM, Noble Paul നോബിള് नोब्ळ्
> wrote:
>> I do not see anything wrong with this .It should have worked . Can you
>> check that dataimport.properties is created (by DIH) in the conf
>> directory? . check the content?
>>
>>
>> are you sure that the query
>>
>> select DId from 2_Doc where ModifiedDate > '${dataimporter.last_index_time}'
>>
>> works with a date format -MM-dd HH:mm:ss . This is the format
>> which DIH sends the date in . If the format is wrong you may need to
>> format it using a dateformat function.
>>
>> see here
>>
>> http://wiki.apache.org/solr/DataImportHandler#head-5675e913396a42eb7c6c5d3c894ada5dadbb62d7
>>
>>
>> The trunk DIH can work with Solr1.3 (you may need to put the DIH jar
>> and slf4j). Can
>> - Show quoted text -
>> On Wed, Mar 4, 2009 at 3:53 AM, Garafola Timothy
>> wrote:
>>> I'm using solr 1.3 and am trying to get a delta-import with the DIH.
>>> Recently the wiki, http://wiki.apache.org/solr/DataImportHandler, was
>>> updated explaining that delta import is a 1.4 feature now but it was
>>> still possible get a delta using the full import example here,
>>> http://wiki.apache.org/solr/DataImportHandlerFaq#fullimportdelta. I
>>> tried this but each time I run DIH, it reimports all rows and updates.
>>>
>>> Below is my data-config.xml. I set rootEntity to false and issued
>>> command=full-import&clean=false&optimize=false through DIH. Am I
>>> doing something wrong here or is the DataImportHandlerFaq incorrect?
>>>
>>>
>>> >> url="jdbc:mysql://pencil-somewhere.com:2/SomeDB" user="someUser"
>>> password="somePassword"/>
>>>
>>> >> query = "select DId from 2_Doc where
>>> ModifiedDate > '${dataimporter.last_index_time}'
>>> and DocType != 'Research Articles'">
>>> >> transformer="RegexTransformer"
>>> query = "SELECT d.DId, d.SiteId,
>>> d.DocTitle, d.DocURL, d.DocDesc,
>>> d.DocType, d.Tags, d.Source,
>>> d.Last90DaysRFIsPercent,
>>> d.ModifiedDate, d.DocGuid, d.Author,
>>> i.Industry FROM 2_Doc d LEFT
>>> OUTER JOIN tmp_DocIndustry i
>>> ON (d.DocId=i.DocId AND
>>> d.SiteId=i.SiteId) where d.DocType != 'Research articles'
>>> and d.DId = '${item.DId}' and
>>> d.ModifiedDate > '${dataimporter.last_index_time}'">
>>>
>>>
>>>
>>> >> ="DocTitle"/>
>>>
>>>
>>> >> regex="^(.{0,800})\b.*$" sourceColName="DocDesc"/>
>>>
>>> >> splitBy=";" sourceColName="Tags"/>
>>>
>>> >> "Last90DaysRFIsPercent" name ="Last90DaysRFIsPercent"/>
>>> >> ="ModifiedDate"/>
>>>
>>>
>>> >> ="Industry" sourceColName="Industry"/>
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>> -Tim
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
>
> --
> -Tim
>
--
--Noble Paul
Re: datadir issue for multicore on latest trunk
I have reopened the issue. We will fix it completely in a day or two. On Wed, Mar 4, 2009 at 6:30 PM, Walter Ferrara wrote: > tried with > > > but no luck, the dataDir parameter seems ignored, no matter what is written > there > > On Wed, Mar 4, 2009 at 12:58 PM, Noble Paul നോബിള് नोब्ळ् < > noble.p...@gmail.com> wrote: > >> looks like a bug. must reopen the issue >> >> On Wed, Mar 4, 2009 at 5:26 PM, Noble Paul നോബിള് नोब्ळ् >> wrote: >> > On Wed, Mar 4, 2009 at 5:24 PM, Walter Ferrara >> wrote: >> >> using: >> >> >> >> >> >> >> >> >> >> doesn't work either >> > >> > dataDir="/multicore/core0" means the path is absolute. >> > where did it create? >> > >> >> >> >> here the output: >> >> >> >> 2009-03-04 12:50:54.890::INFO: Logging to STDERR via >> >> org.mortbay.log.StdErrLog >> >> 2009-03-04 12:50:54.968::INFO: jetty-6.1.3 >> >> 4-mar-2009 12.50.55 org.apache.solr.servlet.SolrDispatchFilter init >> >> INFO: SolrDispatchFilter.init() >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> >> locateInstanceDir >> >> INFO: JNDI not configured for solr (NoInitialContextEx) >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> >> locateInstanceDir >> >> INFO: using system property solr.solr.home: multicore >> >> 4-mar-2009 12.50.55 org.apache.solr.core.CoreContainer$Initializer >> >> initialize >> >> INFO: looking for solr.xml: >> >> d:\DEV\apache-solr-2009-03-03_08-06-53\example\multicore\solr.xml >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> >> INFO: Solr home set to 'multicore/' >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> >> createClassLoader >> >> INFO: Reusing parent classloader >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> >> INFO: Solr home set to 'multicore\core0/' >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrResourceLoader >> >> createClassLoader >> >> INFO: Reusing parent classloader >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrConfig >> >> INFO: Loaded SolrConfig: solrconfig.xml >> >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> >> INFO: Reading Solr Schema >> >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> >> INFO: Schema name=example core zero >> >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader >> load >> >> INFO: created string: org.apache.solr.schema.StrField >> >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> >> INFO: default search field is name >> >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> >> INFO: query parser default operator is OR >> >> 4-mar-2009 12.50.55 org.apache.solr.schema.IndexSchema readSchema >> >> INFO: unique key field: id >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore >> >> INFO: [core0] Opening new SolrCore at multicore\core0/, >> >> dataDir=core0//multicore/core0/ >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore >> >> INFO: JMX monitoring not detected for core: core0 >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener >> >> INFO: [core0] Searching for listeners: >> //listen...@event="firstSearcher"] >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore parseListener >> >> INFO: [core0] Searching for listeners: //listen...@event="newSearcher"] >> >> 4-mar-2009 12.50.55 org.apache.solr.core.SolrCore initIndex >> >> AVVERTENZA: [core0] Solr index directory 'core0\multicore\core0\index' >> >> doesn't exist. Creating new index... >> >> 4-mar-2009 12.50.55 org.apache.solr.update.SolrIndexWriter getDirectory >> >> AVVERTENZA: No lockType configured for core0//multicore/core0/index/ >> >> assuming 'simple' >> >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader >> load >> >> INFO: created standard: org.apache.solr.handler.StandardRequestHandler >> >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader >> load >> >> INFO: created /update: org.apache.solr.handler.XmlUpdateRequestHandler >> >> 4-mar-2009 12.50.55 org.apache.solr.util.plugin.AbstractPluginLoader >> load >> >> INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers >> >> 4-mar-2009 12.50.55 org.apache.solr.search.SolrIndexSearcher >> >> INFO: Opening searc...@1e57e8f main >> >> 4-mar-2009 12.50.55 >> >> org.apache.solr.update.DirectUpdateHandler2$CommitTracker >> >> INFO: AutoCommit: disabled >> >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler >> inform >> >> INFO: Adding >> >> component:org.apache.solr.handler.component.querycompon...@19a32e0 >> >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler >> inform >> >> INFO: Adding >> >> component:org.apache.solr.handler.component.facetcompon...@8238f4 >> >> 4-mar-2009 12.50.55 org.apache.solr.handler.component.SearchHandler >> inform >> >> INFO: Adding >> >> >> component:org.apache.solr.handler.component.morelikethiscompon...@16925b0 >> >> 4-mar-2009 12.50.55 org.apache.solr.handler.compone
Column Specific Query with q parameter
Hi, I am implementing column specific query with q query parameter. for e.g. ?q=prdMainTitle_product_s:math & qt=dismaxrequest The above query doesnt work while if I use the same query with q.alt parameter, it works. ?q=&q.alt= prdMainTitle_product_s:math & qt=dismaxrequest Please suggest, how to achieve this with q query. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Column-Specific-Query-with-q-parameter-tp22345960p22345960.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Column Specific Query with q parameter
The syntax for the q param when using dismax is different from standard. Check this out: http://wiki.apache.org/solr/DisMaxRequestHandler#head-df8184dddf870336839490ba276ea6ac566d0bdf q.alt under dismax is parsed using the standard query parser though: http://wiki.apache.org/solr/DisMaxRequestHandler#head-9d23a23915b7932490069d3262ef7f3625e398ff Using dismax with that query... you could do it using the fq param: ?fq=prdMainTitle_product_s:math&qt=dismaxrequest&q.alt=*:* But make sure you understand how the fq param works; how solr uses its caching... http://wiki.apache.org/solr/CommonQueryParameters#head-6522ef80f22d0e50d2f12ec487758577506d6002 Hope this helps, Matt On Thu, Mar 5, 2009 at 1:30 AM, dabboo wrote: > > Hi, > > I am implementing column specific query with q query parameter. for e.g. > > ?q=prdMainTitle_product_s:math & qt=dismaxrequest > > The above query doesnt work while if I use the same query with q.alt > parameter, it works. > > ?q=&q.alt= prdMainTitle_product_s:math & qt=dismaxrequest > > Please suggest, how to achieve this with q query. > > Thanks, > Amit Garg > -- > View this message in context: > http://www.nabble.com/Column-Specific-Query-with-q-parameter-tp22345960p22345960.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
Any standard/specification for Search ??
Hi Guys, Are you aware of any standard/specification (like JSR 168/286 for portals, CMIS for CMS) for Search engines ? Is there any such specification people are working on currently ? Regards, Sourav CAUTION - Disclaimer * This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS***
Re: Distributed Search
> : > > Ok, so it wouldn't be possible to have a smaller, faster authoritative > : > > shard for near-real-time updates while keeping the entire dataset in a > : > > second shard which is updates less frequently? > > I believe Otis's point is that many people use distributed search across > shards where some are large and mostly static and one is small and > frequently updated with new docs in order to get some performance > advantages out of hte long cache lifes on the larger shard(s) ... but this > typically works best when you only "add" new docs, and don't modify old > ones (or only modify docs added very recently so they're always in the > small shard) while the bigger shards are treated as "archives" that don't > change. > > To be deterministic you can't have the same uniqueKey in multiple shards. Hmm, partitioning by document has a lot of merit, but having this be (configurably) deterministic would seem to enable some interesting features, such as simple 'tagging' by partitioning by document fields. For example, you could have a large essentially read-only index of documents and a separate small index for tags. To tag a document, you would create (or update) a document in the tag index containing the uniqueKey from the main index as well as a multivalued tag field, and whenever you search, you fire off a distributed search across the two shards, but pulling the fields from the main index (eg /solr/select?fq=tag1&shards=main_index/path,tag_index/path&q=*:*). My specific use case is a bit more involved, but if there were either some way to deterministically pick the shard source *or* to dynamically (additively) merge the multiple docs sharing the same uniqueKey from separate shards, it would be quite helpful. The later would provide the general case functionality to have partial document updates, except even more powerful. However, I could get by with just the former - using the main index for all scoring but being able to augment documents for filtering. I'm not a solr expert by any means, so if there is another recommended way to achieve that functionality, I'd love some guidance. Or, if this is just a rare case, I guess it'd be time for me to roll up my sleeves and hack up some solr code. Making QueryComponent configurably deterministic would suffice (eg a "shard.primary=main_index/path" parameter, perhaps? or even just treating the shards parameter as an ordered list with the primary first?). Adding field merging would likely be... more involved though. Thanks in advance for any advice! -pete
Re: How to search the database tables using solr.
Does anybody has any stats to share on how much time does DataImportHandler takes to index a given set of data ? I am currently indexing 18 millions rows in 1.5 - 2 hours by sending xmls to solr. From: Shalin Shekhar Mangar To: solr-user@lucene.apache.org; cra...@ceiindia.com Sent: Wednesday, March 4, 2009 8:15:07 AM Subject: Re: How to search the database tables using solr. On Wed, Mar 4, 2009 at 7:51 PM, Radha C. wrote: > Thanks Shalin, > > We just stepped on solr. This information is very much useful for me. But > before that I want some clear details about where to start.. > I want to test this in my local environment, so I need some basic > information about how to start using this ( database and solr ). Do you > have > some information on this? > I think the easiest way is to start using Solr is with the embedded jetty container. Modify the example/conf/schema.xml file and add your own fields etc. Read through the DataImportHandler wiki page and at the example/example-DIH directory in the solr zip/tarball. If you have a specific doubt/question, ask on the list. -- Regards, Shalin Shekhar Mangar.
