Re: How to implement Spell Checker using Solr?

[anupamxyz]

The error I have been receiving after crawling using Solr is as mentioned
below: 

2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Basic Indexing
Filter (index-basic)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Basic Summarizer
Plug-in (summary-basic)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Site Query 
Filter
(query-site)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Http / Https
Protocol Plug-in (protocol-httpclient)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - HTTP Framework
(lib-http)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Pass-through URL
Normalizer (urlnormalizer-pass)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Regex URL Filter
(urlfilter-regex)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Http Protocol
Plug-in (protocol-http)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - XML Response 
Writer
Plug-in (response-xml)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Regex URL
Normalizer (urlnormalizer-regex)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - OPIC Scoring
Plug-in (scoring-opic)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - CyberNeko HTML
Parser (lib-nekohtml)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Anchor Indexing
Filter (index-anchor)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - URL Query Filter
(query-url)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Regex URL Filter
Framework (lib-regex-filter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - JSON Response
Writer Plug-in (response-json)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Registered
Extension-Points:
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Summarizer
(org.apache.nutch.searcher.Summarizer)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Protocol
(org.apache.nutch.protocol.Protocol)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Analysis
(org.apache.nutch.analysis.NutchAnalyzer)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Field 
Filter
(org.apache.nutch.indexer.field.FieldFilter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - HTML Parse 
Filter
(org.apache.nutch.parse.HtmlParseFilter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Query 
Filter
(org.apache.nutch.searcher.QueryFilter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Search
Results Response Writer (org.apache.nutch.searcher.response.ResponseWriter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch URL
Normalizer (org.apache.nutch.net.URLNormalizer)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch URL Filter
(org.apache.nutch.net.URLFilter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Online 
Search
Results Clustering Plugin (org.apache.nutch.clustering.OnlineClusterer)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Indexing
Filter (org.apache.nutch.indexer.IndexingFilter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Content
Parser (org.apache.nutch.parse.Parser)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Nutch Scoring
(org.apache.nutch.scoring.ScoringFilter)
2011-08-24 15:47:56,225 INFO  plugin.PluginRepository - Ontology Model
Loader (org.apache.nutch.ontology.Ontology)
2011-08-24 15:47:56,241 INFO  indexer.IndexingFilters - Adding
org.apache.nutch.indexer.basic.BasicIndexingFilter
2011-08-24 15:47:56,241 INFO  indexer.IndexingFilters - Adding
org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2011-08-24 15:47:57,366 WARN  mapred.LocalJobRunner - job_local_0001
org.apache.solr.common.SolrException: Internal Server Error

Internal Server Error

request: http://localhost:7001/solr/update?wt=javabin&version=2.2
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:343)
at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:183)
at
org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:217)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48)
at org.apache.nutch.indexer.solr.SolrWriter.close(SolrWriter.java:69)
at
org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:48)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:170)
2011-08-24 15:47:57,882 FATAL solr.SolrIndexer - SolrIndexer:
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
at 
org.apache.nutch.indexer.solr.SolrIndexer.indexSolr(SolrIndexer.java:73)

Re: Need help with spellcheck city name

[anupamxyz]

Can you let me know how did you get the suggestions? Are you using a
file-based spell checker...? Or an Indexed-based one? And what is the exact
structure for them and how to set up the Solr configuration for the Spell
checker...since I am not able to get teh spell checker build.


please guide me.

Thanks,
Anupam

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-spellcheck-city-name-tp1592771p3315722.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to implement Spell Checker using Solr?

[anupamxyz]

I have been able to setup Solr Spell checker on my web application. It is a
file based spell checker that i have implemented. I would like to add that
the same isn't that accurate, since I haven't applied any specific algorithm
for having the most relevant search result. Kindly do let me know in case
you have any issues in implementing the same at your end.

regards,
Anupam

--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-implement-Spell-Checker-using-Solr-tp3268450p3371563.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: File based index doesn't work in spellcheck component

[anupamxyz]

I am using Nutch to crawl and Solr for searching. The search has been
successfully implemented. Now I want a file based Suggestion or a "Do you
mean Feature?" implemented. It is more or less like a Spell checker. For the
same I am making the requisite changes to the SolrConfig.xml and the
Schema.xml for the Solr, but it fails when I am re-indexing it gain to get
the new implementation. Please let me know how that can be corrected and
also, how can I have the suggestion displayed using Jsp over my application.
I can share part of the codes changed later if you intend to help me on
this.

Thanks in advance,
Anupam

--
View this message in context: 
http://lucene.472066.n3.nabble.com/File-based-index-doesn-t-work-in-spellcheck-component-tp489070p3268423.html
Sent from the Solr - User mailing list archive at Nabble.com.

How to implement Spell Checker using Solr?

[anupamxyz]

I am using Nutch to crawl and Solr for searching. The search has been
successfully implemented. Now I want a file based Suggestion or a "Do you
mean Feature?" implemented. It is more or less like a Spell checker. For the
same I am making the requisite changes to the SolrConfig.xml and the
Schema.xml for the Solr, but it fails when I am re-indexing it gain to get
the new implementation. Please let me know how that can be corrected and
also, how can I have the suggestion displayed using Jsp over my application.
I can share part of the codes changed later if you intend to help me on
this.

Thanks in advance,
Anupam

--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-implement-Spell-Checker-using-Solr-tp3268450p3268450.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to implement Spell Checker using Solr?

[anupamxyz]

Both Nutch and Solr can be used as per the need.
http://www.lucidimagination.com/blog/2009/03/09/nutch-solr/ . So the search
is implemented and I am able to search on the values. Now I need the
SpellChecker to be implemented. The changes are exactly as per the ones
listed in http://wiki.apache.org/solr/SpellCheckComponent . I will share the
log details with you by Monday.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-implement-Spell-Checker-using-Solr-tp3268450p3268695.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to implement Spell Checker using Solr?

[anupamxyz]

The changes for Solrconfig.xml in solr is as follows



  
  default
  
  solr.IndexBasedSpellChecker
  
  spell
  
  ./spellchecker
  
  0.7
  
  .0001



  jarowinkler
  lowerfilt
  
  org.apache.lucene.search.spell.JaroWinklerDistance
  ./spellchecker




textSpell


And for the Request handler, I have incorporated the following changes:




  
  true
  
  false
  
  default
  
  false
  
  5
true
  true


  spellcheck

  

The same is failing while crawling. I have reveretd my code for now. But can
try it once again and post the exception that I have been getting while
crawling.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-implement-Spell-Checker-using-Solr-tp3268450p3274069.html
Sent from the Solr - User mailing list archive at Nabble.com.