Re: Solr - Index Concurrency - Is it possible to have multiple threads write to same index?

2012-08-22 Thread ksu wildcats
Thanks for the reply Mikhail. For our needs the speed is more important than flexibility and we have huge text files (ex: blogs / articles ~2 MB size) that needs to be read from our filesystem and then store into the index. We have our app creating separate core per client (dynamically) and there

Re: Solr Custom Filter Factory - How to pass parameters?

2012-08-22 Thread ksu wildcats
Thanks Erick. I tried to do it all at the filter but the problem i am running into doing it at the filter is intercepting the final commit calls or in other words I am unable to figure out when the final commit should happen such that I don't miss out any data. One option I tried is to increase the

Solr - Index Concurrency - Is it possible to have multiple threads write to same index?

2012-08-21 Thread ksu wildcats
We have a webapp that has embedded solr integrated in it. It essentially handles creating separate index (core) per client and it is currently setup such that there can only be one index write operation per core. Say if we have 1 Million documents that needs be to Indexed, our app reads each docume

Re: Solr Custom Filter Factory - How to pass parameters?

2012-08-21 Thread ksu wildcats
Jack Reading through the documentation for UpdateRequestProcessor my understanding is that its good for handling processing of documents before analysis. Is it true that processAdd (where we can have custom logic) is invoked once per document and is invoked before any of the analyzers gets invoke

RE: Solr Custom Filter Factory - How to pass parameters?

2012-08-20 Thread ksu wildcats
Thanks for your help. I was able to get it working with using the parameters from filedtype definition in config files. I am now stuck on next step. Can you please tell if there is a way to identify/intercept last token that gets added to index (across all documents) ? Here is my scenario 1) I ha

RE: Solr Custom Filter Factory - How to pass parameters?

2012-08-20 Thread ksu wildcats
Thanks Markus. Links are helpful. I will give it a try and see if that solves my problem. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Custom-Filter-Factory-How-to-pass-parameters-tp4002217p4002248.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Custom Filter Factory - How to pass parameters?

2012-08-20 Thread ksu wildcats
Thanks Jack. The information I want to pass is the "databasename" into which the analyzed data needs to be inserted. As i was saying earlier, the set up we have is 1) we use embedded solr server with multi cores - embedded into our webapp 2) support one index for each client - each client has a

Solr Custom Filter Factory - How to pass parameters?

2012-08-20 Thread ksu wildcats
We are using SOLR and are in the process of adding custom filter factory to handle the processing of words/tokens to suit our needs. Here is what our custom filter factory does 1) Reads the tokens and does some analysis and writes the result of analysis to database. We are using Embedded Solr wit