Hmmm. Interesting line of thought. Thanks a lot Jay. Will explore this approach. There are lot of duplicate tables though :).
I was about to try a different approach - set up two solar cores, keep reloading config and updating one, merge with the bigger index ... But your approach is worth exploring. Thanks. Jay Hill wrote: > > You could use separate DIH config files for each of your three tables. > This > might be overkill, but it would keep them separate. The DIH is not limited > to one request handler setup, so you could create a unique handler for > each > case with a unique name: > > <requestHandler name="/indexer/table1" > class="org.apache.solr.handler.dataimport.DataImportHandler"> > <lst name="defaults"> > <str name="config">table1-config.xml</str> > </lst> > </requestHandler> > > <requestHandler name="/indexer/table2" > class="org.apache.solr.handler.dataimport.DataImportHandler"> > <lst name="defaults"> > <str name="config">table2-config.xml</str> > </lst> > </requestHandler> > > <requestHandler name="/indexer/table3" > class="org.apache.solr.handler.dataimport.DataImportHandler"> > <lst name="defaults"> > <str name="config">table3-config.xml</str> > </lst> > </requestHandler> > > When you go to ...solr/admin/dataimport.jsp you should see a list of all > DataImportHandlers that are configured, and can select them individually, > if > that works for your needs. > > -Jay > http://www.lucidimagination.com > > On Fri, Oct 9, 2009 at 10:57 AM, solr.searcher > <solr.searc...@gmail.com>wrote: > >> >> Hi all, >> >> First of all, please accept my apologies if this has been asked and >> answered >> before. I tried my best to search and couldn't find anything on this. >> >> The problem I am trying to solve is as follows. I have multiple tables >> with >> identical schema - table_a, table_b, table_c ... and I am trying to >> create >> one big index with the data from each of these tables. The idea was to >> programatically create the data-config file (just changing the table >> name) >> and do a reload-config followed by a full-import with clean set to false. >> In >> other words: >> >> 1. publish the data-config file >> 2. do a reload-config >> 3. do a full-import with clean = false >> 4. commit, optimize >> 5. repeat with new table name >> >> I wanted to then follow the same procedure for delta imports. The problem >> is >> that after I do a reload-config and then do a full-import, the old data >> in >> the index is getting lost. >> >> What am I missing here? Please note that I am new to solr. >> >> INFO: [] webapp=/solr path=/dataimport >> params={command=reload-config&clean=false} status=0 QTime=4 >> Oct 9, 2009 10:17:30 AM org.apache.solr.core.SolrCore execute >> INFO: [] webapp=/solr path=/dataimport >> params={command=full-import&clean=false} status=0 QTime=1 >> Oct 9, 2009 10:17:30 AM org.apache.solr.handler.dataimport.SolrWriter >> readIndexerProperties >> INFO: Read dataimport.properties >> Oct 9, 2009 10:17:30 AM org.apache.solr.handler.dataimport.DataImporter >> doFullImport >> INFO: Starting Full Import >> Oct 9, 2009 10:17:30 AM org.apache.solr.handler.dataimport.SolrWriter >> readIndexerProperties >> INFO: Read dataimport.properties >> Oct 9, 2009 10:17:30 AM >> org.apache.solr.handler.dataimport.JdbcDataSource$1 >> call >> INFO: Creating a connection for entity <blah blah blah> >> Oct 9, 2009 10:17:30 AM >> org.apache.solr.handler.dataimport.JdbcDataSource$1 >> call >> INFO: Time taken for getConnection(): 12 >> Oct 9, 2009 10:17:31 AM org.apache.solr.core.SolrDeletionPolicy onInit >> INFO: SolrDeletionPolicy.onInit: commits:num=1 >> >> >> commit{dir=/blah/blah/index,segFN=segments_1z,version=1255032607825,generation=71,filenames=[segments_1z, >> _cl.cfs] >> Oct 9, 2009 10:17:31 AM org.apache.solr.core.SolrDeletionPolicy >> updateCommits >> INFO: last commit = 1255032607825 >> >> Any help will be greatly appreciated. Is there any other way to >> automaticaly >> slurp data from multiple, identical tables? >> >> Thanks a lot. >> >> -- >> View this message in context: >> http://www.nabble.com/Dynamic-Data-Import-from-multiple-identical-tables-tp25825381p25825381.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/Dynamic-Data-Import-from-multiple-identical-tables-tp25825381p25828773.html Sent from the Solr - User mailing list archive at Nabble.com.
