Norberto, Thanks for your input. What do you mean with "Have you tried connecting to SOLR over HTTP from localhost, therefore avoiding any firewall issues and network latency ? it should work a LOT faster than from a remote site." ?
Here are how our servers lay out: 1) Database ( Oracle ) is running on separate machine 2) Solr master is running on separate machine by itself 3) 6 solr slaves ( these 6 pulll the index from master using rsync) We have a SQL(Oracle) script to post the data/index from Oracle Database machine to Solr Master over http. We wrote those script(Someone in Oracle Database administrator write it). In Solr master configuration we have scripts.conf that like this: user= solr_hostname=localhost solr_port=7001 rsyncd_port=18983 data_dir= webapp_name=solr master_host=localhost master_data_dir=solr/snapshot master_status_dir=solr/status So, basically from Oracle system we launch the Oracle/SQL script posting the data to Solr Master using http://solrmaster/solr/update ( inside the SQL script we put this). We can not do localhost since it's solr is not running on Oracle machine. Another alternative that we think of is to transform XML into CSV and import/export it. How about if LUSQL, some mentioned about this? Is this apps free(open source) application? Do you have any experience with this apps? Thanks All for your valuable suggestions! Francis -----Original Message----- From: Norberto Meijome [mailto:numard...@gmail.com] Sent: Thursday, July 02, 2009 3:01 AM To: solr-user@lucene.apache.org Cc: Francis Yakin Subject: Re: Is there any other way to load the index beside using "http" connection? On Wed, 1 Jul 2009 15:07:12 -0700 Francis Yakin <fya...@liquid.com> wrote: > > We have several thousands of xml files in database that we load it to solr > master The Database uses "http" connection and transfer those files to solr > master. Solr then translate xml files to their lindex. > > We are experiencing issue with close/open connection in the firewall and very > very slow. > > Is there any other way to load the data/index from Database to solr master > beside using http connection, so it means we just scp/ftp the xml file from > Database system to solr master and let solr convert those to lucene indexes? > Francis, after reading the whole thread, it seems you have : - Data source : Oracle DB, on separate location to your SOLR. - Data format : XML output. definitely DIH is a great option, but since you are on 1.2, not available to you (you should look into upgrading if you can!). Have you tried connecting to SOLR over HTTP from localhost, therefore avoiding any firewall issues and network latency ? it should work a LOT faster than from a remote site. Also make sure not to commit until you really needed. Other alternatives are to transform the XML into csv and import it that way. Or write a simple app that will parse the xml and post it directly using the embedded solr method. plenty of options, all of them documented @ solr's site. good luck, b _________________________ {Beto|Norberto|Numard} Meijome "People demand freedom of speech to make up for the freedom of thought which they avoid. " Soren Aabye Kierkegaard I speak for myself, not my employer. Contents may be hot. Slippery when wet. Reading disclaimers makes you go blind. Writing them is worse. You have been Warned.
