No, we do not use DIH. Based on other responses I saw, its seems likely that the issue is in the DIH component somehow.
JRJ -----Original Message----- From: Awasthi, Shishir [mailto:shishir.awas...@baml.com] Sent: Tuesday, October 25, 2011 3:24 PM To: solr-user@lucene.apache.org; Jaeger, Jay - DOT Subject: RE: Loading data to SOLR first time ( taking too long) Ok that makes me feel better. We have around 40 fields being loaded from multiple tables. Other than not commiting every row is there any other setting that you make? Are you also using DataImportHandler? -----Original Message----- From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov] Sent: Tuesday, October 25, 2011 4:03 PM To: 'solr-user@lucene.apache.org' Subject: RE: Loading data to SOLR first time ( taking too long) My goodness. We do 4 million in about 1/2 HOUR (7+ million in 40 minutes). First question: Are you somehow forcing Solr to do a commit for each and every record? If so, that way leads to the house of PAIN. The thing to do next, I suppose, might be to try and figure out whether the issue is in Solr proper, or in the database you are importing from. What does your query against your database look like? How many fields do you have per record (we have around 30, counting copyField destinations) Using a performance monitoring tool, try and find out the CPU utilization, memory utilization, page write rates and physical disk drive queue lengths to narrow down which of the two systems are having the problem (assuming your database is not on the same machine as Solr!) JRJ -----Original Message----- From: Awasthi, Shishir [mailto:shishir.awas...@baml.com] Sent: Tuesday, October 25, 2011 2:57 PM To: solr-user@lucene.apache.org Subject: Loading data to SOLR first time ( taking too long) Hi, I recently started working on SOLR and loaded approximately 4 million records to the solr using DataImportHandler. It took 5 days to complete this process. Can you please suggest how this can be improved? I would like this to be done in less than 6 hrs. Thanks, Shishir ---------------------------------------------------------------------- This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses. References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing. ---------------------------------------------------------------------- This message w/attachments (message) is intended solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. Unless specifically indicated, this message is not an offer to sell or a solicitation of any investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Sender. Subject to applicable law, Sender may intercept, monitor, review and retain e-communications (EC) traveling through its networks/systems and may produce any such EC to regulators, law enforcement, in litigation and as required by law. The laws of the country of each sender/recipient may impact the handling of EC, and EC may be archived, supervised and produced in countries other than the country in which you are located. This message cannot be guaranteed to be secure or free of errors or viruses. References to "Sender" are references to any subsidiary of Bank of America Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a Condition to Any Banking Service or Activity * Are Not Insured by Any Federal Government Agency. Attachments that are part of this EC may have additional important disclosures and disclaimers, which you should read. This message is subject to terms available at the following link: http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you consent to the foregoing.
