I recommend to look at the underlying problem that you try to solve. Writing an own loader requires thorough technical design (eg recoverability in case of errors, stoping in case user requested it, proper multithreading without overloading the cluster etc) - I have not seen many that were well written. Furthermore your performance issue might be due to how you configured solr.
You can multithread in DiH by having multiple dih working on part of the data. Where the data (eg kafka ) does not matter , I really recommend to look in the problem you try to solve and then maybe let your design review here. > Am 31.01.2019 um 11:55 schrieb Srinivas Kashyap <srini...@bamboorose.com>: > > Hello, > > As we all know DIH is single threaded and has it's own issues while indexing. > > Got to know that we can write our own API's to pull data from DB and push it > into solr. One such I heard was Apache Kafka being used for the purpose. > > Can any of you send me the links and guides to use apache kafka to pull data > from DB and push into solr? > > If there are any other alternatives please suggest. > > Thanks and Regards, > Srinivas Kashyap > ________________________________ > DISCLAIMER: > E-mails and attachments from Bamboo Rose, LLC are confidential. > If you are not the intended recipient, please notify the sender immediately > by replying to the e-mail, and then delete it without making copies or using > it in any way. > No representation is made that this email or any attachments are free of > viruses. Virus scanning is recommended and is the responsibility of the > recipient.
