Hi, We have a requirement of huge 250 million data to be indexed and we are using DIH to index the data in batches of 10 million. We are using Solr 4.4. We are using Solr cloud with One zookeeper and two collections with two shards in each in two Tomcat servers. But the indexing fails after 40/ 50 million data being indexed. Please let me know how to mitigate this issue. I am posting the zookeeper log below.
2013-08-20 00:36:32,188 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595] - Est ablished session 0x1409a3945380000 with negotiated timeout 30000 for client /0:0 :0:0:0:0:0:1:53440 2013-08-20 00:36:33,873 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@476] - Processed session termination for sessionid: 0x1409a39453800 00 2013-08-20 00:36:33,873 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxn@1001] - Closed socket connection for client /0:0:0:0:0:0:0:1:53 440 which had sessionid 0x1409a3945380000 2013-08-20 00:36:43,826 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxnFactory@197] - Accepted socket connection from / 127.0.0.1:53449 2013-08-20 00:36:43,826 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0 .1:53449 2013-08-20 00:36:43,872 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595] - Est ablished session 0x1409a3945380001 with negotiated timeout 30000 for client /127 .0.0.1:53449 2013-08-20 00:36:44,013 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@476] - Processed session termination for sessionid: 0x1409a39453800 01 2013-08-20 00:36:44,028 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxn@1001] - Closed socket connection for client / 127.0.0.1:53449 wh ich had sessionid 0x1409a3945380001 2013-08-20 00:36:47,133 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxnFactory@197] - Accepted socket connection from / 127.0.0.1:53455 2013-08-20 00:36:47,148 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0 .1:53455 2013-08-20 00:36:47,164 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595] - Est ablished session 0x1409a3945380002 with negotiated timeout 30000 for client /127 .0.0.1:53455 2013-08-20 00:36:49,754 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@476] - Processed session termination for sessionid: 0x1409a39453800 02 2013-08-20 00:36:49,769 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxn@1001] - Closed socket connection for client / 127.0.0.1:53455 wh ich had sessionid 0x1409a3945380002 2013-08-20 00:36:54,824 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxnFactory@197] - Accepted socket connection from / 127.0.0.1:53464 2013-08-20 00:36:54,839 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0 .1:53464 2013-08-20 00:36:54,870 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595] - Est ablished session 0x1409a3945380003 with negotiated timeout 30000 for client /127 .0.0.1:53464 2013-08-20 00:36:55,011 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@476] - Processed session termination for sessionid: 0x1409a39453800 03 2013-08-20 00:36:55,058 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxn@1001] - Closed socket connection for client / 127.0.0.1:53464 wh ich had sessionid 0x1409a3945380003 2013-08-20 00:37:00,003 [myid:] - INFO [SessionTracker:ZooKeeperServer@325] - E xpiring session 0x1409617c5750005, timeout of 40000ms exceeded 2013-08-20 00:37:00,003 [myid:] - INFO [SessionTracker:ZooKeeperServer@325] - E xpiring session 0x1409617c5750004, timeout of 40000ms exceeded 2013-08-20 00:37:00,003 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@476] - Processed session termination for sessionid: 0x1409617c57500 05 2013-08-20 00:37:00,003 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@476] - Processed session termination for sessionid: 0x1409617c57500 04 2013-08-20 00:37:17,584 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:NIOServerCnxnFactory@197] - Accepted socket connection from / 127.0.0.1:53480 2013-08-20 00:37:17,584 [myid:] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:21 81:ZooKeeperServer@839] - Client attempting to establish new session at /127.0.0 .1:53480 2013-08-20 00:37:17,600 [myid:] - INFO [SyncThread:0:ZooKeeperServer@595] - Est ablished session 0x1409a3945380004 with negotiated timeout 40000 for client /127 .0.0.1:53480 2013-08-20 00:37:17,631 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x1 zxid:0x91e txntype:-1 reqpath:n/a Error Path :/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:17,662 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x3 zxid:0x91f txntype:-1 reqpath:n/a Error Path :/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:48,176 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:delete cxid:0x12 zxid:0x924 txntype:-1 reqpath:n/a Error Pat h:/live_nodes/localhost:8280_solr Error:KeeperErrorCode = NoNode for /live_nodes /localhost:8280_solr 2013-08-20 00:37:48,238 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:delete cxid:0x1c zxid:0x927 txntype:-1 reqpath:n/a Error Pat h:/overseer_elect/leader Error:KeeperErrorCode = NoNode for /overseer_elect/lead er 2013-08-20 00:37:48,254 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x21 zxid:0x929 txntype:-1 reqpath:n/a Error Pat h:/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:48,285 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x22 zxid:0x92a txntype:-1 reqpath:n/a Error Pat h:/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:48,300 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x24 zxid:0x92b txntype:-1 reqpath:n/a Error Pat h:/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:48,316 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x26 zxid:0x92c txntype:-1 reqpath:n/a Error Pat h:/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:delete cxid:0x223 zxid:0x949 txntype:-1 reqpath:n/a Error Pa th:/collections/firstcollection/leaders/shard1 Error:KeeperErrorCode = NoNode fo r /collections/firstcollection/leaders/shard1 2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:delete cxid:0x224 zxid:0x94a txntype:-1 reqpath:n/a Error Pa th:/collections/firstcollection/leaders/shard2 Error:KeeperErrorCode = NoNode fo r /collections/firstcollection/leaders/shard2 2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:delete cxid:0x225 zxid:0x94b txntype:-1 reqpath:n/a Error Pa th:/collections/secondcollection/leaders/shard2 Error:KeeperErrorCode = NoNode f or /collections/secondcollection/leaders/shard2 2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x226 zxid:0x94c txntype:-1 reqpath:n/a Error Pa th:/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x227 zxid:0x94d txntype:-1 reqpath:n/a Error Pa th:/overseer Error:KeeperErrorCode = NodeExists for /overseer 2013-08-20 00:37:51,405 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequ estProcessor@627] - Got user-level KeeperException when processing sessionid:0x1 409a3945380004 type:create cxid:0x229 zxid:0x94e txntype:-1 reqpath:n/a Error Pa th:/overseer Error:KeeperErrorCode = NodeExists for /overseer With Regards, Santanu
