balmukundblr commented on a change in pull request #132:
URL: https://github.com/apache/lucene/pull/132#discussion_r646609916
##########
File path:
lucene/benchmark/src/java/org/apache/lucene/benchmark/byTask/feeds/ReutersContentSource.java
##########
@@ -100,21 +100,24 @@ public void close() throws IOException {
@Override
public DocData getNextDocData(DocData docData) throws NoMoreDataException,
IOException {
- Path f = null;
- String name = null;
- synchronized (this) {
- if (nextFile >= inputFiles.size()) {
- // exhausted files, start a new round, unless forever set to false.
- if (!forever) {
- throw new NoMoreDataException();
- }
- nextFile = 0;
- iteration++;
- }
- f = inputFiles.get(nextFile++);
- name = f.toRealPath() + "_" + iteration;
+ if (docCountArrCreated == false) {
+ docCountArrInit();
}
+ //Extract ThreadIndex from unique ThreadName (at position 12), which is
set with '"IndexThread-"+index', in TaskSequence.java's doParallelTasks()
+ int threadIndex =
Integer.parseInt(Thread.currentThread().getName().substring(12));
+ assert (threadIndex >= 0 && threadIndex < docCountArr.length):"Please
check threadIndex or docCountArr length";
+ int stride = threadIndex + docCountArr[threadIndex] * docCountArr.length;
+ int inFileSize = inputFiles.size();
+
+ //Modulo Operator covers all three possible senarios i.e. 1. If
inputFiles.size() < Num Of Threads 2.inputFiles.size() == Num Of Threads
3.inputFiles.size() > Num Of Threads
+ int fileIndex = stride % inFileSize;
Review comment:
Mike, its already handling in ReutersContentSource.java's setConfig().
Please find the code snippet for the same.
if (inputFiles.size() == 0) {
throw new RuntimeException("No txt files in dataDir:
"+dataDir.toAbsolutePath());
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
