Vikasht34 opened a new issue, #15841:
URL: https://github.com/apache/lucene/issues/15841
### Description
### Description
The TaskExecutor.invokeAll() refactor in Lucene 10 (removal of TaskGroup,
introduction of shared-lambda work-stealing via AtomicInteger) introduced a
behavioral regression compared to Lucene 9.12. When the number of tasks
submitted to
invokeAll() is significantly larger than the executor's thread pool size,
the calling thread steals a disproportionate number of tasks, starving the
executor pool threads.
### Root Cause
Lucene 9.12 (TaskGroup.invokeAll): Each task was submitted as its own
RunnableFuture directly to the executor. Pool threads picked up dedicated tasks
with zero contention. The calling thread ran at most the remaining tasks after
all others were
submitted.
java
```
// 9.12: each pool thread gets a dedicated FutureTask
for (int i = futures.size() - 1; i > 0; i--) {
executor.execute(futures.get(i));
}
futures.get(0).run(); // calling thread runs exactly 1
```
Lucene 10.x (TaskExecutor.invokeAll): All N-1 submissions share a single
lambda that races on an AtomicInteger counter. The calling thread enters a
tight while loop doing getAndIncrement(), winning the CAS race repeatedly
because it is already
running (no wake-up latency), while pool threads must: (1) be scheduled by
the OS, (2) enter the lambda, (3) compete on the same AtomicInteger.
```
java
// 10.x: shared lambda, all threads race on one AtomicInteger
final AtomicInteger taskId = new AtomicInteger(0);
final Runnable work = () -> {
int id = taskId.getAndIncrement();
if (id < count) { futures.get(id).run(); }
};
for (int j = 0; j < count - 1; j++) {
executor.execute(work);
}
// calling thread steals in tight loop:
while ((id = taskId.getAndIncrement()) < count) {
futures.get(id).run();
}
```
### Impact
With a small number of tasks (e.g., 5 segment slices), the regression is
negligible — the calling thread may steal 1 extra task.
With a large number of tasks (e.g., 100 per-leaf tasks as used by OpenSearch
k-NN plugin's rescore), the calling thread steals 10-20+ tasks before pool
threads wake up. This keeps the calling thread busy for the duration of those
tasks, effectively reducing the system's ability to process other requests on
that thread.
### Empirical Evidence
Observed in a mixed ingestion + search workload on ARM/Graviton (aarch64,
JDK 21):
| Metric | Lucene 9.12 | Lucene 10.3.1 |
|---|---|---|
| Calling thread CPU | 2-3% each | 5-7% each |
| Executor pool thread CPU | 10-18% each (8-9 active) | 3-4% each (1-2
active) |
| Thread pool queue at ~300k docs | 0 | 101 |
| Max sustainable throughput | 600k+ docs | ~315k docs (then queue
saturation) |
The workload submits ~100 tasks (one per leaf reader context) to
invokeAll(). In Lucene 9.12, the pool threads did the heavy lifting. In Lucene
10.3.1, the calling thread steals a large fraction of the work.
Stack traces confirm the difference:
- **9.12**: TaskExecutor$TaskGroup$1.run(TaskExecutor.java:120) — pool
thread runs a dedicated FutureTask
- **10.3.1**: TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98) — pool
thread runs shared lambda with getAndIncrement()
### Version
Regression introduced between Lucene 9.12 and 10.0 when TaskGroup was
removed and invokeAll was refactored to use a shared AtomicInteger counter.
### Related Files
- lucene/core/src/java/org/apache/lucene/search/TaskExecutor.java
- Affects any caller of IndexSearcher.getTaskExecutor().invokeAll() with
high task counts
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
