fanruan-tuco opened a new issue, #18521:
URL: https://github.com/apache/iceberg/issues/18521
### Apache Iceberg version
main (development).
### Query engine
Other — Java API reproduction.
### Please describe the bug 🐞
Scan planning is slow with large numbers of files, even with fast manifest
reads:
- `HadoopCatalog` + `HadoopFileIO` on local disk.
- S3 + a caching `FileIO`, with manifest cache hits.
When the result queue is empty and tasks are pending,
`ParallelIterable.hasNext()` sleeps for 10 ms. Results arriving or tasks
completing during that sleep do not wake the consumer. Repeated polling delays
can dominate fast reads.
In a synthetic test with 1,000 one-element tasks, 10 threads, 20 task slots,
and a 1 ms delayed start per task, the original took about 505 ms (50 sleeps)
versus 54 ms with notification-based waiting, approximately 9.4× faster
(medians of 10 runs after 2 warm-ups; not an end-to-end storage benchmark).
Could we wake the consumer on data arrival, completion, failure, or closure
instead of polling?
<details>
<summary>Minimal reproduction against the original implementation (imports
omitted)</summary>
```java
public class ParallelIterablePollingRepro {
public static void main(String[] args) throws Exception {
System.setProperty("iceberg.worker.num-threads", "10");
ScheduledThreadPoolExecutor workers = new
ScheduledThreadPoolExecutor(10) {
@Override
public void execute(Runnable command) {
schedule(command, 1, TimeUnit.MILLISECONDS);
}
};
workers.prestartAllCoreThreads();
try {
List<Iterable<Integer>> inputs = Collections.nCopies(1000, List.of(1));
long start = System.nanoTime();
int count = 0;
try (ParallelIterable<Integer> results = new
ParallelIterable<>(inputs, workers)) {
for (Integer ignored : results) {
count++;
}
}
System.out.printf("items=%d elapsed=%.3f ms%n", count,
(System.nanoTime() - start) / 1_000_000.0);
} finally {
workers.shutdownNow();
workers.awaitTermination(5, TimeUnit.SECONDS);
}
}
}
```
</details>
Related: [#10691
discussion](https://github.com/apache/iceberg/pull/10691#discussion_r1685429472),
[#4596
discussion](https://github.com/apache/iceberg/pull/4596#discussion_r882301640).
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
