[
https://issues.apache.org/jira/browse/LUCENE-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270580#comment-17270580
]
Dawid Weiss commented on LUCENE-9670:
-------------------------------------
I can see how those settings Robert recommended do make sense in certain
situations. But maybe we can come up with global defaults that work across a
range of setups rather than have local tweaks?
The 12 CPUs limit is really arbitrary. I thought at 12 CPUs memory bandwidth
and I/O would be the bottleneck anyway - I can see it's not really the case (on
my machine too). We can increase it. I think it should be related to both the
number of cores *and* available physical memory - if you fork too many JVMs you
risk exhausting physical memory on machines that have, say, 64 CPUs but "only"
64 GB of RAM... If you can tweak it for your setup then the feedback would be
very helpful.
> gradle precommit sometimes fails with "IOException: stream closed" from
> javadoc in nightly benchmarks
> -----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-9670
> URL: https://issues.apache.org/jira/browse/LUCENE-9670
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Priority: Major
>
> I recently added tracking how long {{gradle precommit}} takes each night so
> we can track slowdowns over time.
> But it sometimes fails with:
> {noformat}
> > Task :lucene:join:renderJavadoc FAILED
> Could not read standard output of command '/opt/jdk-15.0.1/bin/javadoc'.
> java.io.IOException: Stream Closed
> at java.base/java.io.FileOutputStream.writeBytes(Native Method)
> at java.base/java.io.FileOutputStream.write(FileOutputStream.java:347)
> at
> java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
> at
> java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)
> at
> org.gradle.process.internal.streams.ExecOutputHandleRunner.forwardContent(ExecOutputHandleRunner.java:68)
> at
> org.gradle.process.internal.streams.ExecOutputHandleRunner.run(ExecOutputHandleRunner.java:53)
> at
> org.gradle.internal.operations.CurrentBuildOperationPreservingRunnable.run(CurrentBuildOperationPreservingRunnable.java:42)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
> at
> org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> at
> org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
> at java.base/java.lang.Thread.run(Thread.java:832) {noformat}
> I'm not sure why ... when I run {{./gradlew precommit}} interactively it
> doesn't seem to do this.
> The nightly tool is quite simple – it just launches a sub-process using
> {{os.system}}: (first to {{git clean}} then to run {{./gradlew precommit)}}:
> https://github.com/mikemccand/luceneutil/blob/master/src/python/runNightlyGradleTestPrecommit.py
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
