MrFlap commented on code in PR #15952:
URL: https://github.com/apache/lucene/pull/15952#discussion_r3120176644
##########
lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsWriter.java:
##########
@@ -96,18 +107,22 @@ public final void merge(MergeState mergeState) throws
IOException {
}
}
- for (FieldInfo fieldInfo : mergeState.mergeFieldInfos) {
- if (fieldInfo.hasVectorValues()) {
- if (mergeState.infoStream.isEnabled("VV")) {
- mergeState.infoStream.message("VV", "merging " +
mergeState.segmentInfo);
- }
+ try {
+ for (FieldInfo fieldInfo : mergeState.mergeFieldInfos) {
+ if (fieldInfo.hasVectorValues()) {
+ if (mergeState.infoStream.isEnabled("VV")) {
+ mergeState.infoStream.message("VV", "merging " +
mergeState.segmentInfo);
+ }
- mergeOneField(fieldInfo, mergeState);
+ mergeOneField(fieldInfo, mergeState);
- if (mergeState.infoStream.isEnabled("VV")) {
- mergeState.infoStream.message("VV", "merge done " +
mergeState.segmentInfo);
+ if (mergeState.infoStream.isEnabled("VV")) {
+ mergeState.infoStream.message("VV", "merge done " +
mergeState.segmentInfo);
+ }
}
}
+ } finally {
+ afterMerge();
}
finish();
Review Comment:
Our desired functionality is to have a set number `N` threads per merge.
This way we won't have resource contention between merges. So if one field
needs 50 singular merge operations, we should see `50 * N` threads spawning.
Wrapping functionality around mergeOneField is difficult because we want to
1. Allocate N threads when each singular merge operation starts.
2. Release N threads when each singular merge operation completes.
3. Be able to release N threads on a specific merge operation when an
exception happens
We would really like to keep it to a "N threads per singular merge
operation" as making it a set number of threads per field merge could lead to
resource contention per thread. We don't really see an easy way to be this
granular about how to detect and allocate resources per merge, since there are
no hooks to see when a singular merge operation happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
