neoremind opened a new pull request, #16119: URL: https://github.com/apache/lucene/pull/16119
Inspired by #13543 for `write(int)`, I was profiling the write path and noticed a similar optimization can be applied to `writeBytes`, this can eliminate unnecessary `growIfNeeded` checks and the large-byte-array bypass checks. **Why this is safe:** 1. `XBufferedOutputStream` uses a fixed 8KB buffer, it never resizes, `growIfNeeded` is pure overhead. 2. The fallback to `super.write(b, off, len)` retains the existing behavior for large byte arrays that should bypass buffering entirely and chunk them later for real I/O. ## Verification Since Lucene indexing is CPU-intensive and the I/O path isn't the time-consuming part, there is no significant processing time difference in luceneutil nightly benchmarks. But I did profiling with JFR and verified the overhead is completely removed. Note that I can only sample `growIfNeeded` since it's a method call, the large-byte-array bypass check is inlined and hard to measure, so the real benefit should be slightly better than what the numbers below show. ### Wikipedia medium 33M docs, single-thread, with dv, wait for commit/merge, use CMS and TieredMergePolicy | | growIfNeeded samples | Total CPU samples | Ratio | |---|---|---|---| | **Baseline** | 26 | 409,250 | 0.01% | See Appendix 2. for detailed call stack captured. [flamegraph link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/bench-index-bench_medium-wikimediumall.lucene.baseline.Lucene104.nd33.3326M-2026.05.23.14.42.03-baseline.html) [log link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/bench_medium.wikimediumall.lucene.baseline.Lucene104.nd33.3326M.log) What to call out is that `writeBytes` is not the dominanting write op in `OutputStreamIndexOutput`, only 5% of total calls. But the savings are still positive. In a 33M wikipedia indexing, it removes 245M unnecessary `growIfNeeded` and large-byte-array bypass checks. I intercepted `OutputStreamIndexOutput` to count every write call (see how-to in Appendix 1). The stats show `writeInt` dominates by call count (61.4%), and `writeBytes` handles nearly half the total bytes despite being only 5% of calls. And 93% of `writeBytes` calls are less than 128 bytes, they all hit the optimization. ``` WriteStats: totalOps=4,875,075,375 totalBytes=26,174,155,590 writeByte: 1,627,290,133 ( 33.4% calls, 6.2% bytes) writeBytes: 245,247,290 ( 5.0% calls, 47.9% bytes, avgLen=51) writeInt: 2,991,534,308 ( 61.4% calls, 45.7% bytes) writeLong: 5,474,411 ( 0.1% calls, 0.2% bytes) writeShort: 5,529,233 ( 0.1% calls, 0.0% bytes) writeBytes size distribution: [1-16]: 164,778,837 [17-128]: 65,456,422 [129-1K]: 14,993,877 [1K-8K]: 1,722 [>8K]: 16,432 ``` ### KNN indexing 1M 1024-dim float32 vectors, wait for commit/merge, use CMS, force merge at last | | growIfNeeded samples | Total CPU samples | Ratio | |---|---|---|---| | **Baseline** | 4 | 1,943,015 | 0.0002% | See Appendix 2. for detailed call stack captured. [flamegraph link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/knn-perf-test.2026-05-22-before.html) [log link](https://neoremind.com/report/log/lucene/OutputStreamIndexOutput-write-bytes/knnPerfTest.py_20260522_070450.log) For knn, `writeBytes` dominates at 78.4% of all write calls, indexing 1M vectors involves tens of millions of `writeBytes` ops, and all of them benefit from the fast path. But since knn is super CPU-intensive with vectorized dot product dominating the profiling, the small improvement diminishes as it's barely visible in the samples, but still a micro-level win. ``` During multi-thread indexing with 1024-dim float32 vector, rerank to improve precision. Wait for commit and merge to complete: WriteStats: totalOps=55,596,839 totalBytes=12,931,045,866 writeByte: 11,985,442 ( 21.6% calls, 0.1% bytes) writeBytes: 43,597,865 ( 78.4% calls, 99.9% bytes, avgLen=296) writeInt: 4,752 ( 0.0% calls, 0.0% bytes) writeLong: 8,574 ( 0.0% calls, 0.0% bytes) writeShort: 206 ( 0.0% calls, 0.0% bytes) writeBytes size distribution: [1-16]: 40,481,940 [17-128]: 7,426 [129-1K]: 32,581 [1K-8K]: 3,075,918 [>8K]: 0 Follow-up merge to 1 segment: WriteStats: totalOps=21,901,875 totalBytes=4,252,427,320 writeByte: 4,719,616 ( 21.5% calls, 0.1% bytes) writeBytes: 17,180,254 ( 78.4% calls, 99.9% bytes, avgLen=247) writeInt: 38 ( 0.0% calls, 0.0% bytes) writeLong: 1,966 ( 0.0% calls, 0.0% bytes) writeShort: 1 ( 0.0% calls, 0.0% bytes) writeBytes size distribution: [1-16]: 16,169,470 [17-128]: 478 [129-1K]: 9,791 [1K-8K]: 1,000,515 [>8K]: 0 ``` ## Write path deep dive `FSDirectory.FSIndexOutput` writes data through a layered stack: ``` Lucene write call (writeByte, writeBytes, writeShort, writeInt, writeLong) → XBufferedOutputStream (heap byte[] buffer, 8KB) → CheckedOutputStream (CRC32 checksum, inline on flush) → FilterOutputStream (chunks writes to ≤ CHUNK_SIZE) → ChannelOutputStream → FileChannelImpl.write(HeapByteBuffer) → sun.nio.ch.IOUtil.write() → getTemporaryDirectBuffer() (thread-local cached DirectByteBuffer) → memcpy heap → direct → pwrite() syscall (kernel page cache → disk) ``` For the common case that small writes that fit in the buffer, do `System.arraycopy`, no unnecessary checks. For large arrays that the buffer can't hold, fall to `BufferedOutputStream.write()` which flushes the buffer and writes directly to the underlying stream, the `FilterOutputStream` in `FSDirectory` then chunks those large writes to less than 8K before hitting the file channel, preventing unwanted larger native buffer allocation. This is true, so I've also updated the stale `CHUNK_SIZE` comment as the original comment is a bit misleading. <img width="2777" height="3326" alt="wite-path-deep-dive" src="https://github.com/user-attachments/assets/e5e84a3c-a95d-4203-a9ff-d43e1c3f2341"; /> ## Appendix 1. Statistical analysis method of lucene write path Write statistics were captured using a modified luceneutil that intercepts all write calls and performs statistical analysis after indexing completes: [luceneutil commit](https://github.com/neoremind/luceneutil/commit/e35084338e2e45f004310f1d8b0f3f4492e76349#diff-9a2d54be8c91660fcf380e6df9944630a045021e5045a05754e1534277296714). ## Appendix 2. Detailed call stacks hitting growIfNeeded captured by JFR <details> <summary>Baseline top stacks of wikipedia 33MM</summary> --- Count: 3 --- java.io.BufferedOutputStream.growIfNeeded() line: 138 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 2 --- java.io.BufferedOutputStream.growIfNeeded() line: 135 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter.merge() line: 666 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279 org.apache.lucene.index.SegmentMerger.mergePoints() line: 197 org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 155 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 --- Count: 2 --- java.io.BufferedOutputStream.growIfNeeded() line: 137 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 2 --- java.io.BufferedOutputStream.growIfNeeded() line: 135 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609 org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209 org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604 org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156 org.apache.lucene.index.PointValuesWriter.flush() line: 317 org.apache.lucene.index.IndexingChain.writePoints() line: 408 org.apache.lucene.index.IndexingChain.flush() line: 318 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 2 --- java.io.BufferedOutputStream.growIfNeeded() line: 138 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609 org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209 org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604 org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156 org.apache.lucene.index.PointValuesWriter.flush() line: 317 org.apache.lucene.index.IndexingChain.writePoints() line: 408 org.apache.lucene.index.IndexingChain.flush() line: 318 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 2 --- java.io.BufferedOutputStream.growIfNeeded() line: 138 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.FieldsConsumer.merge() line: 94 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190 org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223 org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 147 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 138 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 142 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 491 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.finishTerm() line: 576 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 172 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 138 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlock() line: 944 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlocks() line: 636 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.pushTerm() line: 1039 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 995 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 142 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter.merge() line: 666 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279 org.apache.lucene.index.SegmentMerger.mergePoints() line: 197 org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 155 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 138 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.writeLevel1SkipData() line: 535 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 487 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.startDoc() line: 259 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 144 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.FieldsConsumer.merge() line: 94 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190 org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223 org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 147 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 134 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609 org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209 org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604 org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156 org.apache.lucene.index.PointValuesWriter.flush() line: 317 org.apache.lucene.index.IndexingChain.writePoints() line: 408 org.apache.lucene.index.IndexingChain.flush() line: 318 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 137 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 134 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlock() line: 926 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.writeBlocks() line: 666 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.pushTerm() line: 1039 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 995 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 134 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter.merge() line: 666 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.merge() line: 279 org.apache.lucene.index.SegmentMerger.mergePoints() line: 197 org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1ca8e0.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 155 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 142 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.writeLevel1SkipData() line: 535 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.flushDocBlock() line: 487 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.startDoc() line: 259 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 144 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write() line: 161 org.apache.lucene.index.FreqProxTermsWriter.flush() line: 133 org.apache.lucene.index.IndexingChain.flush() line: 361 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 134 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.ByteBuffersDataOutput.copyTo() line: 338 org.apache.lucene.codecs.lucene90.LZ4WithPresetDictCompressionMode$LZ4WithPresetDictCompressor.compress() line: 194 org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.flush() line: 259 org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter.finishDocument() line: 191 org.apache.lucene.index.StoredFieldsConsumer.finishDocument() line: 94 org.apache.lucene.index.IndexingChain.finishStoredFields() line: 565 org.apache.lucene.index.IndexingChain.processDocument() line: 657 org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments() line: 291 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 428 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 137 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValuesRange() line: 1496 org.apache.lucene.util.bkd.BKDWriter.writeHighCardinalityLeafBlockPackedValues() line: 1422 org.apache.lucene.util.bkd.BKDWriter.writeLeafBlockPackedValues() line: 1358 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.writeLeafBlock() line: 859 org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.add() line: 754 org.apache.lucene.util.bkd.BKDWriter$2.visit() line: 609 org.apache.lucene.index.PointValuesWriter$1.visitDocValues() line: 209 org.apache.lucene.util.bkd.BKDWriter.writeField1Dim() line: 604 org.apache.lucene.util.bkd.BKDWriter.writeField() line: 449 org.apache.lucene.codecs.lucene90.Lucene90PointsWriter.writeField() line: 156 org.apache.lucene.index.PointValuesWriter.flush() line: 317 org.apache.lucene.index.IndexingChain.writePoints() line: 408 org.apache.lucene.index.IndexingChain.flush() line: 318 org.apache.lucene.index.DocumentsWriterPerThread.flush() line: 492 org.apache.lucene.index.DocumentsWriter.doFlush() line: 541 org.apache.lucene.index.DocumentsWriter.postUpdate() line: 404 org.apache.lucene.index.DocumentsWriter.updateDocuments() line: 450 org.apache.lucene.index.IndexWriter.updateDocuments() line: 1600 org.apache.lucene.index.IndexWriter.updateDocument() line: 1882 org.apache.lucene.index.IndexWriter.addDocument() line: 1491 perf.IndexThreads$IndexThread.run() line: 419 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 135 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.codecs.lucene104.PForUtil.encode() line: 104 org.apache.lucene.codecs.lucene104.Lucene104PostingsWriter.addPosition() line: 342 org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm() line: 159 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter$TermsWriter.write() line: 988 org.apache.lucene.codecs.lucene103.blocktree.Lucene103BlockTreeTermsWriter.write() line: 390 org.apache.lucene.codecs.FieldsConsumer.merge() line: 94 org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge() line: 190 org.apache.lucene.index.SegmentMerger.mergeTerms() line: 223 org.apache.lucene.index.SegmentMerger$$Lambda.0x000000004f1c48b8.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 147 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 Total CPU samples: 409250 Matched: 26 Percentage: 0.01% </details> <details> <summary>Baseline top stacks of knn 1MM</summary> --- Count: 3 --- java.io.BufferedOutputStream.growIfNeeded() line: 134 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.util.GroupVIntUtil.writeGroupVInts() line: 173 org.apache.lucene.store.DataOutput.writeGroupVInts() line: 358 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeGraph() line: 550 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph() line: 484 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0() line: 451 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda.0x000000003019a9e0.run() line: -1 org.apache.lucene.codecs.KnnVectorsWriter.merge() line: 144 org.apache.lucene.index.SegmentMerger.mergeVectorValues() line: 272 org.apache.lucene.index.SegmentMerger$$Lambda.0x0000000030198000.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 159 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 knn.KnnIndexer$TrackingConcurrentMergeScheduler.doMerge() line: 360 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 --- Count: 1 --- java.io.BufferedOutputStream.growIfNeeded() line: 137 java.io.BufferedOutputStream.write() line: 186 org.apache.lucene.store.OutputStreamIndexOutput.writeBytes() line: 59 org.apache.lucene.store.RateLimitedIndexOutput.writeBytes() line: 60 org.apache.lucene.store.DataOutput.writeBytes() line: 54 org.apache.lucene.util.GroupVIntUtil.writeGroupVInts() line: 173 org.apache.lucene.store.DataOutput.writeGroupVInts() line: 358 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.writeGraph() line: 550 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.buildAndWriteGraph() line: 484 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter.lambda$mergeOneField$0() line: 451 org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter$$Lambda.0x000000003019a9e0.run() line: -1 org.apache.lucene.codecs.KnnVectorsWriter.merge() line: 144 org.apache.lucene.index.SegmentMerger.mergeVectorValues() line: 272 org.apache.lucene.index.SegmentMerger$$Lambda.0x0000000030198000.merge() line: -1 org.apache.lucene.index.SegmentMerger.mergeWithLogging() line: 315 org.apache.lucene.index.SegmentMerger.merge() line: 159 org.apache.lucene.index.IndexWriter.mergeMiddle() line: 5322 org.apache.lucene.index.IndexWriter.merge() line: 4785 org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge() line: 6590 org.apache.lucene.index.ConcurrentMergeScheduler.doMerge() line: 661 knn.KnnIndexer$TrackingConcurrentMergeScheduler.doMerge() line: 360 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run() line: 726 Total CPU samples: 1943015 Matched: 4 Percentage: 0.00% </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
