tang-hi commented on code in PR #966:
URL: https://github.com/apache/lucene/pull/966#discussion_r901054938
##########
lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java:
##########
@@ -230,9 +230,29 @@ final void writeByte(int stream, byte b) {
}
final void writeBytes(int stream, byte[] b, int offset, int len) {
- // TODO: optimize
final int end = offset + len;
- for (int i = offset; i < end; i++) writeByte(stream, b[i]);
+ int streamAddress = streamAddressOffset + stream;
+ int upto = termStreamAddressBuffer[streamAddress];
+ byte[] slice = bytePool.buffers[upto >> ByteBlockPool.BYTE_BLOCK_SHIFT];
+ assert slice != null;
+ int sliceOffset = upto & ByteBlockPool.BYTE_BLOCK_MASK;
+
+ while (slice[sliceOffset] == 0 && offset < end) {
+ slice[sliceOffset++] = b[offset++];
+ (termStreamAddressBuffer[streamAddress])++;
+ }
+
+ while (offset < end) {
+ int offsetAndLength = bytePool.allocKnownSizeSlice(slice, sliceOffset);
+ sliceOffset = offsetAndLength >> 8;
+ int sliceLength = offsetAndLength & 0xff;
+ slice = bytePool.buffer;
+ int writeLength = Math.min(sliceLength - 1, end - offset);
+ System.arraycopy(b, offset, slice, sliceOffset, writeLength);
+ sliceOffset += writeLength;
+ offset += writeLength;
+ termStreamAddressBuffer[streamAddress] = sliceOffset +
bytePool.byteOffset;
+ }
Review Comment:
If we want to set `ByteSliceReader.level` to the right value, may be we
should use one bit in [buffer[byteUpto - 1]
](https://github.com/apache/lucene/blob/6ba759df866289db485d44fd1f75b3eb00f8d99f/lucene/core/src/java/org/apache/lucene/util/ByteBlockPool.java#L268-L269)
as a flag and other bits to represent the customized size.
eg.
if the highest bit is flag,`ByteSliceReader.nextSlice` will check the
highest bit and decide whether to use the old method or calculate
[newSize](https://github.com/apache/lucene/blob/6ba759df866289db485d44fd1f75b3eb00f8d99f/lucene/core/src/java/org/apache/lucene/index/ByteSliceReader.java#L101)
from the remaining bits
But this implementation needs to change a lot and needs to be fully tested.I
think this is best done in another pr.I might do it in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
