dungba88 commented on code in PR #12624: URL: https://github.com/apache/lucene/pull/12624#discussion_r1400633005
########## lucene/core/src/java/org/apache/lucene/util/fst/GrowableByteArrayDataOutput.java: ########## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.util.fst; + +import java.io.IOException; +import org.apache.lucene.store.DataOutput; +import org.apache.lucene.util.Accountable; +import org.apache.lucene.util.ArrayUtil; +import org.apache.lucene.util.RamUsageEstimator; + +// Storing a byte[] for the current node of the FST we are writing. The byte[] will only grow, never +// shrink. +final class GrowableByteArrayDataOutput extends DataOutput implements Accountable { + + private static final long BASE_RAM_BYTES_USED = + RamUsageEstimator.shallowSizeOfInstance(GrowableByteArrayDataOutput.class); + + private static final int INITIAL_SIZE = 1 << 8; + + // holds an initial size of 256 bytes. this byte array will only grow, but not shrink + private byte[] bytes = new byte[INITIAL_SIZE]; + + private int nextWrite; + + @Override + public void writeByte(byte b) { + ensureCapacity(1); + bytes[nextWrite++] = b; + } + + @Override + public void writeBytes(byte[] b, int offset, int len) { + ensureCapacity(len); + System.arraycopy(b, offset, bytes, nextWrite, len); + nextWrite += len; + } + + public int getPosition() { + return nextWrite; + } + + public byte[] getBytes() { + return bytes; + } + + /** Set the position of the byte[], increasing the capacity if needed */ + public void setPosition(int newLen) { + assert newLen >= 0; + if (newLen > nextWrite) { + ensureCapacity(newLen - nextWrite); + } + nextWrite = newLen; Review Comment: Yeah I think don't need to shrink for now. It would also reduce the GC load and the times we need to grow the array (comparing to when we shrink and expand the `byte[]` as it goes). I ran with a custom dictionary of size ~1MB and the maximum node is <256 bytes. For text-based dictionary I think it would depend on the length of the word (which is usually short) and the output (which depends). Large dictionary which has larger node addresses would also require more RAM, which is also ideal for us. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
