[
https://issues.apache.org/jira/browse/LUCENE-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17420745#comment-17420745
]
Adrien Grand commented on LUCENE-10125:
---------------------------------------
This seems to be FSDirectory indeed since this simple patch makes merge times
for points significantly faster:
{noformat}
diff --git a/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java
b/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java
index 965268a5c00..d95784768c2 100644
--- a/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java
+++ b/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java
@@ -39,6 +39,8 @@ import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Future;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.lucene.util.BitUtil;
import org.apache.lucene.util.Constants;
import org.apache.lucene.util.IOUtils;
@@ -381,6 +383,8 @@ public abstract class FSDirectory extends BaseDirectory {
*/
static final int CHUNK_SIZE = 8192;
+ private final byte[] spare = new byte[Long.BYTES];
+
public FSIndexOutput(String name) throws IOException {
this(name, StandardOpenOption.WRITE, StandardOpenOption.CREATE_NEW);
}
@@ -403,6 +407,24 @@ public abstract class FSDirectory extends BaseDirectory {
},
CHUNK_SIZE);
}
+
+ @Override
+ public void writeShort(short i) throws IOException {
+ BitUtil.VH_LE_SHORT.set(spare, 0, i);
+ writeBytes(spare, 0, Short.BYTES);
+ }
+
+ @Override
+ public void writeInt(int i) throws IOException {
+ BitUtil.VH_LE_INT.set(spare, 0, i);
+ writeBytes(spare, 0, Integer.BYTES);
+ }
+
+ @Override
+ public void writeLong(long i) throws IOException {
+ BitUtil.VH_LE_LONG.set(spare, 0, i);
+ writeBytes(spare, 0, Long.BYTES);
+ }
}
@Override
{noformat}
But ideally we'd come up with a better approach that would avoid copying the
data twice, first in this {{spare}} buffer and then again in
{{BufferedOutputStream}}'s buffer?
> Investigate indexing throughput regression on NYC Taxis between 2021-04-12
> and 2021-05-24
> -----------------------------------------------------------------------------------------
>
> Key: LUCENE-10125
> URL: https://issues.apache.org/jira/browse/LUCENE-10125
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Adrien Grand
> Priority: Minor
>
> There's been a significant drop in indexing throughput between 2021-04-12 and
> 2021-05-24 on the NYC Taxis benchmark. Unfortunately several suspects have
> been merged during that period of time so we might need to git bisect to
> figure out which one is responsible for the regression. Interestingly the
> sorted index looks less affected than the non-sorted indexes.
> https://home.apache.org/~mikemccand/lucenebench/sparseResults.html#index_throughput
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
