mikemccand commented on a change in pull request #1912: URL: https://github.com/apache/lucene-solr/pull/1912#discussion_r498242843
########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are + // greater than or equal to 64 are densely populated + // Items closed to the beginning of this list are more likely to have a Review comment: s/`closed`/`close`? ########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are Review comment: s/`sparely`/`sparsely`? ########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are + // greater than or equal to 64 are densely populated + // Items closed to the beginning of this list are more likely to have a + // higher weight. + private final List<T> slots = new ArrayList<>(Long.SIZE); + + // A bitset where ones indicate that the corresponding index in `slots` is taken. + private long usedSlots = 0L; + + ApproximatePriorityQueue() { + for (int i = 0; i < Long.SIZE; ++i) { + slots.add(null); + } + } + + /** + * Add an entry to this queue that has the provided weight. + */ + void add(T entry, long weight) { + if (entry == null) { + throw new NullPointerException(); + } + + final int expectedSlot = Long.numberOfLeadingZeros(weight); + + // If the slot is already taken, we take the next one that is free. + final long freeSlots = ~usedSlots; + int destinationSlot = expectedSlot + Long.numberOfTrailingZeros(freeSlots >>> expectedSlot); Review comment: This magical math is hard to understand :) Can you add some comments, e.g. "this is a faster way to do this more obvious thing: ..."? I think the gist is that you could have used a `for` loop to (linearly) walk, looking for the next free slot, but instead we are doing these bitwise operations to make it O(1) cost? Maybe `assert destinationSlot >= expectedSlot`? ########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are + // greater than or equal to 64 are densely populated + // Items closed to the beginning of this list are more likely to have a + // higher weight. + private final List<T> slots = new ArrayList<>(Long.SIZE); + + // A bitset where ones indicate that the corresponding index in `slots` is taken. + private long usedSlots = 0L; + + ApproximatePriorityQueue() { + for (int i = 0; i < Long.SIZE; ++i) { + slots.add(null); + } + } + + /** + * Add an entry to this queue that has the provided weight. + */ + void add(T entry, long weight) { + if (entry == null) { + throw new NullPointerException(); + } + + final int expectedSlot = Long.numberOfLeadingZeros(weight); + + // If the slot is already taken, we take the next one that is free. + final long freeSlots = ~usedSlots; + int destinationSlot = expectedSlot + Long.numberOfTrailingZeros(freeSlots >>> expectedSlot); + if (destinationSlot < Long.SIZE) { + usedSlots |= 1L << destinationSlot; + slots.set(destinationSlot, entry); Review comment: Maybe `assert slots.get(destinationSlot) == null`? ########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are + // greater than or equal to 64 are densely populated + // Items closed to the beginning of this list are more likely to have a + // higher weight. + private final List<T> slots = new ArrayList<>(Long.SIZE); + + // A bitset where ones indicate that the corresponding index in `slots` is taken. Review comment: Oh how I wish GH and IDEs and Emacs and others would render your \`slots\` corrrectly! ########## File path: lucene/core/src/java/org/apache/lucene/index/DocumentsWriterPerThreadPool.java ########## @@ -112,19 +110,12 @@ private synchronized DocumentsWriterPerThread newWriter() { DocumentsWriterPerThread getAndLock() { synchronized (this) { ensureOpen(); - // Important that we are LIFO here! This way if number of concurrent indexing threads was once high, - // but has now reduced, we only use a limited number of DWPTs. This also guarantees that if we have suddenly - // a single thread indexing - final Iterator<DocumentsWriterPerThread> descendingIterator = freeList.descendingIterator(); - while (descendingIterator.hasNext()) { - DocumentsWriterPerThread perThread = descendingIterator.next(); - if (perThread.tryLock()) { - descendingIterator.remove(); - return perThread; - } + DocumentsWriterPerThread dwpt = freeList.poll(DocumentsWriterPerThread::tryLock); + if (dwpt == null) { + // DWPT is already locked before return by this method: Review comment: Hmm, can you put this comment back in its original place (just before the `return dwpt`? It is confusing here, making me think the "allocate a new DWPT" case has something to do with the locking semantics. ########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are + // greater than or equal to 64 are densely populated + // Items closed to the beginning of this list are more likely to have a + // higher weight. + private final List<T> slots = new ArrayList<>(Long.SIZE); + + // A bitset where ones indicate that the corresponding index in `slots` is taken. + private long usedSlots = 0L; + + ApproximatePriorityQueue() { + for (int i = 0; i < Long.SIZE; ++i) { + slots.add(null); + } + } + + /** + * Add an entry to this queue that has the provided weight. + */ + void add(T entry, long weight) { + if (entry == null) { Review comment: Maybe switch to `assert`? ########## File path: lucene/core/src/java/org/apache/lucene/index/ApproximatePriorityQueue.java ########## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.util.ArrayList; +import java.util.List; +import java.util.ListIterator; +import java.util.function.Predicate; + +/** + * An approximate priority queue, which attempts to poll items by decreasing + * log of the weight, though exact ordering is not guaranteed. + * This class doesn't support null elements. + */ +final class ApproximatePriorityQueue<T> { + + // Indexes between 0 and 63 are sparely populated, and indexes that are + // greater than or equal to 64 are densely populated + // Items closed to the beginning of this list are more likely to have a + // higher weight. + private final List<T> slots = new ArrayList<>(Long.SIZE); + + // A bitset where ones indicate that the corresponding index in `slots` is taken. + private long usedSlots = 0L; + + ApproximatePriorityQueue() { + for (int i = 0; i < Long.SIZE; ++i) { + slots.add(null); + } + } + + /** + * Add an entry to this queue that has the provided weight. + */ + void add(T entry, long weight) { + if (entry == null) { + throw new NullPointerException(); + } + + final int expectedSlot = Long.numberOfLeadingZeros(weight); + + // If the slot is already taken, we take the next one that is free. + final long freeSlots = ~usedSlots; + int destinationSlot = expectedSlot + Long.numberOfTrailingZeros(freeSlots >>> expectedSlot); + if (destinationSlot < Long.SIZE) { + usedSlots |= 1L << destinationSlot; + slots.set(destinationSlot, entry); + } else { + slots.add(entry); Review comment: My box might be the only thing that exercises this line!! Oh hmm, actually, `slots` can be used even when there are fewer than 64 concurrent indexing threads, since the dense 0 .. 63 case is based on `log(ramBytesUsed)`. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org