aishikbh commented on code in PR #12049:
URL: https://github.com/apache/pinot/pull/12049#discussion_r1412170065
##########
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/partition/PartitionFunctionTest.java:
##########
@@ -115,6 +115,82 @@ public void testMurmurPartitioner() {
}
}
+ /**
+ * Unit test for {@link Murmur3PartitionFunction}.
+ * <ul>
+ * <li> Tests that partition values are in expected range. </li>
+ * <li> Tests that toString returns expected string. </li>
+ * <li> Tests that the partition numbers returned by partition function
with null functionConfig and
+ * functionConfig with empty seed value are equal</li>
+ * </ul>
+ */
+ @Test
+ public void testMurmur3Partitioner() {
Review Comment:
added.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Murmur3PartitionFunction.java:
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.partition;
+
+import com.google.common.base.Preconditions;
+import com.google.common.hash.Hashing;
+import java.util.Map;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+
+/**
+ * Implementation of {@link PartitionFunction} which partitions based on 32
bit murmur3 hash
+ */
+public class Murmur3PartitionFunction implements PartitionFunction {
+ private static final String NAME = "Murmur3";
+ private final int _numPartitions;
+ private final int _hashSeed;
+
+ /**
+ * Constructor for the class.
+ * @param numPartitions Number of partitions.
+ * @param functionConfig to extract configurations for the partition
function.
+ */
+ public Murmur3PartitionFunction(int numPartitions, Map<String, String>
functionConfig) {
+ Preconditions.checkArgument(numPartitions > 0, "Number of partitions must
be > 0");
+ _numPartitions = numPartitions;
+ _hashSeed = (functionConfig == null || functionConfig.get("seed") == null)
? 0
Review Comment:
Chose to keep the seed value as 0. If someone wants to add seed value they
can add.
I have added the config keys as mentioned!
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Murmur3PartitionFunction.java:
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.partition;
+
+import com.google.common.base.Preconditions;
+import com.google.common.hash.Hashing;
+import java.util.Map;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+
+/**
+ * Implementation of {@link PartitionFunction} which partitions based on 32
bit murmur3 hash
+ */
+public class Murmur3PartitionFunction implements PartitionFunction {
+ private static final String NAME = "Murmur3";
Review Comment:
I can see that we used the same implementation but for 128 bits
[here](https://github.com/apache/pinot/blob/fe072c63bd61a1fe092c68e15bc11fee03f15911/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java#L31).
So chose to use the same one, but the 32_x86 variant.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Murmur3PartitionFunction.java:
##########
@@ -0,0 +1,453 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.partition;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.hash.Hashing;
+import java.util.Map;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+
+/**
+ * Implementation of {@link PartitionFunction} which partitions based on 32
bit murmur3 hash
+ */
+public class Murmur3PartitionFunction implements PartitionFunction {
+ public static final byte INVALID_CHAR = (byte) '?';
+ private static final String NAME = "Murmur3";
+ private static final String SEED_KEY = "seed";
+ private static final String MURMUR3_VARIANT = "variant";
+ private final int _numPartitions;
+ private final int _hashSeed;
+ private final String _variant;
+
+ /**
+ * Constructor for the class.
+ * @param numPartitions Number of partitions.
+ * @param functionConfig to extract configurations for the partition
function.
+ */
+ public Murmur3PartitionFunction(int numPartitions, Map<String, String>
functionConfig) {
+ Preconditions.checkArgument(numPartitions > 0, "Number of partitions must
be > 0");
+ Preconditions.checkArgument(
+ functionConfig == null || functionConfig.get(MURMUR3_VARIANT) == null
|| functionConfig.get(MURMUR3_VARIANT)
+ .isEmpty() || functionConfig.get(MURMUR3_VARIANT).equals("x86_32")
|| functionConfig.get(MURMUR3_VARIANT)
+ .equals("x64_32"), "Murmur3 variant must be either x86_32 or
x64_32");
+ _numPartitions = numPartitions;
+
+ // default value of the hash seed is 0.
+ _hashSeed =
+ (functionConfig == null || functionConfig.get(SEED_KEY) == null ||
functionConfig.get(SEED_KEY).isEmpty()) ? 0
+ : Integer.parseInt(functionConfig.get(SEED_KEY));
+
+ // default value of the murmur3 variant is x86_32.
+ _variant =
+ (functionConfig == null || functionConfig.get(MURMUR3_VARIANT) == null
|| functionConfig.get(MURMUR3_VARIANT)
+ .isEmpty()) ? "x86_32" : functionConfig.get(MURMUR3_VARIANT);
+ }
+
+ @Override
+ public int getPartition(Object value) {
+ if (_variant.equals("x86_32")) {
+ return (murmurHash332BitsX86(value.toString().getBytes(UTF_8),
_hashSeed) & Integer.MAX_VALUE) % _numPartitions;
Review Comment:
makes sense. This name is a better name for readability.
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/partition/Murmur3PartitionFunction.java:
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.partition;
+
+import com.google.common.base.Preconditions;
+import com.google.common.hash.Hashing;
+import java.util.Map;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+
+/**
+ * Implementation of {@link PartitionFunction} which partitions based on 32
bit murmur3 hash
+ */
+public class Murmur3PartitionFunction implements PartitionFunction {
+ private static final String NAME = "Murmur3";
Review Comment:
@snleee summarised it perfectly.
Just to add : `murmur3` has 3 variants as part of the original
implementation : `x86 32 bits`, `x64 128 bits` and `x86 128 bits` and Guava
supports these implementations. Since we are generating partition numbers `x86
32 bits` should cover most of the use cases.
The issue comes from the users of Debezium. Debezium uses a variant of the
`x64 128` version reduced to 32 bits. I wanted to add that as well, as some
users might want to use `Debezium + Murmur3`. I have kept the `x86 32 bits`
variant as the default as that is part of the original implementation and `x64
32 bits` as the configurable one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
