Jackie-Jiang commented on code in PR #12538:
URL: https://github.com/apache/pinot/pull/12538#discussion_r1552634492
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java:
##########
@@ -44,8 +73,31 @@ public static Object hashPrimaryKey(PrimaryKey primaryKey,
HashFunction hashFunc
return new ByteArray(HashUtils.hashMD5(primaryKey.asBytes()));
case MURMUR3:
return new ByteArray(HashUtils.hashMurmur3(primaryKey.asBytes()));
+ case UUID:
+ return new ByteArray(HashUtils.hashUUID(primaryKey.getValues()));
default:
throw new IllegalArgumentException(String.format("Unrecognized hash
function %s", hashFunction));
}
}
+
+ /**
+ * Concatenates the string representation of all values into a single byte
array. Each element is prepended with its
+ * 4-byte length to ensure no collisions can happen. (sacrifice space
efficiency for correctness)
+ */
+ private static byte[] concatenate(Object[] values) {
+ byte[][] allValueBytes = new byte[values.length][];
+ int totalLen = 0;
+ for (int j = 0; j < allValueBytes.length; j++) {
+ allValueBytes[j] = values[j] == null ?
"null".getBytes(StandardCharsets.UTF_8)
Review Comment:
This is dangerous. This means we treat string "null" and real null the same
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/HashUtils.java:
##########
@@ -36,6 +40,31 @@ public static byte[] hashMD5(byte[] bytes) {
return Hashing.md5().hashBytes(bytes).asBytes();
}
+ /**
+ * Returns a byte array that is a concatenation of the binary representation
of each of the passed UUID values. This
+ * is done by getting a String from each value by calling {@link
Object#toString()}, which is then used to create a
+ * {@link UUID} object. The 16 bytes of each UUID are appended to a buffer
which is then returned in the result.
+ * If any of the values is not a valid UUID, then we return the result of
{@link #concatenate}.
+ */
+ public static byte[] hashUUID(Object[] values) {
+ byte[] result = new byte[values.length * 16];
+ ByteBuffer byteBuffer =
ByteBuffer.wrap(result).order(ByteOrder.BIG_ENDIAN);
+ for (Object value : values) {
+ if (value == null) {
+ return concatenate(values);
Review Comment:
Silently fallback can cause performance issue and is very hard to debug. I'd
suggest not falling back but directly throw exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
