amogh-jahagirdar commented on code in PR #13555:
URL: https://github.com/apache/iceberg/pull/13555#discussion_r2220529650
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java:
##########
@@ -163,6 +165,14 @@ private Spark3Util.CatalogAndIdentifier
catalogAndIdentifier(CaseInsensitiveStri
selector = TAG_PREFIX + tag;
}
+ String groupId =
+ options.getOrDefault(
+ SparkReadOptions.SCAN_TASK_SET_ID,
+ options.get(SparkWriteOptions.REWRITTEN_FILE_SCAN_TASK_SET_ID));
+ if (groupId != null) {
+ selector = REWRITE_PREFIX + groupId.replace("-", "");
Review Comment:
Will add a comment. I'm not sure about a 120 character limit (did some
investigation couldn't find anything related) but even then I think we're still
safe for this rewrite case because for the rewrite case the identifier is the
groupId UUID and then there's the "#" followed by the selector which is another
UUID. This UUID gets mapped to the actual table reference in SparkTableCache
during the compaction job. This combined is 81 characters.
Actually saying this out loud made me realize that there's really no value
to adding the same UUID again for the rewrite case :)
So we should probably just do something like have the selector just be the
word "rewrite"
Then we just have a UUID + "#_rewrite", which is then just 45 bytes.
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java:
##########
@@ -163,6 +165,14 @@ private Spark3Util.CatalogAndIdentifier
catalogAndIdentifier(CaseInsensitiveStri
selector = TAG_PREFIX + tag;
}
+ String groupId =
+ options.getOrDefault(
+ SparkReadOptions.SCAN_TASK_SET_ID,
+ options.get(SparkWriteOptions.REWRITTEN_FILE_SCAN_TASK_SET_ID));
+ if (groupId != null) {
+ selector = REWRITE_PREFIX + groupId.replace("-", "");
Review Comment:
I'm not sure about a 120 character limit (did some investigation couldn't
find anything related) but even then I think we're still safe for this rewrite
case because for the rewrite case the identifier is the groupId UUID and then
there's the "#" followed by the selector which is another UUID. This UUID gets
mapped to the actual table reference in SparkTableCache during the compaction
job. This combined is 81 characters.
Actually saying this out loud made me realize that there's really no value
to adding the same UUID again for the rewrite case :)
So we should probably just do something like have the selector just be the
word "rewrite"
Then we just have a UUID + "#_rewrite", which is then just 45 bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
