amogh-jahagirdar commented on code in PR #13310:
URL: https://github.com/apache/iceberg/pull/13310#discussion_r2191101009
##########
spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkWriteBuilder.java:
##########
@@ -116,16 +119,36 @@ public WriteBuilder overwrite(Filter[] filters) {
@Override
public Write build() {
- // Validate
- Schema writeSchema = validateOrMergeWriteSchema(table, dsSchema,
writeConf);
+ // The write schema should only include row lineage in the output if it's
an overwrite
+ // operation.
+ // In any other case, only null row IDs and sequence numbers would be
produced which
+ // means the row lineage columns can be excluded from the output files
+ boolean writeIncludesRowLineage = TableUtil.supportsRowLineage(table) &&
overwriteFiles;
+ StructType sparkWriteSchema = dsSchema;
+ if (writeIncludesRowLineage) {
+ sparkWriteSchema = sparkWriteSchema.add("_row_id", LongType$.MODULE$);
Review Comment:
use the constant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
