GGraziadei commented on PR #16777:
URL: https://github.com/apache/iceberg/pull/16777#issuecomment-6602189233
The specification states that a write-default expression is evaluated for
any record written after the field was added, provided the writer does not
supply a value. This defines which records receive a default, but leaves the
evaluation frequency of non-deterministic functions ambiguous: once per row,
per write operation, or per file/commit.
For example, ANSI SQL (and Spark) evaluate current_timestamp() once per
statement, so a batch "INSERT INTO t SELECT ..." writing 10M rows to a table
with a created_at TIMESTAMP write-default of current_timestamp() would assign
the same timestamp to every row. In contrast, a streaming sink might evaluate
the expression per record or per checkpoint, producing different timestamps
across rows. Furthermore, after checkpoint replay, re-emitted records could
receive new values, making the write non-idempotent. Consequently, the same
table metadata could produce different results across engines. To avoid this
ambiguity, I suggest explicitly specifying that non-deterministic functions in
write-default expressions must be evaluated at most once per write operation,
consistent with SQL statement-level semantics. Alternatively, if per-record
evaluation is intended, the specification should state this explicitly to
ensure consistent behavior across engines.
It would also be helpful to clarify that a writer unable to resolve or
evaluate a write-default expression must fail the write rather than silently
write NULL or omit the field.
Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
