Copilot commented on code in PR #3662:
URL: https://github.com/apache/iceberg-rust/pull/3662#discussion_r4520123437
##########
crates/iceberg/src/transaction/snapshot/simple.rs:
##########
@@ -128,24 +126,24 @@ impl SimpleSnapshotProducer {
.await
}
- /// Checks that every file is written under the table's default partition
- /// spec with a compatible partition value.
+ /// Checks that every file uses a known partition spec with a compatible
+ /// partition value.
///
/// Runs against the table the manifests are written for; a later rebase
/// does not invalidate manifests already written under a spec the table
/// still has.
fn validate_added_data_files(&self, table: &Table) -> Result<()> {
let metadata = table.metadata();
for data_file in &self.intent.data_files {
- // TODO(#3500): accept any spec known to the table and group the
- // files by spec when writing manifests, instead of requiring the
- // default spec.
- if metadata.default_partition_spec_id() !=
data_file.partition_spec_id {
- return Err(invalid_data!(
- "Data file partition spec id does not match table default
partition spec id"
- ));
- }
- validate_partition_value(data_file.partition(),
metadata.default_partition_type())?;
+ let spec = metadata
+ .partition_spec_by_id(data_file.partition_spec_id)
+ .ok_or_else(|| {
+ invalid_data!("Unknown partition spec id {}",
data_file.partition_spec_id)
+ })?;
+ validate_partition_value(
+ data_file.partition(),
+ &spec.partition_type(metadata.current_schema())?,
Review Comment:
After switching to an unpartitioned spec and dropping the old
identity-partitioned column, this rejects valid files under the retained spec:
`partition_type(current_schema())` produces `Unknown`, which rejects non-null
partition values. Resolve a retained schema containing the spec's source fields
and use it consistently for validation, manifest writing (`simple.rs:228`), and
added-file summaries (`snapshot/mod.rs:262`), following the removed-file
summary path. Add a regression test for appending under an old spec after its
source column is dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
