CTTY commented on code in PR #3662:
URL: https://github.com/apache/iceberg-rust/pull/3662#discussion_r4533943110
##########
crates/iceberg/src/transaction/snapshot/simple.rs:
##########
@@ -128,24 +126,24 @@ impl SimpleSnapshotProducer {
.await
}
- /// Checks that every file is written under the table's default partition
- /// spec with a compatible partition value.
+ /// Checks that every file uses a known partition spec with a compatible
+ /// partition value.
///
/// Runs against the table the manifests are written for; a later rebase
/// does not invalidate manifests already written under a spec the table
/// still has.
fn validate_added_data_files(&self, table: &Table) -> Result<()> {
let metadata = table.metadata();
for data_file in &self.intent.data_files {
- // TODO(#3500): accept any spec known to the table and group the
- // files by spec when writing manifests, instead of requiring the
- // default spec.
- if metadata.default_partition_spec_id() !=
data_file.partition_spec_id {
- return Err(invalid_data!(
- "Data file partition spec id does not match table default
partition spec id"
- ));
- }
- validate_partition_value(data_file.partition(),
metadata.default_partition_type())?;
+ let spec = metadata
+ .partition_spec_by_id(data_file.partition_spec_id)
+ .ok_or_else(|| {
+ invalid_data!("Unknown partition spec id {}",
data_file.partition_spec_id)
Review Comment:
+1 on both points mentioned here
##########
crates/iceberg/src/transaction/snapshot/simple.rs:
##########
@@ -180,28 +178,25 @@ impl SimpleSnapshotProducer {
self.validate_added_data_files(table)?;
- // Allocate every path before writing, so each is recorded as owned
- // even if the write fails part-way: the path is write-once and a
- // partial file is an orphan to clean up like any other.
- //
- // TODO(#3500): group the files by partition spec and write one
- // manifest per group.
+ // BTreeMap keeps added manifests in deterministic partition spec
order.
+ let mut files_by_spec: BTreeMap<i32, Vec<&DataFile>> = BTreeMap::new();
+ for file in &self.intent.data_files {
Review Comment:
we are iterating the data_files from intent twice. Once in
`validate_added_data_files` and another time here. how about having
`validated_added_data_files` return `Result<files_by_spec>` directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
