blackmwk commented on code in PR #2916:
URL: https://github.com/apache/iceberg-rust/pull/2916#discussion_r4548466761
##########
crates/iceberg/src/spec/values/serde.rs:
##########
@@ -473,8 +485,25 @@ pub(crate) mod _serde {
},
RawLiteralEnum::String(v) => match ty {
Type::Primitive(PrimitiveType::String) =>
Ok(Some(Literal::string(v))),
+ // apache-avro deserializes a UUID as a hyphenated string.
+ // UUID values read via `Value` (i.e.
`read_data_files_from_avro`), land here.
+ // The hyphenated restriction matches behaviour with Java.
+ Type::Primitive(PrimitiveType::Uuid) =>
Hyphenated::from_str(&v)
+ .map(|hyphenated_uuid|
Some(Literal::uuid(hyphenated_uuid.into_uuid())))
+ .map_err(|_| {
+ invalid_err_with_reason(
+ "string",
+ "UUID must be a hyphenated 8-4-4-4-12 string",
+ )
+ }),
_ => Err(invalid_err("string")),
},
+ // `Fixed` is only built when writing, so that apache-avro
gets a fixed value.
+ // Avro fixed values are read back as `Bytes`, and uuid values
as `String` or `Bytes`.
+ RawLiteralEnum::Fixed(_) => Err(invalid_err_with_reason(
Review Comment:
I think this is incorrect? Now uuid has been serialized to Fixed, when you
serialize into other formats like json, we should still return uuid?
##########
crates/iceberg/src/spec/manifest/data_file.rs:
##########
@@ -485,10 +485,125 @@ impl std::fmt::Display for DataFileFormat {
mod test {
use crate::ErrorKind;
use crate::spec::{
- DataContentType, DataFileBuilder, DataFileFormat, Literal,
NestedField, PrimitiveType,
- Struct, StructType, Type,
+ DataContentType, DataFileBuilder, DataFileFormat, FormatVersion,
Literal, NestedField,
+ PrimitiveType, Schema, Struct, StructType, Type,
read_data_files_from_avro,
+ write_data_files_to_avro,
};
+ #[test]
Review Comment:
Now we have complete MainfestReader/ManifestWriter, I prefer to remove these
tests in data file, and add complete manifest read/write roundtrip tests using
reader/writer, covering differnt table format versions, partition types with
fixed/uuid data types
##########
crates/iceberg/src/spec/values/tests.rs:
##########
Review Comment:
I think you should also cover the new tests using json serde
##########
crates/iceberg/src/spec/manifest/writer.rs:
##########
@@ -921,4 +921,44 @@ mod tests {
])
.await;
}
+
+ #[tokio::test]
+ async fn
test_write_manifest_with_repeated_uuid_partition_type_as_spec_fixed_uuid() {
Review Comment:
See https://github.com/apache/iceberg-rust/pull/2916/changes#r4548701868
##########
crates/iceberg/src/spec/values/serde.rs:
##########
@@ -473,8 +480,20 @@ pub(crate) mod _serde {
},
RawLiteralEnum::String(v) => match ty {
Type::Primitive(PrimitiveType::String) =>
Ok(Some(Literal::string(v))),
+ Type::Primitive(PrimitiveType::Uuid) =>
Hyphenated::from_str(&v)
Review Comment:
Interesting details from avro library, thx for explaination.
##########
crates/iceberg/src/spec/values/tests.rs:
##########
@@ -1511,6 +1582,101 @@ fn test_datum_ser_deser() {
test_fn(datum);
}
+#[test]
+fn test_datum_uuid_converts_to_avro_fixed_value() {
+ let uuid_value = Uuid::from_u128(0xaaaaaaaa_bbbb_cccc_dddd_eeeeeeeeeeee);
+ let output = to_value(Datum::uuid(uuid_value)).unwrap();
+
+ let expected = Value::Record(vec![
+ ("type".to_string(), Value::String("uuid".to_string())),
+ (
+ "literal".to_string(),
+ Value::Fixed(16, uuid_value.as_bytes().to_vec()),
+ ),
+ ]);
+ assert_eq!(output, expected);
+}
+
+#[test]
+fn test_avro_uuid_value_converts_to_datum_uuid() {
+ let uuid_value = Uuid::from_u128(0xaaaaaaaa_bbbb_cccc_dddd_eeeeeeeeeeee);
+ let avro_value = Value::Record(vec![
+ ("type".to_string(), Value::String("uuid".to_string())),
+ ("literal".to_string(), Value::Uuid(uuid_value)),
+ ]);
+
+ let output = apache_avro::from_value::<Datum>(&avro_value).unwrap();
+
+ assert_eq!(output, Datum::uuid(uuid_value));
+}
+
+#[test]
+fn test_avro_fixed_value_converts_to_datum_uuid() {
Review Comment:
```suggestion
fn test_avro_fixed_16_bytes_value_converts_to_datum_uuid() {
```
##########
crates/iceberg/src/spec/values/serde.rs:
##########
@@ -106,6 +109,7 @@ pub(crate) mod _serde {
Double(f64),
String(String),
Bytes(ByteBuf),
+ Fixed(#[serde(serialize_with =
"apache_avro::serde::fixed::serialize")] ByteBuf),
Review Comment:
I think you are right, adding this annotation is an acceptable temporary
workaround. We may need to rethink the datum/literal ser/de architecture if we
want an avro specific value. Please add comments to explain why we need this
and why this has no affect for other formats.
##########
crates/iceberg/src/spec/values/serde.rs:
##########
Review Comment:
We should remove this branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]