blackmwk commented on code in PR #2916:
URL: https://github.com/apache/iceberg-rust/pull/2916#discussion_r4548466761


##########
crates/iceberg/src/spec/values/serde.rs:
##########
@@ -473,8 +485,25 @@ pub(crate) mod _serde {
                 },
                 RawLiteralEnum::String(v) => match ty {
                     Type::Primitive(PrimitiveType::String) => 
Ok(Some(Literal::string(v))),
+                    // apache-avro deserializes a UUID as a hyphenated string.
+                    // UUID values read via `Value` (i.e. 
`read_data_files_from_avro`), land here.
+                    // The hyphenated restriction matches behaviour with Java.
+                    Type::Primitive(PrimitiveType::Uuid) => 
Hyphenated::from_str(&v)
+                        .map(|hyphenated_uuid| 
Some(Literal::uuid(hyphenated_uuid.into_uuid())))
+                        .map_err(|_| {
+                            invalid_err_with_reason(
+                                "string",
+                                "UUID must be a hyphenated 8-4-4-4-12 string",
+                            )
+                        }),
                     _ => Err(invalid_err("string")),
                 },
+                // `Fixed` is only built when writing, so that apache-avro 
gets a fixed value.
+                // Avro fixed values are read back as `Bytes`, and uuid values 
as `String` or `Bytes`.
+                RawLiteralEnum::Fixed(_) => Err(invalid_err_with_reason(

Review Comment:
   I think this is incorrect? Now uuid has been serialized to Fixed, when you 
serialize into other formats like json, we should still return uuid?



##########
crates/iceberg/src/spec/manifest/data_file.rs:
##########
@@ -485,10 +485,125 @@ impl std::fmt::Display for DataFileFormat {
 mod test {
     use crate::ErrorKind;
     use crate::spec::{
-        DataContentType, DataFileBuilder, DataFileFormat, Literal, 
NestedField, PrimitiveType,
-        Struct, StructType, Type,
+        DataContentType, DataFileBuilder, DataFileFormat, FormatVersion, 
Literal, NestedField,
+        PrimitiveType, Schema, Struct, StructType, Type, 
read_data_files_from_avro,
+        write_data_files_to_avro,
     };
 
+    #[test]

Review Comment:
   Now we have complete MainfestReader/ManifestWriter, I prefer to remove these 
tests in data file, and add complete manifest read/write roundtrip tests using 
reader/writer, covering differnt table format versions, partition types with 
fixed/uuid data types



##########
crates/iceberg/src/spec/values/tests.rs:
##########


Review Comment:
   I think you should also cover the new tests using json serde



##########
crates/iceberg/src/spec/manifest/writer.rs:
##########
@@ -921,4 +921,44 @@ mod tests {
         ])
         .await;
     }
+
+    #[tokio::test]
+    async fn 
test_write_manifest_with_repeated_uuid_partition_type_as_spec_fixed_uuid() {

Review Comment:
   See https://github.com/apache/iceberg-rust/pull/2916/changes#r4548701868



##########
crates/iceberg/src/spec/values/serde.rs:
##########
@@ -473,8 +480,20 @@ pub(crate) mod _serde {
                 },
                 RawLiteralEnum::String(v) => match ty {
                     Type::Primitive(PrimitiveType::String) => 
Ok(Some(Literal::string(v))),
+                    Type::Primitive(PrimitiveType::Uuid) => 
Hyphenated::from_str(&v)

Review Comment:
   Interesting details from avro library, thx for explaination.



##########
crates/iceberg/src/spec/values/tests.rs:
##########
@@ -1511,6 +1582,101 @@ fn test_datum_ser_deser() {
     test_fn(datum);
 }
 
+#[test]
+fn test_datum_uuid_converts_to_avro_fixed_value() {
+    let uuid_value = Uuid::from_u128(0xaaaaaaaa_bbbb_cccc_dddd_eeeeeeeeeeee);
+    let output = to_value(Datum::uuid(uuid_value)).unwrap();
+
+    let expected = Value::Record(vec![
+        ("type".to_string(), Value::String("uuid".to_string())),
+        (
+            "literal".to_string(),
+            Value::Fixed(16, uuid_value.as_bytes().to_vec()),
+        ),
+    ]);
+    assert_eq!(output, expected);
+}
+
+#[test]
+fn test_avro_uuid_value_converts_to_datum_uuid() {
+    let uuid_value = Uuid::from_u128(0xaaaaaaaa_bbbb_cccc_dddd_eeeeeeeeeeee);
+    let avro_value = Value::Record(vec![
+        ("type".to_string(), Value::String("uuid".to_string())),
+        ("literal".to_string(), Value::Uuid(uuid_value)),
+    ]);
+
+    let output = apache_avro::from_value::<Datum>(&avro_value).unwrap();
+
+    assert_eq!(output, Datum::uuid(uuid_value));
+}
+
+#[test]
+fn test_avro_fixed_value_converts_to_datum_uuid() {

Review Comment:
   ```suggestion
   fn test_avro_fixed_16_bytes_value_converts_to_datum_uuid() {
   ```



##########
crates/iceberg/src/spec/values/serde.rs:
##########
@@ -106,6 +109,7 @@ pub(crate) mod _serde {
         Double(f64),
         String(String),
         Bytes(ByteBuf),
+        Fixed(#[serde(serialize_with = 
"apache_avro::serde::fixed::serialize")] ByteBuf),

Review Comment:
   I think you are right, adding this annotation is an acceptable temporary 
workaround. We may need to rethink the datum/literal ser/de architecture if we 
want an avro specific value. Please add comments to explain why we need this 
and why this has no affect for other formats.



##########
crates/iceberg/src/spec/values/serde.rs:
##########


Review Comment:
   We should remove this branch?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to