redpheonixx commented on code in PR #1799:
URL: https://github.com/apache/iceberg-python/pull/1799#discussion_r2001294212
##########
tests/io/test_pyarrow_stats.py:
##########
@@ -470,6 +473,9 @@ def construct_test_table_primitive_types() ->
Tuple[pq.FileMetaData, Union[Table
strings = ["hello", "world"]
uuids = [uuid.uuid3(uuid.NAMESPACE_DNS, "foo").bytes,
uuid.uuid3(uuid.NAMESPACE_DNS, "bar").bytes]
binaries = [b"hello", b"world"]
+ decimal8 = [Decimal("123.45"), Decimal("678.91")]
+ decimal16 = [Decimal("123456789.123456"), Decimal("678912345.678912")]
+ decimal32 = [Decimal("12345678901234.123456"),
Decimal("98765432109870.654321")]
Review Comment:
`decimal8 = pa.array([Decimal('123.45'), Decimal('678.91')],
pa.decimal128(8, 2))`
@kevinjqliu pls check if we can go with this as this have now proper
precision and scale
pls note here that this will accept only int and Decimal values and not
float hence we require the above import Decimal
Do let me know if I am incorrect here or there is any other workaround.
##########
tests/io/test_pyarrow_stats.py:
##########
@@ -485,14 +491,17 @@ def construct_test_table_primitive_types() ->
Tuple[pq.FileMetaData, Union[Table
"strings": strings,
"uuids": uuids,
"binaries": binaries,
+ "decimal8": decimal8,
+ "decimal16": decimal16,
+ "decimal32": decimal32,
},
schema=arrow_schema,
)
metadata_collector: List[Any] = []
with pa.BufferOutputStream() as f:
- with pq.ParquetWriter(f, table.schema,
metadata_collector=metadata_collector) as writer:
+ with pq.ParquetWriter(f, table.schema,
metadata_collector=metadata_collector, store_decimal_as_integer=True) as writer:
Review Comment:
I tried without this parameter but then the physical type for decimal(8, 2)
will be FIXED_LEN_BYTE_ARRAY. There is a check for this as below
```
if expected_physical_type != physical_type_string:
# Allow promotable physical types
# INT32 -> INT64 and FLOAT -> DOUBLE are safe type casts
if (physical_type_string == "INT32" and expected_physical_type
== "INT64") or (
physical_type_string == "FLOAT" and expected_physical_type
== "DOUBLE"
):
pass
else:
raise ValueError(
f"Unexpected physical type {physical_type_string} for
{iceberg_type}, expected {expected_physical_type}"
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
