jayceslesar commented on issue #1120:
URL:
https://github.com/apache/iceberg-python/issues/1120#issuecomment-3016726263
Maybe I am interpreting this wrong but I did attempt a fix... it comes down
to replacing
`pa.field("partition", pa_record_struct, nullable=False),` with
`pa.field("partition", pa.map_(pa.string(), pa.string()), nullable=False),`
[here](https://github.com/apache/iceberg-python/blob/5e975d569e243f1e67e8021a6f941cd77b08d926/pyiceberg/table/inspect.py#L284).
This gets us how I believe I am interpreting your expectation, and also makes
sense to me because now the operation respects the current manifest, instead of
the schema we used to create which would result in the output linked in your
notebook of `{'timestamp': 17, 'category': None}` when we really should just be
seeing `{'timestamp': 17}`
I get pretty confused because when running an existing integration test
(`test_inspect_partitions_partitioned`), we get an error of
```
def check_pyiceberg_df_equals_spark_df(df: pa.Table, spark_df:
DataFrame) -> None:
lhs = df.to_pandas().sort_values("spec_id")
rhs = spark_df.toPandas().sort_values("spec_id")
for column in df.column_names:
for left, right in zip(lhs[column].to_list(),
rhs[column].to_list()):
> assert left == right, f"Difference in column {column}:
{left} != {right}"
E AssertionError: Difference in column partition:
[('dt_month', '612')] != {'dt_month': 612, 'dt_day': None}
E assert [('dt_month', '612')] == {'dt_month': 612, 'dt_day':
None}
E Full diff:
E - {'dt_day': None, 'dt_month': 612}
E + [('dt_month', '612')]
tests/integration/test_inspect_table.py:589: AssertionError
```
And either I am not understanding something or spark is also giving us this
`None` back. It is more likely that I am not understanding so any info/help is
appreciated!
I also recognize that the type of the `partition` column in the table is now
different but one issue at a time 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
