Fokko commented on code in PR #1644:
URL: https://github.com/apache/iceberg-python/pull/1644#discussion_r2175497964
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1814,9 +1814,12 @@ def struct(
array = self._cast_if_needed(field, field_array)
field_arrays.append(array)
fields.append(self._construct_field(field, array.type))
- elif field.optional:
+ elif field.optional or field.initial_default is not None:
arrow_type = schema_to_pyarrow(field.field_type,
include_field_ids=self._include_field_ids)
- field_arrays.append(pa.nulls(len(struct_array),
type=arrow_type))
+ if field.initial_default is None:
+ field_arrays.append(pa.nulls(len(struct_array),
type=arrow_type))
+ else:
+ field_arrays.append(pa.repeat(field.initial_default,
len(struct_array)))
fields.append(self._construct_field(field, arrow_type))
else:
raise ResolveError(f"Field is required, and could not be found
in the file: {field}")
Review Comment:
I've added a comment, but different than you suggested.
```
When an optional field is added, the defaults may be null and should be
explicitly set
```
I don't think this is true, for V2 you can just add an optional field:
Hive:
```sql
ALTER TABLE aa_monthly ADD cadence STRING
```
Here you set the null implicitly.
In V3, with a required field with default value:
```sql
ALTER TABLE aa_monthly ADD cadence string NOT NULL DEFAULT 'UNDEFINED'
```
However, it is also fine to make it optional:
```sql
ALTER TABLE aa_monthly ADD cadence string DEFAULT 'UNDEFINED'
```
But that's a questionable data modelling approach :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
