ismailsimsek commented on issue #1728:
URL:
https://github.com/apache/iceberg-python/issues/1728#issuecomment-3102079276
@Fokko @grbinho probably related to identifier field.
Im also getting similar error with iceberg scheme which has identifier fields
Code:
```
if schema:
# If an Iceberg schema is provided, convert it to a PyArrow schema and
use it for parsing.
parse_options = pa_json.ParseOptions(explicit_schema=schema.as_arrow())
return pa_json.read_json(json_lines_buffer, parse_options=parse_options)
```
Error:
```
ERROR: failed to consume events in python
Mismatch in fields:
┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ┃ Table field ┃ Dataframe field
┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ ❌ │ 1: id: required long │ 1: id: optional long
│
│ ✅ │ 2: first_name: optional string │ 2: first_name: optional string
│
│ ✅ │ 3: last_name: optional string │ 3: last_name: optional string
│
│ ✅ │ 4: email: optional string │ 4: email: optional string
│
│ ✅ │ 5: __op: optional string │ 5: __op: optional string
│
│ ✅ │ 6: __table: optional string │ 6: __table: optional string
│
│ ✅ │ 7: __source_ts_ms: optional long │ 7: __source_ts_ms: optional long
│
│ ✅ │ 8: __ts_ms: optional long │ 8: __ts_ms: optional long
│
│ ✅ │ 9: _consumed_at: optional │ 9: _consumed_at: optional
│
│ │ timestamptz │ timestamptz
│
│ ✅ │ 10: _dbz_event_key: optional │ 10: _dbz_event_key: optional
│
│ │ string │ string
│
│ ✅ │ 11: _dbz_event_key_hash: optional │ 11: _dbz_event_key_hash: optional
│
│ │ string │ string
│
└────┴────────────────────────────────────┴────────────────────────────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
