joshuarobinson opened a new issue, #6435:
URL: https://github.com/apache/iceberg/issues/6435
### Feature Request / Improvement
In reading manifests for a table for a table scan in PyIceberg 0.2.0, I get
an EOFError.
Table was originally written in June 2022 with the most recent version of
Spark/Iceberg at that time. (3.2.3 and 0.14.1 I think)
Full tracebook of the error:
```
Traceback (most recent call last):
File "/read.py", line 16, in <module>
print(tbl.scan(selected_fields=("time",
"icao24","call_sign","origin_country")).to_arrow().to_pandas().head())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line
350, in to_arrow
for task in self.plan_files():
File
"/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line
335, in plan_files
yield from (FileScanTask(file) for file in matching_partition_files)
File
"/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line
335, in <genexpr>
yield from (FileScanTask(file) for file in matching_partition_files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line
149, in <genexpr>
return (entry.data_file for entry in live_entries(input_file))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line
145, in <genexpr>
return (entry for entry in read_manifest_entry(input_file) if
entry.status != ManifestEntryStatus.DELETED)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line
139, in read_manifest_entry
for record in reader:
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py",
line 178, in __next__
return self.__next__()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py",
line 170, in __next__
return next(self.block)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py",
line 106, in __next__
return self.reader.read(self.block_decoder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py",
line 283, in read
result[pos] = field.read(decoder)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py",
line 283, in read
result[pos] = field.read(decoder)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py",
line 267, in read
return self.option.read(decoder)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py",
line 329, in read
read_items[key] = self.value.read(decoder)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py",
line 234, in read
return decoder.read_bytes()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/decoder.py",
line 122, in read_bytes
return self.read(self.read_int())
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/decoder.py",
line 58, in read
raise EOFError
EOFError
```
The table can be read successfully with Trino.
for interest, the schema of my table is
```
table {
1: time: optional long
2: icao24: optional string
3: callsign: optional string
4: origin_country: optional string
5: time_position: optional double
6: last_contact: optional long
7: longitude: optional double
8: latitude: optional double
9: baro_altitude: optional double
10: on_ground: optional boolean
11: velocity: optional double
12: true_track: optional double
13: vertical_rate: optional double
14: geo_altitude: optional double
15: squawk: optional string
16: spi: optional boolean
17: position_source: optional long
}
```
### Query engine
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]