joshuarobinson opened a new issue, #6435:
URL: https://github.com/apache/iceberg/issues/6435

   ### Feature Request / Improvement
   
   In reading manifests for a table for a table scan in PyIceberg 0.2.0, I get 
an EOFError.
   
   Table was originally written in June 2022 with the most recent version of 
Spark/Iceberg at that time. (3.2.3 and 0.14.1 I think)
   
   Full tracebook of the error:
   ```
   Traceback (most recent call last):
     File "/read.py", line 16, in <module>
       print(tbl.scan(selected_fields=("time", 
"icao24","call_sign","origin_country")).to_arrow().to_pandas().head())
             
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File 
"/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 
350, in to_arrow
       for task in self.plan_files():
     File 
"/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 
335, in plan_files
       yield from (FileScanTask(file) for file in matching_partition_files)
     File 
"/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 
335, in <genexpr>
       yield from (FileScanTask(file) for file in matching_partition_files)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line 
149, in <genexpr>
       return (entry.data_file for entry in live_entries(input_file))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line 
145, in <genexpr>
       return (entry for entry in read_manifest_entry(input_file) if 
entry.status != ManifestEntryStatus.DELETED)
              
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line 
139, in read_manifest_entry
       for record in reader:
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py", 
line 178, in __next__
       return self.__next__()
              ^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py", 
line 170, in __next__
       return next(self.block)
              ^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py", 
line 106, in __next__
       return self.reader.read(self.block_decoder)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py", 
line 283, in read
       result[pos] = field.read(decoder)
                     ^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py", 
line 283, in read
       result[pos] = field.read(decoder)
                     ^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py", 
line 267, in read
       return self.option.read(decoder)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py", 
line 329, in read
       read_items[key] = self.value.read(decoder)
                         ^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/reader.py", 
line 234, in read
       return decoder.read_bytes()
              ^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/decoder.py", 
line 122, in read_bytes
       return self.read(self.read_int())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/decoder.py", 
line 58, in read
       raise EOFError
   EOFError
   ```
   
   The table can be read successfully with Trino.
   
   for interest, the schema of my table is 
   ```
   table {
     1: time: optional long
     2: icao24: optional string
     3: callsign: optional string
     4: origin_country: optional string
     5: time_position: optional double
     6: last_contact: optional long
     7: longitude: optional double
     8: latitude: optional double
     9: baro_altitude: optional double
     10: on_ground: optional boolean
     11: velocity: optional double
     12: true_track: optional double
     13: vertical_rate: optional double
     14: geo_altitude: optional double
     15: squawk: optional string
     16: spi: optional boolean
     17: position_source: optional long
   }
   ```
   
   ### Query engine
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to