bschindler opened a new issue, #43614:
URL: https://github.com/apache/arrow/issues/43614
### Describe the bug, including details regarding any error messages,
version, and platform.
FixedShapeTensorArray.to_numpy_ndarray() fails with numpy arrays of type
string
with pyarrow 16, whereas pyarrow 15 works. As a result, we are unable to
load our feather files that were created with older pyarrow versions.
While we should not have stored those strings this way, we have now a
significant library of files that contain such constructs.
Code to load the file:
```
from pathlib import Path
from typing import Any
import pyarrow as pa
def _extract_element_feather(column: Any):
import pyarrow as pa
if type(column) == pa.lib.FixedShapeTensorArray:
return column.to_numpy_ndarray()
else:
return column.as_py()
with pa.ipc.open_file(Path("out.feather")) as reader:
table = reader.get_batch(0)
features = {
k.name: _extract_element_feather(table[k.name])
for k in table.schema
}
print(features)
```
Test file:
[out.feather.zip](https://github.com/user-attachments/files/16540720/out.feather.zip)
Pyarrow 16:
```
File "/Users/bschindler/Library/Application
Support/JetBrains/PyCharm2024.1/scratches/scratch_41.py", line 17, in <module>
features = {
File "/Users/bschindler/Library/Application
Support/JetBrains/PyCharm2024.1/scratches/scratch_41.py", line 18, in <dictcomp>
k.name: _extract_element_feather(table[k.na
me])
File "/Users/bschindler/Library/Application
Support/JetBrains/PyCharm2024.1/scratches/scratch_41.py", line 11, in
_extract_element_feather
return column.to_numpy_ndarray()
File "pyarrow/array.pxi", line 4240, in
pyarrow.lib.FixedShapeTensorArray.to_numpy_ndarray
File "pyarrow/array.pxi", line 4265, in
pyarrow.lib.FixedShapeTensorArray.to_tensor
File "pyarrow/error.pxi", line 154, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: string is not valid data type for a tensor
```
Pyarrow 15:
```
{'data': array([['Hello World']], dtype=object)}
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
