agramfort opened a new issue, #47238:
URL: https://github.com/apache/arrow/issues/47238
### Describe the bug, including details regarding any error messages,
version, and platform.
I have pandas Dataframes that can contain lists of floats that sometimes can
be NaNs. If all the entries are NaNs `pa.Table.from_pandas` infers a list<item:
null> but this only happens if it's all NaNs. If one entry is a regular float
it becomes list<item: double> (which is what I expect). Interesting if you have
a list of np.float32 then you get a list<item: float> even if it's all NaNs.
See:
```import numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame(
{
"col": [[np.nan]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
df = pd.DataFrame(
{
"col": [[np.float64(np.nan)]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
df = pd.DataFrame(
{
"col": [[np.float32(np.nan)]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
df = pd.DataFrame(
{
"col": [[1.0, np.nan]],
}
)
print(pa.Table.from_pandas(df).schema.field(0).type)
```
<img width="635" height="725" alt="Image"
src="https://github.com/user-attachments/assets/55b05e43-1536-41fc-957f-4b7ba3fceaa7";
/>
Is there a way to enforce that I get always list<item: double> as
technically isinstance(np.nan, float) is True.
thanks
cc @alanhdu
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
