viirya opened a new issue, #50429:
URL: https://github.com/apache/arrow/issues/50429
### Describe the enhancement requested
Follow-up of #50326. GH-50327 makes `Array.to_pylist()` convert without
per-element Scalars, but the `maps_as_pydicts` option still routes to the
Scalar-based path: every map row allocates a `MapScalar`, converts its keys via
per-element `as_py`, and builds the dict in Python.
Converting maps to dicts is the natural consumption pattern for engines
whose map values are Python dicts (e.g. Spark's Arrow-serialized Python UDFs
currently receive association lists from `to_pylist()` and rebuild a dict per
row in pure Python — one of the dominant remaining costs in that path).
Proposal: thread `maps_as_pydicts` through the scalar-free `_getitem_py`
mechanism introduced in GH-50327 so `MapArray` builds the dict directly from
the flattened keys/items children:
- default (`None`) semantics unchanged (association lists);
- `'lossy'`/`'strict'` build the dict in one pass; when the dict size shows
duplicate keys, the row is redone with the careful per-key loop so the warning
('lossy') and `KeyError` ('strict') semantics stay exactly identical to
`MapScalar.as_py`;
- invalid values still raise the same `ValueError` when a map value is
converted;
- the option propagates through nested types (list/struct/map values) as
before, and unspecialized types keep the exact Scalar fallback.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
