djouallah opened a new issue, #53290:
URL: https://github.com/apache/arrow/issues/53290
### Describe the enhancement requested
Add an option to the CSV reader to read rows that have fewer fields than the
number of columns, and fill the missing fields with null. pandas' C engine does
this when `names` is given, and DuckDB has `read_csv(..., null_padding = true)`.
Today the width is fixed by `column_names`, or by the first row. A shorter
row is an error, and `invalid_row_handler` can only skip it or stop. It cannot
keep the row.
```python
import io
import pyarrow as pa
from pyarrow import csv
data = b"a,b\n1,2,3,4\n5,6,7,8\n"
names = ["w", "x", "y", "z"]
csv.read_csv(io.BytesIO(data),
read_options=csv.ReadOptions(column_names=names))
# ArrowInvalid: CSV parse error: Expected 4 columns, got 2: a,b
csv.read_csv(io.BytesIO(data),
read_options=csv.ReadOptions(autogenerate_column_names=True))
# ArrowInvalid: CSV parse error: Expected 2 columns, got 4: 1,2,3,4
csv.read_csv(io.BytesIO(data),
read_options=csv.ReadOptions(column_names=names),
parse_options=csv.ParseOptions(invalid_row_handler=lambda r:
"skip"))
# {'w': [1, 5], 'x': [2, 6], 'y': [3, 7], 'z': [4, 8]} -- the short row is
dropped, not padded
```
Expected, with the new option:
```
{'w': ['a', '1', '5'], 'x': ['b', '2', '6'], 'y': [None, '3', '7'], 'z':
[None, '4', '8']}
```
Real files have this shape. For example, the AEMO (Australian electricity
market) daily CSV files mix record types: the first row is a short header
record, and the data rows are wider. pandas with `engine="pyarrow"` cannot read
them, because pandas uses Arrow's reader and Arrow takes the width from the
first row. pandas marked this as an Arrow limitation:
https://github.com/pandas-dev/pandas/issues/71034
The same request was #27832, closed as stale in January 2026. #28421 added
`invalid_row_handler`, but it can only skip or error.
Tested with pyarrow 26.0.0, Python 3.12, Linux.
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
