mmcdermott opened a new issue, #43626: URL: https://github.com/apache/arrow/issues/43626
### Describe the enhancement requested I tried to check existing issues and did not see this discussed anywhere previously, but if I missed something I apologize. I am involved with the [MEDS project](https://github.com/Medical-Event-Data-Standard/meds/blob/main/src/meds/schema.py) which uses parquet files as its storage format with apache pyarrow. We have a number of schemas where only a subset of columns are mandatory, but other columns may be either optional (e.g., if a column by that name is present, it must have a certain type), or some schemas can accept additional columns of arbitrary types. It would be nice if there were a way to handle this directly in the notion of a PyArrow schema, e.g., instead of something like this ```python label_schema = pa.schema( [ ("patient_id", pa.int64()), ("prediction_time", pa.timestamp("us")), ("boolean_value", pa.bool_()), # Optional ("integer_value", pa.int64()), # Optional ("float_value", pa.float64()), # Optional ("categorical_value", pa.string()), # Optional ] ) ``` Where we need to dynamically filter the columns to those present on the fly, we could have something like this: ```python label_schema = pa.schema( [ ("patient_id", pa.int64()), ("prediction_time", pa.timestamp("us")), ("boolean_value", pa.optional(pa.bool_())), ("integer_value", pa.optional(pa.int64())), ("float_value", pa.optional(pa.float64())), ("categorical_value", pa.optional(pa.string())), ] ) ``` And then when using something like `df.to_arrow().cast(label_schema)` the system naturally errors if mandatory columns are missing and casts optional columns that are present to their mandatory types but doesn't error if an optional column is missing. ### Component(s) Parquet, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
