rahulsmahadev opened a new pull request, #17544:
URL: https://github.com/apache/iceberg/pull/17544
## Summary
`max-file-group-input-files` is a `rewrite_data_files` option that is read
and validated by `SizeBasedFileRewritePlanner` but is missing from
`validOptions()`, so passing it is rejected.
`SizeBasedFileRewritePlanner.maxGroupCount()` reads and validates the
option, but neither `SizeBasedFileRewritePlanner.validOptions()` nor
`BinPackRewriteFilePlanner.validOptions()` lists it.
`RewriteDataFilesSparkAction.validateAndInitOptions()` fails on any key outside
the union of valid options, so `CALL system.rewrite_data_files(table => 't',
options => map('max-file-group-input-files', '100'))` throws even though the
option is implemented.
This also makes the option behave differently across engines: Flink exposes
it through `RewriteDataFiles.maxFileGroupInputFiles()` where it works, while
Spark rejects it. The option was added in #14837 and `validOptions()` was not
updated at the time.
## Test plan
- Add `MAX_FILE_GROUP_INPUT_FILES` to
`SizeBasedFileRewritePlanner.validOptions()`
- Update the exact-set assertions in `TestSizeBasedFileRewritePlanner`,
`TestBinPackRewriteFilePlanner`, and `TestBinPackRewritePositionDeletePlanner`
- `./gradlew :iceberg-core:test --tests "org.apache.iceberg.actions.*"`
passes (53 tests), plus spotless and checkstyle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
