hubgeter opened a new pull request, #70510: URL: https://github.com/apache/doris/pull/70510
### Background Iceberg `rewrite_data_files` selects and groups input files, then runs an INSERT-SELECT for each group under one shared rewrite transaction. Each task clones the caller's session, and the backend writer reads its rollover target from the task's query options. ### What problem does this PR solve? Issue Number: None Related PR: #64304 (introduced the regression), #56413 (original implementation) PR Type: Bugfix Problem Summary: The procedure's `target-file-size-bytes` reaches file selection but is lost before the output write. For example, a 128 MiB procedure target with the default session value still uses the backend's 1 GiB writer fallback. Carry the connector-resolved value, including the 512 MiB action default, with every rewrite group and apply it only to that task's cloned session before planning the write. ### Details - **Frequency:** Rewrites that produce groups and whose effective action target differs from the inherited writer option or backend fallback. - **Root cause:** The connector SPI migration omitted the legacy per-task writer-target assignment. Neutral groups carried paths and statistics, while tasks inherited the original session unchanged. - **Why existing tests missed it:** Planning tests checked file paths and statistics; task tests covered cancellation without checking the serialized writer options. - **Fix:** Add immutable connector-resolved task-session overrides to rewrite groups and apply them through the existing session-variable setter. Keep procedure parsing and default resolution inside the connector. Advance the connector API to 31.0 and record the rewrite-group surface. - **Other paths and branches:** Ordinary INSERT keeps its existing session/fallback behavior. `branch-4.2` at `1bb4a49bddc8a505003d2067cc56f0dc3432f23f` already passes the action target through its legacy executor and [sets it on the task session](https://github.com/apache/doris/blob/1bb4a49bddc8a505003d2067cc56f0dc3432f23f/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/rewrite/RewriteGroupTask.java#L137), so this fix needs no 4.2 backport. No Thrift field or BE implementation changes are needed. ### Testing - `./build.sh --fe -j4`: passed locally, including FE packaging, Checkstyle and connector architecture gates. - `run-fe-ut.sh`: all 154 SPI tests, 53 related connector tests and 12 rewrite-driver tests passed locally (219 cases). Both SPI surface checks passed. - The seven task-session tests are being rerun; their results will be updated on this draft. - Final Checkstyle passed for the changed FE modules and the latest task test fixture. The tests check the actual `SessionVariable.toThrift()` field and its presence bit, default and explicit values, multiple groups, values above the 32-bit range, immutable settings, invalid input and concurrent task-session isolation. Physical output-file distributions and end-to-end SQL rewrite have not been measured locally; file sizes remain approximate rollover targets. ### Release note Iceberg `rewrite_data_files` now applies `target-file-size-bytes`, including its default 512 MiB value, to output writers. Output file counts may change when the effective target differs from the previously inherited session limit or backend fallback. ### Check List (For Author) - Behavior changed: Yes (see Problem Summary and Release note). - Does this need documentation: No; this restores the documented procedure behavior. - Pre-submission status: Not ready: task-session test verification is still running. End-to-end SQL and remote CI also remain pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests and this description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
