nimesh1601 opened a new pull request, #17842:
URL: https://github.com/apache/iceberg/pull/17842
## Summary
Backport the branch write option added to Spark 4.1 in #15288 to the Spark
**3.5** and **4.0** modules.
`SparkWriteOptions.BRANCH` (`"branch"`) lets a DataFrame write target a
table branch via
`.option("branch", "...")`, mirroring the existing `SparkReadOptions.BRANCH`
read option that has
been available in these modules since #5150.
## Motivation
`SparkReadOptions` exposes a `branch` read option in v3.5/v4.0/v4.1, but the
corresponding write
option only exists in **v4.1** (added by #15288). This leaves v3.5 and v4.0
users unable to target a
branch through the DataFrame writer option — they can only use the
`branch_<name>` identifier suffix
or a session-level WAP branch. This PR closes that read/write asymmetry for
the older modules.
## Change
`SparkWriteConf.branch()` now resolves the write option first:
- If `branch` is set in both the table identifier and the write option, they
must match
(otherwise a `ValidationException` is raised, symmetric to the existing
read-side check in
`SparkReadConf.branch()`).
- The explicit option/identifier branch takes precedence over the session
WAP branch.
- When no option/identifier branch is set, the existing WAP-branch
resolution is unchanged.
This is intentionally the **minimal** backport: it does not pull in the
larger v4.1
`SparkTableUtil` read/write resolution refactor from #15288 (which is
v4.1-only and touched ~19
files). This mirrors the minimal-backport approach used in #16245.
## Tests
Added `writeBranchOption` and `writeBranchOptionConflictsWithIdentifier` to
`TestSparkWriteConf`
in both v3.5 and v4.0. Existing WAP-branch tests are unaffected because the
new code is a no-op
when the `branch` option is absent.
- `./gradlew -DsparkVersions=3.5,4.0
:iceberg-spark:iceberg-spark-3.5_2.12:test
:iceberg-spark:iceberg-spark-4.0_2.13:test --tests
"org.apache.iceberg.spark.TestSparkWriteConf"` — green
- `spotlessApply` / `spotlessCheck` — clean
## Related
- #15288 — Spark 4.1: Align handling of branches in reads and writes
(introduced the v4.1 write option)
- #16245 — precedent for a minimal branch backport to v3.4/v3.5/v4.0
- #5150 — original branch read option
---
**AI Disclosure**
- Model: Claude Opus 4.8
- Platform/Tool: GitHub Copilot CLI
- Human Oversight: partially reviewed
- Prompt Summary: Investigate why the Spark branch write option
(SparkWriteOptions.BRANCH) exists only in Spark 4.1, then backport it to the
Spark 3.5 and 4.0 modules following the v4.1 (#15288) semantics and the
minimal-backport pattern from #16245, with tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
