anuragmantri commented on code in PR #17877: URL: https://github.com/apache/iceberg/pull/17877#discussion_r4009193184
########## .agents/skills/add-spark-version/SKILL.md: ########## @@ -0,0 +1,132 @@ +--- +name: add-spark-version +description: Add initial support for a new Apache Spark minor version in Apache Iceberg, including the versioned source tree, Gradle wiring, CI and publication decisions, compatibility fixes, and validation. Use for a new Spark line; do not use for patch-version upgrades within an existing line. +--- + +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Add a Spark Version + +Introduce a new Spark line without losing the history of the versioned Spark sources or accidentally changing which artifacts Iceberg builds and publishes by default. + +Keep the workflow agent-neutral: base decisions on repository state and use standard Git, shell, and Gradle commands rather than product-specific features. + +## Establish the target + +Before editing, identify: + +- the target Spark line and full dependency version; +- the supported Scala and JDK versions; +- the closest existing `spark/v<source>` directory to copy; +- whether the target artifacts are in Maven Central or require a temporary repository; and +- whether this is preview-only source support or a version that should be built and published by default. + +Inspect the current tree and search for every version registry rather than assuming the file list is unchanged: + +```bash +git status --short --branch +find spark -maxdepth 1 -type d -name 'v*' -print +rg -n 'knownSparkVersions|defaultSparkVersions|sparkVersions|spark[0-9]+' \ + gradle.properties gradle settings.gradle build.gradle spark jmh.gradle .github dev site +``` + +Use a clean branch based on the latest `upstream/main`, preferably in an isolated worktree when another Spark branch is active. Stop if the target version directory or a matching upstream contribution already exists. + +## Protect an existing target branch + +Before reusing a target branch, inspect both its local ref and the corresponding ref on the chosen remote. If both exist at different commits, stop and determine which tip must be preserved. Fetch a remote-only branch before continuing. + +If the target branch exists, create a uniquely named local backup from its exact tip before any reset, rebase, or force-push. After resolving the existing target ref, replace the placeholders and run: + +```bash +target_branch="<target-branch>" +existing_target_ref="<resolved-local-or-remote-ref>" +original_target_sha=$(git rev-parse "${existing_target_ref}^{commit}") +backup_branch="${target_branch}-backup-$(date +%Y%m%d-%H%M%S)" + +git branch "${backup_branch}" "${original_target_sha}" +test "$(git rev-parse "${backup_branch}^{commit}")" = "${original_target_sha}" +``` + +Let `git branch` fail rather than overwrite an existing backup name. Keep the backup until the rewritten target branch is published and its remote tip is verified. Push the backup only when explicitly requested. If the target branch does not exist, create it normally from the refreshed base. + +## Preserve source history + +Create the new versioned tree with two mechanical commits. Replace the placeholders with concrete version lines before running the commands. + +```bash +git mv spark/v<source> spark/v<target> +git add -A -- spark +git commit -m "Spark: Move <source> as <target>" + +cp -R spark/v<target> spark/v<source> +git add -A -- spark +git commit -m "Spark: Copy back <target> as <source>" +``` + +Keep these commits free of compatibility edits so Git can follow the move. Use `git add -A -- spark`; narrower old/new pathspecs may fail while one side of the rename is absent. Never delete an existing target tree to force this sequence. + +## Wire the new line + +Make the support commit after the mechanical commits. Check the current equivalents of all of these integration points: + +- `gradle/libs.versions.toml`: add the full Spark dependency version. +- `spark/v<target>/build.gradle`: update the Spark line, version-catalog key, Scala/JDK constraints, module coordinates, exclusions, and optional integrations. +- `gradle.properties`: add the line to `knownSparkVersions`. Change `defaultSparkVersions` only when the new line should become the default. +- `settings.gradle`, `spark/build.gradle`, and `jmh.gradle`: register the core, extensions, runtime, and benchmark projects. +- `.gitignore`: cover generated benchmark or warehouse paths for the new tree. +- `.github/workflows/spark-ci.yml` and `.github/workflows/cve-scan.yml`: add supported combinations while respecting the current matrix-size and JDK/Scala exclusions. +- `dev/stage-binaries.sh` and snapshot publication workflows: add the line only when artifacts should be published. Preview source support and release publication are separate decisions. Review Comment: This section separates preview support from release publication decisions, which is the right distinction, but nothing here or in the discovery command above mentions `.gitattributes`. The `export-ignore` entries there are what actually gate whether a preview version tree is included in `git archive` source releases. #18024, "Include Spark 4.2 in source releases," was entirely about removing `/spark/v4.2` and `/spark/v4.2/**` from `.gitattributes`. Could this get called out explicitly, and could `.gitattributes` be added to the discovery search paths in "Establish the target"? ########## .agents/skills/add-spark-version/SKILL.md: ########## @@ -0,0 +1,132 @@ +--- +name: add-spark-version +description: Add initial support for a new Apache Spark minor version in Apache Iceberg, including the versioned source tree, Gradle wiring, CI and publication decisions, compatibility fixes, and validation. Use for a new Spark line; do not use for patch-version upgrades within an existing line. +--- + +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Add a Spark Version + +Introduce a new Spark line without losing the history of the versioned Spark sources or accidentally changing which artifacts Iceberg builds and publishes by default. + +Keep the workflow agent-neutral: base decisions on repository state and use standard Git, shell, and Gradle commands rather than product-specific features. + +## Establish the target + +Before editing, identify: + +- the target Spark line and full dependency version; +- the supported Scala and JDK versions; +- the closest existing `spark/v<source>` directory to copy; +- whether the target artifacts are in Maven Central or require a temporary repository; and +- whether this is preview-only source support or a version that should be built and published by default. + +Inspect the current tree and search for every version registry rather than assuming the file list is unchanged: + +```bash +git status --short --branch +find spark -maxdepth 1 -type d -name 'v*' -print +rg -n 'knownSparkVersions|defaultSparkVersions|sparkVersions|spark[0-9]+' \ + gradle.properties gradle settings.gradle build.gradle spark jmh.gradle .github dev site Review Comment: This `rg` pattern doesn't match `sparkScalaVersions`, the map inside `project(':iceberg-bom')` in root `build.gradle` that controls which Scala variants of a Spark line get included in the BOM. I checked: `echo "sparkScalaVersions" | grep -E 'knownSparkVersions|defaultSparkVersions|sparkVersions|spark[0-9]+'` exits 1. Could the pattern (and the "Wire the new line" file list below) include it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
