wypoon commented on PR #12721: URL: https://github.com/apache/iceberg/pull/12721#issuecomment-2811124141
@danielcweeks thank you for looking into this. I have considered something along the lines of your suggestion before. I had implemented something similar in https://github.com/apache/iceberg/pull/2512, following the input of the community at the time, when we first added support for Spark 3.1. The idea was to build using Spark 3.0 but test using Spark 3.1 as well by defining a configuration within the module. I believe this is your idea here as well: build using Hive 2, and test using Hive 2, 3 and 4. The problem with that was that by not building against Spark 3.1, we failed to discover some issue (the `RepartitionByExpression` constructor changed signature in 3.1 and I'd failed to replace one occurrence of `RepartitionByExpression` by the necessary reflection in a code path that was not exercised by tests). Subsequently someone tried to add tests that exercised that code path and I had to put up https://github.com/apache/iceberg/pull/2954. As you know, subsequently the community decided to follow a different approach when th e time came to support Spark 3.2. Considering this, I really think it is best to build the module against all versions and test accordingly. While I may flushed out issues in the process of working through this PR to achieve a source set that appears to build and run successfully with Hive 2, 3, and 4, that is not a guarantee that new issues may not manifest when we upgrade the Hive 4 version, for example, that would be caught by building against it. (That said, I tried to follow the suggestion you outlined, but it didn't quite work as-is, and I tried to fix it, but without success thus far due to my limited Gradle knowledge. I'm continuing to look into it.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
