Kontinuation commented on PR #12347: URL: https://github.com/apache/iceberg/pull/12347#issuecomment-2670709563
I found that it is not easy to upgrade the parquet dependency to the (not-released-yet) next version, because parquet-hadoop now uses a FileSystem API introduced in Hadoop 3: https://github.com/apache/parquet-java/pull/3079. Upgrading parquet dependencies to the latest SNAPSHOT version results in the following failure when running tests in `iceberg-data`: ``` 'org.apache.hadoop.fs.FutureDataInputStreamBuilder org.apache.hadoop.fs.FileSystem.openFile(org.apache.hadoop.fs.Path)' java.lang.NoSuchMethodError: 'org.apache.hadoop.fs.FutureDataInputStreamBuilder org.apache.hadoop.fs.FileSystem.openFile(org.apache.hadoop.fs.Path)' at org.apache.parquet.hadoop.util.HadoopInputFile.newStream(HadoopInputFile.java:114) at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:925) at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:710) at org.apache.iceberg.parquet.ReadConf.newReader(ReadConf.java:194) at org.apache.iceberg.parquet.ReadConf.<init>(ReadConf.java:76) ``` We have to remove Hadoop 2 support and migrate to Hadoop 3 for all submodules. There is a stale PR working on this: https://github.com/apache/iceberg/pull/10932. I found that https://github.com/apache/iceberg/issues/10940 was closed as completed but there are still lots of submodule depending on Hadoop 2. I'd like to know how should we proceed to upgrade the parquet package. Should we upgrade dependencies to Hadoop 2 to Hadoop 3 to unblock the parquet upgrade? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
