venkata91 opened a new pull request, #18947:
URL: https://github.com/apache/iceberg/pull/18947

   ## What Changed
   
   - `ParquetIO.file(OutputFile)` now wraps a `HadoopOutputFile` in a new 
`HadoopFileSystemOutputFile` that creates the stream through the file's own 
`FileSystem`. Before, it used Parquet's `HadoopOutputFile.fromPath`, which 
looks the `FileSystem` up again from the path and Hadoop `Configuration`, so it 
ignored the instance the Iceberg file was created with.
   - Removed the `ParquetIO.file(OutputFile, ParquetConfiguration)` overload. 
`ParquetWriter` no longer keeps the `ParquetConfiguration` field and calls 
`ParquetIO.file(output)` instead, so a Hadoop `Configuration` is no longer 
built from the Parquet write config when opening the output stream.
   - Added `TestParquet.writeUsesFileSystemOfHadoopOutputFile`. It wraps the 
local `FileSystem` in a counting `FilterFileSystem`, checks that the write 
calls `create` exactly once on that `FileSystem`, and checks that the file 
reads back with one record.
   
   ## Risk Assessment
   
   ✅ Low: The change is small and package-private. I checked its create, 
block-size and path semantics against parquet-hadoop's HadoopOutputFile 
bytecode (1.18.1 jar, the closest one cached; the build pins 1.18.2), and 
nothing in ParquetFileWriter depends on parquet's own HadoopOutputFile type, so 
there is no API change and default-config behavior is unchanged.
   
   ## Testing
   
   I ran the real Iceberg Parquet writer and reader against a local Hadoop 
filesystem, on both the target commit and the base. With the fix, a write 
through a HadoopOutputFile built on a counting FilterFileSystem calls that 
filesystem's create exactly once and the file reads back. With the base's main 
code the same test fails (creates == 0). Writing through HadoopFileIO 
round-trips correctly. A plain create on an existing file is rejected and 
leaves the original rows intact, while overwrite() replaces them. The new 
output file reports the same path, supportsBlockSize and defaultBlockSize as 
parquet-hadoop's own HadoopOutputFile. All of TestParquet and 
TestParquetEncryption pass. The scratch test was removed from the worktree; a 
copy and the logs are in the evidence directory.
   
   - Live validation: ✅ go - 6 of 6 scenarios driven live against the product
   
   | Scenario | Result | Live | Evidence |
   | --- | --- | --- | --- |
   | Parquet write via HadoopOutputFile built with a custom (counting) 
FileSystem on a local temp file calls that FS's create once and the file is 
readable | ✅ pass | live | TestParquet.writeUsesFileSystemOfHadoopOutputFile 
passed on target (target-tests.log, TEST-...TestParquet.xml) |
   | Adversarial: same write with base-commit ParquetIO bypasses the provided 
FileSystem | ✅ pass | live | base-regression-test.log: expected 1 but was 0 |
   | HadoopFileIO local write/read round trip still works (3 rows written and 
read back) | ✅ pass | live | Scratch output 'NM roundtrip read: [1, 2, 
3]' |
   | Adversarial: create without overwrite on an existing file fails and leaves 
original data; overwrite() replaces it | ✅ pass | live | Scratch output 
'create-on-existing threw: UncheckedIOException' then 'after 
overwrite read: [7, 8]' |
   | New output file reports the same path, supportsBlockSize and 
defaultBlockSize as parquet-hadoop's HadoopOutputFile for the same local 
path/conf | ✅ pass | live | Scratch output lines 'NM ours' vs 'NM 
parquet-hadoop' |
   | Encrypted Parquet writes/reads still work (TestParquetEncryption) and 
existing TestParquet suite passes | ✅ pass | live | TestParquetEncryption 3/3, 
TestParquet 37/37 in target-tests.log |
   
   
   <details>
   <summary>Evidence: Target run log (TestParquet, TestParquetEncryption, 
scratch scenarios)</summary>
   
   ```text
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current
                                    Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--    
 0   9 43764    9  4125    0     0  19764      0  0:00:02 --:--:--  0:00:02 
19736 100 43764  100 43764    0     0   191k      0 --:--:-- --:--:-- --:--:--  
191k
   Initialized native services in: ~/.gradle/native
   Initialized jansi services in: ~/.gradle/native
   The client will now receive all logging from the daemon (pid: 2911512). The 
daemon log file: ~/.gradle/daemon/8.14.6/daemon-2911512.out.log
   Starting 8th build in daemon [uptime: 7 mins 16.104 secs, performance: 100%, 
GC rate: 0.00/s, heap usage: 0% of 1.5 GiB]
   Using 15 worker leases.
   Configuration on demand is an incubating feature.
   Not watching {run-worktree} since the file system is not supported
   Watching the file system is configured to be enabled if available
   File system watching is active
   Starting Build
   Settings evaluated using settings file '{run-worktree}/settings.gradle'.
   Using local directory build cache for the root build (location = 
~/.gradle/caches/build-cache-1, remove unused entries = after 7 days).
   Projects loaded. Root project using build file '{run-worktree}/build.gradle'.
   Included projects: [root project 'iceberg', project ':iceberg-aliyun', 
project ':iceberg-api', project ':iceberg-arrow', project ':iceberg-aws', 
project ':iceberg-aws-bundle', project ':iceberg-azure', project 
':iceberg-azure-bundle', project ':iceberg-bigquery', project ':iceberg-bom', 
project ':iceberg-bundled-guava', project ':iceberg-common', project 
':iceberg-core', project ':iceberg-data', project ':iceberg-dell', project 
':iceberg-delta-lake', project ':iceberg-flink', project ':iceberg-gcp', 
project ':iceberg-gcp-bundle', project ':iceberg-hive-metastore', project 
':iceberg-kafka-connect', project ':iceberg-mr', project ':iceberg-nessie', 
project ':iceberg-open-api', project ':iceberg-orc', project 
':iceberg-parquet', project ':iceberg-snowflake', project ':iceberg-spark', 
project ':iceberg-flink:iceberg-flink-1.20', project 
':iceberg-flink:iceberg-flink-2.2', project ':iceberg-flink:iceberg-flink-2.3', 
project ':iceberg-flink:iceberg-flink-runtime-1.20', project ':iceberg
 -flink:iceberg-flink-runtime-2.2', project 
':iceberg-flink:iceberg-flink-runtime-2.3', project 
':iceberg-kafka-connect:iceberg-kafka-connect', project 
':iceberg-kafka-connect:iceberg-kafka-connect-events', project 
':iceberg-kafka-connect:iceberg-kafka-connect-runtime', project 
':iceberg-kafka-connect:iceberg-kafka-connect-transforms', project 
':iceberg-spark:iceberg-spark-4.2_2.13', project 
':iceberg-spark:iceberg-spark-extensions-4.2_2.13', project 
':iceberg-spark:iceberg-spark-runtime-4.2_2.13']
   Task path ':iceberg-parquet:test' matched project ':iceberg-parquet'
   
   > Configure project :
   Evaluating root project 'iceberg' using build file 
'{run-worktree}/build.gradle'.
   Starting process 'command 'git''. Working directory: {run-worktree} Command: 
git describe --tags --always --first-parent --abbrev=7 --match=apache-iceberg-* 
HEAD
   Successfully started process 'command 'git''
   Starting process 'command 'git''. Working directory: {run-worktree} Command: 
git status --porcelain
   Successfully started process 'command 'git''
   Generating iceberg-build.properties from git
   Starting process 'command 'git''. Working directory: {run-worktree} Command: 
git branch --show-current
   Successfully started process 'command 'git''
   
   > Configure project :iceberg-parquet
   Evaluating project ':iceberg-parquet' using build file 
'{run-worktree}/parquet/build.gradle'.
   Task name matched 'test'
   Selected primary task 'test' from project :iceberg-parquet
   
   > Configure project :iceberg-api
   Evaluating project ':iceberg-api' using build file 
'{run-worktree}/api/build.gradle'.
   
   > Configure project :iceberg-bundled-guava
   Evaluating project ':iceberg-bundled-guava' using build file 
'{run-worktree}/bundled-guava/build.gradle'.
   
   > Configure project :iceberg-core
   Evaluating project ':iceberg-core' using build file 
'{run-worktree}/core/build.gradle'.
   
   > Configure project :iceberg-common
   Evaluating project ':iceberg-common' using build file 
'{run-worktree}/common/build.gradle'.
   All projects evaluated.
   Tasks to be executed: [task ':generateGitProperties', task ':buildInfo', 
task ':iceberg-bundled-guava:compileJava', task 
':iceberg-bundled-guava:processResources', task 
':iceberg-bundled-guava:classes', task 
':iceberg-bundled-guava:compileTestJava', task 
':iceberg-bundled-guava:shadowJar', task ':iceberg-api:compileJava', task 
':iceberg-common:compileJava', task ':iceberg-core:compileJava', task 
':iceberg-parquet:compileJava', task ':iceberg-parquet:processResources', task 
':iceberg-parquet:classes', task ':iceberg-parquet:jar', task 
':iceberg-parquet:compileTestFixturesJava', task 
':iceberg-parquet:processTestFixturesResources', task 
':iceberg-parquet:testFixturesClasses', task 
':iceberg-parquet:testFixturesJar', task ':iceberg-api:processResources', task 
':iceberg-api:classes', task ':iceberg-api:jar', task 
':iceberg-api:compileTestJava', task ':iceberg-api:processTestResources', task 
':iceberg-api:testClasses', task ':iceberg-api:testJar', task 
':iceberg-common:processResources
 ', task ':iceberg-common:classes', task ':iceberg-common:jar', task 
':iceberg-core:processResources', task ':iceberg-core:classes', task 
':iceberg-core:jar', task ':iceberg-core:compileTestJava', task 
':iceberg-core:processTestResources', task ':iceberg-core:testClasses', task 
':iceberg-core:testJar', task ':iceberg-parquet:compileTestJava', task 
':iceberg-parquet:processTestResources', task ':iceberg-parquet:testClasses', 
task ':iceberg-parquet:test']
   Tasks that were excluded: []
   Resolve mutations for :generateGitProperties (Thread[Execution 
worker,5,main]) started.
   Resolve mutations for :iceberg-bundled-guava:compileJava (Thread[Execution 
worker Thread 2,5,main]) started.
   Resolve mutations for :iceberg-parquet:processResources (Thread[Execution 
worker Thread 3,5,main]) started.
   Resolve mutations for :iceberg-api:processResources (Thread[Execution worker 
Thread 4,5,main]) started.
   Resolve mutations for :iceberg-common:processResources (Thread[Execution 
worker Thread 5,5,main]) started.
   :generateGitProperties (Thread[Execution worker,5,main]) started.
   :iceberg-parquet:processResources (Thread[Execution worker Thread 3,5,main]) 
started.
   :iceberg-api:processResources (Thread[Execution worker Thread 6,5,main]) 
started.
   :iceberg-common:processResources (Thread[Execution worker Thread 5,5,main]) 
started.
   Resolve mutations for :iceberg-core:processResources (Thread[Execution 
worker Thread 4,5,main]) started.
   :iceberg-bundled-guava:compileJava (Thread[Execution worker Thread 
2,5,main]) started.
   :iceberg-core:processResources (Thread[Execution worker Thread 8,5,main]) 
started.
   
   > Task :iceberg-parquet:processResources NO-SOURCE
   Skipping task ':iceberg-parquet:processResources' as it has no source files 
and no previous output files.
   
   > Task :iceberg-api:processResources NO-SOURCE
   Skipping task ':iceberg-api:processResources' as it has no source files and 
no previous output files.
   
   > Task :iceberg-common:processResources NO-SOURCE
   Skipping task ':iceberg-common:processResources' as it has no source files 
and no previous output files.
   Resolve mutations for :iceberg-parquet:processTestFixturesResources 
(Thread[Execution worker Thread 3,5,
   
   ... [28134 bytes truncated] ...
   
   apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.gz]
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.zstd]
   
   TestParquet > testRowGroupSizeConfigurableWithWriter() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > perColumnCompressionCodecForNonExistentColumnIsIgnored() 
STANDARD_ERROR
       [Test worker] WARN org.apache.iceberg.parquet.Parquet - Skipping 
per-column compression config for missing field: non_existent_field
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.snappy]
   
   TestParquet > testPerColumnDictionaryEncodingForNonExistentColumnIsIgnored() 
STANDARD_ERROR
       [Test worker] WARN org.apache.iceberg.parquet.Parquet - Skipping 
dictionary encoding config for missing field: non_existent_field
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > adaptiveBloomFilterSizingShrinksFile() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
       [Test worker] INFO 
org.apache.parquet.column.values.bloomfilter.AdaptiveBlockSplitBloomFilter - 
The number of distinct values in [id] is approximately 5, the optimal bloom 
filter can accept 216500 distinct values, byte size is 262144.
   
   TestParquet > deleteRowGroupSizeLargerThanIntegerMax() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > testMetricsMissingColumnStatisticsInRowGroups() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > testPerColumnDictionaryEncoding() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > testRowGroupSizeConfigurable() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
       [Test worker] WARN org.apache.parquet.hadoop.InternalParquetRecordWriter 
- Too much memory used: Store {
        [intCol] optional int32 intCol = 1 {
         r:0 bytes
         d:0 bytes
          data: FallbackValuesWriter{
          data: initial: DictionaryValuesWriter{
          data: initial: dict:400
          data: initial: values:400
          data: initial:}
   
          data: fallback: PLAIN CapacityByteArrayOutputStream 0 slabs, 0 bytes
          data:}
   
          pages: ColumnChunkPageWriter ConcatenatingByteBufferCollector 0 
slabs, 0 bytes
          total: 400/800
        }
       }
   
   
   TestParquet > booleanNotEqualFilter() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
       [Test worker] INFO org.apache.parquet.filter2.compat.FilterCompat - 
Filtering using predicate: noteq(flag, true)
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- RecordReader initialized will read a total of 3 records.
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- at row 0. reading next block
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new decompressor [.gz]
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- block read in memory in 0 ms. row count = 3
   
   TestParquet > testTwoLevelList() STANDARD_ERROR
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- RecordReader initialized will read a total of 1 records.
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- at row 0. reading next block
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- block read in memory in 0 ms. row count = 1
   
   TestParquet > testColumnStatisticsEnabled() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > rowGroupSizeLargerThanIntegerMax() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > testPerColumnDictionaryEncodingViaBuilder() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > testPerColumnDictionaryEncodingNestedField() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > testGeospatialFooterMetricsSkipParquetBounds() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > perColumnCompressionCodec() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.zstd]
   
   TestParquet > testNumberOfBytesWritten() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquet > timestampNanoFilterRespectsNanoseconds() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
       [Test worker] INFO org.apache.parquet.filter2.compat.FilterCompat - 
Filtering using predicate: gteq(ts, 1704067200000000500)
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- RecordReader initialized will read a total of 5 records.
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- at row 0. reading next block
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new decompressor [.gz]
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- block read in memory in 0 ms. row count = 5
   
   TestParquetEncryption > testReadEncryptedFileWithoutAADPrefix() 
STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquetEncryption > testReadEncryptedFileWithoutKeys() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
   
   TestParquetEncryption > testReadEncryptedFile() STANDARD_ERROR
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new compressor [.gz]
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- RecordReader initialized will read a total of 100 records.
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- at row 0. reading next block
       [Test worker] INFO org.apache.hadoop.io.compress.CodecPool - Got 
brand-new decompressor [.gz]
       [Test worker] INFO org.apache.parquet.hadoop.InternalParquetRecordReader 
- block read in memory in 0 ms. row count = 100
   
   Gradle Test Executor 16 finished executing tests.
   
   > Task :iceberg-parquet:test
   Finished generating test XML results (0.001 secs) into: 
{run-worktree}/parquet/build/test-results/test
   Generating HTML test report...
   Finished generating test html results (0.003 secs) into: 
{run-worktree}/parquet/build/reports/tests/test
   Stored cache entry for task ':iceberg-parquet:test' with cache key 
42035343ff53f52dda8bf660815f9885
   
   [Incubating] Problems report is available at: 
file://{run-worktree}/build/reports/problems/problems-report.html
   
   Deprecated Gradle features were used in this build, making it incompatible 
with Gradle 9.0.
   
   You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.
   
   For more on this, please refer to 
https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
   
   BUILD SUCCESSFUL in 13s
   22 actionable tasks: 13 executed, 9 from cache
   Some of the file system contents retained in the virtual file system are on 
file systems that Gradle doesn't support watching. The relevant state was 
discarded to ensure changes to these locations are properly detected. You can 
override this by explicitly enabling file system watching.
   ```
   </details>
   <details>
   <summary>Evidence: Base regression run (test fails on base: expected 1 but 
was 0)</summary>
   
   ```text
   Configuration on demand is an incubating feature.
   > Task :iceberg-parquet:processResources NO-SOURCE
   > Task :iceberg-api:processResources NO-SOURCE
   > Task :iceberg-core:processResources NO-SOURCE
   > Task :iceberg-common:processResources NO-SOURCE
   > Task :iceberg-core:processTestResources UP-TO-DATE
   > Task :iceberg-parquet:processTestFixturesResources UP-TO-DATE
   > Task :iceberg-parquet:processTestResources NO-SOURCE
   > Task :generateGitProperties
   > Task :buildInfo
   > Task :iceberg-api:processTestResources
   > Task :iceberg-bundled-guava:compileJava UP-TO-DATE
   > Task :iceberg-bundled-guava:processResources NO-SOURCE
   > Task :iceberg-bundled-guava:classes UP-TO-DATE
   > Task :iceberg-bundled-guava:compileTestJava NO-SOURCE
   > Task :iceberg-bundled-guava:shadowJar UP-TO-DATE
   > Task :iceberg-common:compileJava UP-TO-DATE
   > Task :iceberg-common:classes UP-TO-DATE
   > Task :iceberg-common:jar UP-TO-DATE
   > Task :iceberg-api:compileJava UP-TO-DATE
   > Task :iceberg-api:classes UP-TO-DATE
   > Task :iceberg-api:jar UP-TO-DATE
   > Task :iceberg-api:compileTestJava UP-TO-DATE
   > Task :iceberg-api:testClasses
   > Task :iceberg-core:compileJava UP-TO-DATE
   > Task :iceberg-core:classes UP-TO-DATE
   > Task :iceberg-core:jar UP-TO-DATE
   
   > Task :iceberg-api:testJar
   file '{run-worktree}/build/iceberg-build.properties' will be copied to 
'iceberg-build.properties', overwriting file 
'{run-worktree}/api/build/resources/test/iceberg-build.properties', which has 
already been copied there.
   
   > Task :iceberg-parquet:compileJava FROM-CACHE
   > Task :iceberg-parquet:classes UP-TO-DATE
   > Task :iceberg-core:compileTestJava UP-TO-DATE
   > Task :iceberg-core:testClasses UP-TO-DATE
   > Task :iceberg-core:testJar UP-TO-DATE
   > Task :iceberg-parquet:jar
   > Task :iceberg-parquet:compileTestFixturesJava FROM-CACHE
   > Task :iceberg-parquet:testFixturesClasses UP-TO-DATE
   > Task :iceberg-parquet:testFixturesJar UP-TO-DATE
   
   > Task :iceberg-parquet:compileTestJava
   Note: Some input files use or override a deprecated API.
   Note: Recompile with -Xlint:deprecation for details.
   
   > Task :iceberg-parquet:testClasses
   
   > Task :iceberg-parquet:test FAILED
   
   TestParquet > writeUsesFileSystemOfHadoopOutputFile() FAILED
       org.opentest4j.AssertionFailedError: 
       expected: 1
        but was: 0
           at 
app//org.apache.iceberg.parquet.TestParquet.writeUsesFileSystemOfHadoopOutputFile(TestParquet.java:1099)
   
   Failures:
     
org.apache.iceberg.parquet.TestParquet.writeUsesFileSystemOfHadoopOutputFile()
   
   1 test completed, 1 failed
   
   [Incubating] Problems report is available at: 
file://{run-worktree}/build/reports/problems/problems-report.html
   
   FAILURE: Build failed with an exception.
   
   * What went wrong:
   Execution failed for task ':iceberg-parquet:test'.
   > There were failing tests. See the report at: 
file://{run-worktree}/parquet/build/reports/tests/test/index.html
   
   * Try:
   > Run with --scan to get full insights.
   
   Deprecated Gradle features were used in this build, making it incompatible 
with Gradle 9.0.
   
   You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.
   
   For more on this, please refer to 
https://docs.gradle.org/8.14.6/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.
   
   BUILD FAILED in 7s
   22 actionable tasks: 6 executed, 2 from cache, 14 up-to-date
   ```
   </details>
   <details>
   <summary>Evidence: Scratch scenario test source</summary>
   
   ```text
   /*
    * Licensed to the Apache Software Foundation (ASF) under one
    * or more contributor license agreements.  See the NOTICE file
    * distributed with this work for additional information
    * regarding copyright ownership.  The ASF licenses this file
    * to you under the Apache License, Version 2.0 (the
    * "License"); you may not use this file except in compliance
    * with the License.  You may obtain a copy of the License at
    *
    *   http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing,
    * software distributed under the License is distributed on an
    * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    * KIND, either express or implied.  See the License for the
    * specific language governing permissions and limitations
    * under the License.
    */
   package org.apache.iceberg.parquet;
   
   import static org.apache.iceberg.types.Types.NestedField.required;
   import static org.assertj.core.api.Assertions.assertThat;
   import static org.assertj.core.api.Assertions.assertThatThrownBy;
   
   import java.io.File;
   import java.nio.file.Path;
   import java.util.List;
   import org.apache.hadoop.conf.Configuration;
   import org.apache.iceberg.Schema;
   import org.apache.iceberg.data.GenericRecord;
   import org.apache.iceberg.data.Record;
   import org.apache.iceberg.data.parquet.GenericParquetWriter;
   import org.apache.iceberg.data.parquet.InternalReader;
   import org.apache.iceberg.hadoop.HadoopFileIO;
   import org.apache.iceberg.io.CloseableIterable;
   import org.apache.iceberg.io.FileAppender;
   import org.apache.iceberg.relocated.com.google.common.collect.Lists;
   import org.apache.iceberg.types.Types;
   import org.apache.parquet.io.OutputFile;
   import org.junit.jupiter.api.Test;
   import org.junit.jupiter.api.io.TempDir;
   
   class TestNmScratchWrite {
     private static final Schema SCHEMA = new Schema(required(1, "id", 
Types.LongType.get()));
     @TempDir Path temp;
   
     private void write(HadoopFileIO io, String loc, boolean overwrite, long... 
ids) throws Exception {
       Parquet.WriteBuilder b =
           
Parquet.write(io.newOutputFile(loc)).schema(SCHEMA).createWriterFunc(GenericParquetWriter::create);
       if (overwrite) {
         b = b.overwrite();
       }
       try (FileAppender<Record> a = b.build()) {
         for (long id : ids) {
           Record r = GenericRecord.create(SCHEMA);
           r.setField("id", id);
           a.add(r);
         }
       }
     }
   
     private List<Long> read(HadoopFileIO io, String loc) throws Exception {
       List<Long> out = Lists.newArrayList();
       try (CloseableIterable<Record> it =
           Parquet.read(io.newInputFile(loc)).project(SCHEMA)
               .createReaderFunc(s -> InternalReader.create(SCHEMA, 
s)).build()) {
         for (Record r : it) {
           out.add((Long) r.get(0));
         }
       }
       return out;
     }
   
     @Test
     void hadoopFileIORoundTripCreateAndOverwrite() throws Exception {
       HadoopFileIO io = new HadoopFileIO(new Configuration());
       String loc = new File(temp.toFile(), "a.parquet").toURI().toString();
       write(io, loc, false, 1L, 2L, 3L);
       System.out.println("NM roundtrip read: " + read(io, loc));
       assertThat(read(io, loc)).containsExactly(1L, 2L, 3L);
   
       assertThatThrownBy(() -> write(io, loc, false, 9L))
           .satisfies(e -> System.out.println("NM create-on-existing threw: " + 
e));
       assertThat(read(io, loc)).containsExactly(1L, 2L, 3L);
   
       write(io, loc, true, 7L, 8L);
       System.out.println("NM after overwrite read: " + read(io, loc));
       assertThat(read(io, loc)).containsExactly(7L, 8L);
     }
   
     @Test
     void outputFileMetadataMatchesParquetHadoopOutputFile() throws Exception {
       Configuration conf = new Configuration();
       String loc = new File(temp.toFile(), "b.parquet").getAbsolutePath();
       OutputFile ours = ParquetIO.file(new 
HadoopFileIO(conf).newOutputFile(loc));
       OutputFile theirs =
           org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(new 
org.apache.hadoop.fs.Path(loc), conf);
       System.out.println("NM ours: path=" + ours.getPath() + " 
supportsBlockSize=" + ours.supportsBlockSize()
           + " defaultBlockSize=" + ours.defaultBlockSize() + " class=" + 
ours.getClass().getName());
       System.out.println("NM parquet-hadoop: path=" + theirs.getPath() + " 
supportsBlockSize="
           + theirs.supportsBlockSize() + " defaultBlockSize=" + 
theirs.defaultBlockSize());
       assertThat(ours.getPath()).isEqualTo(theirs.getPath());
       
assertThat(ours.supportsBlockSize()).isEqualTo(theirs.supportsBlockSize());
       assertThat(ours.defaultBlockSize()).isEqualTo(theirs.defaultBlockSize());
     }
   }
   ```
   </details>
   <details>
   <summary>Evidence: Scratch scenario output</summary>
   
   ```text
   NM ours: path=file:/tmp/junit-.../b.parquet supportsBlockSize=false 
defaultBlockSize=33554432 
class=org.apache.iceberg.parquet.ParquetIO$HadoopFileSystemOutputFile
   NM parquet-hadoop: path=file:/tmp/junit-.../b.parquet 
supportsBlockSize=false defaultBlockSize=33554432
   NM roundtrip read: [1, 2, 3]
   NM create-on-existing threw: java.io.UncheckedIOException: Failed to create 
Parquet file
   NM after overwrite read: [7, 8]
   ```
   </details>
   
   - Evidence: TestParquet JUnit XML (local file: 
<code>~/.no-mistakes/evidence/01M7EBHZ3M1YRZ9N4PS8EKC29F/TEST-org.apache.iceberg.parquet.TestParquet.xml</code>)
   
   ## Pipeline
   
   Updates from [git push 
no-mistakes](https://github.com/kunchenguid/no-mistakes)
   
   <!-- no-mistakes-pipeline-attestation:v1 
{"head_sha":"d4de555cf518fef893bd609aefb66eb53c9b1247","steps":[{"step":"intent","status":"completed"},{"step":"rebase","status":"skipped"},{"step":"review","status":"completed"},{"step":"test","status":"completed"},{"step":"document","status":"completed"},{"step":"lint","status":"completed"},{"step":"push","status":"completed"},{"step":"pr","status":"running"},{"step":"ci","status":"pending"}],"live_validation":{"verdict":"go","live":6,"total":6}}
 -->
   
   <details>
   <summary>✅ **Intent** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>⏭️ **Rebase** - skipped</summary>
   
   Step was skipped.
   </details>
   
   <details>
   <summary>⚠️ **Review** - 1 info</summary>
   
   - ℹ️ 
`parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java:118` - 
FileSystem resolution for writes no longer sees writer-level config. The old 
ParquetIO.file(output, conf) built a Hadoop Configuration from the writer&#39;s 
ParquetConfiguration, which is the HadoopOutputFile conf plus every 
WriteBuilder.set/setAll entry, including table properties (Parquet.java:193, 
514-516), and called FileSystem.get with it. The new code uses 
hfile.getFileSystem(), resolved from the file&#39;s own conf. The two only 
differ when the FS cache is disabled for the scheme 
(fs.&lt;scheme&gt;.impl.disable.cache=true) and FS settings are passed as table 
or writer properties. In that case the old code created a fresh, never-closed 
FileSystem per data file with those overrides applied; the new code reuses the 
FileIO&#39;s FS. This fits the author&#39;s stated design and arguably fixes a 
leak, so no action is needed. It is noted only for the PR discussion.
   </details>
   
   <details>
   <summary>✅ **Test** - passed</summary>
   
   ✅ No issues found.
   - Live validation: ✅ go - 6 of 6 scenarios driven live against the product
   
   | Scenario | Result | Live | Evidence |
   | --- | --- | --- | --- |
   | Parquet write via HadoopOutputFile built with a custom (counting) 
FileSystem on a local temp file calls that FS&#39;s create once and the file is 
readable | ✅ pass | live | TestParquet.writeUsesFileSystemOfHadoopOutputFile 
passed on target (target-tests.log, TEST-...TestParquet.xml) |
   | Adversarial: same write with base-commit ParquetIO bypasses the provided 
FileSystem | ✅ pass | live | base-regression-test.log: expected 1 but was 0 |
   | HadoopFileIO local write/read round trip still works (3 rows written and 
read back) | ✅ pass | live | Scratch output &#39;NM roundtrip read: [1, 2, 
3]&#39; |
   | Adversarial: create without overwrite on an existing file fails and leaves 
original data; overwrite() replaces it | ✅ pass | live | Scratch output 
&#39;create-on-existing threw: UncheckedIOException&#39; then &#39;after 
overwrite read: [7, 8]&#39; |
   | New output file reports the same path, supportsBlockSize and 
defaultBlockSize as parquet-hadoop&#39;s HadoopOutputFile for the same local 
path/conf | ✅ pass | live | Scratch output lines &#39;NM ours&#39; vs &#39;NM 
parquet-hadoop&#39; |
   | Encrypted Parquet writes/reads still work (TestParquetEncryption) and 
existing TestParquet suite passes | ✅ pass | live | TestParquetEncryption 3/3, 
TestParquet 37/37 in target-tests.log |
   
   - <code>`./gradlew :iceberg-parquet:test --tests 
org.apache.iceberg.parquet.TestParquet --tests 
org.apache.iceberg.parquet.TestParquetEncryption --tests 
org.apache.iceberg.parquet.TestNmScratchWrite --offline` on target commit (37 + 
3 + 2 tests, 0 failures)</code>
   - <code>`./gradlew :iceberg-parquet:test --tests 
org.apache.iceberg.parquet.TestParquet.writeUsesFileSystemOfHadoopOutputFile` 
with parquet/src/main reverted to base 234d55774 (fails: expected 1 but was 
0)</code>
   - `Temporary scratch test (removed afterwards): HadoopFileIO local 
write/read round trip, create on existing file rejected, overwrite() replaces 
contents`
   - `Temporary scratch test: ParquetIO.file(HadoopFileIO output) 
getPath/supportsBlockSize/defaultBlockSize compared against parquet-hadoop 
HadoopOutputFile.fromPath for the same path and conf`
   </details>
   
   <details>
   <summary>✅ **Document** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Lint** - passed</summary>
   
   ✅ No issues found.
   </details>
   
   <details>
   <summary>✅ **Push** - passed</summary>
   
   ✅ No issues found.
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to